# Citation Styles (CSL)

Journal-specific Citation Style Language files for pandoc citeproc rendering.
Source: https://github.com/citation-style-language/styles (zotero/styles).

## Bundled CSLs

| File | Use for | Notes |
|------|---------|-------|
| `european-radiology.csl` | European Radiology, EURE | Dependent on `springer-basic-brackets.csl` (must be in same dir) |
| `cardiovascular-and-interventional-radiology.csl` | CVIR | Dependent on `springer-vancouver-brackets.csl` |
| `radiology.csl` | Radiology (RSNA) | Independent. Also acceptable fallback for Radiology: AI when no dedicated CSL exists |
| `american-journal-of-roentgenology.csl` | AJR | Independent |
| `korean-journal-of-radiology.csl` | KJR | Independent. Parenthesised numbers `(1, 2)`, et-al after 6 (first 6 + et al) — **not** superscript |
| `american-medical-association.csl` | AMA Manual of Style 11th ed. — JAMA family, and any journal citing "AMA style" | Independent. Superscript, et-al after 6 (first 3 + et al), DOI kept |
| `liver-international.csl` | Liver International (Wiley) | AMA-style superscript: et-al after 6 (first 3 + et al), no PMID, DOI kept. Also a fallback for Wiley/AMA "first-3-et-al" superscript journals |
| `journal-of-cachexia-sarcopenia-and-muscle.csl` | JCSM | Independent. Superscript, et-al after 6 (first 6 + et al) |
| `nature.csl` | Nature portfolio | Independent. Superscript, et-al after 5 (first 1 + et al) |
| `journal-of-korean-medical-science.csl` | JKMS | Dependent on `nlm-citation-sequence.csl` (must be in same dir) |
| `journal-of-korean-medical-science-strict.csl` | JKMS, strict variant | Independent. Superscript NLM citation-sequence, et-al after 6 (first 6 + et al) |
| `vancouver.csl` | Generic Vancouver (brackets) | Fallback when journal CSL unavailable (e.g., JVIR, Radiology: AI) |
| `vancouver-superscript.csl` | Generic Vancouver (superscript) | Alternative fallback |
| `springer-basic-brackets.csl` | Parent of European Radiology | Do not use directly — keep co-located |
| `springer-vancouver-brackets.csl` | Parent of CVIR | Do not use directly — keep co-located |
| `nlm-citation-sequence.csl` | Parent of JKMS | Do not use directly — keep co-located |

## Missing — use fallback

- **Radiology: Artificial Intelligence (RYAI)**: no dedicated CSL on zotero/styles as of 2026-04. Use `radiology.csl` (parent journal, identical RSNA house style).
- **Journal of Vascular and Interventional Radiology (JVIR)**: no dedicated CSL. Use `vancouver.csl` and verify against latest author guidelines before submission.

## Updating

Only the files below are verbatim upstream styles, so only these may be refreshed by slug:

```bash
cd "$(dirname "$0")"
for s in european-radiology radiology american-journal-of-roentgenology \
         cardiovascular-and-interventional-radiology korean-journal-of-radiology \
         american-medical-association journal-of-cachexia-sarcopenia-and-muscle nature \
         journal-of-korean-medical-science nlm-citation-sequence \
         springer-basic-brackets springer-vancouver-brackets; do
  curl -fsSL -o "${s}.csl" "https://www.zotero.org/styles/${s}"
done
```

**Do not refresh these by filename** — they are locally renamed or locally modified copies whose
`<id>` does not match their filename, so fetching `zotero.org/styles/<filename>` would replace them
with a different style:

| File | Actual `<id>` slug | Caught by the check below? |
|------|--------------------|---------------------------|
| `vancouver.csl` | `nlm-citation-sequence` | yes |
| `vancouver-superscript.csl` | `nlm-citation-sequence-superscript` | yes |
| `liver-international.csl` | `nlm-citation-sequence-superscript` (locally retitled) | yes |
| `journal-of-korean-medical-science-strict.csl` | `journal-of-korean-medical-science-strict` | **no** — the slug matches the filename, but the style is locally authored (its `<title>` is a description, not a journal name). Confirm the slug resolves upstream before refreshing it |

Check which files are safe to refresh (filename must equal the `<id>` slug):
```bash
for f in *.csl; do
  slug=$(sed -n 's:.*<id>.*/\([^/<]*\)</id>.*:\1:p' "$f" | head -1)
  [ "$slug" = "${f%.csl}" ] || echo "LOCAL VARIANT: $f -> $slug"
done
```

Verify dependent-parent links if an upstream publisher reorganizes:
```bash
grep -H independent-parent *.csl
```

# Cross-reference QC — symptom triage

`check_xref.py --strict` writes a 3-way matrix to `qc/xref_audit.json` that
classifies every Table/Figure label across (a) in-text citations, (b) body
captions in `## Tables` / `## Figures` / `## Figure Legends` /
`## Supplementary {Tables,Figures}`, and (c) caption paragraphs in the
rendered DOCX (via `python-docx`).

| Status | Meaning | Severity | Fix |
|---|---|---|---|
| `OK` | cited + body caption + DOCX caption all present, caption text agrees (Jaccard ≥ 0.40) | — | none |
| `MISSING_DOCX` | cited but no caption with that label in the rendered DOCX | **P0 blocker** | drop the citation if the figure/table was retired, or re-add it to the build pipeline and rebuild DOCX |
| `MISSING_BODY` | cited but no caption definition in the markdown body sections | **P0 blocker**, with one exception — see below | add the caption under `## Tables` / `## Figures` in `manuscript.md`, then re-render |
| `MISMATCH` | label exists in both body and DOCX but caption text disagrees (Jaccard < 0.40) | **P0 blocker** | reconcile body vs build script — body caption is the SSOT, update the build pipeline to match, never the reverse |
| `UNCITED` | caption defined or rendered but never cited in main text | warn | either delete the caption or add a citation; never ship UNCITED on a clean run |
| `NOT_CITED_NO_BODY` | label appears only in DOCX (rare; legacy artifact) | warn | clean up the build pipeline; the DOCX is leaking captions from a previous draft |

### `MISSING_BODY` names two different situations

The row above describes the one people mean by it — **build SSOT drift**: the float is rendered in
the DOCX, but nothing in `manuscript.md` defines its caption, so the build pipeline is the only
place that knows the text. That is a P0 under every policy, including
`--allow-separate-attachments`. No attachment style makes a build script an acceptable single
source of truth for a caption.

The same verdict is also returned when **no `--docx` was supplied at all**. Then there is no
rendered artifact to have drifted *from*, and the run genuinely cannot distinguish

- a caption you forgot to write, from
- a float that lives in a **separate supplement file** this invocation never saw — the normal
  packaging for radiology and most medical journals.

**`--allow-separate-attachments` downgrades that second case**, because the flag is exactly the
declaration that some floats live outside the main document. But it is an *excuse*, not a check:
the run prints

```
[check_xref] WARN: 2 MISSING_BODY row(s) EXCUSED WITHOUT EVIDENCE under --allow-separate-attachments:
           Figure:S-S1, Table:S-S1
           No --docx was supplied, so nothing here was actually checked.
```

and records the count in `summary.downgraded_unchecked`, separately from
`summary.downgraded_proven_absent` — the `MISSING_DOCX` rows a supplied DOCX actually proved absent
from the rendered output.

**Run once with `--docx` before submitting.** That is what converts the excuse into evidence: a
float genuinely absent from the rendered output becomes `MISSING_DOCX` (still downgraded, now
proven), and a caption nobody wrote becomes visible again.

| invocation | separate-supplement float | forgotten caption |
|---|---|---|
| no flag | **blocks** | **blocks** |
| flag, no `--docx` | passes — excused without evidence | passes — **this is the cost of the flag** |
| flag + `--docx` | passes — proven absent from the DOCX | **blocks** as `MISSING_BODY` (in DOCX) or surfaces as `MISSING_DOCX` you must explain |

## Why this exists

Internal consistency in `/self-review` Phase 2.5 does NOT catch
cross-reference defects because both the body prose and the build script
can echo their own divergent SSOTs cleanly — each looks self-consistent in
isolation. The failure it misses: the body cites a supplementary table as a
sensitivity analysis while the rendered DOCX carries a diagnostics table under
that number, further supplement numbers mismatch the same way, and some are
cited but absent from the DOCX entirely. The 3-way matrix between citations, body captions,
and DOCX captions is the only place those drifts surface.

## Pipeline placement

Always run **after** the DOCX build (Workflow A step 2 or after
`render_pandoc.sh`) and **before** the final submission gate. If
`python-docx` is unavailable, the script falls back to a body-only audit
(citations vs body captions) with a stderr warning; install with
`pip install python-docx` for full coverage.

# Journal Reference-Style Specs (CSL acceptance criteria)

Zotero-sourced CSL files are **not** auto-validated against each journal's author
guide. A dependent (stub) CSL inherits its parent's format, which can differ from
what the journal requires. This table is the **acceptance criteria** for
`scripts/check_csl_render.py` — run it BEFORE submission so format mismatches are
caught at build time, not in the portal proof PDF.

## How to use

```bash
# Validate a CSL against a journal spec before rendering the manuscript:
python scripts/check_csl_render.py --csl citation_styles/<file>.csl \
    --bib refs.bib --journal <key>
# Exits non-zero on mismatch (in-text format / DOI / abbreviation).
```

If a journal's stub CSL fails (parent format ≠ author guide), create a
`-strict` variant (see `journal-of-korean-medical-science-strict.csl`) and use it
for the pandoc render while keeping the original for Zotero CWYW.

## Spec table

| key | Journal | in-text | DOI | journal name | et-al | date | issue | CSL status |
|-----|---------|---------|-----|--------------|-------|------|-------|-----------|
| **jkms** | J Korean Med Sci | **superscript** | **none** | **NLM abbrev** | ≤6 then et al | **year only** | include `(n)` | stub→**use `-strict`** ✅ verified 2026-06-03 |
| radiology | Radiology (RSNA) | paren `(1)` | yes | abbrev | — | — | — | full; ⚠ VERIFY vs author guide |
| ajr | Am J Roentgenol | superscript | none | abbrev | — | — | — | full; ⚠ VERIFY |
| kjr | Korean J Radiol | superscript | none | abbrev | — | — | — | full; ⚠ VERIFY |
| eur-radiol | European Radiology | bracket `[1]` | yes | full ok | — | — | — | stub→springer-basic; ⚠ VERIFY |
| cvir | Cardiovasc Intervent Radiol | bracket `[1]` | yes | full ok | — | — | — | stub→springer-vancouver; ⚠ VERIFY |

**Legend:** ✅ verified against author guide · ⚠ inherited from Zotero, not yet
confirmed against the journal's Instructions for Authors. Confirm and update the
row (and `check_csl_render.py::SPECS`) when you next submit to that journal.

## Known pitfalls (from JKMS submission, 2026-06-03)

1. **Journal abbreviation needs metadata.** CSL `container-title form="short"` only
   works if the `.bib` entry has a `shortjournal` (BibLaTeX) / `journalAbbreviation`
   (Zotero) field. Without it, the full name is printed regardless of CSL. Use
   `scripts/fill_journal_abbrev.py` to populate NLM abbreviations from PubMed.
2. **Title proper nouns.** pandoc/citeproc applies sentence-case to titles and will
   lowercase proper nouns (`Fleischner Society` → `fleischner society`) unless the
   title is double-braced `title = {{...}}`. Pull authoritative titles (with
   subtitle + proper-noun casing) from PubMed efetch ArticleTitle.
3. **Name particles.** `de Torres`, `von Elm`, `de Bock` get demoted to
   `Torres JP de` unless braced `{de Torres}` in the family field. `demote-non-dropping-particle="never"`
   alone is insufficient because citeproc auto-splits lowercase particles.
4. **DOI hyperlinks survive surgical docx edits.** If you swap reference text in a
   Word file, a `<w:hyperlink>` carrying the DOI persists as a separate element
   (python-docx `p.runs` doesn't see it). Remove hyperlinks explicitly. See
   `~/.claude/rules/submission-portal-verification.md` §1.

## Related
- `scripts/check_csl_render.py` — acceptance test
- `scripts/fill_journal_abbrev.py` — NLM abbreviation injection
- `citation_styles/journal-of-korean-medical-science-strict.csl` — JKMS author-guide-faithful variant
- `~/.claude/rules/manuscript-references.md` — hybrid pandoc/Zotero workflow

#!/usr/bin/env python3
"""Publisher markup in a .bib title corrupts the rendered bibliography.

CrossRef ships titles with markup — `<scp>WHO</scp>`, `<i>IDH</i>`, `<sub>1</sub>` —
and a DOI-add stores them verbatim. Downstream, Better BibTeX either escapes the tags
(`{$<$}scp{$>$}`) or strips them without restoring the space they were standing in, and
the reference list renders as garbage:

    The 2021 {$<$}scp{$>$}WHO{$<$}/scp{$>$} Classification of Tumors...
    Glioma Groups Based on 1p/19q,IDH, andTERTPromoter Mutations

Nothing catches this. `/verify-refs` checks whether the reference is *true* (DOI, authors);
`check_citation_keys` checks whether the key *resolves*. Neither looks at the title as it
will be printed, so the corruption is found — if it is found at all — by eyeballing the
rendered document, which is exactly the reading nobody does on the reference list.

Verdicts:
  TITLE_MARKUP (major)  raw or escaped publisher markup survives in the title
  TITLE_FUSION (major)  a tag was stripped without restoring its space, welding two words
                        together (`andTERT`, `,IDH`)

The fusion check is deliberately narrow: it fires on an English function word or a comma
welded to an acronym, not on any lowercase-then-uppercase transition — `mRNA`, `hTERT`,
`nnU-Net`, `pH`, `1,2-dichloroethane` are ordinary and must not be flagged. The point of a
gate is that a clean run means something.

Usage:
    check_bib_title_markup.py --bib refs.bib [--out qc/bib_title_markup.json] [--strict]

Exit 0 when every title is clean (or, without --strict, always). Stdlib only.
"""

from __future__ import annotations

import argparse
import json
import re
import sys
from pathlib import Path

# Raw HTML/XML tags and their BibTeX-escaped forms. `{$<$}` is what BBT writes when it
# escapes a `<` it does not understand.
MARKUP = re.compile(
    r"</?(?:i|b|em|strong|scp|sub|sup|span|p|br)\b[^>]*>"   # raw tags
    r"|\{\$<\$\}|\{\$>\$\}"                                  # BBT-escaped angle brackets
    r"|&lt;|&gt;|&amp;(?![a-z]+;)",                          # HTML entities
    re.IGNORECASE,
)

# A tag stripped without restoring its space welds an English function word — or a comma —
# directly onto the acronym the tag was wrapping. `and<i>TERT</i>` -> `andTERT`.
FUNCTION_WORDS = (
    "and|or|the|in|of|with|for|by|on|at|from|to|as|an|a|is|are|was|were|via|per|"
    "between|among|versus|vs"
)
FUSION_WORD = re.compile(rf"\b({FUNCTION_WORDS})([A-Z]{{2,}})", re.UNICODE)
# A comma glued straight onto a letter. Legitimate titles always put a space after a comma;
# a numeric comma (`1,2-dichloroethane`, `10,000`) is not matched because a digit follows.
FUSION_COMMA = re.compile(r",(?=[A-Za-z])")

ENTRY = re.compile(r"@(\w+)\s*\{\s*([^,]+),(.*?)\n\}", re.DOTALL)
TITLE = re.compile(r"^\s*(?:title|booktitle)\s*=\s*[{\"](.+?)[}\"]\s*,?\s*$", re.MULTILINE | re.IGNORECASE)


def findings_for(key: str, title: str) -> list[dict]:
    out: list[dict] = []
    for m in MARKUP.finditer(title):
        out.append(
            {
                "verdict": "TITLE_MARKUP",
                "severity": "major",
                "key": key,
                "match": m.group(0),
                "title": title,
                "detail": (
                    f"publisher markup {m.group(0)!r} survives in the title of `{key}`; it will "
                    "render literally (or be stripped, welding the neighbouring words). Unwrap the "
                    "tag in the reference manager — do not hand-edit the rendered bibliography."
                ),
            }
        )
    for m in FUSION_WORD.finditer(title):
        out.append(
            {
                "verdict": "TITLE_FUSION",
                "severity": "major",
                "key": key,
                "match": m.group(0),
                "title": title,
                "detail": (
                    f"`{m.group(0)}` in the title of `{key}`: a markup tag was stripped without "
                    "restoring the space it occupied, welding a word onto an acronym. Restore the "
                    "space at the source (the reference manager), not in the rendered output."
                ),
            }
        )
    if FUSION_COMMA.search(title):
        m = FUSION_COMMA.search(title)
        ctx = title[max(0, m.start() - 12) : m.start() + 12]
        out.append(
            {
                "verdict": "TITLE_FUSION",
                "severity": "major",
                "key": key,
                "match": ctx,
                "title": title,
                "detail": (
                    f"comma glued to a word in the title of `{key}` (…{ctx}…) — the space a stripped "
                    "tag was holding. Restore it at the source."
                ),
            }
        )
    return out


def audit(bib: Path) -> dict:
    text = bib.read_text(encoding="utf-8", errors="replace")
    findings: list[dict] = []
    titles = 0
    for _, key, body in ENTRY.findall(text):
        m = TITLE.search(body)
        if not m:
            continue
        titles += 1
        findings.extend(findings_for(key.strip(), m.group(1).strip()))

    return {
        "detector": "check_bib_title_markup",
        "bib": str(bib),
        "titles_checked": titles,
        "findings": findings,
        "summary": {
            "TITLE_MARKUP": sum(1 for f in findings if f["verdict"] == "TITLE_MARKUP"),
            "TITLE_FUSION": sum(1 for f in findings if f["verdict"] == "TITLE_FUSION"),
        },
        "submission_safe": not findings,
    }


def main() -> int:
    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument("--bib", required=True, type=Path, help="the .bib file to lint")
    ap.add_argument("--out", type=Path, help="write the JSON audit record here")
    ap.add_argument("--strict", action="store_true", help="exit 1 if any title is corrupted")
    ap.add_argument("--quiet", action="store_true")
    a = ap.parse_args()

    if not a.bib.is_file():
        raise SystemExit(f"not found: {a.bib}")

    rep = audit(a.bib)
    if a.out:
        a.out.parent.mkdir(parents=True, exist_ok=True)
        a.out.write_text(json.dumps(rep, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")

    if not a.quiet:
        print(f"{a.bib.name}: {rep['titles_checked']} title(s) checked")
        for f in rep["findings"]:
            print(f"  [{f['severity'].upper()}] {f['verdict']}: {f['detail']}")
        if not rep["findings"]:
            print("  OK — no publisher markup or tag-strip fusion in any title")

    return 1 if (a.strict and rep["findings"]) else 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
"""check_citation_keys.py — Validate pandoc-style [@bibkey] citations against a .bib file.

Reports:
  - keys cited in markdown but missing from .bib (UNDEFINED) — always fail
  - keys present in .bib but never cited (UNUSED) — warn by default,
    suppress with --allow-unused, or escalate with --strict-unused

Usage:
  check_citation_keys.py manuscript.md references.bib
  check_citation_keys.py manuscript.md references.bib --allow-unused
  check_citation_keys.py manuscript.md references.bib --strict-unused

Why --allow-unused exists:
  During early drafting, the .bib often holds a working set of candidate
  references that have not yet been cited in the manuscript. UNUSED output
  is noise in that phase and makes the diagnostic harder to read. Pass
  --allow-unused to suppress UNUSED reporting entirely; UNDEFINED remains a
  hard failure.

Why --strict-unused exists:
  At submission-package freeze time, UNUSED entries usually represent
  forgotten edits (a citation was removed from the manuscript but the .bib
  entry remains). Pass --strict-unused to treat any UNUSED entry as a
  build failure so the freeze gate catches it.
"""
from __future__ import annotations

import argparse
import re
import sys
from pathlib import Path

# pandoc citation syntax: [@key], [@key, p. 3], [@key1; @key2], [-@key] (suppress author)
# A key is alnum + : . _ - / + (per pandoc docs) — but pandoc counts internal punctuation as part of
# the key ONLY when a letter or digit follows it. The old pattern dropped that condition, so a
# citation ending a sentence — which is most of them — swallowed the full stop:
#
#     "...as previously reported @Smith2023."   ->  key "Smith2023."
#
# and the tool then reported the SAME reference as UNDEFINED (no `Smith2023.` in the .bib) and as
# UNUSED (nothing cited `Smith2023`) in a single run. Trailing `;` `,` `]` already terminated the
# match; `.` `-` `/` `+` leaked, and `.` is the one that ends English sentences. `@Sec.2a` stays one
# key, because there the period is followed by an alphanumeric — pandoc's own rule, now encoded.
CITE_RE = re.compile(r"(?<![A-Za-z0-9_])-?@([A-Za-z][\w]*(?:[:.\-/+][\w]+)*)")
BIB_KEY_RE = re.compile(r"^@\w+\s*\{\s*([^,\s]+)\s*,", re.MULTILINE)

# Quarto / pandoc-crossref cross-references share the `@` sigil but are not bibliography keys:
# `@fig-flow`, `@tbl-baseline`, `@sec-methods`, `@eq-lik`, and the older `@fig:flow` form. They
# resolve against the document's own labels, and `quarto render` compiles a manuscript full of them
# without complaint — while this checker read every one as an undefined reference and exited 1. The
# repo scaffolds `manuscript/index.qmd` itself (`scripts/init_project.py`), so the toolkit was
# failing manuscripts in the shape it generates.
#
# Excluded from the UNDEFINED verdict, never dropped silently: they are counted and reported. A key
# with one of these prefixes that IS defined in the .bib resolves normally and never reaches this
# exclusion, so a real bibliography entry named `fig-...` is unaffected.
CROSSREF_PREFIXES = (
    "fig", "tbl", "eq", "sec", "lst", "thm", "lem", "cor", "prp", "cnj",
    "def", "exm", "exr", "sol", "rem", "alg", "tip", "nte", "wrn", "imp", "cau",
)
CROSSREF_RE = re.compile(rf"^(?:{'|'.join(CROSSREF_PREFIXES)})[-:]\w", re.IGNORECASE)


def extract_md_keys(md_path: Path) -> set[str]:
    text = md_path.read_text(encoding="utf-8")
    # strip code fences to avoid false positives
    text = re.sub(r"```.*?```", "", text, flags=re.DOTALL)
    text = re.sub(r"`[^`\n]+`", "", text)
    return set(CITE_RE.findall(text))


def extract_bib_keys(bib_path: Path) -> set[str]:
    text = bib_path.read_text(encoding="utf-8", errors="replace")
    return set(BIB_KEY_RE.findall(text))


def main() -> int:
    parser = argparse.ArgumentParser(
        description=__doc__,
        formatter_class=argparse.RawDescriptionHelpFormatter,
    )
    parser.add_argument("markdown", type=Path, help="Path to manuscript.md")
    parser.add_argument("bib", type=Path, help="Path to references.bib")
    group = parser.add_mutually_exclusive_group()
    group.add_argument(
        "--allow-unused",
        action="store_true",
        help="Suppress UNUSED reporting entirely (drafting mode).",
    )
    group.add_argument(
        "--strict-unused",
        action="store_true",
        help="Treat any UNUSED entry as a build failure (submission gate).",
    )
    args = parser.parse_args()

    if not args.markdown.exists():
        print(f"ERROR: markdown not found: {args.markdown}", file=sys.stderr)
        return 2
    if not args.bib.exists():
        print(f"ERROR: bib not found: {args.bib}", file=sys.stderr)
        return 2

    cited = extract_md_keys(args.markdown)
    defined = extract_bib_keys(args.bib)

    # Split the unresolved keys before judging them: a Quarto cross-reference resolves against the
    # document's own labels, not the bibliography, so calling it an undefined citation fails a
    # manuscript that `quarto render` compiles without complaint. Reported, never silently dropped.
    unresolved = sorted(cited - defined)
    crossrefs = [k for k in unresolved if CROSSREF_RE.match(k)]
    undefined = [k for k in unresolved if not CROSSREF_RE.match(k)]
    unused = sorted(defined - cited)

    print(f"[check_citation_keys] cited={len(cited)} defined={len(defined)}")
    if crossrefs:
        print(f"\nCROSS-REFERENCES ({len(crossrefs)}) — Quarto/pandoc-crossref labels, not "
              f"bibliography keys; resolved by the renderer, not checked here:")
        for k in crossrefs:
            print(f"  [@{k}]")
    if undefined:
        print(f"\nUNDEFINED ({len(undefined)}) — cited in markdown but not in .bib:")
        for k in undefined:
            print(f"  [@{k}]")
    show_unused = unused and not args.allow_unused
    if show_unused:
        label = "UNUSED" if not args.strict_unused else "UNUSED (--strict-unused: treated as failure)"
        print(f"\n{label} ({len(unused)}) — defined in .bib but never cited:")
        for k in unused:
            print(f"  {k}")
    if not undefined and not show_unused:
        if args.allow_unused and unused:
            print(f"OK: all cited keys defined ({len(unused)} UNUSED suppressed by --allow-unused).")
        else:
            print("OK: all cited keys defined and all defined keys used.")

    exit_code = 0
    if undefined:
        exit_code = 1
    elif unused and args.strict_unused:
        exit_code = 1
    return exit_code


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
"""CSL acceptance test — render a sample and verify in-text format / DOI / journal
abbreviation against the target journal's author-guide spec.

Motivation: Zotero-sourced CSL files are not validated against each journal's
author guide. A "dependent" (stub) CSL inherits its parent's format, which may
differ from what the journal actually requires (e.g. JKMS author guide mandates
superscript Arabic numerals + NLM abbreviations + no DOI, but the Zotero
journal-of-korean-medical-science.csl points to nlm-citation-sequence which
renders parenthetical (1), keeps DOI, and prints full journal names).

This script renders a 2-citation sample through pandoc + the CSL and checks:
  - in-text format: superscript | bracket | paren
  - DOI present in reference list
  - journal name: abbreviated vs full
  - et-al rule (>=N authors collapses)
Compares against expected spec (from REFERENCE_STYLE_SPECS.md or CLI flags) and
exits non-zero on mismatch — run this BEFORE submission, not after the proof PDF.

Exit codes:
  0  output matches journal spec
  1  spec mismatch (in-text / DOI / abbreviation)
  2  environment / input error (pandoc or python-docx missing, bib not found,
     pandoc render failed) — reported with a clear message, never a raw traceback

Usage:
  python check_csl_render.py --csl path/to.csl --bib refs.bib \\
      --expect-intext superscript --expect-doi 0 --expect-abbrev yes
  # or pull expected spec by journal key:
  python check_csl_render.py --csl ... --bib ... --journal jkms
"""
import argparse, subprocess, tempfile, re, os, sys, json
from pathlib import Path

# python-docx is required for the superscript check. Import is guarded at the top
# so a missing dependency is a clear, actionable message (exit 2) rather than an
# ImportError traceback raised deep inside analyze().
try:
    from docx import Document
except ImportError:  # pragma: no cover - environment-dependent
    Document = None

# Minimal built-in spec table (extend via REFERENCE_STYLE_SPECS.md).
# intext: superscript|bracket|paren ; doi: 0|1 ; abbrev: yes|no
SPECS = {
    "jkms":      {"intext": "superscript", "doi": 0, "abbrev": "yes", "note": "verified 2026-06-03"},
    "radiology": {"intext": "paren",       "doi": 1, "abbrev": "yes", "note": "VERIFY against author guide"},
    "ajr":       {"intext": "superscript", "doi": 0, "abbrev": "yes", "note": "VERIFY"},
    "kjr":       {"intext": "superscript", "doi": 0, "abbrev": "yes", "note": "VERIFY"},
    "eur-radiol":{"intext": "bracket",     "doi": 1, "abbrev": "no",  "note": "Springer; VERIFY"},
    "cvir":      {"intext": "bracket",     "doi": 1, "abbrev": "no",  "note": "Springer; VERIFY"},
}

SAMPLE = ("Risk is elevated [@A; @B].\n\n# References\n")


class RenderError(RuntimeError):
    """Environment/input failure that should exit 2 with a clear message."""


def _read_bib(bib: str) -> str:
    """Read the .bib file, raising a clear RenderError if it is missing/unreadable."""
    p = Path(bib)
    if not p.exists():
        raise RenderError(f"bib file not found: {bib}")
    try:
        return p.read_text(encoding="utf-8")
    except OSError as exc:
        raise RenderError(f"could not read bib file {bib}: {exc}") from exc


def render(csl: str, bib: str, fmt: str, first: str, second: str, outdir: str) -> str:
    """Render the 2-citation SAMPLE through pandoc+CSL into ``outdir``.

    ``first``/``second`` are the two citekeys to substitute (passed explicitly so
    this function is standalone-callable — no module globals). The input markdown
    and the output file live under ``outdir`` so the caller's TemporaryDirectory
    cleans everything up; nothing leaks. Raises RenderError if pandoc is missing
    or returns non-zero, so a failed render can never be silently analyzed as if
    it had succeeded.
    """
    md_path = os.path.join(outdir, "sample.md")
    out_path = os.path.join(outdir, f"out.{fmt}")
    Path(md_path).write_text(SAMPLE.replace("@A", first).replace("@B", second), encoding="utf-8")
    try:
        proc = subprocess.run(
            ["pandoc", md_path, "--citeproc", f"--bibliography={bib}",
             f"--csl={csl}", "-o", out_path],
            capture_output=True, text=True,
        )
    except FileNotFoundError as exc:
        raise RenderError(
            "pandoc not found on PATH. Install pandoc to run the CSL render check."
        ) from exc
    if proc.returncode != 0:
        raise RenderError(
            f"pandoc failed (exit {proc.returncode}) rendering {fmt}: "
            f"{proc.stderr.strip()[:500]}"
        )
    return out_path


def analyze(csl: str, bib: str) -> dict:
    # Validate inputs first (bib path), so a missing bib is reported clearly and
    # independently of whether the optional python-docx parser is installed.
    keys = re.findall(r"@\w+\{([^,]+),", _read_bib(bib))
    first, second = (keys + ["A", "B"])[:2]
    if Document is None:
        raise RenderError(
            "python-docx is required for the in-text superscript check "
            "(pip install python-docx)."
        )
    with tempfile.TemporaryDirectory(prefix="csl_render_") as tmp:
        docx = render(csl, bib, "docx", first, second, tmp)
        txt_out = render(csl, bib, "plain", first, second, tmp)
        txt = Path(txt_out).read_text(encoding="utf-8") if os.path.exists(txt_out) else ""
        # in-text format
        d = Document(docx)
        sup = sum(1 for p in d.paragraphs for r in p.runs
                  if r.font.superscript and re.search(r"\d", r.text))
    body = txt.split("References")[0] if "References" in txt else txt
    intext = ("superscript" if sup > 0
              else "bracket" if re.search(r"\[\d", body)
              else "paren" if re.search(r"\(\d", body)
              else "unknown")
    doi = 1 if re.search(r"doi|10\.\d{4}/", txt, re.I) else 0
    # crude abbrev check: presence of a long journal word vs none
    full = bool(re.search(r"\b(Annals|Journal of|American Journal|European|Radiology\.)", txt))
    return {"intext": intext, "doi": doi, "abbrev_full_detected": full,
            "superscript_runs": sup}

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--csl", required=True)
    ap.add_argument("--bib", required=True)
    ap.add_argument("--journal", help="spec key (jkms, radiology, ...)")
    ap.add_argument("--expect-intext", choices=["superscript", "bracket", "paren"])
    ap.add_argument("--expect-doi", type=int, choices=[0, 1])
    ap.add_argument("--expect-abbrev", choices=["yes", "no"])
    a = ap.parse_args()
    exp = dict(SPECS.get(a.journal, {})) if a.journal else {}
    if a.expect_intext: exp["intext"] = a.expect_intext
    if a.expect_doi is not None: exp["doi"] = a.expect_doi
    if a.expect_abbrev: exp["abbrev"] = a.expect_abbrev
    try:
        got = analyze(a.csl, a.bib)
    except RenderError as exc:
        print(f"ERROR: {exc}", file=sys.stderr)
        sys.exit(2)
    print(json.dumps({"detector": "check_csl_render", "csl": os.path.basename(a.csl), "expected": exp, "got": got}, indent=2))
    fails = []
    if exp.get("intext") and got["intext"] != exp["intext"]:
        fails.append(f"in-text {got['intext']} != expected {exp['intext']}")
    if "doi" in exp and got["doi"] != exp["doi"]:
        fails.append(f"DOI {got['doi']} != expected {exp['doi']}")
    if exp.get("abbrev") == "yes" and got["abbrev_full_detected"]:
        fails.append("journal names appear FULL — need NLM abbreviation "
                     "(fill_journal_abbrev.py to add shortjournal)")
    if fails:
        print("FAIL:", "; ".join(fails), file=sys.stderr)
        sys.exit(1)
    print("PASS — CSL output matches journal spec")

if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""Duplicate-bibliography gate for a built manuscript (manage-refs / sync-submission).

A manuscript whose markdown carries BOTH inline `[@key]` citations AND a hand-typed
`## References` numbered list will, when built with pandoc `--citeproc`, render TWO
reference lists: the literal hand-typed one plus a citeproc-generated bibliography
appended at the end (often after the figure/table legends, where it is easy to
miss). A reviewer reads it as "the same reference is listed twice." A cross-
reference QC pass (check_xref) does not catch it because each entry is individually
valid; only a duplicate-list scan does.

This detector reads the BUILT artifact (docx via stdlib zipfile, or a rendered
md/txt) and fires when the reference list is duplicated. Signals, any of which is
load-bearing:

  DUP_REF_HEADING     two or more reference-section headings
                      (References / Bibliography / Works Cited).
  REF_NUMBER_RESTART  a numbered reference-entry sequence that restarts
                      (entry "1." appears two or more times) — the signature of a
                      second list concatenated after the first.
  REF_SIGNATURE_DUP   the (first-author surname, year) signature of >= 3 distinct
                      references each appears two or more times — a whole list
                      repeated, not a coincidental same-author-same-year pair.

A single duplicated (surname, year) with no other signal is reported as a
DUP_REF_ENTRY flag (Minor: verify — could be two distinct same-year papers by one
author), never a Major on its own.

Motivation: a built submission docx rendered the 15-entry hand-typed reference list
and then a second 15-entry citeproc bibliography after the legends; a co-author
flagged "a reference is repeated twice." See
~/.claude/rules/manuscript-references.md (Hybrid hand-list + citeproc section).

INPUTS  (one of)
  --docx PATH    built .docx (read via stdlib zipfile; word/document.xml)
  --text PATH    rendered markdown / plain text

OUTPUT  reconciliation table (stdout) + optional JSON:
  {source, ref_entries, headings, claims[{verdict, severity, detail, where}], summary}

Stdlib-only (re / json / zipfile / argparse / pathlib). Exit codes: 0 clean/
report-only, 1 Major with --strict, 2 input/usage error.
"""
from __future__ import annotations

import argparse
import json
import re
import sys
import zipfile
from pathlib import Path

HEADING_RE = re.compile(
    r"^\s*#{0,6}\s*\**\s*(references|reference list|bibliography|works cited|literature cited)\s*\**\s*:?\s*$",
    re.I,
)
# A reference entry begins with an author "Surname I" token (capitalized surname
# followed by an uppercase initial), optionally preceded by a list number. This
# does NOT depend on a text-level number, because Word auto-numbered lists keep
# the number in list formatting (numPr), not in the paragraph text — so a hand
# list and a citeproc list render entries that both start at the surname.
ENTRY_RE = re.compile(
    r"^\s*(?P<num>\d{1,3})?\s*[.\)]?\s*"
    r"(?P<surname>[A-Z][A-Za-zÀ-ɏ'\-]{1,})\s*,?\s+[A-Z]"
)
YEAR_RE = re.compile(r"\b(19|20)\d{2}\b")
SURNAME_RE = re.compile(r"[A-Z][A-Za-zÀ-ɏ'\-]+")
DOI_RE = re.compile(r"10\.\d{4,9}/[^\s\"'<>]+", re.I)


def _docx_paragraphs(path: Path) -> list[str]:
    try:
        with zipfile.ZipFile(path, "r") as z:
            xml = z.read("word/document.xml").decode("utf-8", errors="replace")
    except (zipfile.BadZipFile, KeyError, OSError):
        return []
    # Paragraph boundary = </w:p>. Strip remaining tags inside each paragraph.
    paras = re.split(r"</w:p>", xml)
    out = []
    for p in paras:
        txt = re.sub(r"<[^>]+>", "", p)
        txt = re.sub(r"\s+", " ", txt).strip()
        if txt:
            out.append(txt)
    return out


def _text_lines(path: Path) -> list[str]:
    raw = path.read_text(encoding="utf-8", errors="replace")
    return [ln.strip() for ln in raw.splitlines() if ln.strip()]


def analyze(lines: list[str], source: str) -> dict:
    headings = [i for i, ln in enumerate(lines) if HEADING_RE.match(ln)]

    # Reference entries: numbered lines whose tail looks like a citation (has a
    # 4-digit year somewhere). Restrict to lines that begin with a small integer
    # and contain a capitalized author-like token to avoid eating body numerals.
    entries = []
    for i, ln in enumerate(lines):
        m = ENTRY_RE.match(ln)
        if not m:
            continue
        if not YEAR_RE.search(ln):
            continue
        num = int(m.group("num")) if m.group("num") else None
        surname = m.group("surname").lower()
        year = YEAR_RE.search(ln).group(0)
        dm = DOI_RE.search(ln)
        entries.append({"line": i, "num": num, "tail": ln,
                        "sig": (surname, year),
                        "doi": dm.group(0).lower() if dm else None})

    claims = []

    # Signal 1: duplicate reference-section heading.
    if len(headings) >= 2:
        claims.append({
            "verdict": "DUP_REF_HEADING", "severity": "Major",
            "detail": f"{len(headings)} reference-section headings found "
                      f"(lines {', '.join(str(h + 1) for h in headings)}); a built "
                      f"manuscript should have exactly one bibliography.",
            "where": ", ".join(str(h + 1) for h in headings),
        })

    # Signal 2: numbered sequence restarts (entry '1' appears >= 2 times).
    ones = [e for e in entries if e["num"] == 1]
    if len(ones) >= 2 and len(entries) >= 4:
        claims.append({
            "verdict": "REF_NUMBER_RESTART", "severity": "Major",
            "detail": f"numbered reference list restarts: entry '1.' appears "
                      f"{len(ones)} times (lines {', '.join(str(o['line'] + 1) for o in ones)}) "
                      f"across {len(entries)} numbered entries — a second list is "
                      f"concatenated after the first.",
            "where": ", ".join(str(o["line"] + 1) for o in ones),
        })

    # Signal 3: (surname, year) signature of >= 3 distinct refs each duplicated.
    sigs = [e["sig"] for e in entries if e["sig"]]
    counts: dict = {}
    for s in sigs:
        counts[s] = counts.get(s, 0) + 1
    dup_sigs = [s for s, c in counts.items() if c >= 2]
    # DOI duplicates reinforce the signal.
    dois = [e["doi"] for e in entries if e["doi"]]
    dup_dois = {d for d in dois if dois.count(d) >= 2}

    if len(dup_sigs) >= 3:
        ex = ", ".join(f"{s[0].title()} {s[1]}" for s in dup_sigs[:4])
        claims.append({
            "verdict": "REF_SIGNATURE_DUP", "severity": "Major",
            "detail": f"{len(dup_sigs)} distinct references each appear >=2x in the "
                      f"reference list (e.g. {ex}{'…' if len(dup_sigs) > 4 else ''})"
                      + (f"; {len(dup_dois)} DOI(s) also duplicated" if dup_dois else "")
                      + " — the bibliography is duplicated.",
            "where": f"{len(entries)} numbered entries",
        })
    elif dup_sigs:
        ex = ", ".join(f"{s[0].title()} {s[1]}" for s in dup_sigs)
        claims.append({
            "verdict": "DUP_REF_ENTRY", "severity": "Minor",
            "detail": f"{len(dup_sigs)} reference signature(s) duplicated ({ex}). "
                      f"Verify these are not two distinct same-year papers by the "
                      f"same first author.",
            "where": ex,
        })

    n_major = sum(1 for c in claims if c["severity"] == "Major")
    return {
        "source": source,
        "ref_entries": len(entries),
        "headings": len(headings),
        "claims": claims,
        "summary": {
            "n_claims": len(claims),
            "n_major": n_major,
            "verdict": "MAJOR_CANDIDATE" if n_major else "OK",
        },
    }


def render(result: dict) -> str:
    lines = ["| Check | Severity | Detail |", "|---|---|---|"]
    for c in result["claims"]:
        lines.append(f"| {c['verdict']} | {c['severity']} | {c['detail']} |")
    if len(lines) == 2:
        lines.append("| (none) | — | single reference list; no duplication detected |")
    return "\n".join(lines)


def main() -> int:
    ap = argparse.ArgumentParser(
        description="Duplicate-bibliography gate for a built manuscript.")
    ap.add_argument("--docx", help="built .docx (read via stdlib zipfile)")
    ap.add_argument("--text", help="rendered markdown / plain text")
    ap.add_argument("--out", help="write JSON artifact to this path")
    ap.add_argument("--strict", action="store_true", help="exit 1 if any Major finding")
    ap.add_argument("--quiet", action="store_true", help="suppress stdout table")
    args = ap.parse_args()

    if not args.docx and not args.text:
        sys.stderr.write("ERROR: one of --docx / --text is required\n")
        return 2
    if args.docx:
        p = Path(args.docx)
        if not p.is_file():
            sys.stderr.write(f"ERROR: not a file: {p}\n")
            return 2
        lines = _docx_paragraphs(p)
        source = str(p)
    else:
        p = Path(args.text)
        if not p.is_file():
            sys.stderr.write(f"ERROR: not a file: {p}\n")
            return 2
        lines = _text_lines(p)
        source = str(p)

    result = analyze(lines, source)

    if not args.quiet:
        print("=" * 42)
        print(" Reference Duplication (build-time gate)")
        print("=" * 42)
        print(render(result))
        print()
        s = result["summary"]
        if s["n_major"]:
            print(f"MAJOR candidate: bibliography appears duplicated "
                  f"({result['ref_entries']} numbered entries, {result['headings']} headings).")
        else:
            print(f"OK: single reference list "
                  f"({result['ref_entries']} numbered entries, {result['headings']} headings).")

    if args.out:
        Path(args.out).parent.mkdir(parents=True, exist_ok=True)
        Path(args.out).write_text(json.dumps({"detector": "check_reference_duplication", **result}, indent=2), encoding="utf-8")
        if not args.quiet:
            print(f"\nwrote {args.out}")

    return 1 if (args.strict and result["summary"]["n_major"]) else 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
"""check_xref.py — Manuscript ↔ rendered DOCX cross-reference QC.

Catches the failure mode where in-text references to Tables / Figures /
Supplementary Tables / Supplementary Figures point to labels that either
(a) do not exist in the rendered DOCX, (b) have no caption definition in
the manuscript body, or (c) carry a caption text in the rendered DOCX
that disagrees with the body's caption definition.

The failure it catches: the body cites a supplementary table as a
sensitivity analysis while the rendered DOCX carries a diagnostics table
under that number, further numbers mismatch the same way, and some are
cited but absent from the DOCX. Internal consistency checks do not catch
this because the build script carries its own legacy SSOT.

Inputs
------
  --md PATH         manuscript markdown (the in-text citation source)
  --docx PATH       rendered DOCX (optional but recommended)
  --out PATH        JSON audit (default: qc/xref_audit.json)
  --strict          exit 1 on any non-OK finding (submission gate)
  --quiet           suppress stdout summary table
  --vN-docx-md5 PATH
                    v_N (previous version) docx path. When supplied:
                    (a) asserts the new --docx MD5 differs from this docx
                        (identity = unmodified seed copy);
                    (b) when --vN-md is also supplied, computes the
                        markdown-only diff and verifies each diff line
                        appears verbatim in the new docx body XML.
  --vN-md PATH      v_N markdown path (used by --vN-docx-md5 diff check).

v_(N+1) docx regeneration check
-------------------------------
Defense-in-depth at build time (complements verify_package_integrity.py
--assert-vN-docx-changed which runs at submission time). If a markdown
body change exists between v_N and v_(N+1) but the v_(N+1) docx is a
byte-identical copy of v_N, the change will silently revert at peer
review. The check fails fast at the QC stage.

Output
------
  qc/xref_audit.json with submission_safe boolean and per-label rows.
  When --vN-docx-md5 is used, the JSON also carries a `vN_docx_check`
  block with `identical_bytes` and `diff_line_misses` fields.

Exit codes
----------
  0  all OK (or non-strict and only warnings)
  1  --strict and at least one non-OK finding
     OR v_N docx identity / diff-miss assertion failure
  2  argument / IO error

Dependencies
------------
  python-docx (only if --docx is passed). Falls back to body-only audit
  with a warning if python-docx is unavailable.
"""
from __future__ import annotations

import argparse
import json
import re
import sys
from dataclasses import dataclass, field, asdict
from pathlib import Path
from typing import Optional


# ---------------------------------------------------------------------------
# Regex
# ---------------------------------------------------------------------------

# In-text citation token. Captures:
#   group 1: "Supplementary " | "Supp " | "" (supplementary marker)
#   group 2: Table | Figure
#   group 3: number, possibly S-prefixed, with optional letter suffix (e.g. S4, 2A)
CITE_RE = re.compile(
    r"(?<![A-Za-z])(Supplementary\s+|Supp\s+|Supp\.\s+|S\.\s*)?"
    r"(Table|Figure|Fig\.|Fig)\s+"
    r"(S?\d+[A-Za-z]?)",
    re.IGNORECASE,
)

# The same mention written the way people actually write it: a PLURAL kind word carrying a list of
# numbers — "Figures 1 and 2", "Tables 1-3", "Figures 2, 4 and 5". CITE_RE cannot see any of these
# (the plural "s" breaks its `\s+`), and the consequence was not a missed note but a DISABLED GATE:
# a float cited only this way scored UNCITED rather than MISSING_DOCX, and UNCITED is not in
# `blocking_statuses`. Two manuscripts identical in meaning got opposite verdicts —
#   "Figures 1 and 2"       -> exit 0, submission_safe
#   "Figure 1 and Figure 2" -> exit 1, SUBMISSION BLOCKED
# — so writing ordinary English turned the blocker off, while the repetitive form this tool's own
# examples happen to use kept it on.
CITE_LIST_RE = re.compile(
    r"(?<![A-Za-z])(Supplementary\s+|Supp\s+|Supp\.\s+|S\.\s*)?"
    r"(Tables|Figures|Figs\.|Figs)\s+"
    r"(S?\d+[A-Za-z]?(?:\s*(?:,|&|–|—|-|and|to|through)\s*S?\d+[A-Za-z]?)*)",
    re.IGNORECASE,
)

# One token inside a numlist, with an optional "A-B" range tail. A range must be EXPANDED: reading
# "Figures 1-3" as {1, 3} silently drops Figure 2 — the same missing-blocker outcome, one number in.
# The separators must match what CITE_LIST_RE accepts as a JOIN, or a form it lets through gets read
# as two endpoints and the interior is dropped: "Figures 3 to 5" scoring {3, 5} loses Figure 4 exactly
# as an unexpanded "3-5" would.
CITE_TOKEN_RE = re.compile(
    r"(S?)(\d+)([A-Za-z]?)(?:\s*(?:[–—-]|to|through)\s*(S?)(\d+)([A-Za-z]?))?", re.IGNORECASE)

# Caption definition (start of line, optional bold markdown). Matches:
#   Table 1. Caption text...
#   **Table 1.** Caption...
#   Supplementary Table S4. Caption...
CAPTION_RE = re.compile(
    r"^\s*(?:\*\*)?\s*"
    r"(Supplementary\s+|Supp\s+|Supp\.\s+)?"
    r"(Table|Figure|Fig\.|Fig)\s+"
    r"(S?\d+[A-Za-z]?)"
    r"\s*[.:]\s*(?:\*\*)?\s*"
    r"(.+?)\s*$",
    re.IGNORECASE | re.MULTILINE,
)

# Section header patterns for the manuscript body's caption sections.
# Longer alternatives MUST come first so that "FIGURE LEGENDS" is not
# truncated to "FIGURE" by an earlier short match.
# Markdown bold wrappers (``## **FIGURE LEGENDS**``) are tolerated.
CAPTION_SECTION_RE = re.compile(
    r"^#{1,3}\s+\*{0,2}"
    r"(Supplementary\s+Tables?|Supplementary\s+Figures?|Supplementary\s+Materials?|"
    r"Figure\s+Legends?|Table\s+(?:Captions?|Legends?)|"
    r"Tables?|Figures?)"
    r"\*{0,2}",
    re.IGNORECASE | re.MULTILINE,
)


# ---------------------------------------------------------------------------
# Data classes
# ---------------------------------------------------------------------------

@dataclass
class Label:
    """Canonical (kind, supplementary, number) tuple for a Table/Figure."""
    kind: str           # "Table" | "Figure"
    supplementary: bool
    number: str         # "1", "S4", "2A"

    @property
    def key(self) -> str:
        prefix = "S-" if self.supplementary else ""
        return f"{self.kind}:{prefix}{self.number}"

    @property
    def display(self) -> str:
        supp = "Supplementary " if self.supplementary else ""
        return f"{supp}{self.kind} {self.number}"


@dataclass
class Caption:
    label: Label
    text: str
    source: str  # "body" | "docx"


@dataclass
class Finding:
    label: str
    status: str          # OK | MISSING_DOCX | MISSING_BODY | MISMATCH | UNCITED | NOT_CITED_NO_BODY
    cited: bool
    in_body: bool
    in_docx: Optional[bool]   # None if --docx not provided
    body_caption: Optional[str]
    docx_caption: Optional[str]
    note: str = ""


# ---------------------------------------------------------------------------
# Parsing
# ---------------------------------------------------------------------------

def _normalize(kind_raw: str, supp_raw: Optional[str], number: str) -> Label:
    kind = "Figure" if kind_raw.lower().startswith("fig") else "Table"
    supplementary = bool(supp_raw) or number.upper().startswith("S")
    # Normalize number capitalization (S4a -> S4A, lowercase letter suffix uppercase)
    if len(number) > 1 and number[-1].isalpha():
        number = number[:-1].upper() + number[-1].upper()
    else:
        number = number.upper() if number.upper().startswith("S") else number
    return Label(kind=kind, supplementary=supplementary, number=number)


def extract_citations(md_text: str, body_caption_offsets: list[tuple[int, int]]) -> list[Label]:
    """Extract in-text citations excluding ranges that are caption sections."""
    # Mask caption sections so caption first lines don't double as citations
    masked = list(md_text)
    for start, end in body_caption_offsets:
        for i in range(start, min(end, len(masked))):
            masked[i] = " "
    text = "".join(masked)

    # Strip fenced code
    text = re.sub(r"```.*?```", "", text, flags=re.DOTALL)
    text = re.sub(r"`[^`\n]+`", "", text)

    labels: list[Label] = []
    for m in CITE_RE.finditer(text):
        labels.append(_normalize(m.group(2), m.group(1), m.group(3)))
    # Plural mentions carrying a numlist. Singular "Figure" is a prefix of plural "Figures", so the
    # two patterns cannot both match the same span and no citation is double-counted.
    for m in CITE_LIST_RE.finditer(text):
        supp, kind, numlist = m.group(1), m.group(2), m.group(3)
        # Normalise the plural kind word to the singular the label vocabulary uses.
        singular = "Table" if kind.lower().startswith("table") else "Figure"
        for supp_a, num_a, suf_a, supp_b, num_b, suf_b in CITE_TOKEN_RE.findall(numlist):
            if num_b and not suf_a and not suf_b and int(num_b) >= int(num_a):
                # A true range: expand it. A lettered endpoint ("2A-2C") is a panel range, not a
                # float range, so it is left to its endpoints rather than invented over.
                for n in range(int(num_a), int(num_b) + 1):
                    labels.append(_normalize(singular, supp, f"{supp_a}{n}"))
            else:
                labels.append(_normalize(singular, supp, f"{supp_a}{num_a}{suf_a}"))
                if num_b:
                    labels.append(_normalize(singular, supp, f"{supp_b}{num_b}{suf_b}"))
    return labels


def find_caption_section_ranges(md_text: str) -> list[tuple[int, int]]:
    """Return (start, end) byte offsets for body caption sections."""
    headers = list(CAPTION_SECTION_RE.finditer(md_text))
    ranges: list[tuple[int, int]] = []
    for i, h in enumerate(headers):
        start = h.start()
        end = headers[i + 1].start() if i + 1 < len(headers) else len(md_text)
        # If next non-caption section header appears (## something else), cut there
        # Find next ^#{1,3}\s heading after start that is NOT a caption section
        next_section = re.search(r"\n#{1,3}\s+\S", md_text[h.end():end])
        if next_section:
            tentative_end = h.end() + next_section.start()
            # Only cut if the next heading is not itself a caption section
            after = md_text[h.end() + next_section.start():end]
            if not CAPTION_SECTION_RE.match(after):
                end = tentative_end
        ranges.append((start, end))
    return ranges


def extract_body_captions(md_text: str) -> dict[str, Caption]:
    """Extract caption definitions from the manuscript body's caption sections."""
    ranges = find_caption_section_ranges(md_text)
    captions: dict[str, Caption] = {}
    for start, end in ranges:
        chunk = md_text[start:end]
        for m in CAPTION_RE.finditer(chunk):
            label = _normalize(m.group(2), m.group(1), m.group(3))
            text = m.group(4).strip().rstrip("*").strip()
            # Keep first definition per label (later ones likely continuation lines)
            if label.key not in captions:
                captions[label.key] = Caption(label=label, text=text, source="body")
    return captions


def extract_docx_captions(docx_path: Path) -> dict[str, Caption]:
    """Extract caption paragraphs from a rendered DOCX using python-docx."""
    try:
        from docx import Document  # type: ignore
    except ImportError:
        print(
            "[check_xref] WARNING: python-docx not installed; "
            "skipping rendered-DOCX audit. Install with: pip install python-docx",
            file=sys.stderr,
        )
        return {}

    doc = Document(str(docx_path))
    captions: dict[str, Caption] = {}
    for para in doc.paragraphs:
        text = para.text.strip()
        if not text:
            continue
        m = CAPTION_RE.match(text)
        if not m:
            continue
        label = _normalize(m.group(2), m.group(1), m.group(3))
        caption_text = m.group(4).strip()
        if label.key not in captions:
            captions[label.key] = Caption(label=label, text=caption_text, source="docx")

    # Also scan tables (Word can put captions in adjacent paragraphs that are inside cells
    # or before the table). The paragraph scan above is usually sufficient.
    return captions


# ---------------------------------------------------------------------------
# Reconciliation
# ---------------------------------------------------------------------------

def _tokens(text: str) -> set[str]:
    return set(re.findall(r"[A-Za-z][A-Za-z0-9]+", text.lower()))


def caption_agreement(a: str, b: str, threshold: float = 0.4) -> tuple[bool, float]:
    """Heuristic agreement: Jaccard token overlap >= threshold = agree."""
    ta, tb = _tokens(a), _tokens(b)
    if not ta or not tb:
        return False, 0.0
    inter = len(ta & tb)
    union = len(ta | tb)
    j = inter / union if union else 0.0
    return j >= threshold, j


def reconcile(
    citations: list[Label],
    body: dict[str, Caption],
    docx: Optional[dict[str, Caption]],
) -> list[Finding]:
    cited_keys = {lbl.key for lbl in citations}
    body_keys = set(body.keys())
    docx_keys = set(docx.keys()) if docx is not None else set()
    all_keys = cited_keys | body_keys | docx_keys

    findings: list[Finding] = []
    for key in sorted(all_keys, key=_sort_key):
        is_cited = key in cited_keys
        in_body = key in body_keys
        in_docx = (key in docx_keys) if docx is not None else None

        body_text = body[key].text if in_body else None
        docx_text = docx[key].text if (docx is not None and key in docx_keys) else None

        # Panel-letter fallback: "Figure 2A" cite resolves to "Figure 2" caption.
        panel_note = ""
        if is_cited and not in_body:
            base = _strip_panel(key)
            if base and base in body_keys:
                in_body = True
                body_text = body[base].text
                panel_note = f"panel reference; resolved to {base.replace(':', ' ')}"
        if is_cited and in_docx is False:
            base = _strip_panel(key)
            if base and base in docx_keys:
                in_docx = True
                docx_text = docx[base].text  # type: ignore[index]
                panel_note = (panel_note + "; " if panel_note else "") + \
                    f"panel reference resolved to {base.replace(':', ' ')} in DOCX"

        status, note = _classify(is_cited, in_body, in_docx, body_text, docx_text)
        if panel_note and status == "OK":
            note = panel_note

        findings.append(Finding(
            label=key,
            status=status,
            cited=is_cited,
            in_body=in_body,
            in_docx=in_docx,
            body_caption=body_text,
            docx_caption=docx_text,
            note=note,
        ))
    return findings


def _strip_panel(key: str) -> Optional[str]:
    """Map 'Figure:2A' -> 'Figure:2' (panel letter strip). Returns None if no change."""
    kind, num = key.split(":", 1)
    m = re.match(r"^(S?)(\d+)([A-Z])$", num)
    if m and m.group(3):
        return f"{kind}:{m.group(1)}{m.group(2)}"
    return None


def _classify(
    cited: bool,
    in_body: bool,
    in_docx: Optional[bool],
    body_text: Optional[str],
    docx_text: Optional[str],
) -> tuple[str, str]:
    if not cited:
        if in_body or in_docx:
            return "UNCITED", "defined or rendered but never cited in main text"
        return "NOT_CITED_NO_BODY", ""

    # Cited
    if in_docx is False:
        return "MISSING_DOCX", "cited but absent from rendered DOCX"
    if not in_body and in_docx is None:
        return "MISSING_BODY", "cited but no caption definition in markdown body sections"
    if not in_body and in_docx:
        return "MISSING_BODY", "cited and rendered, but no body caption definition (build SSOT drift risk)"

    # Both body and docx (or body only when in_docx is None)
    if body_text and docx_text:
        agree, j = caption_agreement(body_text, docx_text)
        if not agree:
            return "MISMATCH", f"caption text disagrees (Jaccard={j:.2f})"
    return "OK", ""


def _sort_key(key: str) -> tuple:
    # "Table:S4" -> ("Table", 1, 4) ; "Figure:2A" -> ("Figure", 0, 2, "A")
    kind, num = key.split(":", 1)
    is_supp = 1 if num.startswith("S") else 0
    digits = num.lstrip("S")
    m = re.match(r"^(\d+)([A-Z]?)$", digits)
    if m:
        return (kind, is_supp, int(m.group(1)), m.group(2))
    return (kind, is_supp, 999, digits)


# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------

def render_summary(findings: list[Finding], cited_count: int) -> str:
    counts: dict[str, int] = {}
    for f in findings:
        counts[f.status] = counts.get(f.status, 0) + 1

    lines = []
    lines.append(f"\n[check_xref] in-text citations: {cited_count}, unique labels: {len(findings)}")
    lines.append("Status summary: " + ", ".join(f"{k}={v}" for k, v in sorted(counts.items())))
    lines.append("")
    lines.append(f"  {'LABEL':<14} {'STATUS':<18} {'CITED':<6} {'BODY':<5} {'DOCX':<5} NOTE")
    for f in findings:
        docx_mark = "—" if f.in_docx is None else ("✓" if f.in_docx else "✗")
        lines.append(
            f"  {f.label:<14} {f.status:<18} "
            f"{'✓' if f.cited else '✗':<6} "
            f"{'✓' if f.in_body else '✗':<5} "
            f"{docx_mark:<5} {f.note}"
        )
    return "\n".join(lines)


def _md5_of(path: Path) -> str:
    import hashlib
    h = hashlib.md5()
    with path.open("rb") as fh:
        for chunk in iter(lambda: fh.read(65536), b""):
            h.update(chunk)
    return h.hexdigest()


def _docx_body_text(docx_path: Path) -> str:
    """Concatenate all w:t text content from a docx for substring search."""
    import zipfile
    try:
        with zipfile.ZipFile(docx_path, "r") as z:
            xml = z.read("word/document.xml").decode("utf-8", errors="replace")
    except (zipfile.BadZipFile, KeyError, OSError):
        return ""
    # Strip XML tags — we only need text content for verbatim grep.
    return re.sub(r"<[^>]+>", " ", xml)


def _markdown_diff_lines(vN_md: Path, new_md: Path) -> list[str]:
    """Lines present in new_md but not in vN_md (added/changed lines).

    Trimmed to non-trivial substrings (≥40 chars after stripping markdown
    metacharacters) so that the verbatim grep is meaningful.
    """
    old_lines = set(vN_md.read_text(encoding="utf-8").splitlines())
    out: list[str] = []
    for ln in new_md.read_text(encoding="utf-8").splitlines():
        if ln in old_lines:
            continue
        # Strip leading markdown noise (headers, list markers) and YAML.
        clean = re.sub(r"^[\s#>*\-_+~`|]+", "", ln).strip()
        if len(clean) >= 40:
            out.append(clean)
    return out


def run_vN_docx_check(
    new_docx: Path,
    vN_docx: Path,
    new_md: Path | None,
    vN_md: Path | None,
) -> dict:
    """Returns {identical_bytes, diff_line_misses, error?}.

    identical_bytes is True if v_N and new docx have the same MD5.
    diff_line_misses lists v_N→v_(N+1) markdown additions that do NOT
    appear in the new docx body XML.
    """
    out: dict = {
        "vN_docx": str(vN_docx),
        "new_docx": str(new_docx),
        "identical_bytes": False,
        "diff_line_misses": [],
    }
    if not vN_docx.is_file():
        out["error"] = f"v_N docx not found: {vN_docx}"
        return out
    if not new_docx.is_file():
        out["error"] = f"new docx not found: {new_docx}"
        return out
    if _md5_of(vN_docx) == _md5_of(new_docx):
        out["identical_bytes"] = True
        return out  # Identity already disqualifies — no point checking diff.

    if new_md is not None and vN_md is not None:
        if not new_md.is_file() or not vN_md.is_file():
            out["error"] = "v_N md or new md not found"
            return out
        diff_lines = _markdown_diff_lines(vN_md, new_md)
        body_text = _docx_body_text(new_docx)
        # Light normalization: collapse runs of whitespace.
        body_norm = re.sub(r"\s+", " ", body_text).lower()
        misses: list[str] = []
        for diff in diff_lines:
            needle = re.sub(r"\s+", " ", diff).lower()
            if needle not in body_norm:
                misses.append(diff[:120])
        out["diff_line_misses"] = misses
    return out


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    parser.add_argument("--md", required=True, type=Path, help="manuscript.md path")
    parser.add_argument("--docx", type=Path, default=None, help="rendered DOCX path (optional)")
    parser.add_argument("--out", type=Path, default=Path("qc/xref_audit.json"))
    parser.add_argument("--strict", action="store_true", help="exit 1 on any non-OK finding")
    parser.add_argument("--quiet", action="store_true")
    parser.add_argument(
        "--allow-separate-attachments", action="store_true",
        help="Declare that some figures/tables are submitted as separate attachment "
             "files rather than inline (the norm in radiology and most medical "
             "journals). Downgrades two things from FAIL to WARN: MISSING_DOCX, where "
             "a supplied --docx proved the float is not in the rendered main document; "
             "and MISSING_BODY where NO --docx was supplied, so nothing was checked and "
             "the float is excused on your word. Those two are reported separately "
             "because only the first has evidence behind it. Still FAIL regardless: "
             "MISMATCH, and MISSING_BODY where the float IS in the rendered DOCX but "
             "nothing in the markdown defines its caption — that is SSOT drift, which "
             "no attachment policy makes acceptable. Run with --docx before submitting.",
    )
    parser.add_argument(
        "--vN-docx-md5",
        dest="vN_docx_md5",
        type=Path,
        default=None,
        help=(
            "v_N docx path. Asserts the new --docx MD5 != this docx. "
            "Identity = unmodified seed copy = FAIL."
        ),
    )
    parser.add_argument(
        "--vN-md",
        dest="vN_md",
        type=Path,
        default=None,
        help=(
            "v_N manuscript markdown path (companion to --vN-docx-md5). "
            "When supplied, every line added in v_(N+1) markdown must "
            "appear verbatim in the new docx body XML; missing diff "
            "lines fail."
        ),
    )
    args = parser.parse_args()

    if not args.md.exists():
        print(f"ERROR: markdown not found: {args.md}", file=sys.stderr)
        return 2

    md_text = args.md.read_text(encoding="utf-8")
    section_ranges = find_caption_section_ranges(md_text)
    citations = extract_citations(md_text, section_ranges)
    body_captions = extract_body_captions(md_text)

    docx_captions: Optional[dict[str, Caption]] = None
    if args.docx is not None:
        if not args.docx.exists():
            print(f"ERROR: docx not found: {args.docx}", file=sys.stderr)
            return 2
        docx_captions = extract_docx_captions(args.docx)

    findings = reconcile(citations, body_captions, docx_captions)

    # Submission safety: any cited label whose status is not OK or UNCITED is a blocker.
    #
    # --allow-separate-attachments is a declaration by the operator: "some of this
    # manuscript's floats are submitted as separate attachment files, not inline."
    # That is the normal packaging for radiology and most medical journals. Under it,
    # two situations are downgraded to warnings — and they are NOT equally well
    # evidenced, so they are reported apart:
    #
    #   MISSING_DOCX                    a DOCX was supplied and PROVED the float is not
    #                                   in the rendered main document. Evidence exists.
    #   MISSING_BODY with in_docx None  no DOCX was supplied, so nothing was checked.
    #                                   The float may equally be a caption someone
    #                                   forgot to write. Excused on the operator's word.
    #
    # MISSING_BODY with in_docx True stays a blocker under every policy: the float IS
    # rendered and nothing in the markdown defines its caption, so the build pipeline
    # is the only place that knows the text. That is SSOT drift, and no attachment
    # policy makes it acceptable. MISMATCH likewise.
    if args.allow_separate_attachments:
        blockers = [
            f for f in findings
            if f.status == "MISMATCH"
            or (f.status == "MISSING_BODY" and f.in_docx is True)
        ]
        proven_absent = [f for f in findings if f.status == "MISSING_DOCX"]
        unchecked = [
            f for f in findings
            if f.status == "MISSING_BODY" and f.in_docx is None
        ]
        warnings = proven_absent + unchecked
    else:
        blocking_statuses = {"MISSING_DOCX", "MISSING_BODY", "MISMATCH"}
        blockers = [f for f in findings if f.status in blocking_statuses]
        proven_absent = []
        unchecked = []
        warnings = []
    submission_safe = len(blockers) == 0

    vN_check: dict | None = None
    vN_check_failed = False
    if args.vN_docx_md5 is not None:
        if args.docx is None:
            print(
                "ERROR: --vN-docx-md5 requires --docx (the new manuscript docx).",
                file=sys.stderr,
            )
            return 2
        vN_check = run_vN_docx_check(
            new_docx=args.docx,
            vN_docx=args.vN_docx_md5,
            new_md=args.md,
            vN_md=args.vN_md,
        )
        if vN_check.get("error"):
            print(f"ERROR: vN docx check: {vN_check['error']}", file=sys.stderr)
            return 2
        if vN_check["identical_bytes"]:
            vN_check_failed = True
        if vN_check["diff_line_misses"]:
            vN_check_failed = True

    payload = {
        "version": "1.2",
        "manuscript": str(args.md),
        "docx": str(args.docx) if args.docx else None,
        "policy": {
            "allow_separate_attachments": bool(args.allow_separate_attachments),
        },
        "summary": {
            "total_in_text_citations": len(citations),
            "unique_labels": len(findings),
            "ok": sum(1 for f in findings if f.status == "OK"),
            "missing_docx": sum(1 for f in findings if f.status == "MISSING_DOCX"),
            "missing_body": sum(1 for f in findings if f.status == "MISSING_BODY"),
            "mismatch": sum(1 for f in findings if f.status == "MISMATCH"),
            "uncited": sum(1 for f in findings if f.status == "UNCITED"),
            "blockers": len(blockers),
            "warnings": len(warnings),
            # The two downgrades kept apart, because a consumer that treats them alike
            # cannot tell a float the DOCX proved absent from one nothing ever checked.
            "downgraded_proven_absent": len(proven_absent),
            "downgraded_unchecked": len(unchecked),
        },
        "downgraded_unchecked_labels": [f.label for f in unchecked],
        "submission_safe": submission_safe and not vN_check_failed,
        "findings": [asdict(f) for f in findings],
        "vN_docx_check": vN_check,
    }

    args.out.parent.mkdir(parents=True, exist_ok=True)
    args.out.write_text(json.dumps({"detector": "check_xref", **payload}, indent=2, ensure_ascii=False), encoding="utf-8")

    if not args.quiet:
        print(render_summary(findings, len(citations)))
        print(f"\n[check_xref] wrote {args.out}")
        # The two downgrades are printed apart because their evidence differs, and the
        # weaker one has to stay visible. Merging them into one count is how an excused
        # row starts reading like a checked one.
        if proven_absent:
            print(
                f"[check_xref] WARN: {len(proven_absent)} MISSING_DOCX row(s) downgraded under "
                f"--allow-separate-attachments: " + ", ".join(f.label for f in proven_absent) + "\n"
                f"           The DOCX was read and does not contain them, which is what a "
                f"separate attachment looks like."
            )
        if unchecked:
            print(
                f"[check_xref] WARN: {len(unchecked)} MISSING_BODY row(s) EXCUSED WITHOUT "
                f"EVIDENCE under --allow-separate-attachments:\n"
                f"           " + ", ".join(f.label for f in unchecked) + "\n"
                f"           No --docx was supplied, so nothing here was actually checked. Each "
                f"of these is either a float in a\n"
                f"           separate supplement file — which is what you declared — or a caption "
                f"nobody wrote. This run cannot\n"
                f"           tell them apart, and passed them on your word.\n"
                f"           Run again with --docx <rendered.docx> before submitting: a float "
                f"genuinely absent from the rendered\n"
                f"           output becomes MISSING_DOCX (still downgraded, but now proven), and "
                f"a caption you forgot becomes visible."
            )
        if not submission_safe:
            print(f"[check_xref] SUBMISSION BLOCKED: {len(blockers)} cross-reference defect(s).")
            drift = [f for f in blockers
                     if f.status == "MISSING_BODY" and f.in_docx is True]
            if drift and args.allow_separate_attachments:
                print(
                    f"[check_xref] NOTE: {len(drift)} of those are MISSING_BODY that "
                    f"--allow-separate-attachments does NOT excuse "
                    f"({', '.join(f.label for f in drift)}).\n"
                    f"           These floats ARE in the rendered DOCX while nothing in the "
                    f"markdown defines their caption, so the\n"
                    f"           build pipeline is the only place that knows the text. That is "
                    f"SSOT drift, not attachment style."
                )
        if vN_check is not None:
            if vN_check["identical_bytes"]:
                print(
                    "[check_xref] FAIL: v_(N+1) docx is byte-identical to "
                    "v_N docx — unmodified seed copy, regenerate via "
                    "pandoc / Zotero CWYW."
                )
            elif vN_check["diff_line_misses"]:
                print(
                    f"[check_xref] FAIL: {len(vN_check['diff_line_misses'])} "
                    "markdown diff line(s) absent from new docx body. "
                    "v_(N+1) docx body did not pick up the markdown edits."
                )
                for miss in vN_check["diff_line_misses"][:5]:
                    print(f"    - {miss}")

    block_exit = args.strict and not submission_safe
    if block_exit or vN_check_failed:
        return 1
    return 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
"""Inject NLM journal abbreviations (+ optionally normalize author case / titles)
into a BibLaTeX .bib from PubMed authoritative metadata.

Why: CSL `container-title form="short"` prints the full journal name unless the
.bib entry carries a `shortjournal` field. Most Zotero/BBT exports omit it, so
NLM-style journals (JKMS, AJR, KJR, NEJM, ...) render the full title regardless
of CSL. This script resolves each entry's DOI → PMID → PubMed esummary `source`
(the NLM Title Abbreviation) and writes `shortjournal = {...}`.

Also (optional, --titles): pulls efetch ArticleTitle to double-brace titles
(preserving proper-noun casing + subtitle), and flags ALL-CAPS author fields.

Authoritative source: PubMed (esummary `source` = NLM abbreviation; efetch
ArticleTitle = full title). Falls back to CrossRef `short-container-title` when a
DOI has no PubMed record. Never invents abbreviations.

Usage:
  python fill_journal_abbrev.py --bib refs.bib --out refs_nlm.bib            # shortjournal only
  python fill_journal_abbrev.py --bib refs.bib --out refs_nlm.bib --titles   # + title double-brace
"""
import argparse, re, json, time, urllib.request, urllib.parse, sys

EUTILS = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils"
UA = {"User-Agent": "medsci-manage-refs/1.0"}

def _get(url):
    with urllib.request.urlopen(urllib.request.Request(url, headers=UA), timeout=30) as r:
        return r.read().decode()

def doi_to_pmid(doi):
    try:
        term = urllib.parse.quote(f'"{doi}"[AID] OR "{doi}"[DOI]')
        j = json.loads(_get(f"{EUTILS}/esearch.fcgi?db=pubmed&term={term}&retmode=json"))
        ids = j.get("esearchresult", {}).get("idlist", [])
        return ids[0] if ids else None
    except Exception:
        return None

def pmids_summary(pmids):
    """return {pmid: {'source':abbrev, 'title':...}}"""
    if not pmids: return {}
    j = json.loads(_get(f"{EUTILS}/esummary.fcgi?db=pubmed&id={','.join(pmids)}&retmode=json"))
    res = j.get("result", {})
    return {p: {"source": res.get(p, {}).get("source")} for p in pmids}

def parse_entries(bib):
    # `finditer` yields Match objects, and every caller treats these as the entry TEXT. Returning the
    # matches meant `re.match(..., b)` received a Match where it wanted a string, and the script died
    # with `TypeError: expected string or bytes-like object, got 're.Match'` on the FIRST entry — so
    # it had never run at all, while `check_csl_render.py` names it to the user as the remedy for a
    # missing `shortjournal`.
    return [m.group(0) for m in re.finditer(r"@\w+\{[^@]*?\n\}", bib, re.S)]

def field(block, name):
    # The trailing comma is optional: BibTeX does not require one after an entry's LAST field, and
    # `doi` is very often exactly that. Requiring it returned None for those entries, which here
    # means the DOI is never resolved and the abbreviation is never filled — silently, on the
    # entries most likely to need it. Same defect class as the reference parser fixed in #445.
    m = re.search(rf"{name}\s*=\s*\{{(.+?)\}}\s*,?", block, re.S)
    return m.group(1).strip() if m else None

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--bib", required=True)
    ap.add_argument("--out", required=True)
    ap.add_argument("--titles", action="store_true", help="also double-brace titles via efetch")
    a = ap.parse_args()
    bib = open(a.bib, encoding="utf-8").read()

    # 1) collect DOIs → PMIDs
    blocks = parse_entries(bib)
    doi_by_key = {}
    for b in blocks:
        key = re.match(r"@\w+\{([^,]+),", b).group(1).strip()
        doi = field(b, "doi") or field(b, "DOI")
        if doi: doi_by_key[key] = doi.strip()
    pmid_by_key = {}
    for k, doi in doi_by_key.items():
        pmid_by_key[k] = doi_to_pmid(doi); time.sleep(0.4)
    pmids = [p for p in pmid_by_key.values() if p]
    summ = pmids_summary(pmids)

    # 2) inject shortjournal
    out = bib; n = 0
    for k, pmid in pmid_by_key.items():
        if not pmid: continue
        abbrev = summ.get(pmid, {}).get("source")
        if not abbrev: continue
        # find this entry, add shortjournal after journaltitle/journal if absent
        pat = re.compile(r"(@\w+\{" + re.escape(k) + r",.*?)(\n\})", re.S)
        m = pat.search(out)
        if not m or "shortjournal" in m.group(1): continue
        block = m.group(1)
        inj = re.sub(r"(\n\t?journal(?:title)?\s*=\s*\{[^}]*\},)",
                     r"\1\n\tshortjournal = {" + abbrev + "},", block, count=1)
        if inj == block:  # no journal field found; append before closing
            inj = block + "\n\tshortjournal = {" + abbrev + "},"
        out = out.replace(block, inj, 1); n += 1

    open(a.out, "w", encoding="utf-8").write(out)
    print(f"shortjournal injected: {n}/{len(doi_by_key)} entries "
          f"(DOIs resolved to PMID: {len(pmids)})")
    caps = [re.match(r'@\w+\{([^,]+),', b).group(1) for b in blocks
            if (au := field(b, "author")) and re.search(r"\{?[A-Z]{3,}\}?,\s*\{?[A-Z]{3,}", au or "")]
    if caps:
        print("WARN: ALL-CAPS author fields (fix case manually or re-pull):", caps)
    if a.titles:
        print("NOTE: --titles efetch double-brace not auto-applied; "
              "use the efetch ArticleTitle pattern documented in REFERENCE_STYLE_SPECS.md")

if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""inject_zotero_cwyw.py — Inject native Zotero CWYW field codes into a .docx.

Wraps the vendored ``_vendor_citation_writer.insert_citations`` and patches its
``zotero_to_csl_json`` to use Zotero's native ``?format=csljson`` endpoint, so
webpage / report / non-journal items map correctly (the upstream
_ITEM_TYPE_MAP fallback to ``"article"`` silently drops URL/accessDate fields).

Workflow:
  1. Document body must already use pandoc-style ``[@KEY]`` markers (use
     ``md_marker_convert.py`` to convert ``[N]`` first).
  2. Local Zotero must be running with the connector API at
     ``http://localhost:23119/api/users/<USER_ID>/items/<KEY>``.
  3. Output .docx contains live ``ADDIN ZOTERO_ITEM`` / ``ADDIN ZOTERO_BIBL``
     field codes. Open in Word → Zotero tab → **Add/Edit Bibliography** once
     to populate the bibliography (see Known limitation #1).

Usage:
  inject_zotero_cwyw.py --input markers.docx --output cwyw.docx \\
    --user-id 16613550 --keys ABC123,DEF456,...
  inject_zotero_cwyw.py --input markers.docx --output cwyw.docx \\
    --user-id 16613550 --keys-from keys.txt

Known limitations (carry-over from an active meta-analysis project validation, 2026-05-01):
  - First build: BIBL field is an empty stub. User must run "Add/Edit
    Bibliography" once in Word; subsequent Refresh keeps it in sync.
  - Surgical post-build patches (regex on ``[N]``) are unsafe — Zotero
    rendered superscripts can collide. For ref additions, regenerate the
    whole .docx from the markdown SSOT.

Anti-Hallucination:
  - Item metadata is fetched live from Zotero — never invented.
  - On any HTTP failure for any key, abort with a non-zero exit so partial
    builds with missing items never reach the user.
"""

from __future__ import annotations

import argparse
import hashlib
import json
import sys
import urllib.request
from pathlib import Path

HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(HERE))

import _vendor_citation_writer as _cw  # noqa: E402
from _vendor_citation_writer import insert_citations  # noqa: E402


def _native_zotero_to_csl_json(item_data: dict, user_id: str) -> dict:
    """Replacement for upstream ``zotero_to_csl_json`` — fetches Zotero's own
    CSL-JSON serialization which handles webpage / report / etc. correctly.
    """
    key = item_data.get("key", "")
    if not key:
        raise ValueError("item_data missing 'key' — cannot fetch CSL-JSON.")
    url = f"http://localhost:23119/api/users/{user_id}/items/{key}?format=csljson"
    with urllib.request.urlopen(url, timeout=10) as r:
        payload = json.loads(r.read())
    csl = payload[0] if isinstance(payload, list) else payload
    csl.pop("citation-key", None)
    csl["id"] = int(hashlib.md5(key.encode()).hexdigest()[:8], 16)
    csl["_uris"] = [f"http://zotero.org/users/{user_id}/items/{key}"]
    return csl


# Patch the vendored module's symbol — insert_citations() resolves via global
# lookup so the patched function is used.
_cw.zotero_to_csl_json = _native_zotero_to_csl_json


def fetch_item(user_id: str, key: str) -> dict:
    url = f"http://localhost:23119/api/users/{user_id}/items/{key}"
    with urllib.request.urlopen(url, timeout=10) as r:
        full = json.loads(r.read())
    return full["data"]


def load_keys(args) -> list[str]:
    if args.keys:
        return [k.strip() for k in args.keys.split(",") if k.strip()]
    if args.keys_from:
        text = Path(args.keys_from).read_text(encoding="utf-8")
        return [line.strip() for line in text.splitlines()
                if line.strip() and not line.strip().startswith("#")]
    sys.exit("ERROR: provide --keys or --keys-from.")


def main() -> int:
    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    ap.add_argument("--input", required=True, type=Path,
                    help=".docx with [@KEY] markers in body text.")
    ap.add_argument("--output", required=True, type=Path,
                    help="Output .docx with CWYW field codes.")
    ap.add_argument("--user-id", required=True,
                    help="Zotero numeric user ID (see Settings → Sync).")
    src = ap.add_mutually_exclusive_group()
    src.add_argument("--keys", help="Comma-separated Zotero item keys.")
    src.add_argument("--keys-from", help="File with one Zotero key per line.")
    args = ap.parse_args()

    if not args.input.exists():
        sys.exit(f"ERROR: input not found: {args.input}")

    keys = load_keys(args)
    if not keys:
        sys.exit("ERROR: no keys supplied.")

    print(f"[inject_cwyw] fetching {len(keys)} items from Zotero...", file=sys.stderr)
    item_data: dict[str, dict] = {}
    for k in keys:
        try:
            item_data[k] = fetch_item(args.user_id, k)
        except Exception as e:
            sys.exit(f"ERROR: Zotero fetch failed for key {k}: {e}")

    saved, n_cited = insert_citations(
        document_path=str(args.input),
        item_data=item_data,
        user_id=args.user_id,
        output_path=str(args.output),
    )
    print(f"[inject_cwyw] {n_cited} unique citations → {saved}", file=sys.stderr)
    print(f"[inject_cwyw] next: open {args.output} in Word → Zotero tab → "
          f"Add/Edit Bibliography (first build only) or Refresh.", file=sys.stderr)
    return 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
"""md_marker_convert.py — Convert Vancouver-style ``[N]`` / ``[N, M]`` markers in
markdown or .docx into pandoc-style ``[@KEY]`` / ``[@KEY1, @KEY2]`` citekeys
(or back), driven by an explicit N→key mapping.

Two directions:
  --to-keys     ``[N]``  → ``[@KEY]``   (default)
  --to-numbers  ``[@KEY]`` → ``[N]``    (round-trip / debug)

Inputs:
  --input  FILE        path to .md or .docx
  --output FILE        write converted file (extension must match input)
  --map    FILE        N↔key mapping (JSON ``{"1": "ABC123", ...}`` or 2-column CSV ``n,key``)
  --active-ns CSV      optional comma list of N's to convert; markers outside
                       this set are left untouched. Use to stage partial
                       conversion (e.g. sample-only build).

Numbers without a mapping (or outside ``--active-ns``) are left as plain
``[N]`` so a downstream Zotero refresh / hand-edit can finalize them.

Anti-Hallucination:
  - The mapping is the single source of truth. Never invents keys.
  - If a marker contains a number not in the map AND no ``--active-ns`` is
    given, it is left untouched and reported on stderr (exit 0). The caller
    must fix the map or accept the partial conversion.

Origin: generalized from the a per-project ``build_zotero_docx.py`` replacer
(2026-05-01), validated on a 21-reference manuscript.
"""

from __future__ import annotations

import argparse
import csv
import json
import re
import sys
from pathlib import Path

# [N], [N, M], [N,M,...], possibly with internal whitespace
_NUM_RE = re.compile(r"\[(\d+(?:\s*,\s*\d+)*)\]")
# Pandoc citekey group: [@KEY], [@K1, @K2], [@K1; @K2]
_KEY_RE = re.compile(r"\[(@[A-Za-z0-9_]+(?:\s*[,;]\s*@[A-Za-z0-9_]+)*)\]")


def load_map(path: Path) -> dict[int, str]:
    raw = path.read_text(encoding="utf-8").strip()
    if path.suffix.lower() == ".json" or raw.startswith("{"):
        data = json.loads(raw)
        return {int(k): str(v).strip() for k, v in data.items() if str(v).strip()}
    out: dict[int, str] = {}
    for row in csv.reader(raw.splitlines()):
        if not row or row[0].strip().lower() in {"n", "number", ""}:
            continue
        if len(row) < 2:
            continue
        out[int(row[0].strip())] = row[1].strip()
    return out


def parse_active(spec: str | None, full_keys: set[int]) -> set[int]:
    if not spec:
        return full_keys
    return {int(x.strip()) for x in spec.split(",") if x.strip()}


def make_num_to_key(n_to_key: dict[int, str], active: set[int]):
    unmapped: set[int] = set()

    def repl(m: re.Match) -> str:
        nums = [int(x.strip()) for x in m.group(1).split(",")]
        if not all(n in active for n in nums):
            return m.group(0)
        keys: list[str] = []
        for n in nums:
            k = n_to_key.get(n)
            if k is None:
                unmapped.add(n)
                return m.group(0)
            keys.append(f"@{k}")
        return "[" + ", ".join(keys) + "]"

    return repl, unmapped


def make_key_to_num(n_to_key: dict[int, str]):
    key_to_n = {v: n for n, v in n_to_key.items()}
    unknown: set[str] = set()

    def repl(m: re.Match) -> str:
        keys = [k.strip().lstrip("@") for k in re.split(r"[,;]", m.group(1))]
        nums: list[int] = []
        for k in keys:
            n = key_to_n.get(k)
            if n is None:
                unknown.add(k)
                return m.group(0)
            nums.append(n)
        return "[" + ", ".join(str(n) for n in nums) + "]"

    return repl, unknown


def transform_text(text: str, repl) -> str:
    # Auto-pick regex based on whether the text contains '@' citekeys.
    pattern = _KEY_RE if "[@" in text else _NUM_RE
    return pattern.sub(repl, text)


def convert_markdown(src: Path, dst: Path, repl) -> int:
    text = src.read_text(encoding="utf-8")
    pattern = _KEY_RE if "[@" in text else _NUM_RE
    n = sum(1 for _ in pattern.finditer(text))
    new = pattern.sub(repl, text)
    dst.write_text(new, encoding="utf-8")
    return n


def convert_docx(src: Path, dst: Path, repl, direction: str) -> int:
    try:
        from docx import Document  # type: ignore
    except ImportError:
        sys.exit("ERROR: python-docx is required for .docx input. `pip install python-docx`")
    doc = Document(str(src))
    pattern = _KEY_RE if direction == "to-numbers" else _NUM_RE
    n = 0

    def process(p):
        nonlocal n
        for run in p.runs:
            if "[" not in run.text:
                continue
            n += sum(1 for _ in pattern.finditer(run.text))
            run.text = pattern.sub(repl, run.text)

    for p in doc.paragraphs:
        process(p)
    for table in doc.tables:
        for row in table.rows:
            for cell in row.cells:
                for p in cell.paragraphs:
                    process(p)
    dst.parent.mkdir(parents=True, exist_ok=True)
    doc.save(str(dst))
    return n


def main() -> int:
    ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    ap.add_argument("--input", required=True, type=Path)
    ap.add_argument("--output", required=True, type=Path)
    ap.add_argument("--map", required=True, type=Path)
    ap.add_argument("--active-ns", default=None,
                    help="Comma-separated N's to convert (default: all in map).")
    direction = ap.add_mutually_exclusive_group()
    direction.add_argument("--to-keys", action="store_true", default=True,
                           help="[N] → [@KEY] (default).")
    direction.add_argument("--to-numbers", action="store_true",
                           help="[@KEY] → [N] (round-trip).")
    args = ap.parse_args()

    if not args.input.exists():
        sys.exit(f"ERROR: input not found: {args.input}")
    if args.input.suffix != args.output.suffix:
        sys.exit("ERROR: --input and --output must share the same extension (.md or .docx).")

    n_to_key = load_map(args.map)
    if not n_to_key:
        sys.exit("ERROR: empty mapping.")

    if args.to_numbers:
        repl, unknown = make_key_to_num(n_to_key)
        direction = "to-numbers"
    else:
        active = parse_active(args.active_ns, set(n_to_key))
        repl, unknown = make_num_to_key(n_to_key, active)
        direction = "to-keys"

    if args.input.suffix == ".docx":
        seen = convert_docx(args.input, args.output, repl, direction)
    else:
        seen = convert_markdown(args.input, args.output, repl)

    print(f"[md_marker_convert] direction={direction} markers_seen={seen} → {args.output}",
          file=sys.stderr)
    if unknown:
        print(f"[md_marker_convert] WARNING: unmapped tokens left untouched: "
              f"{sorted(unknown)}", file=sys.stderr)
    return 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env bash
# pre_submission_gate.sh — manage-refs four-stage pre-submission orchestration.
#
# Chains the existing scripts in the order required before a manuscript DOCX
# leaves the project. Each stage delegates to the canonical script under
# /manage-refs or /verify-refs — this wrapper does NOT reimplement any check.
#
# Stages (run in order; first failure aborts the chain):
#   1. check_citation_keys.py  — markdown [@bibkey] ↔ refs.bib key matching
#   2. check_bib_title_markup.py — publisher markup / tag-strip fusion in .bib titles
#   3. verify_refs.py --strict — refs.bib ↔ PubMed/CrossRef entry verification
#   4. render_pandoc.sh        — (only when --docx is not provided) render DOCX
#   5. check_xref.py --strict  — manuscript ↔ rendered DOCX cross-reference QC
#                                (optionally with --allow-separate-attachments)
#
# Usage:
#   pre_submission_gate.sh --md MD --bib BIB [--docx DOCX] [--journal CSL]
#                          [--allow-separate-attachments] [--qc-dir DIR]
#
# Exit codes:
#   0 — all stages PASS; qc/pre_submission_gate.json contains submission_safe:true
#   1 — at least one stage FAIL; qc/pre_submission_gate.json contains submission_safe:false
#   2 — usage / missing-input error
#
# The script writes qc/pre_submission_gate.json with one record per stage
# (status, exit_code, stderr_excerpt) plus a top-level submission_safe boolean.

set -uo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
VERIFY_REFS_DIR="${SCRIPT_DIR}/../../verify-refs/scripts"

usage() {
  cat >&2 <<'EOF'
Usage: pre_submission_gate.sh --md MD --bib BIB [options]

Required:
  --md PATH                   Manuscript markdown SSOT
  --bib PATH                  BibTeX bibliography

Optional:
  --docx PATH                 Pre-built DOCX. If omitted, render via render_pandoc.sh.
  --journal CSL_KEY           Journal CSL key (default: vancouver)
  --allow-separate-attachments
                              Forwarded to check_xref.py. Declares that some floats
                              are submitted as separate attachment files. Downgrades
                              MISSING_DOCX (proven absent from a supplied --docx) and,
                              when no --docx was supplied, MISSING_BODY (excused
                              without evidence — see summary.downgraded_unchecked).
                              A MISSING_BODY whose float IS in the DOCX still fails.
  --qc-dir PATH               Output artifact directory (default: qc/)
  -h, --help                  Show this help

Outputs:
  <qc-dir>/pre_submission_gate.json   summary
  <qc-dir>/reference_audit.json       verify_refs output
  <qc-dir>/xref_audit.json            check_xref output

Exit: 0 = pass, 1 = fail, 2 = bad input.
EOF
  exit 2
}

MD=""
BIB=""
DOCX=""
JOURNAL="vancouver"
ALLOW_SEPARATE_ATTACHMENTS=0
QC_DIR="qc"

while [[ $# -gt 0 ]]; do
  case "$1" in
    --md)       MD="$2"; shift 2 ;;
    --bib)      BIB="$2"; shift 2 ;;
    --docx)     DOCX="$2"; shift 2 ;;
    --journal)  JOURNAL="$2"; shift 2 ;;
    --allow-separate-attachments)
                ALLOW_SEPARATE_ATTACHMENTS=1; shift ;;
    --qc-dir)   QC_DIR="$2"; shift 2 ;;
    -h|--help)  usage ;;
    *)
      echo "ERROR: unknown argument: $1" >&2
      usage
      ;;
  esac
done

[[ -z "$MD"  ]] && { echo "ERROR: --md is required" >&2;  usage; }
[[ -z "$BIB" ]] && { echo "ERROR: --bib is required" >&2; usage; }
[[ -f "$MD"  ]] || { echo "ERROR: markdown not found: $MD" >&2; exit 2; }
[[ -f "$BIB" ]] || { echo "ERROR: refs.bib not found: $BIB" >&2; exit 2; }

mkdir -p "$QC_DIR"

ARTIFACT="$QC_DIR/pre_submission_gate.json"
TIMESTAMP="$(date -u +%Y-%m-%dT%H:%M:%SZ)"

# Stage helpers ---------------------------------------------------------------

declare -a STAGE_NAMES=()
declare -a STAGE_STATUSES=()
declare -a STAGE_EXITS=()
declare -a STAGE_NOTES=()

record_stage() {
  local name="$1" status="$2" exit_code="$3" note="$4"
  STAGE_NAMES+=("$name")
  STAGE_STATUSES+=("$status")
  STAGE_EXITS+=("$exit_code")
  STAGE_NOTES+=("$note")
}

write_artifact() {
  local safe="$1"
  python3 - "$ARTIFACT" "$TIMESTAMP" "$safe" \
    "${STAGE_NAMES[@]}" "::sep::" \
    "${STAGE_STATUSES[@]}" "::sep::" \
    "${STAGE_EXITS[@]}" "::sep::" \
    "${STAGE_NOTES[@]}" <<'PY'
import json, sys
out_path, ts, safe = sys.argv[1], sys.argv[2], sys.argv[3] == "true"
rest = sys.argv[4:]

def split_groups(values, sep="::sep::"):
    groups = []
    cur = []
    for v in values:
        if v == sep:
            groups.append(cur); cur = []
        else:
            cur.append(v)
    groups.append(cur)
    return groups

names, statuses, exits, notes = split_groups(rest)
gates = [
    {"stage": n, "status": s, "exit_code": int(e), "note": no}
    for n, s, e, no in zip(names, statuses, exits, notes)
]
payload = {
    "version": "1.0",
    "timestamp": ts,
    "submission_safe": safe,
    "gates": gates,
}
with open(out_path, "w", encoding="utf-8") as f:
    json.dump(payload, f, indent=2, ensure_ascii=False)
PY
}

abort_with_failure() {
  write_artifact "false"
  echo "[pre_submission_gate] FAIL (artifact: $ARTIFACT)" >&2
  exit 1
}

# Stage 1: citation keys ------------------------------------------------------

echo "[pre_submission_gate] stage 1/5: check_citation_keys.py"
STAGE1_LOG="$(mktemp)"
if python3 "$SCRIPT_DIR/check_citation_keys.py" "$MD" "$BIB" > "$STAGE1_LOG" 2>&1; then
  record_stage "check_citation_keys" "PASS" 0 "$(tail -n 1 "$STAGE1_LOG")"
  cat "$STAGE1_LOG"
else
  rc=$?
  record_stage "check_citation_keys" "FAIL" "$rc" "$(tail -n 5 "$STAGE1_LOG" | tr '\n' '; ')"
  cat "$STAGE1_LOG"
  rm -f "$STAGE1_LOG"
  abort_with_failure
fi
rm -f "$STAGE1_LOG"

# Stage 2: bib title markup ---------------------------------------------------
# verify_refs proves the reference is TRUE; this proves it will PRINT. CrossRef ships
# markup in titles (<scp>WHO</scp>, <i>IDH</i>) and a DOI-add stores it verbatim, so the
# rendered list can read "andTERTPromoter" while every other gate is green.

echo "[pre_submission_gate] stage 2/5: check_bib_title_markup.py --strict"
STAGE1B_LOG="$(mktemp)"
if python3 "$SCRIPT_DIR/check_bib_title_markup.py" --bib "$BIB" \
     --out "$QC_DIR/bib_title_markup.json" --strict > "$STAGE1B_LOG" 2>&1; then
  record_stage "check_bib_title_markup" "PASS" 0 "$(tail -n 1 "$STAGE1B_LOG")"
  cat "$STAGE1B_LOG"
else
  rc=$?
  record_stage "check_bib_title_markup" "FAIL" "$rc" "$(tail -n 5 "$STAGE1B_LOG" | tr '\n' '; ')"
  cat "$STAGE1B_LOG"
  rm -f "$STAGE1B_LOG"
  abort_with_failure
fi
rm -f "$STAGE1B_LOG"

# Stage 3: verify_refs.py against refs.bib ------------------------------------

echo "[pre_submission_gate] stage 3/5: verify_refs.py --strict"
PROJECT_ROOT="$(dirname "$QC_DIR")"
[[ -z "$PROJECT_ROOT" || "$PROJECT_ROOT" == "." ]] && PROJECT_ROOT="$(pwd)"
STAGE2_LOG="$(mktemp)"
if python3 "$VERIFY_REFS_DIR/verify_refs.py" "$BIB" \
     --project-root "$PROJECT_ROOT" --strict > "$STAGE2_LOG" 2>&1; then
  record_stage "verify_refs" "PASS" 0 "$(grep -E '^\[verify-refs\]|"counts"' "$STAGE2_LOG" | tail -n 1)"
  cat "$STAGE2_LOG"
else
  rc=$?
  record_stage "verify_refs" "FAIL" "$rc" "$(tail -n 5 "$STAGE2_LOG" | tr '\n' '; ')"
  cat "$STAGE2_LOG"
  rm -f "$STAGE2_LOG"
  abort_with_failure
fi
rm -f "$STAGE2_LOG"

# Stage 3: render DOCX if missing --------------------------------------------

if [[ -z "$DOCX" ]]; then
  echo "[pre_submission_gate] stage 4/5: render_pandoc.sh -j $JOURNAL"
  DERIVED_DOCX="${MD%.md}.docx"
  STAGE3_LOG="$(mktemp)"
  # -S: skip render_pandoc's own pre-render audit — stage 2 already ran verify_refs --strict.
  if bash "$SCRIPT_DIR/render_pandoc.sh" -S -j "$JOURNAL" -i "$MD" -b "$BIB" -o "$DERIVED_DOCX" \
       > "$STAGE3_LOG" 2>&1; then
    DOCX="$DERIVED_DOCX"
    record_stage "render_pandoc" "PASS" 0 "rendered $DERIVED_DOCX"
    cat "$STAGE3_LOG"
  else
    rc=$?
    record_stage "render_pandoc" "FAIL" "$rc" "$(tail -n 5 "$STAGE3_LOG" | tr '\n' '; ')"
    cat "$STAGE3_LOG"
    rm -f "$STAGE3_LOG"
    abort_with_failure
  fi
  rm -f "$STAGE3_LOG"
else
  echo "[pre_submission_gate] stage 3/4: skipped (pre-built DOCX: $DOCX)"
  record_stage "render_pandoc" "SKIPPED" 0 "pre-built DOCX supplied"
fi

[[ -f "$DOCX" ]] || { echo "ERROR: DOCX not found after stage 3: $DOCX" >&2; abort_with_failure; }

# Stage 4: check_xref.py ------------------------------------------------------

echo "[pre_submission_gate] stage 5/5: check_xref.py --strict"
XREF_OUT="$QC_DIR/xref_audit.json"
STAGE4_ARGS=(--md "$MD" --docx "$DOCX" --out "$XREF_OUT" --strict)
if [[ "$ALLOW_SEPARATE_ATTACHMENTS" == "1" ]]; then
  STAGE4_ARGS+=(--allow-separate-attachments)
fi
STAGE4_LOG="$(mktemp)"
if python3 "$SCRIPT_DIR/check_xref.py" "${STAGE4_ARGS[@]}" > "$STAGE4_LOG" 2>&1; then
  record_stage "check_xref" "PASS" 0 "$(grep -E 'wrote|WARN|SUBMISSION' "$STAGE4_LOG" | tr '\n' '; ')"
  cat "$STAGE4_LOG"
else
  rc=$?
  record_stage "check_xref" "FAIL" "$rc" "$(grep -E 'BLOCKED|wrote' "$STAGE4_LOG" | tr '\n' '; ')"
  cat "$STAGE4_LOG"
  rm -f "$STAGE4_LOG"
  abort_with_failure
fi
rm -f "$STAGE4_LOG"

# All stages passed -----------------------------------------------------------

write_artifact "true"
echo "[pre_submission_gate] PASS (artifact: $ARTIFACT)"
exit 0

#!/usr/bin/env bash
# render_pandoc.sh — Pandoc citeproc wrapper for manuscript rendering with journal CSL.
# (Renamed from render_manuscript.sh on 2026-05-01 when relocated to /manage-refs.)
#
# Usage:
#   render_manuscript.sh -j <journal> -i <input.md> -b <refs.bib> [-o <out.docx>] [-t <reference.docx>] [-- <extra pandoc args>]
#
# Example:
#   render_manuscript.sh -j european-radiology -i manuscript.md -b references.bib -o manuscript_v4.docx
#
# Citation syntax in markdown body: [@bibkey] or [@key1; @key2]
# References section is auto-generated by pandoc citeproc — do NOT hand-write a References list.

set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
CSL_DIR="${SCRIPT_DIR}/../citation_styles"

usage() {
  cat >&2 <<EOF
Usage: $(basename "$0") -j <journal> -i <input.md> -b <refs.bib> [-o <out>] [-t <reference.docx>] [-f <format>]

Options:
  -j  Journal CSL name (without .csl). Examples: european-radiology, radiology,
      american-journal-of-roentgenology, cardiovascular-and-interventional-radiology,
      korean-journal-of-radiology, vancouver. See \${CSL_DIR}/README.md.
  -i  Input markdown manuscript
  -b  BibTeX bibliography file
  -o  Output path (default: <input>.docx). Format inferred from extension.
  -t  Optional Word reference .docx for styles/fonts
  -f  Force output format (docx|pdf|html). Default: from -o extension or docx.
  -S  Skip the pre-render reference audit (verify-refs) gate (NOT recommended).
  -h  Help

Pass-through: any args after '--' go directly to pandoc.

The pre-render gate delegates to /verify-refs (when found alongside, or via
\$MEDSCI_VERIFY_REFS) and blocks the render on fabricated/mismatched citations.
EOF
  exit 1
}

JOURNAL=""
INPUT=""
BIB=""
OUTPUT=""
REFDOC=""
FORMAT=""
SKIP_AUDIT=0

while getopts ":j:i:b:o:t:f:hS" opt; do
  case "$opt" in
    j) JOURNAL="$OPTARG" ;;
    i) INPUT="$OPTARG" ;;
    b) BIB="$OPTARG" ;;
    o) OUTPUT="$OPTARG" ;;
    t) REFDOC="$OPTARG" ;;
    f) FORMAT="$OPTARG" ;;
    S) SKIP_AUDIT=1 ;;
    h|*) usage ;;
  esac
done
shift $((OPTIND - 1))

[[ -z "$JOURNAL" || -z "$INPUT" || -z "$BIB" ]] && usage

CSL_FILE="${CSL_DIR}/${JOURNAL}.csl"
if [[ ! -f "$CSL_FILE" ]]; then
  echo "ERROR: CSL not found: $CSL_FILE" >&2
  echo "Available styles:" >&2
  ls -1 "${CSL_DIR}" | grep '\.csl$' | sed 's/\.csl$//' | sed 's/^/  /' >&2
  exit 2
fi
[[ -f "$INPUT" ]] || { echo "ERROR: input not found: $INPUT" >&2; exit 2; }
[[ -f "$BIB" ]]   || { echo "ERROR: bib not found: $BIB" >&2; exit 2; }

# Pre-render reference audit gate — delegates to /verify-refs and blocks the render on
# fabricated / mismatched citations (e.g., #2..#N family-name hallucinations or author-count
# mismatches) before they reach the PDF/DOCX. Best-effort: if verify-refs is not installed
# alongside (and $MEDSCI_VERIFY_REFS is unset) the gate is skipped with a warning, so a
# standalone manage-refs copy still renders. Opt out explicitly with -S.
if [[ "$SKIP_AUDIT" -eq 0 ]]; then
  VR="${MEDSCI_VERIFY_REFS:-${SCRIPT_DIR}/../../verify-refs/scripts/verify_refs.py}"
  if [[ -f "$VR" ]] && command -v python3 >/dev/null 2>&1; then
    echo "[render] pre-render reference audit (verify-refs) on $BIB ..." >&2
    set +e
    python3 "$VR" "$BIB" >&2
    audit_rc=$?
    set -e
    if [[ "$audit_rc" -eq 0 ]]; then
      echo "[render] reference audit: clean" >&2
    elif [[ "$audit_rc" -eq 1 ]]; then
      echo "ERROR: reference audit found FABRICATED / MISMATCH / duplicate citations (see qc/reference_audit.json)." >&2
      echo "       Fix the .bib (or the Zotero record, since BBT overwrites manual .bib edits), or pass -S to skip (NOT recommended)." >&2
      exit 3
    else
      echo "[render] WARNING: reference audit did not complete (exit $audit_rc — e.g., no network, bad input, or no references); continuing render." >&2
    fi
  else
    echo "[render] (verify-refs not found alongside; skipping pre-render audit — set \$MEDSCI_VERIFY_REFS or pass -S)" >&2
  fi
fi

if [[ -z "$OUTPUT" ]]; then
  OUTPUT="${INPUT%.*}.docx"
fi
if [[ -z "$FORMAT" ]]; then
  case "${OUTPUT##*.}" in
    docx) FORMAT="docx" ;;
    pdf)  FORMAT="pdf" ;;
    html) FORMAT="html" ;;
    *)    FORMAT="docx" ;;
  esac
fi

# Run after citeproc: source locations are needed to resolve citations, but must
# not survive as custom document properties in an artifact shared with others.
ARGS=(--citeproc --csl="$CSL_FILE" --bibliography="$BIB"
      --lua-filter="${SCRIPT_DIR}/strip_source_metadata.lua" -t "$FORMAT" -o "$OUTPUT")
[[ -n "$REFDOC" && -f "$REFDOC" ]] && ARGS+=(--reference-doc="$REFDOC")

echo "[render] journal=$JOURNAL csl=$(basename "$CSL_FILE") in=$INPUT bib=$BIB out=$OUTPUT" >&2
pandoc "${ARGS[@]}" "$@" "$INPUT"
echo "[render] ok → $OUTPUT" >&2

# SPDX-License-Identifier: MIT
# Copyright (c) 2026 Ali Soroush
# Vendored from https://github.com/alisoroushmd/zotero-mcp
#   src/zotero_mcp/citation_writer.py @ ed5dfb71
# Imported into medsci-skills 2026-05-01 (originally vendored into an active meta-analysis
# the same day, relocated here after an active meta-analysis project validation). No functional
# modifications — `inject_zotero_cwyw.py` patches `zotero_to_csl_json` at import
# time to use Zotero's native `?format=csljson` endpoint, which handles
# webpage / report / non-journal item types correctly. See ../NOTICE.md.
# Full license: ../LICENSE.zotero-mcp

"""Citation writer -- builds Word documents with live Zotero field codes.

Creates .docx files containing ADDIN ZOTERO_ITEM and ADDIN ZOTERO_BIBL
field codes that the Zotero Word plugin recognizes as live citations.
"""

from __future__ import annotations

import hashlib
import json
import re
import uuid
from dataclasses import dataclass, field
from pathlib import Path

from docx import Document
from docx.oxml import OxmlElement
from docx.oxml.ns import qn

# ---------------------------------------------------------------------------
# 1. Citation Parser
# ---------------------------------------------------------------------------

_CITATION_RE = re.compile(r"\[@([A-Za-z0-9]+(?:,\s*@[A-Za-z0-9]+)*)\]")


@dataclass
class TextBlock:
    """A segment of parsed text -- either plain text or a citation group."""

    kind: str  # "text" or "citation"
    content: str  # raw text for "text", empty for "citation"
    keys: list[str] = field(default_factory=list)
    numbers: list[int] = field(default_factory=list)


def parse_citations(text: str) -> tuple[list[TextBlock], dict[str, int]]:
    """Parse text with [@KEY] markers into blocks with Vancouver numbering.

    Supports single ``[@KEY]`` and grouped ``[@KEY1, @KEY2]`` citations.
    Numbers are assigned sequentially by order of first appearance.
    Same key reuses its number.

    Args:
        text: Input text containing citation markers.

    Returns:
        Tuple of (list of TextBlocks, dict mapping item_key -> vancouver_number).
    """
    blocks: list[TextBlock] = []
    mapping: dict[str, int] = {}
    counter = 0
    last_end = 0

    for match in _CITATION_RE.finditer(text):
        start, end = match.span()

        # Text before this citation
        if start > last_end:
            blocks.append(TextBlock("text", text[last_end:start]))

        # Parse keys from the match group
        raw_keys = match.group(1)
        keys = [k.strip().lstrip("@") for k in raw_keys.split(",")]

        numbers: list[int] = []
        for key in keys:
            if key not in mapping:
                counter += 1
                mapping[key] = counter
            numbers.append(mapping[key])

        blocks.append(TextBlock("citation", "", keys, numbers))
        last_end = end

    # Trailing text
    if last_end < len(text):
        blocks.append(TextBlock("text", text[last_end:]))

    return blocks, mapping


# ---------------------------------------------------------------------------
# 2. CSL-JSON Converter
# ---------------------------------------------------------------------------

_ITEM_TYPE_MAP: dict[str, str] = {
    "journalArticle": "article-journal",
    "book": "book",
    "bookSection": "chapter",
    "conferencePaper": "paper-conference",
    "report": "report",
    "thesis": "thesis",
}

_FIELD_MAP: dict[str, str] = {
    "title": "title",
    "publicationTitle": "container-title",
    "volume": "volume",
    "issue": "issue",
    "pages": "page",
    "DOI": "DOI",
    "ISSN": "ISSN",
    "ISBN": "ISBN",
    "abstractNote": "abstract",
    "url": "URL",
    "publisher": "publisher",
}


def _parse_date(date_str: str) -> dict:
    """Parse a date string into CSL-JSON date format.

    Handles ``2024``, ``2024-03``, ``2024-03-15``, and ``/``-separated
    variants.

    Args:
        date_str: Date string from Zotero.

    Returns:
        Dict with ``date-parts`` key.
    """
    if not date_str:
        return {"date-parts": [[]]}
    parts_str = re.split(r"[-/\s]", date_str)
    parts = [int(p) for p in parts_str if p.isdigit()]
    return {"date-parts": [parts]} if parts else {"date-parts": [[]]}


def zotero_to_csl_json(item_data: dict, user_id: str) -> dict:
    """Convert Zotero item data to CSL-JSON format.

    Maps Zotero fields to CSL-JSON fields for embedding in Word field codes.

    Args:
        item_data: Zotero item metadata dict.
        user_id: Zotero user ID for URI construction.

    Returns:
        CSL-JSON dict suitable for embedding in a Zotero field code.
    """
    item_key = item_data.get("key", "")
    item_type = item_data.get("itemType", "")

    csl: dict = {
        "type": _ITEM_TYPE_MAP.get(item_type, "article"),
        "id": int(hashlib.md5(item_key.encode()).hexdigest()[:8], 16),
        "_uris": [f"http://zotero.org/users/{user_id}/items/{item_key}"],
    }

    # Map simple fields
    for zotero_field, csl_field in _FIELD_MAP.items():
        value = item_data.get(zotero_field)
        if value:
            csl[csl_field] = value

    # Creators -- authors only
    creators = item_data.get("creators", [])
    authors = [
        {"family": c.get("lastName", ""), "given": c.get("firstName", "")}
        for c in creators
        if c.get("creatorType") == "author"
    ]
    if authors:
        csl["author"] = authors

    # Date
    date_str = item_data.get("date", "")
    if date_str:
        csl["issued"] = _parse_date(date_str)

    return csl


# ---------------------------------------------------------------------------
# 3. Field Code Builder
# ---------------------------------------------------------------------------


def _make_run() -> OxmlElement:
    """Create a new ``w:r`` element."""
    return OxmlElement("w:r")


def _make_fld_char(fld_char_type: str) -> OxmlElement:
    """Create a ``w:fldChar`` element with the given type.

    Args:
        fld_char_type: One of ``begin``, ``separate``, ``end``.

    Returns:
        A ``w:r`` element containing the fldChar.
    """
    run = _make_run()
    fld_char = OxmlElement("w:fldChar")
    fld_char.set(qn("w:fldCharType"), fld_char_type)
    run.append(fld_char)
    return run


def _make_instr_text(text: str) -> OxmlElement:
    """Create a ``w:r`` element containing ``w:instrText``.

    Args:
        text: The instruction text content.

    Returns:
        A ``w:r`` element with instrText child.
    """
    run = _make_run()
    instr = OxmlElement("w:instrText")
    instr.set(qn("xml:space"), "preserve")
    instr.text = text
    run.append(instr)
    return run


def _make_superscript_run(text: str) -> OxmlElement:
    """Create a ``w:r`` element with superscript text.

    Args:
        text: Display text to render as superscript.

    Returns:
        A ``w:r`` element with superscript formatting.
    """
    run = _make_run()
    rpr = OxmlElement("w:rPr")
    vert_align = OxmlElement("w:vertAlign")
    vert_align.set(qn("w:val"), "superscript")
    rpr.append(vert_align)
    run.append(rpr)
    t_elem = OxmlElement("w:t")
    t_elem.set(qn("xml:space"), "preserve")
    t_elem.text = text
    run.append(t_elem)
    return run


def add_citation_field(paragraph, citation_json: dict, display_text: str) -> None:
    """Insert a Zotero citation field code into a Word paragraph.

    Builds the XML structure:
      begin fldChar -> instrText -> separate fldChar -> display run -> end fldChar

    Args:
        paragraph: A python-docx Paragraph object.
        citation_json: The full citation JSON dict for the field code.
        display_text: Text to display (typically the Vancouver number).
    """
    p_elem = paragraph._element
    json_str = json.dumps(citation_json, ensure_ascii=False)
    instr_content = f"ADDIN ZOTERO_ITEM CSL_CITATION {json_str}"

    p_elem.append(_make_fld_char("begin"))
    p_elem.append(_make_instr_text(instr_content))
    p_elem.append(_make_fld_char("separate"))
    p_elem.append(_make_superscript_run(display_text))
    p_elem.append(_make_fld_char("end"))


def add_bibliography_field(paragraph) -> None:
    """Insert a Zotero bibliography field code into a Word paragraph.

    Args:
        paragraph: A python-docx Paragraph object.
    """
    p_elem = paragraph._element
    instr_content = 'ADDIN ZOTERO_BIBL {"uncited":[],"custom":[]} CSL_BIBLIOGRAPHY'

    p_elem.append(_make_fld_char("begin"))
    p_elem.append(_make_instr_text(instr_content))
    p_elem.append(_make_fld_char("separate"))
    p_elem.append(_make_fld_char("end"))


# ---------------------------------------------------------------------------
# 4. Document Assembler
# ---------------------------------------------------------------------------

# Regex for inline markdown formatting
_BOLD_RE = re.compile(r"\*\*(.+?)\*\*")
_ITALIC_RE = re.compile(r"\*(.+?)\*")


def _add_formatted_text(paragraph, text: str) -> None:
    """Add text with basic markdown formatting (bold/italic) to a paragraph.

    Processes ``**bold**`` and ``*italic*`` markers. Text without markers
    is added as plain runs.

    Args:
        paragraph: A python-docx Paragraph object.
        text: Text that may contain markdown bold/italic markers.
    """
    # Split on bold markers first
    parts = _BOLD_RE.split(text)
    for i, part in enumerate(parts):
        if not part:
            continue
        if i % 2 == 1:
            # Bold segment
            run = paragraph.add_run(part)
            run.bold = True
        else:
            # Check for italic within non-bold segments
            italic_parts = _ITALIC_RE.split(part)
            for j, ipart in enumerate(italic_parts):
                if not ipart:
                    continue
                if j % 2 == 1:
                    run = paragraph.add_run(ipart)
                    run.italic = True
                else:
                    paragraph.add_run(ipart)


def build_document(
    content: str,
    item_data: dict[str, dict],
    user_id: str,
    output_path: str,
) -> str:
    """Build a Word document with live Zotero citations.

    Args:
        content: Markdown text with ``[@KEY]`` citation markers.
        item_data: Dict mapping item keys to their Zotero metadata.
        user_id: Zotero user ID for URI construction.
        output_path: Where to save the .docx file.

    Returns:
        Absolute path of the saved file.
    """
    doc = Document()

    # Parse global citation numbering across the whole document
    _, key_to_number = parse_citations(content)

    # Split into paragraphs on blank lines
    raw_paragraphs = re.split(r"\n\n+", content.strip())

    for raw_para in raw_paragraphs:
        raw_para = raw_para.strip()
        if not raw_para:
            continue

        # Detect headings
        heading_match = re.match(r"^(#{1,3})\s+(.+)$", raw_para)
        if heading_match:
            level = len(heading_match.group(1))
            heading_text = heading_match.group(2)
            doc.add_heading(heading_text, level=level)
            continue

        # Regular paragraph -- parse citations within it
        # Use global key_to_number for consistent numbering across paragraphs
        blocks, _ = parse_citations(raw_para)
        for block in blocks:
            if block.kind == "citation":
                block.numbers = [key_to_number[k] for k in block.keys if k in key_to_number]

        paragraph = doc.add_paragraph()

        for block in blocks:
            if block.kind == "text":
                _add_formatted_text(paragraph, block.content)
            elif block.kind == "citation":
                # Build the citation field code
                citation_items = []
                for key in block.keys:
                    if key in item_data:
                        csl = zotero_to_csl_json(item_data[key], user_id)
                        uris = csl.pop("_uris", [])
                        citation_items.append(
                            {
                                "id": csl["id"],
                                "uris": uris,
                                "itemData": csl,
                            }
                        )

                # Vancouver display: e.g. "1" or "1,2"
                display = ",".join(str(n) for n in block.numbers)

                citation_json = {
                    "citationID": f"cite_{uuid.uuid4().hex[:8]}",
                    "properties": {
                        "formattedCitation": display,
                        "plainCitation": display,
                        "noteIndex": 0,
                    },
                    "citationItems": citation_items,
                    "schema": (
                        "https://github.com/citation-style-language"
                        "/schema/raw/master/csl-citation.json"
                    ),
                }
                add_citation_field(paragraph, citation_json, display)

    # Add References heading and bibliography
    doc.add_heading("References", level=1)
    bib_para = doc.add_paragraph()
    add_bibliography_field(bib_para)

    # Save
    output = Path(output_path).resolve()
    doc.save(str(output))
    return str(output)


# ---------------------------------------------------------------------------
# 5. In-place Citation Insertion (preserves existing document formatting)
# ---------------------------------------------------------------------------


def _paragraph_full_text(paragraph) -> str:
    """Extract full text from a paragraph including all runs.

    Args:
        paragraph: A python-docx Paragraph object.

    Returns:
        Concatenated text of all runs.
    """
    return "".join(run.text for run in paragraph.runs)


def _has_citation_markers(text: str) -> bool:
    """Check whether text contains [@KEY] citation markers.

    Args:
        text: Text to check.

    Returns:
        True if citation markers are found.
    """
    return bool(_CITATION_RE.search(text))


def _rebuild_paragraph_with_citations(
    paragraph,
    blocks: list[TextBlock],
    item_data: dict[str, dict],
    user_id: str,
) -> None:
    """Replace a paragraph's content with citation field codes in-place.

    Clears existing runs and rebuilds with text blocks and Zotero field codes.
    Preserves the paragraph's style (heading level, alignment, spacing, etc.).

    Args:
        paragraph: A python-docx Paragraph object to modify.
        blocks: Parsed TextBlocks from parse_citations.
        item_data: Dict mapping item keys to Zotero metadata.
        user_id: Zotero user ID for URI construction.
    """
    # Preserve paragraph style before clearing
    style = paragraph.style

    # Remove all existing runs from the paragraph XML
    p_elem = paragraph._element
    for child in list(p_elem):
        if child.tag == qn("w:r"):
            p_elem.remove(child)

    # Restore style
    paragraph.style = style

    for block in blocks:
        if block.kind == "text":
            _add_formatted_text(paragraph, block.content)
        elif block.kind == "citation":
            citation_items = []
            for key in block.keys:
                if key in item_data:
                    csl = zotero_to_csl_json(item_data[key], user_id)
                    uris = csl.pop("_uris", [])
                    citation_items.append(
                        {
                            "id": csl["id"],
                            "uris": uris,
                            "itemData": csl,
                        }
                    )

            display = ",".join(str(n) for n in block.numbers)

            citation_json = {
                "citationID": f"cite_{uuid.uuid4().hex[:8]}",
                "properties": {
                    "formattedCitation": display,
                    "plainCitation": display,
                    "noteIndex": 0,
                },
                "citationItems": citation_items,
                "schema": (
                    "https://github.com/citation-style-language/schema/raw/master/csl-citation.json"
                ),
            }
            add_citation_field(paragraph, citation_json, display)


def insert_citations(
    document_path: str,
    item_data: dict[str, dict],
    user_id: str,
    output_path: str | None = None,
) -> tuple[str, int]:
    """Insert Zotero citation field codes into an existing Word document.

    Opens the document, scans all paragraphs for [@KEY] markers, replaces
    them with live Zotero field codes, and appends a bibliography if one
    is not already present. All other document formatting (styles, headers,
    footers, images, tables, page layout) is preserved.

    Args:
        document_path: Path to the existing .docx file.
        item_data: Dict mapping item keys to their Zotero metadata.
        user_id: Zotero user ID for URI construction.
        output_path: Where to save. If None, overwrites the original.

    Returns:
        Tuple of (saved file path, number of citation markers replaced).
    """
    doc = Document(document_path)
    save_to = Path(output_path or document_path).resolve()

    # First pass: collect all citation keys across the entire document
    # for consistent Vancouver numbering
    all_text_parts: list[str] = []
    for paragraph in doc.paragraphs:
        text = _paragraph_full_text(paragraph)
        if _has_citation_markers(text):
            all_text_parts.append(text)

    # Also scan tables
    for table in doc.tables:
        for row in table.rows:
            for cell in row.cells:
                for paragraph in cell.paragraphs:
                    text = _paragraph_full_text(paragraph)
                    if _has_citation_markers(text):
                        all_text_parts.append(text)

    if not all_text_parts:
        doc.save(str(save_to))
        return str(save_to), 0

    # Build global numbering from concatenated text
    combined = "\n\n".join(all_text_parts)
    _, key_to_number = parse_citations(combined)
    citation_count = len(key_to_number)

    # Second pass: rebuild paragraphs that contain citation markers
    def _process_paragraph(paragraph) -> None:
        text = _paragraph_full_text(paragraph)
        if not _has_citation_markers(text):
            return
        blocks, _ = parse_citations(text)
        # Remap to global numbering
        for block in blocks:
            if block.kind == "citation":
                block.numbers = [key_to_number[k] for k in block.keys if k in key_to_number]
        _rebuild_paragraph_with_citations(paragraph, blocks, item_data, user_id)

    for paragraph in doc.paragraphs:
        _process_paragraph(paragraph)

    for table in doc.tables:
        for row in table.rows:
            for cell in row.cells:
                for paragraph in cell.paragraphs:
                    _process_paragraph(paragraph)

    # Add bibliography if not already present
    has_bibliography = False
    for paragraph in doc.paragraphs:
        if "ADDIN ZOTERO_BIBL" in paragraph._element.xml:
            has_bibliography = True
            break

    if not has_bibliography:
        doc.add_heading("References", level=1)
        bib_para = doc.add_paragraph()
        add_bibliography_field(bib_para)

    doc.save(str(save_to))
    return str(save_to), citation_count

#!/usr/bin/env bash
# Regression test for skills/manage-refs/scripts/check_bib_title_markup.py.
#
# The positives are the two titles that actually shipped into a rendered reference list as
# garbage. The negatives are the reason the fusion rule is narrow: `mRNA`, `hTERT`, `nnU-Net`
# and `1,2-dichloroethane` are ordinary scientific typography, and a gate that cries wolf on
# them is worse than no gate.
set -u

REPO_ROOT="$(cd "$(dirname "$0")/../../.." && pwd)"
V="$REPO_ROOT/skills/manage-refs/scripts/check_bib_title_markup.py"
TMP="$(mktemp -d)"
trap 'rm -rf "$TMP"' EXIT

pass=0
fail=0
ck() {
  local label="$1" expected="$2" actual="$3"
  if [ "$expected" = "$actual" ]; then
    printf '  PASS  %-52s exit=%s\n' "$label" "$actual"
    pass=$((pass + 1))
  else
    printf '  FAIL  %-52s expected=%s actual=%s\n' "$label" "$expected" "$actual"
    fail=$((fail + 1))
  fi
}

# --- corrupted: raw CrossRef markup, BBT-escaped markup, and tag-strip fusion ---
cat > "$TMP/dirty.bib" <<'BIB'
@article{louis2021who,
  title = {The 2021 <scp>WHO</scp> Classification of Tumors of the Central Nervous System},
  author = {Louis, David N.},
  year = {2021},
}
@article{brat2021escaped,
  title = {The 2021 {$<$}scp{$>$}WHO{$<$}/scp{$>$} Classification: A Summary},
  author = {Brat, Daniel J.},
  year = {2021},
}
@article{eckel2015glioma,
  title = {Glioma Groups Based on 1p/19q,IDH, andTERTPromoter Mutations in Tumors},
  author = {{Eckel-Passow}, Jeanette E.},
  year = {2015},
}
BIB

# --- clean: ordinary scientific typography that must NOT fire ---
cat > "$TMP/clean.bib" <<'BIB'
@article{isensee2021nnunet,
  title = {nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation},
  author = {Isensee, Fabian},
  year = {2021},
}
@article{sahin2021mrna,
  title = {mRNA-based COVID-19 vaccines and hTERT expression in 1,2-dichloroethane exposure},
  author = {Sahin, Ugur},
  year = {2021},
}
@article{plain2020title,
  title = {Radiomics and machine learning for glioma grading, staging, and outcome prediction},
  author = {Smith, John},
  year = {2020},
}
BIB

# 1) corrupted titles fail under --strict
python3 "$V" --bib "$TMP/dirty.bib" --strict --quiet > /dev/null 2>&1
ck "markup + fusion titles fail (--strict)" 1 "$?"

# 2) clean titles pass
python3 "$V" --bib "$TMP/clean.bib" --strict --quiet > /dev/null 2>&1
ck "clean scientific typography passes (--strict)" 0 "$?"

# 3) each corruption shape is reported, on the right key
python3 "$V" --bib "$TMP/dirty.bib" --out "$TMP/d.json" --quiet > /dev/null 2>&1
python3 - "$TMP/d.json" <<'PY'
import json, sys
r = json.load(open(sys.argv[1]))
by_key = {}
for f in r["findings"]:
    by_key.setdefault(f["key"], set()).add(f["verdict"])
assert "TITLE_MARKUP" in by_key.get("louis2021who", set()), "raw <scp> not caught"
assert "TITLE_MARKUP" in by_key.get("brat2021escaped", set()), "BBT-escaped {$<$} not caught"
assert "TITLE_FUSION" in by_key.get("eckel2015glioma", set()), "andTERT / ,IDH fusion not caught"
assert r["titles_checked"] == 3, r["titles_checked"]
assert r["submission_safe"] is False
PY
ck "every corruption shape reported on its own key" 0 "$?"

# 4) the false-positive guards, asserted individually — this is what makes the gate usable
python3 -B - "$V" <<'PY'
import importlib.util, sys
spec = importlib.util.spec_from_file_location("m", sys.argv[1])
m = importlib.util.module_from_spec(spec); sys.modules["m"] = m; spec.loader.exec_module(m)
clean = [
    "nnU-Net: a self-configuring method",
    "mRNA-based vaccines",
    "hTERT promoter mutations",
    "Exposure to 1,2-dichloroethane and 10,000 participants",
    "pH-sensitive nanoparticles for siRNA delivery",
    "Comparison of CT and MRI in glioma",
]
for t in clean:
    f = m.findings_for("k", t)
    assert not f, f"false positive on {t!r}: {[x['verdict'] for x in f]}"
dirty = ["Based on 1p/19q,IDH, andTERT Promoter", "The 2021 <scp>WHO</scp> Classification"]
for t in dirty:
    assert m.findings_for("k", t), f"missed corruption in {t!r}"
PY
ck "no false positives on mRNA / hTERT / nnU-Net / 1,2-" 0 "$?"

# 5) corruption is reported but tolerated without --strict
python3 "$V" --bib "$TMP/dirty.bib" --quiet > /dev/null 2>&1
ck "corruption tolerated without --strict" 0 "$?"

# 6) the JSON envelope names the detector (repo-wide artifact contract)
python3 - "$TMP/d.json" <<'PY'
import json, sys
assert json.load(open(sys.argv[1]))["detector"] == "check_bib_title_markup"
PY
ck "JSON envelope self-identifies" 0 "$?"

echo "----"
echo "test_bib_title_markup: $pass passed, $fail failed"
[ "$fail" -eq 0 ]

#!/usr/bin/env bash
# Regression test: where a citation key ENDS, and what an `@` is not.
#
# Two defects, both of which made the checker fail correct manuscripts.
#
# 1. A citation that ends a sentence swallowed the full stop. The key pattern allowed `.` anywhere,
#    but pandoc counts internal punctuation as part of a key only when a letter or digit follows it.
#    So "...as previously reported @Smith2023." parsed as key `Smith2023.` — and the tool then
#    reported the SAME reference as UNDEFINED (no `Smith2023.` in the .bib) and as UNUSED (nothing
#    cited `Smith2023`) in one run. A citation ending a sentence is most of them.
#
# 2. Quarto / pandoc-crossref cross-references were read as bibliography keys. `@fig-flow`,
#    `@tbl-baseline`, `@sec-methods` resolve against the document's own labels; `quarto render`
#    compiles them without complaint. This checker called every one an undefined reference and
#    exited 1 — on a manuscript in the shape the repo's own `init_project.py` scaffolds.
#
# They are excluded from the verdict but REPORTED, so nothing is silently dropped, and a .bib entry
# that genuinely happens to be named `fig-...` resolves before the exclusion is ever consulted.
set -u

REPO_ROOT="$(cd "$(dirname "$0")/../../.." && pwd)"
K="$REPO_ROOT/skills/manage-refs/scripts/check_citation_keys.py"
WORK="$(mktemp -d -t citekey_test.XXXXXX)"
trap 'rm -rf "$WORK"' EXIT

pass=0
fail=0
ck() {
  local label="$1" expected="$2" actual="$3"
  if [ "$expected" = "$actual" ]; then
    printf '  PASS  %-52s %s\n' "$label" "$actual"
    pass=$((pass + 1))
  else
    printf '  FAIL  %-52s expected=%s actual=%s\n' "$label" "$expected" "$actual"
    fail=$((fail + 1))
  fi
}

cat > "$WORK/refs.bib" <<'EOF'
@article{Smith2023,
  author = {Smith, John},
  title  = {A title},
  year   = {2023}
}
EOF

cat > "$WORK/figkey.bib" <<'EOF'
@article{fig-consort,
  author = {Real, Author},
  title  = {A paper someone keyed as fig-consort},
  year   = {2024}
}
EOF

run() {  # run <md> <bib> [flags...] -> exit code, output at $WORK/<md>.out
  local md="$1" bib="$2"; shift 2
  python3 "$K" "$WORK/$md" "$WORK/$bib" "$@" > "$WORK/$md.out" 2>&1
  echo $?
}
section_count() {  # section_count <md> <SECTION WORD>
  grep -oE "^$2 \([0-9]+\)" "$WORK/$1.out" | grep -oE '[0-9]+' || echo 0
}

# --- fixtures ---------------------------------------------------------------------------------
printf 'Prior work established this [@Smith2023].\n\nAnother sentence cites @Smith2023.\n' \
  > "$WORK/sentence_end.md"

printf 'A key with internal punctuation: @Sec.2a is one key.\n' > "$WORK/internal_punct.md"

cat > "$WORK/quarto.qmd" <<'EOF'
---
title: Example
---

Prior work established this @Smith2023.

![Study flow](flow.png){#fig-flow}

As shown in @fig-flow, enrolment proceeded. See also @tbl-baseline and @sec-methods.
EOF

printf 'Cites [@Smith2023] and [@Ghost2024].\n' > "$WORK/undefined.md"
printf 'Cites [@fig-consort].\n' > "$WORK/figkey.md"

echo "==== 1. a citation ending a sentence is the same citation ===="
ck "sentence-final @key: exit 0"        0 "$(run sentence_end.md refs.bib)"
ck "  no UNDEFINED"                     0 "$(section_count sentence_end.md UNDEFINED)"
ck "  no UNUSED"                        0 "$(section_count sentence_end.md UNUSED)"
ck "  counted once, not twice"          "cited=1" \
   "$(grep -oE 'cited=[0-9]+' "$WORK/sentence_end.md.out")"

echo "==== 2. pandoc's actual rule: internal punctuation followed by alphanumeric stays ===="
run internal_punct.md refs.bib > /dev/null
ck "@Sec.2a is ONE key, not truncated"  "[@Sec.2a]" \
   "$(grep -oE '\[@Sec\.2a\]' "$WORK/internal_punct.md.out" | head -1)"

echo "==== 3. Quarto cross-references are not bibliography keys ===="
ck "Quarto manuscript: exit 0"          0 "$(run quarto.qmd refs.bib)"
ck "  3 cross-references REPORTED"      3 "$(section_count quarto.qmd CROSS-REFERENCES)"
ck "  0 undefined"                      0 "$(section_count quarto.qmd UNDEFINED)"

echo "==== NEGATIVE CONTROLS ===="
ck "a genuinely undefined key still fails" 1 "$(run undefined.md refs.bib)"
ck "  and it is named"                  "[@Ghost2024]" \
   "$(grep -oE '\[@Ghost2024\]' "$WORK/undefined.md.out" | head -1)"
ck "a real .bib key named fig-* resolves" 0 "$(run figkey.md figkey.bib)"
ck "  not diverted to CROSS-REFERENCES" 0 "$(section_count figkey.md CROSS-REFERENCES)"
ck "UNUSED detection still works"       1 "$(run undefined.md refs.bib --strict-unused)"

echo
echo "  passed=$pass failed=$fail"
[ "$fail" -eq 0 ] || { for f in sentence_end.md quarto.qmd undefined.md figkey.md; do
    echo "--- $f"; cat "$WORK/$f.out"; done; exit 1; }
echo "OK: the full stop is punctuation, a cross-reference is not a citation, and a ghost key still fails."

#!/usr/bin/env bash
# Regression test for the hardened CSL acceptance-check (manage-refs/check_csl_render.py).
# Guards the robustness fixes (no module globals, checked subprocess, no temp leak,
# guarded python-docx import, guarded bib read). Synthetic, PII-free fixtures.
#
# CI-safe: the deepest path (real pandoc render → docx superscript parse) needs
# pandoc, which CI does not install. The error-handling paths this test asserts
# do NOT need pandoc, and the no-pandoc branch is exercised exactly when pandoc is
# absent (i.e. in CI), so fix #2 (clean "pandoc not found") is covered there while
# fixes #1/#3/#4 (the happy path) are covered wherever pandoc is present (locally).
set -u

HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SCRIPT="$HERE/../scripts/check_csl_render.py"
CSL="$HERE/../citation_styles/vancouver.csl"
BIB="$HERE/fixtures/csl_render_sample.bib"

[[ -f "$SCRIPT" ]] || { echo "ENV-ERR: script missing: $SCRIPT" >&2; exit 2; }
[[ -f "$CSL"    ]] || { echo "ENV-ERR: csl missing: $CSL" >&2; exit 2; }
[[ -f "$BIB"    ]] || { echo "ENV-ERR: fixture bib missing: $BIB" >&2; exit 2; }

fail=0
pass() { printf '  PASS  %s\n' "$1"; }
bad()  { printf '  FAIL  %s\n' "$1"; fail=$((fail+1)); }

echo "test_csl_render:"

# 1. Missing bib -> clean error (fix: guarded bib read), exit 2, no traceback.
err="$(python3 "$SCRIPT" --csl "$CSL" --bib /nonexistent_csl_render.bib 2>&1)"; rc=$?
if [[ $rc -eq 2 && "$err" == *"bib file not found"* && "$err" != *"Traceback"* ]]; then
  pass "missing bib -> clean error, exit 2"
else
  bad "missing bib (rc=$rc): $err"
fi

# 2. Valid bib: pandoc present -> happy path renders, exit 0 + JSON.
#    pandoc absent (e.g. CI) -> clean 'pandoc not found' error, exit 2.
out="$(python3 "$SCRIPT" --csl "$CSL" --bib "$BIB" 2>&1)"; rc=$?
if command -v pandoc >/dev/null 2>&1; then
  if [[ $rc -eq 0 && "$out" == *'"got"'* && "$out" != *"Traceback"* ]]; then
    pass "valid bib + pandoc -> renders, exit 0, JSON emitted"
  else
    bad "valid bib + pandoc (rc=$rc): $out"
  fi
else
  if [[ $rc -eq 2 && "$out" == *"pandoc not found"* && "$out" != *"Traceback"* ]]; then
    pass "valid bib, no pandoc -> clean 'pandoc not found', exit 2"
  else
    bad "valid bib, no pandoc (rc=$rc): $out"
  fi
fi

# 3. No leftover temp dirs from this run (fix: TemporaryDirectory cleanup).
if ls -d "${TMPDIR:-/tmp}"/csl_render_* >/dev/null 2>&1; then
  bad "temp dir leak: csl_render_* left behind"
else
  pass "no temp-dir leak"
fi

# The production wrapper must keep citation rendering intact while removing
# source locations from custom Word properties. A real render exercises filter
# ordering: stripping bibliography before citeproc would lose the reference.
if command -v pandoc >/dev/null 2>&1; then
  work="$(mktemp -d)"
  trap 'rm -rf "$work"' EXIT
  printf 'A synthetic citation [@sample2020].\n' > "$work/manuscript.md"
  printf '@article{sample2020, author={Example, A.}, title={Synthetic study}, journal={Example Journal}, year={2020}}\n' > "$work/refs.bib"
  bash "$HERE/../scripts/render_pandoc.sh" -S -j vancouver -i "$work/manuscript.md" \
    -b "$work/refs.bib" -o "$work/manuscript.docx" >/dev/null 2>&1
  python3 - "$work/manuscript.docx" <<'PY'
import sys, zipfile
with zipfile.ZipFile(sys.argv[1]) as z:
    body = z.read("word/document.xml").decode()
    custom = z.read("docProps/custom.xml").decode() if "docProps/custom.xml" in z.namelist() else ""
assert "Synthetic study" in body, "citeproc did not render the reference"
assert 'name="csl"' not in custom and 'name="bibliography"' not in custom
PY
  [[ $? -eq 0 ]] && pass "rendered references survive; source-path properties do not" \
    || bad "source metadata removal broke rendering or missed a property"
else
  echo "  SKIP  source-property render check (pandoc unavailable)"
fi

if [[ $fail -eq 0 ]]; then echo "  OK"; exit 0; else echo "  $fail check(s) failed"; exit 1; fi

#!/usr/bin/env bash
# Regression test: fill_journal_abbrev.py must survive its own first entry.
#
# `parse_entries` returned the Match objects from `re.finditer`, while every caller treats the result
# as the entry TEXT. So the first line of work —
#
#     key = re.match(r"@\w+\{([^,]+),", b).group(1)
#
# — raised `TypeError: expected string or bytes-like object, got 're.Match'`, on the FIRST entry, for
# every input. The script had never run. Meanwhile `check_csl_render.py` names it to the user as the
# remedy when a journal spec needs `shortjournal`, and `manage-refs/SKILL.md` lists it in the tool
# table. A tool that cannot start is worse than a missing one: it is advertised.
#
# The second defect only becomes visible once the first is fixed. `field()` required a trailing comma
# after the value, and BibTeX makes that comma optional on an entry's LAST field — which `doi` very
# often is. The DOI would have come back None, no PMID lookup would happen, and the run would report
# "0/1 entries" while exiting 0. Same defect class as the reference parser fixed in #445.
#
# Network-free: asserts the parsing contract, never PubMed.
set -u

REPO_ROOT="$(cd "$(dirname "$0")/../../.." && pwd)"
F="$REPO_ROOT/skills/manage-refs/scripts/fill_journal_abbrev.py"

pass=0
fail=0
ck() {
  local label="$1" expected="$2" actual="$3"
  if [ "$expected" = "$actual" ]; then
    printf '  PASS  %-52s %s\n' "$label" "$actual"
    pass=$((pass + 1))
  else
    printf '  FAIL  %-52s expected=%s actual=%s\n' "$label" "$expected" "$actual"
    fail=$((fail + 1))
  fi
}

probe() {  # probe <mode> -> one line of output
  python3 - "$F" "$1" <<'PY'
import importlib.util, re, sys
spec = importlib.util.spec_from_file_location("fja", sys.argv[1])
m = importlib.util.module_from_spec(spec)
sys.modules["fja"] = m
spec.loader.exec_module(m)
mode = sys.argv[2]

DOI_LAST = """@article{Rinella2023,
\tauthor = {Rinella, Mary E.},
\tjournal = {Hepatology},
\tyear = {2023},
\tdoi = {10.1097/HEP.0000000000000520}
}
"""
DOI_MID = """@article{Smith2023,
\tauthor = {Smith, John},
\tdoi = {10.1000/xyz789},
\tjournal = {Radiology},
\tyear = {2023}
}
"""
NO_DOI = """@article{Doe2023,
\tauthor = {Doe, Jane},
\tjournal = {Radiology},
\tyear = {2023}
}
"""
TWO = DOI_LAST + "\n" + DOI_MID

if mode == "type":
    print(type(m.parse_entries(DOI_LAST)[0]).__name__)
elif mode == "caller":
    # exactly what main() does first; this is where it used to raise TypeError
    try:
        b = m.parse_entries(DOI_LAST)[0]
        print(re.match(r"@\w+\{([^,]+),", b).group(1).strip())
    except TypeError as e:
        print(f"TypeError: {e}")
elif mode == "count":
    print(len(m.parse_entries(TWO)))
elif mode == "doi_last":
    print(m.field(m.parse_entries(DOI_LAST)[0], "doi") or "None")
elif mode == "doi_mid":
    print(m.field(m.parse_entries(DOI_MID)[0], "doi") or "None")
elif mode == "doi_absent":
    print(m.field(m.parse_entries(NO_DOI)[0], "doi") or "None")
elif mode == "journal_mid":
    print(m.field(m.parse_entries(DOI_MID)[0], "journal") or "None")
PY
}

echo "==== the script must survive its own first entry ===="
ck "parse_entries yields text, not Match"  str        "$(probe type)"
ck "the caller's first line does not raise" Rinella2023 "$(probe caller)"
ck "two entries parse as two"              2          "$(probe count)"

echo "==== a value in the LAST field is still a value ===="
ck "doi as the entry's last field"  "10.1097/HEP.0000000000000520" "$(probe doi_last)"
ck "doi mid-entry, trailing comma"  "10.1000/xyz789"               "$(probe doi_mid)"
ck "journal mid-entry"              "Radiology"                    "$(probe journal_mid)"

echo "==== NEGATIVE CONTROL — absent stays absent ===="
ck "no doi field: None, not invented" None "$(probe doi_absent)"

echo
echo "  passed=$pass failed=$fail"
[ "$fail" -eq 0 ] || exit 1
echo "OK: the tool starts, reads its last field, and does not invent a DOI it was not given."

#!/usr/bin/env bash
# Regression test for the duplicate-bibliography gate (manage-refs / sync-submission).
# Synthetic, PII-free fixtures: a rendered manuscript with TWO reference lists
# (a hand-typed list + a second list concatenated after the figure legends, the
# pandoc-citeproc-duplication pattern) vs a clean single-list manuscript.
# Stdlib-only (python3).
set -u

HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SCRIPT="$HERE/../scripts/check_reference_duplication.py"
DUP="$HERE/fixtures/refdup_text.md"
CLEAN="$HERE/fixtures/refclean_text.md"
OUT="$(mktemp -t refdup_XXXX).json"
trap 'rm -f "$OUT"' EXIT

fail=0
check() { local label="$1"; shift
    if "$@" >/dev/null 2>&1; then printf '  PASS  %s\n' "$label"
    else printf '  FAIL  %s\n' "$label"; fail=$((fail+1)); fi
}
has_verdict() { python3 -c "
import json
d=json.load(open('$OUT'))
assert any(c['verdict']=='$1' for c in d['claims']), '$1 not found'
"; }

[[ -f "$SCRIPT" ]] || { echo "ENV-ERR: script missing" >&2; exit 2; }

echo "test_reference_duplication:"

# (1) duplicated bibliography -> Major -> exit 1 under --strict
python3 "$SCRIPT" --text "$DUP" --out "$OUT" --strict --quiet >/dev/null 2>&1
check "exit 1 under --strict (Major present)" test "$?" -eq 1
check "JSON artifact written" test -s "$OUT"
check "DUP_REF_HEADING detected (two References headings)" has_verdict DUP_REF_HEADING
check "REF_NUMBER_RESTART detected (entry '1.' twice)"     has_verdict REF_NUMBER_RESTART
check "REF_SIGNATURE_DUP detected (whole list repeated)"   has_verdict REF_SIGNATURE_DUP

# (2) single bibliography -> exit 0, no claim
python3 "$SCRIPT" --text "$CLEAN" --out "$OUT" --strict --quiet >/dev/null 2>&1
check "exit 0 on single reference list" test "$?" -eq 0
check "no claims on clean fixture" bash -c "
python3 -c \"import json; d=json.load(open('$OUT')); assert not d['claims']\"
"

if [[ "$fail" -eq 0 ]]; then echo "  ALL PASS"; else echo "  $fail FAILED"; fi
exit "$fail"

#!/usr/bin/env bash
# Regression tests for check_xref.py --vN-docx-md5 / --vN-md flags.
#
# Builds minimal synthetic docx files via zipfile (no pandoc/Word required).

set -uo pipefail

REPO_ROOT="$(cd "$(dirname "$0")/../../.." && pwd)"
SCRIPT="$REPO_ROOT/skills/manage-refs/scripts/check_xref.py"
TMP="$(mktemp -d -t vN_docx_xref.XXXXXX)"
trap 'rm -rf "$TMP"' EXIT

[[ -f "$SCRIPT" ]] || { echo "ENV-ERR: script missing" >&2; exit 2; }

fail=0
ran=0
assert_exit() {
    local label="$1" expected="$2" actual="$3"
    ran=$((ran + 1))
    if [[ "$expected" == "$actual" ]]; then
        printf '  PASS  %-50s exit=%s\n' "$label" "$actual"
    else
        printf '  FAIL  %-50s expected=%s actual=%s\n' "$label" "$expected" "$actual"
        fail=$((fail + 1))
    fi
}

# Helper: build a minimal docx with given body text.
build_docx() {
    local body_text="$1"
    local out="$2"
    python3 - "$body_text" "$out" <<'PY'
import sys, zipfile
body_text, out = sys.argv[1], sys.argv[2]
doc_xml = (
    '<?xml version="1.0" encoding="UTF-8" standalone="yes"?>\n'
    '<w:document xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main">'
    '<w:body><w:p><w:r><w:t xml:space="preserve">' + body_text + '</w:t></w:r></w:p>'
    '</w:body></w:document>'
)
content_types = (
    '<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
    '<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types">'
    '<Default Extension="xml" ContentType="application/xml"/>'
    '<Override PartName="/word/document.xml" '
    'ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.document.main+xml"/>'
    '</Types>'
)
rels = (
    '<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
    '<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
    '<Relationship Id="rId1" '
    'Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/officeDocument" '
    'Target="word/document.xml"/></Relationships>'
)
with zipfile.ZipFile(out, "w", zipfile.ZIP_DEFLATED) as z:
    z.writestr("[Content_Types].xml", content_types)
    z.writestr("_rels/.rels", rels)
    z.writestr("word/document.xml", doc_xml)
PY
}

# --------------------------------------------------------------------------
# Setup: minimal manuscript markdown + docx
# --------------------------------------------------------------------------
VN_MD="$TMP/v1.md"
cat > "$VN_MD" <<'EOF'
# Manuscript

## Methods
The cohort included 100 patients.

## Figure Legends
**Figure 1.** Pipeline overview.
EOF
NEW_MD="$TMP/v2.md"
cat > "$NEW_MD" <<'EOF'
# Manuscript

## Methods
The cohort included 100 patients enrolled across three sites in this prospective study.

## Figure Legends
**Figure 1.** Pipeline overview.
EOF

VN_DOCX="$TMP/v1.docx"
NEW_DOCX_REGEN="$TMP/v2_regen.docx"
NEW_DOCX_COPY="$TMP/v2_copy.docx"
NEW_DOCX_STALE="$TMP/v2_stale.docx"

build_docx "The cohort included 100 patients. Figure 1 Pipeline overview." "$VN_DOCX"
# Regenerated docx: contains the new diff line verbatim.
build_docx "The cohort included 100 patients enrolled across three sites in this prospective study. Figure 1 Pipeline overview." "$NEW_DOCX_REGEN"
# "Copy" docx: byte-identical to v_N.
cp "$VN_DOCX" "$NEW_DOCX_COPY"
# Stale docx: different bytes than v_N but missing the new markdown line.
build_docx "Some text that differs from v_N but lacks the markdown diff." "$NEW_DOCX_STALE"

# --------------------------------------------------------------------------
# Case 1: regenerated docx contains markdown diff. PASS.
# --------------------------------------------------------------------------
python3 "$SCRIPT" --md "$NEW_MD" --docx "$NEW_DOCX_REGEN" \
    --vN-docx-md5 "$VN_DOCX" --vN-md "$VN_MD" \
    --out "$TMP/c1.json" --quiet
assert_exit "case 1: regenerated docx, diff propagated" 0 $?

# --------------------------------------------------------------------------
# Case 2: identical bytes. FAIL.
# --------------------------------------------------------------------------
python3 "$SCRIPT" --md "$NEW_MD" --docx "$NEW_DOCX_COPY" \
    --vN-docx-md5 "$VN_DOCX" --vN-md "$VN_MD" \
    --out "$TMP/c2.json" --quiet
assert_exit "case 2: identical bytes (FAIL)" 1 $?
python3 - "$TMP/c2.json" <<'PY' || fail=$((fail + 1))
import json, sys
with open(sys.argv[1]) as fh: rep = json.load(fh)
assert rep["vN_docx_check"]["identical_bytes"] is True, rep
PY

# --------------------------------------------------------------------------
# Case 3: different bytes but missing diff line. FAIL.
# --------------------------------------------------------------------------
python3 "$SCRIPT" --md "$NEW_MD" --docx "$NEW_DOCX_STALE" \
    --vN-docx-md5 "$VN_DOCX" --vN-md "$VN_MD" \
    --out "$TMP/c3.json" --quiet
assert_exit "case 3: different bytes, missing diff (FAIL)" 1 $?
python3 - "$TMP/c3.json" <<'PY' || fail=$((fail + 1))
import json, sys
with open(sys.argv[1]) as fh: rep = json.load(fh)
assert rep["vN_docx_check"]["identical_bytes"] is False, rep
assert rep["vN_docx_check"]["diff_line_misses"], rep
PY

# --------------------------------------------------------------------------
# Case 4: --vN-docx-md5 without --docx. Should error (exit 2).
# --------------------------------------------------------------------------
python3 "$SCRIPT" --md "$NEW_MD" \
    --vN-docx-md5 "$VN_DOCX" --vN-md "$VN_MD" \
    --out "$TMP/c4.json" --quiet 2>/dev/null
assert_exit "case 4: --vN-docx-md5 without --docx" 2 $?

echo ""
echo "ran=$ran fail=$fail"
[[ $fail -eq 0 ]]

#!/usr/bin/env bash
# Regression test: a plural float mention is a citation.
#
# `check_xref` recognised only the singular "Figure 1". The plural "s" breaks its `\s+`, so
# "Figures 1 and 2" — the way people actually write it — matched NOTHING. The consequence was not a
# missed note, it was a disabled gate: a float cited only that way scored UNCITED instead of
# MISSING_DOCX, and UNCITED is not in `blocking_statuses`. Two manuscripts identical in meaning got
# opposite verdicts:
#
#     "Figures 1 and 2"        -> exit 0, submission_safe: true   <- Figure 2 absent from the DOCX
#     "Figure 1 and Figure 2"  -> exit 1, SUBMISSION BLOCKED
#
# So ordinary English turned the submission blocker off, while the repetitive form this tool's own
# examples happen to use kept it on. The invariant this test pins is that equivalence: the same
# package must get the same verdict however the citation is phrased.
set -u

REPO_ROOT="$(cd "$(dirname "$0")/../../.." && pwd)"
X="$REPO_ROOT/skills/manage-refs/scripts/check_xref.py"
WORK="$(mktemp -d -t xref_plural_test.XXXXXX)"
trap 'rm -rf "$WORK"' EXIT

pass=0
fail=0
ck() {
  local label="$1" expected="$2" actual="$3"
  if [ "$expected" = "$actual" ]; then
    printf '  PASS  %-52s %s\n' "$label" "$actual"
    pass=$((pass + 1))
  else
    printf '  FAIL  %-52s expected=%s actual=%s\n' "$label" "$expected" "$actual"
    fail=$((fail + 1))
  fi
}

# A DOCX containing ONLY Figure 1's caption, so Figure 2 is genuinely absent from the rendered file.
mk_docx() {  # mk_docx <path> <caption text...>
  python3 - "$@" <<'PY'
import sys, zipfile
out, caps = sys.argv[1], sys.argv[2:]
body = "".join(f"<w:p><w:r><w:t>{c}</w:t></w:r></w:p>" for c in caps)
with zipfile.ZipFile(out, "w", zipfile.ZIP_DEFLATED) as z:
    z.writestr("[Content_Types].xml", '<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
        '<Types xmlns="http://schemas.openxmlformats.org/package/2006/content-types">'
        '<Default Extension="xml" ContentType="application/xml"/>'
        '<Default Extension="rels" ContentType="application/vnd.openxmlformats-package.relationships+xml"/>'
        '<Override PartName="/word/document.xml" ContentType="application/vnd.openxmlformats-officedocument.wordprocessingml.document.main+xml"/></Types>')
    z.writestr("_rels/.rels", '<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
        '<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
        '<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/officeDocument" Target="word/document.xml"/></Relationships>')
    z.writestr("word/document.xml", '<?xml version="1.0" encoding="UTF-8" standalone="yes"?>'
        '<w:document xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main"><w:body>'
        + body + '</w:body></w:document>')
PY
}

mk_md() {  # mk_md <path> <citation sentence> [extra legend lines...]
  local path="$1" cite="$2"; shift 2
  {
    echo "## Results"; echo
    echo "$cite"; echo
    echo "## Figure Legends"; echo
    echo "**Figure 1.** Study flow diagram."; echo
    echo "**Figure 2.** Kaplan-Meier survival curves."
    for extra in "$@"; do echo; echo "$extra"; done
  } > "$path"
}

run() {  # run <md> <docx> -> exit code; JSON at $WORK/<md>.json
  python3 "$X" --md "$WORK/$1" --docx "$WORK/$2" --strict --out "$WORK/$1.json" \
    > "$WORK/$1.out" 2>&1
  echo $?
}
safe() { python3 -c "import json;print(json.load(open('$WORK/$1.json'))['submission_safe'])"; }
status_of() {  # status_of <md> <label>   e.g. status_of plural.md Figure:2
  python3 -c "
import json, sys
d = json.load(open(sys.argv[1]))
for f in d['findings']:
    if f['label'] == sys.argv[2]:
        print(f['status']); break
else:
    print('ABSENT')" "$WORK/$1.json" "$2"
}

mk_docx "$WORK/partial.docx" "Figure 1. Study flow diagram."
mk_docx "$WORK/full.docx" "Figure 1. Study flow diagram." "Figure 2. Kaplan-Meier survival curves."

mk_md "$WORK/plural.md"     "As shown in Figures 1 and 2, the effect held."
mk_md "$WORK/repeated.md"   "As shown in Figure 1 and Figure 2, the effect held."
mk_md "$WORK/range.md"      "As shown in Figures 1-2, the effect held."
mk_md "$WORK/wordrange.md"  "As shown in Figures 1 to 2, the effect held."
mk_md "$WORK/uncited.md"    "The analysis is described in the Methods."

echo "==== the equivalence: phrasing must not change the verdict ===="
rc_plural="$(run plural.md partial.docx)"
rc_repeat="$(run repeated.md partial.docx)"
ck "plural 'Figures 1 and 2' blocks"        1 "$rc_plural"
ck "repetitive form blocks (unchanged)"     1 "$rc_repeat"
ck "same verdict either way"                "$rc_repeat" "$rc_plural"
ck "plural: submission_safe false"          False "$(safe plural.md)"
ck "plural: Figure 2 is MISSING_DOCX"       MISSING_DOCX "$(status_of plural.md Figure:2)"

echo "==== ranges expand; an endpoint-only read would drop the interior ===="
ck "'Figures 1-2' blocks"                   1 "$(run range.md partial.docx)"
ck "'Figures 1 to 2' blocks"                1 "$(run wordrange.md partial.docx)"

echo "==== NEGATIVE CONTROLS ===="
ck "a genuinely uncited figure stays UNCITED" UNCITED "$(run uncited.md full.docx >/dev/null; status_of uncited.md Figure:2)"
rc_ok="$(run plural.md full.docx)"
ck "complete package: exit 0"               0 "$rc_ok"
ck "complete package: submission_safe true" True "$(safe plural.md)"
ck "no double count (singular is a prefix of plural)" 2 \
   "$(run repeated.md full.docx >/dev/null; grep -o 'in-text citations: [0-9]*' "$WORK/repeated.md.out" | grep -o '[0-9]*')"

echo
echo "  passed=$pass failed=$fail"
[ "$fail" -eq 0 ] || { for f in plural repeated uncited; do echo "--- $f"; cat "$WORK/$f.md.out"; done; exit 1; }
echo "OK: 'Figures 1 and 2' and 'Figure 1 and Figure 2' are the same claim, and both block."

#!/usr/bin/env bash
# Regression test: --allow-separate-attachments, and the exact shape of the amnesty it grants.
#
# The situation this exists for: a submission whose supplementary tables and figures are
# separate attachment files — the norm in radiology and most medical journals — checked in
# markdown-only mode. Those floats are cited, have no caption in the manuscript body, and
# there is no rendered DOCX to look them up in, so `_classify` returns MISSING_BODY and the
# run once printed SUBMISSION BLOCKED on a correctly packaged submission.
#
# MISSING_BODY carries two different situations under one name and only one of them is the
# SSOT drift that every triage table in this repo describes:
#
#   in_docx is True  -> the float IS rendered but has no body caption. Real drift. P0.
#   in_docx is None  -> no DOCX was supplied, so there is nothing to have drifted FROM.
#
# The maintainer's decision (2026-07-29) is that --allow-separate-attachments downgrades the
# second case: with the flag set the operator has declared those floats live outside the main
# document, and a gate that is red on correct work is a gate that gets skipped.
#
# The cost of that decision is a real amnesty — in markdown-only mode a caption nobody wrote is
# indistinguishable from an attachment, and both now pass. So what this suite pins is not just
# "it passes" but that the amnesty stays VISIBLE and BOUNDED:
#   - the excused rows are named, and labelled EXCUSED WITHOUT EVIDENCE, apart from the
#     MISSING_DOCX rows that a supplied DOCX actually proved absent;
#   - the JSON separates the two counts, so a consumer cannot read one as the other;
#   - MISSING_BODY with the float present in the DOCX still BLOCKS. That is SSOT drift, and no
#     attachment policy makes it acceptable.
set -u

REPO_ROOT="$(cd "$(dirname "$0")/../../.." && pwd)"
S="$REPO_ROOT/skills/manage-refs/scripts/check_xref.py"
TMP="$(mktemp -d)"
trap 'rm -rf "$TMP"' EXIT

pass=0
fail=0
ck() {
  local label="$1" expected="$2" actual="$3"
  if [ "$expected" = "$actual" ]; then
    printf '  PASS  %-58s %s\n' "$label" "$actual"
    pass=$((pass + 1))
  else
    printf '  FAIL  %-58s expected=%s actual=%s\n' "$label" "$expected" "$actual"
    fail=$((fail + 1))
  fi
}

EXCUSED='EXCUSED WITHOUT EVIDENCE'

# --- a correctly packaged submission: supplement lives in separate attachment files ---
cat > "$TMP/sep.md" <<'MD'
# METHODS

We measured the thing (Table 1). Details are in Supplementary Table S1 and
Supplementary Figure S1, submitted as separate attachment files.

# TABLES

**Table 1.** Baseline characteristics of the study population.
MD

# --- genuine SSOT drift: the float IS rendered, but the body defines no caption ---
cat > "$TMP/drift.md" <<'MD'
# METHODS

We measured the thing (Table 1) and Supplementary Table S1.

# TABLES

**Table 1.** Baseline characteristics of the study population.
MD

python3 - "$TMP" <<'PY'
import sys
from docx import Document
tmp = sys.argv[1]

# Renders Table 1 only — the supplement is a separate attachment.
d = Document()
for t in ["METHODS",
          "We measured the thing (Table 1). Details are in Supplementary Table S1 and "
          "Supplementary Figure S1.",
          "Table 1. Baseline characteristics of the study population."]:
    d.add_paragraph(t)
d.save(f"{tmp}/main.docx")

# Renders Supplementary Table S1 too, while drift.md defines no caption for it.
d = Document()
for t in ["METHODS",
          "We measured the thing (Table 1) and Supplementary Table S1.",
          "Table 1. Baseline characteristics of the study population.",
          "Supplementary Table S1. Sensitivity analyses by centre."]:
    d.add_paragraph(t)
d.save(f"{tmp}/drift.docx")
PY

run() { python3 "$S" --md "$1" ${2:+--docx "$2"} --out "$TMP/out.json" \
          --allow-separate-attachments --strict > "$TMP/log.txt" 2>&1; echo $?; }

# 1) The declared case: markdown-only, floats live in separate attachment files.
rc=$(run "$TMP/sep.md" "")
ck "markdown-only separate-supplement PASSES under the flag" 0 "$rc"
grep -q "$EXCUSED" "$TMP/log.txt"; ck "...and the amnesty is stated, not silent" 0 "$?"
# The labels must appear in the EXCUSED block. Grepping the whole log passes vacuously —
# every label is already printed in the findings table above.
awk "/$EXCUSED/,/Run again with --docx/" "$TMP/log.txt" | grep -q 'Table:S-S1'
ck "...and the excused rows are NAMED in that block" 0 "$?"
grep -q 'nothing here was actually checked' "$TMP/log.txt"
ck "...and it says nothing was checked, not that it verified them" 0 "$?"
python3 -c "
import json,sys
s=json.load(open('$TMP/out.json'))['summary']
sys.exit(0 if s.get('downgraded_unchecked')==2 and s.get('downgraded_proven_absent')==0 else 1)" 2>/dev/null
ck "...and the JSON counts it as unchecked, not as proven-absent" 0 "$?"

# 2) A supplied DOCX turns the same rows into an EVIDENCED downgrade, not an excuse.
rc=$(run "$TMP/sep.md" "$TMP/main.docx")
ck "with --docx the same package PASSES too" 0 "$rc"
grep -q "$EXCUSED" "$TMP/log.txt"; ck "...and nothing is excused without evidence" 1 "$?"
grep -q 'MISSING_DOCX row(s) downgraded' "$TMP/log.txt"
ck "...it is reported as a proven-absent downgrade instead" 0 "$?"
python3 -c "
import json,sys
s=json.load(open('$TMP/out.json'))['summary']
sys.exit(0 if s.get('downgraded_proven_absent')==2 and s.get('downgraded_unchecked')==0 else 1)" 2>/dev/null
ck "...and the JSON keeps the two apart" 0 "$?"

# 3) The bound on the amnesty: real SSOT drift still blocks WITH the flag set.
rc=$(run "$TMP/drift.md" "$TMP/drift.docx")
ck "rendered-but-undefined caption still BLOCKS under the flag" 1 "$rc"
grep -q "$EXCUSED" "$TMP/log.txt"; ck "...and is not excused" 1 "$?"
grep -q 'does NOT excuse' "$TMP/log.txt"; ck "...and the run says why it is different" 0 "$?"

# 4) Without the flag nothing is downgraded at all — the declaration has to be made.
rc=$(python3 "$S" --md "$TMP/sep.md" --out "$TMP/out.json" --strict > "$TMP/log.txt" 2>&1; echo $?)
ck "no flag: the same package still blocks (opt-in preserved)" 1 "$rc"

echo "----"
echo "test_xref_separate_supplement: $pass passed, $fail failed"
[ "$fail" -eq 0 ]