lit-sync
Sync research references from .bib files to Zotero library + Obsidian literature notes. Extract cross-cutting concept notes when enough literature accumulates. Works after /search-lit or standalone.
Install
npx skills add https://github.com/Aperivue/medsci-skills/tree/main/skills/lit-sync
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install aperivue-medsci-skills@llmmart
git clone https://github.com/Aperivue/medsci-skills.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole aperivue/medsci-skills collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Literature Sync: Zotero + Obsidian Pipeline
Takes the .bib output of /search-lit (or any user-specified .bib file) and
synchronizes the references into the Zotero library and Obsidian literature notes.
When enough literature notes accumulate, extracts cross-cutting concept notes.
Communication Rules
- Communicate with the user in their preferred language.
- Vault layout — honor what exists, default to English. Before creating notes, detect the
vault's existing layout: if the vault already uses a particular folder structure (including a
Korean one such as
02 연구/문헌/and02 연구/개념노트/), honor it — never silently rename a user's folders. For a new or unclear vault, default to the English foldersLiterature/andConcepts/with the English note templates below. - A Korean opt-in variant (Korean folder layout + Korean-heading templates) lives in
references/locale/ko/note_templates.md— use it when the vault is Korean-structured or the user prefers Korean notes.
When to Use
- After
/search-litcompletes — sync the produced .bib into Zotero + Obsidian. - Bulk-register references from an existing .bib into Zotero + Obsidian.
- Tidy the
references/folder inside a project workspace. - On explicit concept-extraction request → extract cross-cutting concepts from existing literature notes.
Prerequisites
- Project owner only —
/lit-syncis an owner-scoped operation perdocs/zotero_policy.md. Collaborators consume the committedmanuscript/_src/refs.bibsnapshot read-only. - Zotero desktop 7.x + Better BibTeX plugin installed.
- Better BibTeX "Keep updated" auto-export configured to
<project>/manuscript/_src/refs.bib(owner setup checklist indocs/zotero_policy.md§Setup). - Zotero MCP server available (skip the Zotero phase if not connected; auto-export refresh still fires once Zotero is reopened).
- Obsidian CLI or direct file writing to the Obsidian vault.
- Obsidian vault path: configured in user's environment (e.g.,
$OBSIDIAN_VAULT).
Artifact Contract
Per docs/artifact_contract.md, /lit-sync is the sole writer of:
| Artifact | Writer | Readers |
|---|---|---|
manuscript/_src/refs.bib |
/lit-sync (via Better BibTeX auto-export trigger) |
/write-paper, /verify-refs, /manage-refs |
references/zotero_collection.json |
/lit-sync |
/verify-refs, /sync-submission |
Direct hand edits to refs.bib are drift — revert on sight.
Pipeline Overview
.bib file (or /search-lit output)
│
▼ Phase 1: Parse
Extract DOI, PMID, title, authors, journal, year
│
▼ Phase 2: Zotero Sync (owner)
Dedupe → zotero_add_by_doi → place in collection → pin citekey
│
▼ Phase 2.5: refs.bib snapshot refresh
Trigger Better BibTeX auto-export → verify manuscript/_src/refs.bib mtime updated
│
▼ Phase 2.7: Fulltext Retrieval (opt-in)
Disk OA PDFs via /fulltext-retrieval + in-library via find_available_pdf.js → reconcile report
│
▼ Phase 3: Obsidian Literature Notes
Create Literature/{citekey}.md (empty note OK — fill later with highlights)
│
▼ Phase 4: Concept Extraction (conditional)
≥10 literature notes → scan for cross-cutting concepts → propose concept notes
Phase 1: Parse BibTeX
Input
The user-specified .bib file path, or the .bib just produced by /search-lit.
Process
# Parse .bib entries with regex.
# Extract per entry:
# - citekey — read it from the entry, never compose one.
# Better BibTeX keys look like `smithDeepLearningRadiology2024`
# (author + title words + year). A key shaped like `Smith_2024_Validation`
# or `smith2024validation` was almost certainly invented rather than read,
# and will not resolve against the library. See Step 3.2 §Citekey provenance.
# - doi
# - pmid
# - title
# - authors (first + last minimum)
# - journal
# - year
# - volume, number, pages (if present)
Log any parse failures and skip those entries.
Phase 2: Zotero Sync
Step 2.1: Determine project collection
Identify the project from the current working directory or from an explicit user override. Reuse an existing collection key if one is recorded; otherwise create a new collection.
Collection mapping: Check existing Zotero collections for the current project.
If no collection exists, create one with zotero_create_collection. Record the
collection key for future use.
Step 2.2: Dedupe + add
For each entry:
- Use
zotero_search_itemsto search by DOI or title — if already present, skip. This search-first step is what prevents duplicates;zotero_add_by_doidoes not dedupe by itself (it fetches CrossRef and creates the item), so never skip the search. - Otherwise call
zotero_add_by_doi(when a DOI is available) orzotero_add_by_url(falling back to the PubMed URL when no DOI is available).zotero_add_by_doiaccepts anattach_modeargument that governs the OA child-PDF attach attempt at add time (the installed server treatslinked_urlas "bookmark the PDF URL"; other values download/import). Set it when you want a PDF attached during the add. Exact accepted values are server-version-specific — verify against the connected server. Do not usezotero_add_from_fileto attach a PDF to an item added here: it has no parent-item argument and would create a duplicate parent item.
- Use
zotero_manage_collectionsto place the item in the project collection.
Step 2.3: Result report
Zotero Sync:
Added: 8 papers (new)
Skipped: 3 papers (already in library)
Failed: 1 paper (no DOI/PMID)
Collection: RFA-Meta (TZQEP4NH)
If the Zotero MCP is not connected, skip this entire phase and proceed to Phase 3.
Always write references/zotero_collection.json in the project workspace:
{
"schema_version": 1,
"status": "synced",
"collection": "RFA-Meta",
"collection_key": "TZQEP4NH",
"added": 8,
"skipped": 3,
"failed": 1
}
If Zotero is unavailable, write the same file with status: "skipped" and a
human-readable reason.
Phase 2.5: refs.bib snapshot refresh
Better BibTeX "Keep updated" auto-export normally refreshes manuscript/_src/refs.bib within seconds of a Zotero change. This phase verifies the snapshot actually updated before downstream skills consume it.
Step 2.5.1: Resolve path
Read SSOT.yaml → truth.refs_bib. Default: manuscript/_src/refs.bib. If absent (legacy project), fall back to manuscript/_src/refs.bib and emit a WARN recommending SSOT migration.
Step 2.5.1b: Precondition assertion (early-exit, do NOT poll)
Before entering the 10s polling loop in Step 2.5.2, verify both preconditions. If either fails, abort Phase 2.5 with setup instructions instead of waiting for a timeout that will never resolve.
Better BibTeX is answering. Probe the running plugin, not a file on disk:
curl -s -m 5 -o /dev/null -w "%{http_code}" \ http://127.0.0.1:23119/better-bibtex/json-rpc # expect 200A non-200 means Zotero is closed or BBT has not finished starting. Retry once after Zotero's window is up; BBT registers its endpoint a few seconds after the app does.
⚠️ Do not gate on
~/Zotero/better-bibtex/read-only.json. Current BBT releases keep auto-export registrations in their own store, so that file is routinely[]on a perfectly healthy install. Treating an empty list as "not configured" skips this phase on working setups — and a skipped Phase 2.5 is how a stalerefs.biband an invented citekey reach a manuscript.On failure print:
Phase 2.5 skipped: Better BibTeX did not answer on
127.0.0.1:23119(HTTP<code>). Open Zotero, wait for it to finish loading, then re-run/lit-sync.Target refs.bib exists. The resolved
truth.refs_bibpath from Step 2.5.1 must exist on disk (even empty is OK — BBT will overwrite). On failure print:Phase 2.5 skipped: target snapshot
<path>not found. Configure BBT auto-export with "On Change" to the SSOT path, then re-run.
In either early-exit, set refs_bib_refreshed: false + reason: "precondition:<which>" in the Step 2.5.3 JSON and return control to the caller. Record it and tell the user; nothing downstream enforces it. verify_refs.py has never read this flag, and the sentence that said it did was the only thing standing between a stale refs.bib and a manuscript.
Step 2.5.2: Verify refresh
After Phase 2 adds items:
- Capture
stat -f "%m" manuscript/_src/refs.bibbefore Zotero writes. - Wait up to 10s (Better BibTeX debounce). Poll mtime.
- If mtime unchanged after 10s:
- Prompt user to check Zotero is running and BBT export is "Keep updated".
- If BBT auto-export path is wrong, print the expected path (
<project>/manuscript/_src/refs.bib) and refer todocs/zotero_policy.md§Setup. - As last resort, offer manual export:
File → Export Library → Better BibTeX → target path.
- Once mtime advances, grep for the newly added citekeys. All must be present; if any is missing, report as failure (do NOT fabricate entries).
Step 2.5.3: Record in zotero_collection.json
Append to the JSON written in Step 2.3:
{
"refs_bib_path": "manuscript/_src/refs.bib",
"refs_bib_mtime": "2026-04-24T14:32:11Z",
"refs_bib_refreshed": true,
"citekeys_verified": ["smithDeepLearningRadiology2024", "..."]
}
If refresh failed, set refs_bib_refreshed: false and include reason. The flag records whether the export ran. It is a note to the reader, not a gate.
Phase 2.7: Fulltext Retrieval (opt-in, owner-only)
Run only when the user asks for full text (e.g. "download the PDFs", "fetch full
text", or a worklist supplied with that intent). Default /lit-sync stays metadata-only
and network-light — do not auto-run this phase. Runs after items are in Zotero (Phase 2)
and the snapshot is verified (Phase 2.5), before Obsidian notes (Phase 3).
There are two complementary retrieval routes; offer both and reconcile them in one report:
Route A — disk OA PDFs (for downstream skills)
Delegate to the /fulltext-retrieval engine (do not re-implement the OA cascade or
import its code; invoke it by path). Resolve the engine as:
ENGINE="${MEDSCI_SKILLS_ROOT:-$HOME/workspace/medsci-skills}/skills/fulltext-retrieval/fetch_oa.py"
python3 "$ENGINE" <worklist> -o pdfs/ -e <contact-email> --report pdfs/retrieval_report.json
<worklist> is the DOI/PMID(/Title) list — the Phase-1 .bib DOIs, the worklist supplied
in the standalone mode below, or the project collection's DOIs. Output: pdfs/*.pdf for
/meta-analysis and pdf_to_md.py, plus
pdfs/retrieval_report.json (schema 2: retrieval status/source, source_identity,
and file_sha256). Keep the distinction between having a file and assessing its identity.
Route B — in-library PDFs (Zotero-native, higher yield, proxy-aware)
Emit ${MEDSCI_SKILLS_ROOT:-$HOME/workspace/medsci-skills}/skills/fulltext-retrieval/references/find_available_pdf.js
for the user to paste into Zotero (Tools → Developer → Run JavaScript) with the project
collection selected. It triggers Zotero's own addAvailablePDF/addAvailablePDFs, which
reuse the user's OpenURL resolver / institutional proxy — so it typically retrieves more
than OA-only, while no credentials or institutional identifiers enter this skill. The
no-code equivalent is right-click → "Find Available PDF". This route is user-initiated and
session-dependent; record its {attached, missing} summary from the printed JSON.
Report
Merge Route A's pdfs/retrieval_report.json (and the user-reported Route B summary) into
references/fulltext_retrieval.json (owner of this file is /lit-sync):
{
"schema_version": 2,
"retrieved_oa_disk": [{"doi": "...", "source": "unpaywall", "file": "...",
"file_sha256": "...", "title_match": "match",
"source_identity": {"status": "unresolved", "reason": "identifier_not_found"}}],
"retrieved_zotero_native": [{"doi": "...", "via": "addAvailablePDF"}],
"not_retrieved": [{"doi": "...", "journal": "..."}],
"institutional_fallback": ["<DOIs needing institutional access / ILL / author contact>"],
"title_mismatch_flagged": ["<DOIs whose downloaded PDF title did not match>"],
"identity_review_needed": ["<DOIs with conflict, unresolved, unavailable, or absent identity evidence>"]
}
Also append a short fulltext block (counts) to references/zotero_collection.json.
Copy each Route A source_identity object in full (abbreviated above), its file hash,
and the identity-status counts. retrieved_oa_disk counts file retrieval, not verified
papers. A consistent status is advisory corroboration, not claim verification; a
changed file hash needs re-assessment. Route B attachments and legacy reports without
identity evidence remain unassessed and appear in identity_review_needed until reviewed.
not_retrieved DOIs are candidates for institutional access, interlibrary loan, or author
contact — never bypass paywalls or access controls from this skill.
Phase 3: Obsidian Literature Notes
Step 3.1: Check existing literature notes
# Default English layout; substitute the vault's existing folder if one is present
# (e.g. "02 연구/문헌/" for a Korean-structured vault — see references/locale/ko/note_templates.md).
ls "$VAULT/Literature/" | grep -v "📊" | wc -l
Step 3.2: Create literature notes
For each .bib entry, create Literature/{citekey}.md (or the vault's existing literature folder).
Skip if the file already exists (never overwrite).
Citekey provenance — the note filename is a claim about the library
A literature note's filename and its citekey: field assert that an entry with that key
exists in Zotero. Every downstream use depends on it: [@key] in a manuscript, [[key]]
between notes, the Zotero Integration plugin writing {{citekey}}.md into the same folder.
A key that resolves to nothing turns all three into dead ends at once — and the note still
looks correct, which is why this goes unnoticed for months.
So the key is read, never composed:
- Take it from the
.bibentry, or ask Better BibTeX (item.searchover json-rpc — seereferences/bbt_lookup.md). - If the paper is not in Zotero, add it first (
zotero_add_by_doi) and let BBT mint the key. Phase 2 owns that step for a reason: a note written ahead of its library entry has no key to be right about. - If it cannot be added (no DOI, offline), write the note with
citekey: ""and the tag_needs-citekey. An empty field is recoverable; an invented one is not, because nothing downstream can tell it apart from a real key.
Verify before finishing:
python3 scripts/check_citekey_provenance.py --vault "$VAULT" --bib "$REFS_BIB"
Every reported INVENTED is a note whose key exists nowhere — fix it here rather than
letting it reach a manuscript.
Template
---
notetype: literature
citekey: "{citekey}"
title: "{title}"
authors: "{authors}"
journal: "{journal}"
year: {year}
doi: "{doi}"
pmid: "{pmid}"
created: "{today}"
tags:
- type/literature
- _unread
---
# {title}
## Bibliographic info
- **Authors**: {authors}
- **Journal**: {journal}{volume_issue_pages}
- **Year**: {year}
- **DOI**: [{doi}](https://doi.org/{doi})
{pmid_line}
## Key points (in my own words)
## My thoughts
## Related notes
- [[Research Hub]]
- [[Papers & Reviews]]
-
-
(For a Korean-structured vault, use the Korean-heading template in references/locale/ko/note_templates.md and the vault's own hub-note names.)
Rules:
notetype: literature— compatible with the Zotero Integration template._unreadtag — change to_readlater after the user reads the PDF in Zotero and adds highlights.- Leave
## Key pointsand## My thoughtsblank — the user fills these in personally. ## Related notescontains 2 hub links + 2 empty slots (reserved for later concept-note linking).- If a PMID is available, add a PubMed link.
Step 3.3: Result report
Obsidian Literature Notes:
Created: 8 notes (new)
Skipped: 3 notes (already exist)
Location: Literature/
Total in vault: 12 literature notes
Phase 4: Concept Extraction (conditional)
Trigger condition
Run this phase only when there are ≥10 literature notes in the vault. If fewer exist, print a status message like "N literature notes — concept extraction unlocks at ≥10" and stop.
Step 4.1: Cross-cutting concept scan
Read all files under Literature/*.md (or the vault's existing literature folder):
- Extract keywords from each paper's title, journal, and tags.
- Extract major concepts from the .bib entry titles.
- Identify concepts that co-occur across ≥3 literature notes.
Step 4.2: Filtering (5 exclusion rules)
Exclude from concept candidates:
- Model names (GPT-4, Claude, etc.).
- Dataset names (MedQA, ImageNet, etc.).
- Journal names.
- Institution names.
- Generic technique names (too unspecific).
Whatever remains becomes a concept-note candidate.
Step 4.3: Draft concept note
Create Concepts/{concept name}.md (or the vault's existing concept-note folder):
---
title: "{concept name}"
type: concept
tags:
- concept
- {domain tag}
aliases:
- {alternative name}
related_papers:
- "[[{lit-note-1}]]"
- "[[{lit-note-2}]]"
- "[[{lit-note-3}]]"
status: 🌱Seedling
---
# {concept name}
## Definition (My Understanding)
> TODO: write in your own words
## Why it matters
{why the concept matters in this domain — AI supplies a draft}
## Per-paper perspectives
- **[[{lit-note-1}]]**: {this paper's angle}
- **[[{lit-note-2}]]**: {a different angle}
- **[[{lit-note-3}]]**: {comparison / complement}
## Related concepts
- [[{another concept}]]
## Open questions
- {open question 1}
- {open question 2}
## Related notes
- [[Research Hub]]
- [[{related project hub}]]
- [[{lit-note-1}]]
- [[{lit-note-2}]]
(For a Korean-structured vault, use the Korean-heading concept template in references/locale/ko/note_templates.md.)
Key rules:
- Keep the
## Definitionsection as a> TODOmarker — the 2nd-layer note only becomes meaningful once the user writes the definition in their own words. statusalways starts at🌱Seedling.- At least 4 wikilinks under
## Related notes(vault convention).
Step 4.4: Propose to the user
Concept-note candidates (≥3 papers cross-referenced):
1. {Concept A} (4 papers)
2. {Concept B} (3 papers)
3. {Concept C} (5 papers)
Create? (all / selected / skip)
Create only after user confirmation. Auto-draft but always confirm.
Standalone Modes
This skill can run without a fresh .bib file.
Concept extraction only
On an explicit concept-extraction request, scan existing
Literature/*.md (or the vault's existing literature folder) and run only Phase 4.
References tidy
On a "tidy this project's references" request, locate .bib files inside the
workspace and run Phase 1–3.
Zotero sync only
On a "sync Zotero" request, diff the Zotero collection against the .bib file
and add whatever is missing.
PMID-list ingestion (no .bib)
When the user supplies a list of PMIDs (e.g., from a HANDOFF or a colleague), resolve PMIDs to DOIs via PubMed esummary first, then enter Phase 2 with the DOIs:
PMIDS="12345,67890,..."
curl -s "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed&id=${PMIDS}&retmode=json" \
| jq -r '.result | to_entries[] | select(.key != "uids") | "\(.value.uid)\t\(.value.elocationid)\t\(.value.title)"'
For each resolved DOI, search-first with zotero_search_items, then call
zotero_add_by_doi — the search is what dedupes (add-by-doi alone does not). For items
already in the library (detected via zotero_search_items by DOI), use
zotero_manage_collections to attach them to the project collection without re-adding —
re-adding by URL/PubMed-URL would bypass the search dedup and create duplicates. Record both
added and existing items in references/zotero_collection.json.
If a PMID has no DOI in PubMed (rare; older papers, non-indexed), fall back to
zotero_add_by_url with the PubMed URL and mark the entry as no_doi: true.
Worklist ingestion (DOI/PMID/Title; no .bib)
When the user supplies a worklist file (a .tsv/.csv/.md table with a DOI column,
optional PMID/Title, or a plain DOI-per-line list — e.g. an SR include set), enter
Phase 2 directly from it: resolve any PMID-only rows to DOIs (esummary above), then run the
search-first dedupe + add loop. The same worklist file feeds Phase 2.7 Route A
(fetch_oa.py reads .tsv/.csv/.md/plain natively), so no reformatting is needed.
Safety Rules
- Never overwrite literature notes — the user may have added highlights or personal notes.
- Never auto-fill
## Definitionof a concept note — keep the TODO marker; the essence of the 2nd-layer note is the user's own wording. - Skip Zotero for entries without a DOI — ask the user to add those manually.
- Gracefully skip Zotero when the MCP is not connected — Obsidian notes are
created independently; but do NOT hand-edit
refs.bibto compensate (violates artifact contract). - Always record the collection key — report the key to the user when a new collection is created.
- Never write
refs.bibdirectly. Only Better BibTeX auto-export may write that file. If auto-export is broken, fix the Zotero setup rather than writing the file from this skill. - Owner-only execution. If the current user is a collaborator (no Zotero access per
SSOT.yamlreference_manager.required_for), abort with instructions to flag[@NEW:topic]placeholders in the manuscript and notify the owner. - Fulltext boundary (Phase 2.7). Retrieve full text only via OA APIs (the
/fulltext-retrievalengine) and the user-run Zotero "Find Available PDF" snippet (which uses the user's own proxy config). Never automate authenticated browser sessions, never bypass paywalls/access controls, and never hard-code institutional proxies, credentials, or hosts into this skill.not_retrieveditems are routed to institutional access / ILL / author contact, not worked around.
Anti-Hallucination
- Never fabricate DOIs, PMIDs, or citation metadata. All bibliographic data must come from the .bib file or API responses.
- Never auto-fill the "Definition (My Understanding)" section of concept notes. This must be written by the user.
- Never overwrite existing literature notes. User highlights and annotations may be present.
- If a DOI lookup fails, report the failure rather than guessing the metadata.
Files (medsci-skills)
-
references
-
locale
-
ko
-
note_templates.md 2.5 KB
# Korean (ko) locale — Obsidian vault layout + note templates > Opt-in Korean variant for `/lit-sync`. The skill defaults to English folder names > (`Literature/`, `Concepts/`) and English note headings. Use this layout when the user's > Obsidian vault already follows a Korean structure (the skill honors an existing layout) or > when the user explicitly prefers Korean notes. ## Vault folder layout (Korean) - Literature notes: `02 연구/문헌/{citekey}.md` - Concept notes: `02 연구/개념노트/{concept name}.md` ```bash # Step 3.1 — count existing literature notes ls "$VAULT/02 연구/문헌/" | grep -v "📊" | wc -l ``` ## Literature note template (Korean headings) ```markdown --- notetype: literature citekey: "{citekey}" title: "{title}" authors: "{authors}" journal: "{journal}" year: {year} doi: "{doi}" pmid: "{pmid}" created: "{today}" tags: - type/literature - _unread --- # {title} ## 서지 정보 - **저자**: {authors} - **저널**: {journal}{volume_issue_pages} - **연도**: {year} - **DOI**: [{doi}](https://doi.org/{doi}) {pmid_line} ## 핵심 내용 (내 언어로) ## 내 생각 ## 관련 노트 - [[🗺️ 연구 종합]] - [[🗺️ 논문과 리뷰]] - - ``` - Leave `## 핵심 내용` and `## 내 생각` blank — the user fills these in personally. - `## 관련 노트` contains 2 hub links + 2 empty slots (reserved for later concept-note linking). ## Concept note template (Korean headings) ```markdown --- title: "{concept name}" type: concept tags: - 🧠개념 - {domain tag} aliases: - {English/Korean alternative name} related_papers: - "[[{lit-note-1}]]" - "[[{lit-note-2}]]" - "[[{lit-note-3}]]" status: 🌱Seedling --- # {concept name} ## 정의 (My Understanding) > TODO: write in your own words ## 왜 중요한가 {why the concept matters in this domain — AI supplies a draft} ## 논문별 관점 - **[[{lit-note-1}]]**: {this paper's angle} - **[[{lit-note-2}]]**: {a different angle} - **[[{lit-note-3}]]**: {comparison / complement} ## 관련 개념 - [[{another concept}]] ## 열린 질문 - {open question 1} - {open question 2} ## 관련 노트 - [[🗺️ 연구 종합]] - [[{related project hub}]] - [[{lit-note-1}]] - [[{lit-note-2}]] ``` - Keep the `## 정의` section as a `> TODO` marker — the 2nd-layer note only becomes meaningful once the user writes the definition in their own words. - Never auto-fill `## 정의` of a concept note — keep the TODO marker. - At least 4 wikilinks under `## 관련 노트` (vault convention).
-
-
-
bbt_lookup.md 2.2 KB
# Asking Better BibTeX for a citekey Better BibTeX mints the citekey; nothing else is entitled to guess it. When a `.bib` snapshot is stale or absent, ask the running plugin instead of composing a key from the author and year. ## Is BBT answering? ```bash curl -s -m 5 -o /dev/null -w "%{http_code}\n" \ http://127.0.0.1:23119/better-bibtex/json-rpc # 200 = ready ``` A non-200 means Zotero is closed, or BBT has not finished starting — it registers the endpoint a few seconds after the app window appears. Do not read `~/Zotero/better-bibtex/read-only.json` to decide this: current releases keep their auto-export registrations elsewhere, so that file is routinely `[]` on a healthy install. ## Look up one paper ```bash curl -s -m 20 -X POST http://127.0.0.1:23119/better-bibtex/json-rpc \ -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","method":"item.search","params":["<title words or first author>"],"id":1}' \ | python3 -c 'import json,sys for i in json.load(sys.stdin)["result"]: print(i["citation-key"], "|", i.get("DOI", ""), "|", i.get("title", "")[:70])' ``` Match on DOI when the search returns more than one hit. Author-and-year alone will happily match the wrong paper in a library that holds several from the same group. ## Dump the whole library Useful for auditing an existing vault, where per-note lookups would take hours: ```bash curl -s -m 120 "http://127.0.0.1:23119/better-bibtex/export/library?/1/library.bibtex" \ -o library.bib ``` The `?/1/` is the library id, and the extension picks the translator. Other URL shapes (`?libraryID=1&translator=…`, a `.csljson` extension) return an error page of a few dozen bytes rather than an export — check the byte count before trusting the file. Feed that dump straight to the provenance gate: ```bash python3 scripts/check_citekey_provenance.py --vault "$VAULT" --bib library.bib --strict ``` ## What the keys look like BBT's default pattern is author + title words + year: `smithDeepLearningRadiology2024`. Keys shaped like `Smith_2024_Validation`, `smith2024validation`, or `Smith_2024_38471102` do not come from BBT. They come from something that needed a key and made one up — and they resolve against nothing.
-
-
scripts
-
check_citekey_provenance.py 7.6 KB
#!/usr/bin/env python3 """Check that every literature note's citekey exists in the reference library. A literature note filename and its ``citekey:`` field assert that an entry with that key exists in Zotero. When the key was composed rather than read, the note still looks correct while ``[@key]`` resolves to nothing, ``[[key]]`` points at no file, and the Zotero Integration plugin writes a second note under the real key. This reports the keys that exist nowhere, and — where the note carries a DOI the library also has — the real key it should have carried. Verdicts OK citekey is present in the library INVENTED citekey absent, but the note's DOI resolves to a real key (fixable here) UNRESOLVED neither citekey nor DOI is in the library (the paper was never added) NO_CITEKEY note has no citekey (recoverable; not a violation) FILENAME citekey is real but the filename disagrees with it Exit status is 0 unless --strict is given, matching the repo's gate convention. Usage check_citekey_provenance.py --vault ~/Vault/Literature --bib refs.bib check_citekey_provenance.py --vault ~/Vault --live --json audit.json --strict """ from __future__ import annotations import argparse import json import re import sys import urllib.error import urllib.request from pathlib import Path BBT_JSONRPC = "http://127.0.0.1:23119/better-bibtex/json-rpc" FRONTMATTER_RE = re.compile(r"\A---\n(.*?)\n---", re.S) BIB_ENTRY_RE = re.compile(r"@\w+\{([^,]+),(.*?)(?=\n@|\Z)", re.S) BIB_DOI_RE = re.compile(r"\bdoi\s*=\s*[{\"]([^}\"]+)", re.I) def field(frontmatter: str, name: str) -> str: m = re.search(rf'^{name}:\s*"?([^"\n]*)"?\s*$', frontmatter, re.M) return m.group(1).strip() if m else "" def load_bib(paths: list[Path]) -> tuple[set[str], dict[str, str]]: """Return (citekeys, doi -> citekey) across every .bib given.""" keys: set[str] = set() doi_to_key: dict[str, str] = {} for p in paths: try: text = p.read_text(encoding="utf-8", errors="ignore") except OSError as exc: print(f"warning: cannot read {p}: {exc}", file=sys.stderr) continue for key, body in BIB_ENTRY_RE.findall(text): key = key.strip() keys.add(key) doi = BIB_DOI_RE.search(body) if doi: doi_to_key.setdefault(doi.group(1).strip().lower(), key) return keys, doi_to_key def load_live() -> tuple[set[str], dict[str, str]]: """Ask the running Better BibTeX for the whole library. Falls back to an empty library (with a warning) when Zotero is not up, so the check degrades to whatever .bib files were supplied instead of dying. """ url = "http://127.0.0.1:23119/better-bibtex/export/library?/1/library.bibtex" try: with urllib.request.urlopen(url, timeout=120) as resp: text = resp.read().decode("utf-8", errors="ignore") except (urllib.error.URLError, OSError) as exc: print( f"warning: Better BibTeX did not answer ({exc}); " "open Zotero for a live check", file=sys.stderr, ) return set(), {} keys: set[str] = set() doi_to_key: dict[str, str] = {} for key, body in BIB_ENTRY_RE.findall(text): key = key.strip() keys.add(key) doi = BIB_DOI_RE.search(body) if doi: doi_to_key.setdefault(doi.group(1).strip().lower(), key) return keys, doi_to_key def iter_notes(root: Path): for path in sorted(root.rglob("*.md")): if any(part.startswith(".") for part in path.parts): continue try: head = path.read_text(encoding="utf-8", errors="ignore")[:4000] except OSError: continue m = FRONTMATTER_RE.match(head) if not m: continue fm = m.group(1) if field(fm, "notetype") != "literature": continue yield path, fm def main() -> int: ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) ap.add_argument("--vault", required=True, type=Path, help="vault root or the literature folder inside it") ap.add_argument("--bib", type=Path, nargs="*", default=[], help="one or more .bib snapshots to check against") ap.add_argument("--live", action="store_true", help="also query the running Better BibTeX for the full library") ap.add_argument("--json", type=Path, help="write the full audit here") ap.add_argument("--strict", action="store_true", help="exit 1 when any INVENTED note is found") args = ap.parse_args() if not args.vault.exists(): print(f"error: vault not found: {args.vault}", file=sys.stderr) return 2 if not args.bib and not args.live: print("error: give --bib and/or --live; there is nothing to check against", file=sys.stderr) return 2 keys, doi_to_key = load_bib(list(args.bib)) if args.live: live_keys, live_dois = load_live() keys |= live_keys for doi, key in live_dois.items(): doi_to_key.setdefault(doi, key) if not keys: print("error: reference library is empty — every note would be reported as " "unresolved, which says nothing. Check the .bib path or open Zotero.", file=sys.stderr) return 2 rows = [] counts = {"OK": 0, "INVENTED": 0, "UNRESOLVED": 0, "NO_CITEKEY": 0, "FILENAME": 0} for path, fm in iter_notes(args.vault): citekey = field(fm, "citekey") doi = field(fm, "doi").lower() suggestion = doi_to_key.get(doi, "") if doi else "" if not citekey: verdict = "NO_CITEKEY" elif citekey in keys: verdict = "OK" if path.stem == citekey else "FILENAME" suggestion = citekey if verdict == "FILENAME" else "" elif suggestion: verdict = "INVENTED" else: verdict = "UNRESOLVED" counts[verdict] += 1 if verdict != "OK": rows.append({ "file": str(path), "verdict": verdict, "citekey": citekey, "doi": doi, "suggested_citekey": suggestion, }) total = sum(counts.values()) print(f"literature notes checked: {total} (library: {len(keys)} keys)") for verdict in ("OK", "FILENAME", "INVENTED", "UNRESOLVED", "NO_CITEKEY"): if counts[verdict]: print(f" {verdict:<11} {counts[verdict]}") for row in rows[:20]: name = Path(row["file"]).name arrow = f" -> {row['suggested_citekey']}" if row["suggested_citekey"] else "" print(f" [{row['verdict']}] {name}: {row['citekey'] or '(none)'}{arrow}") if len(rows) > 20: print(f" ... {len(rows) - 20} more (use --json for the full list)") if args.json: args.json.parent.mkdir(parents=True, exist_ok=True) args.json.write_text( json.dumps( { "detector": "check_citekey_provenance", "counts": counts, "findings": rows, }, indent=2, ensure_ascii=False, ), encoding="utf-8", ) print(f"audit written: {args.json}") if counts["INVENTED"]: print(f"\n{counts['INVENTED']} note(s) carry a citekey that exists nowhere. " "Each one is a citation that will not resolve.") if args.strict and counts["INVENTED"]: return 1 return 0 if __name__ == "__main__": sys.exit(main())
-
-
tests
-
citekey_provenance_challenge.sh 4.6 KB
#!/usr/bin/env bash # Challenge card for skills/lit-sync/scripts/check_citekey_provenance.py. # # Run with no argument from CI; pass a path to point it at a mutated copy (see the # self-test below, which is how this card earns the right to be believed). # # The script decides whether a literature note's citekey is real. Its verdicts are what a # user acts on, and its --strict exit code is what a gate acts on. Both are asserted here # against a fixture built at runtime, so the card carries no committed vault to drift. # # The boundary that matters and is easy to get wrong: --strict fails on INVENTED only. # UNRESOLVED and FILENAME are reported but do not fail — a paper that was never added to # the library is not the same defect as a citekey that was composed. set -uo pipefail HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" SCRIPT="${1:-$HERE/../scripts/check_citekey_provenance.py}" [ -f "$SCRIPT" ] || { echo "cannot find check_citekey_provenance.py at $SCRIPT"; exit 2; } WORK="$(mktemp -d)" trap 'rm -rf "$WORK"' EXIT PASS=0 FAIL=0 check() { # check <label> <expected> <actual> if [ "$2" = "$3" ]; then printf ' PASS %s\n' "$1"; PASS=$((PASS+1)) else printf ' FAIL %s (expected %s, got %s)\n' "$1" "$2" "$3"; FAIL=$((FAIL+1)); fi } note() { # note <path> <citekey-or-empty> <doi-or-empty> <notetype> mkdir -p "$(dirname "$1")" { echo "---" [ -n "$2" ] && echo "citekey: \"$2\"" [ -n "$3" ] && echo "doi: \"$3\"" echo "notetype: $4" echo "---" echo echo "body" } > "$1" } # ---------------------------------------------------------------- fixture VAULT="$WORK/vault" BIB="$WORK/refs.bib" cat > "$BIB" <<'BIB' @article{realKey2020, title = {A real entry}, doi = {10.1000/real}, } @article{otherKey2021, title = {Another real entry}, doi = {10.1000/other}, } BIB note "$VAULT/realKey2020.md" "realKey2020" "10.1000/real" literature # OK note "$VAULT/wrongName.md" "otherKey2021" "10.1000/other" literature # FILENAME note "$VAULT/composedKey2021.md" "composedKey2021" "10.1000/other" literature # INVENTED (doi resolves) note "$VAULT/neverAdded2019.md" "neverAdded2019" "10.1000/absent" literature # UNRESOLVED note "$VAULT/noKey.md" "" "10.1000/real" literature # NO_CITEKEY note "$VAULT/concept.md" "notAKey" "" concept # skipped: not literature note "$VAULT/.trash/hidden.md" "alsoNotAKey" "" literature # skipped: dot-path OUT="$WORK/out.txt" python3 "$SCRIPT" --vault "$VAULT" --bib "$BIB" --json "$WORK/audit.json" > "$OUT" 2>&1 rc=$? check "exit 0 without --strict" 0 "$rc" count() { python3 -c " import json,sys d=json.load(open('$WORK/audit.json')) print(d['counts'].get('$1',0)) "; } check "OK counted once" 1 "$(count OK)" check "FILENAME counted once" 1 "$(count FILENAME)" check "INVENTED counted once" 1 "$(count INVENTED)" check "UNRESOLVED counted once" 1 "$(count UNRESOLVED)" check "NO_CITEKEY counted once" 1 "$(count NO_CITEKEY)" total=$(python3 -c " import json; d=json.load(open('$WORK/audit.json')); print(sum(d['counts'].values())) ") check "non-literature and dot-path notes skipped" 5 "$total" sugg=$(python3 -c " import json d=json.load(open('$WORK/audit.json')) print(next(r['suggested_citekey'] for r in d['findings'] if r['verdict']=='INVENTED')) ") check "INVENTED names the real key from the DOI" "otherKey2021" "$sugg" # ------------------------------------------------- --strict fires on INVENTED python3 "$SCRIPT" --vault "$VAULT" --bib "$BIB" --strict > /dev/null 2>&1 check "--strict exits 1 when INVENTED present" 1 "$?" # --------------------------------- --strict does NOT fire without INVENTED VAULT2="$WORK/vault2" note "$VAULT2/realKey2020.md" "realKey2020" "10.1000/real" literature # OK note "$VAULT2/neverAdded2019.md" "neverAdded2019" "10.1000/absent" literature # UNRESOLVED note "$VAULT2/wrongName.md" "otherKey2021" "10.1000/other" literature # FILENAME python3 "$SCRIPT" --vault "$VAULT2" --bib "$BIB" --strict > /dev/null 2>&1 check "--strict exits 0 for UNRESOLVED/FILENAME alone" 0 "$?" # ------------------------------------------------------------ input guards python3 "$SCRIPT" --vault "$WORK/does-not-exist" --bib "$BIB" > /dev/null 2>&1 check "missing vault exits 2" 2 "$?" python3 "$SCRIPT" --vault "$VAULT" > /dev/null 2>&1 check "no --bib and no --live exits 2" 2 "$?" : > "$WORK/empty.bib" python3 "$SCRIPT" --vault "$VAULT" --bib "$WORK/empty.bib" > /dev/null 2>&1 check "empty library exits 2 instead of condemning every note" 2 "$?" printf '\n%d passed, %d failed\n' "$PASS" "$FAIL" [ "$FAIL" -eq 0 ] -
citekey_provenance_challenge_selftest.sh 3.4 KB
#!/usr/bin/env bash # Self-test for the citekey-provenance challenge card. # # A card that passes proves it RAN. It proves nothing about whether it would catch a # defect. So: mutate the detector five ways, each removing a behaviour the card claims to # assert, and require the card to FAIL every time. If any mutation slips through green, # the card is blind on that axis and this exits 1. # # The five mutations are the mistakes this detector is actually exposed to: # 1. --strict fails on UNRESOLVED too (a paper never added != a composed citekey) # 2. the empty-library guard is dropped (an empty .bib condemns every note, saying nothing) # 3. the notetype filter is dropped (concept notes get judged as literature notes) # 4. the DOI suggestion is disabled (INVENTED silently degrades to UNRESOLVED) # 5. the filename check is dropped (a note whose filename disagrees reads as OK) set -uo pipefail HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" CARD="$HERE/citekey_provenance_challenge.sh" SRC="$HERE/../scripts/check_citekey_provenance.py" [ -f "$CARD" ] || { echo "missing card: $CARD"; exit 2; } [ -f "$SRC" ] || { echo "missing detector: $SRC"; exit 2; } WORK="$(mktemp -d)" trap 'rm -rf "$WORK"' EXIT python3 - "$SRC" "$CARD" "$WORK" <<'PY' import pathlib, subprocess, sys src_path, card, work = sys.argv[1], sys.argv[2], sys.argv[3] src = pathlib.Path(src_path).read_text(encoding="utf-8") MUTATIONS = { "--strict fires on UNRESOLVED too": ( 'if args.strict and counts["INVENTED"]:', 'if args.strict and (counts["INVENTED"] or counts["UNRESOLVED"]):', ), "empty-library guard removed": ( " return 2\n\n rows = []", " pass\n\n rows = []", ), "notetype filter removed": ( 'if field(fm, "notetype") != "literature":\n continue', "if False:\n continue", ), "DOI suggestion disabled": ( 'suggestion = doi_to_key.get(doi, "") if doi else ""', 'suggestion = ""', ), "filename check removed": ( 'verdict = "OK" if path.stem == citekey else "FILENAME"', 'verdict = "OK"', ), } # The card must pass against the real detector, or the mutations below prove nothing. real = subprocess.run(["bash", card, src_path], capture_output=True, text=True) if real.returncode != 0: print("FAIL: the card does not pass against the unmutated detector") print(real.stdout[-2000:]) sys.exit(1) print(" PASS card is green against the real detector") blind = [] for label, (old, new) in MUTATIONS.items(): n = src.count(old) if n != 1: print(f" FAIL {label}: anchor matched {n} time(s) — the detector changed, " "so this self-test no longer tests what it claims") blind.append(label) continue mutant = pathlib.Path(work) / "mutant.py" mutant.write_text(src.replace(old, new), encoding="utf-8") r = subprocess.run(["bash", card, str(mutant)], capture_output=True, text=True) if r.returncode == 0: print(f" FAIL {label}: the card stayed GREEN — it is blind on this axis") blind.append(label) else: fired = sum(1 for l in r.stdout.splitlines() if l.strip().startswith("FAIL")) print(f" PASS {label}: card failed as it must ({fired} assertion(s) fired)") if blind: print(f"\n{len(blind)} mutation(s) went undetected: {', '.join(blind)}") sys.exit(1) print(f"\nall {len(MUTATIONS)} mutations caught") PY -
test_poll_logic.sh 3.2 KB
#!/usr/bin/env bash # Regression test for /lit-sync Phase 2.5 mtime-polling logic. # # Four synthetic scenarios (origin: ~/.local/cache/phase1b_b_dryrun/ dry-run, # 2026-04-24). Runs in an isolated tmpdir; does not touch any real Zotero or # project files. macOS (stat -f) and Linux (stat -c) compatible. # # Usage: bash skills/lit-sync/tests/test_poll_logic.sh # Exit: 0 all pass, 1 any scenario fails. set -u TMP=$(mktemp -d) trap 'rm -rf "$TMP"' EXIT # Inline poll script — same logic shipped with lit-sync Phase 2.5 guidance. POLL="$TMP/poll.sh" cat > "$POLL" <<'POLL_EOF' #!/usr/bin/env bash TARGET="$1" TIMEOUT="${2:-10}" if stat -f "%m" /dev/null >/dev/null 2>&1; then STAT_CMD='stat -f %m' else STAT_CMD='stat -c %Y' fi BEFORE=$($STAT_CMD "$TARGET") START=$(date +%s) while true; do NOW=$($STAT_CMD "$TARGET") if [[ "$NOW" != "$BEFORE" ]]; then ELAPSED=$(( $(date +%s) - START )) echo "DETECTED mtime change after ${ELAPSED}s" exit 0 fi if (( $(date +%s) - START >= TIMEOUT )); then echo "TIMEOUT after ${TIMEOUT}s" exit 1 fi sleep 0.5 done POLL_EOF chmod +x "$POLL" PASS=0 FAIL=0 run_scenario() { local name="$1" window="$2" write_at="$3" expected_exit="$4" local bib="$TMP/refs_${name//[^a-z0-9]/_}.bib" echo "@misc{test, title={x}}" > "$bib" # Age the file so mtime isn't "now" (macOS second-granularity could cause # the initial BEFORE to match a sub-second later write). touch -t 202001010000 "$bib" if [[ "$write_at" != "none" ]]; then ( sleep "$write_at" && echo "@misc{test, title={y}}" > "$bib" ) & WRITER_PID=$! else WRITER_PID="" fi "$POLL" "$bib" "$window" >/dev/null local actual=$? [[ -n "$WRITER_PID" ]] && wait "$WRITER_PID" 2>/dev/null if [[ "$actual" == "$expected_exit" ]]; then echo " PASS [$name] exit=$actual (expected $expected_exit)" PASS=$((PASS+1)) else echo " FAIL [$name] exit=$actual (expected $expected_exit)" FAIL=$((FAIL+1)) fi } echo "Phase 2.5 polling regression (4 scenarios)" echo " Tmpdir: $TMP" echo # Timing slack. The poller reads `date +%s`, so both START and the `>= TIMEOUT` # comparison are whole seconds: the deadline can fire up to ~1s early on top of any # sleep overshoot. Each scenario therefore needs its write and its deadline several # seconds apart, not one or two. # # Widening a DETECT scenario's window costs no runtime — it exits the moment it sees # the mtime change, so the window is only a deadline. Narrowing a TIMEOUT scenario's # window shortens it. The slack column is the gap these margins buy. # # window write_at slack # 1. BBT write within window → detect run_scenario "detect-within-window" 10 3 0 # 10 3 7s # 2. No write, short window → timeout run_scenario "timeout-silent" 3 none 1 # 3 none n/a # 3. BBT debounce, late-but-within write → detect run_scenario "debounce-late" 14 8 0 # 14 8 6s # 4. Slow BBT, write outside window → timeout (Phase 2.5 fallback prompt) run_scenario "slow-bbt-timeout" 2 8 1 # 2 8 6s echo echo "Summary: $PASS passed, $FAIL failed" [[ $FAIL -eq 0 ]]
-
-
SKILL.md 23 KB
--- name: lit-sync description: Sync research references from .bib files to Zotero library + Obsidian literature notes. Extract cross-cutting concept notes when enough literature accumulates. Works after /search-lit or standalone. triggers: lit-sync, 문헌 동기화, 레퍼런스 정리, 개념 노트 추출, lit sync, Zotero 동기화, reference sync, 참고문헌 옵시디언 tools: Read, Write, Edit, Bash, Grep, Glob model: inherit --- # Literature Sync: Zotero + Obsidian Pipeline Takes the `.bib` output of `/search-lit` (or any user-specified .bib file) and synchronizes the references into the Zotero library and Obsidian literature notes. When enough literature notes accumulate, extracts cross-cutting concept notes. ## Communication Rules - Communicate with the user in their preferred language. - **Vault layout — honor what exists, default to English.** Before creating notes, detect the vault's existing layout: if the vault already uses a particular folder structure (including a Korean one such as `02 연구/문헌/` and `02 연구/개념노트/`), **honor it — never silently rename a user's folders**. For a new or unclear vault, default to the English folders `Literature/` and `Concepts/` with the English note templates below. - A Korean opt-in variant (Korean folder layout + Korean-heading templates) lives in `references/locale/ko/note_templates.md` — use it when the vault is Korean-structured or the user prefers Korean notes. ## When to Use - After `/search-lit` completes — sync the produced .bib into Zotero + Obsidian. - Bulk-register references from an existing .bib into Zotero + Obsidian. - Tidy the `references/` folder inside a project workspace. - On explicit concept-extraction request → extract cross-cutting concepts from existing literature notes. ## Prerequisites - **Project owner only** — `/lit-sync` is an owner-scoped operation per `docs/zotero_policy.md`. Collaborators consume the committed `manuscript/_src/refs.bib` snapshot read-only. - Zotero desktop 7.x + Better BibTeX plugin installed. - Better BibTeX "Keep updated" auto-export configured to `<project>/manuscript/_src/refs.bib` (owner setup checklist in `docs/zotero_policy.md` §Setup). - Zotero MCP server available (skip the Zotero phase if not connected; auto-export refresh still fires once Zotero is reopened). - Obsidian CLI or direct file writing to the Obsidian vault. - Obsidian vault path: configured in user's environment (e.g., `$OBSIDIAN_VAULT`). ## Artifact Contract Per `docs/artifact_contract.md`, `/lit-sync` is the **sole writer** of: | Artifact | Writer | Readers | |---|---|---| | `manuscript/_src/refs.bib` | `/lit-sync` (via Better BibTeX auto-export trigger) | `/write-paper`, `/verify-refs`, `/manage-refs` | | `references/zotero_collection.json` | `/lit-sync` | `/verify-refs`, `/sync-submission` | Direct hand edits to `refs.bib` are drift — revert on sight. ## Pipeline Overview ``` .bib file (or /search-lit output) │ ▼ Phase 1: Parse Extract DOI, PMID, title, authors, journal, year │ ▼ Phase 2: Zotero Sync (owner) Dedupe → zotero_add_by_doi → place in collection → pin citekey │ ▼ Phase 2.5: refs.bib snapshot refresh Trigger Better BibTeX auto-export → verify manuscript/_src/refs.bib mtime updated │ ▼ Phase 2.7: Fulltext Retrieval (opt-in) Disk OA PDFs via /fulltext-retrieval + in-library via find_available_pdf.js → reconcile report │ ▼ Phase 3: Obsidian Literature Notes Create Literature/{citekey}.md (empty note OK — fill later with highlights) │ ▼ Phase 4: Concept Extraction (conditional) ≥10 literature notes → scan for cross-cutting concepts → propose concept notes ``` --- ## Phase 1: Parse BibTeX ### Input The user-specified .bib file path, or the .bib just produced by `/search-lit`. ### Process ```python # Parse .bib entries with regex. # Extract per entry: # - citekey — read it from the entry, never compose one. # Better BibTeX keys look like `smithDeepLearningRadiology2024` # (author + title words + year). A key shaped like `Smith_2024_Validation` # or `smith2024validation` was almost certainly invented rather than read, # and will not resolve against the library. See Step 3.2 §Citekey provenance. # - doi # - pmid # - title # - authors (first + last minimum) # - journal # - year # - volume, number, pages (if present) ``` Log any parse failures and skip those entries. --- ## Phase 2: Zotero Sync ### Step 2.1: Determine project collection Identify the project from the current working directory or from an explicit user override. Reuse an existing collection key if one is recorded; otherwise create a new collection. **Collection mapping**: Check existing Zotero collections for the current project. If no collection exists, create one with `zotero_create_collection`. Record the collection key for future use. ### Step 2.2: Dedupe + add For each entry: 1. Use `zotero_search_items` to search by DOI or title — if already present, skip. This search-first step is what prevents duplicates; `zotero_add_by_doi` does **not** dedupe by itself (it fetches CrossRef and creates the item), so never skip the search. 2. Otherwise call `zotero_add_by_doi` (when a DOI is available) or `zotero_add_by_url` (falling back to the PubMed URL when no DOI is available). - `zotero_add_by_doi` accepts an `attach_mode` argument that governs the **OA child-PDF attach attempt at add time** (the installed server treats `linked_url` as "bookmark the PDF URL"; other values download/import). Set it when you want a PDF attached during the add. Exact accepted values are server-version-specific — verify against the connected server. Do **not** use `zotero_add_from_file` to attach a PDF to an item added here: it has no parent-item argument and would create a duplicate parent item. 3. Use `zotero_manage_collections` to place the item in the project collection. ### Step 2.3: Result report ``` Zotero Sync: Added: 8 papers (new) Skipped: 3 papers (already in library) Failed: 1 paper (no DOI/PMID) Collection: RFA-Meta (TZQEP4NH) ``` If the Zotero MCP is not connected, skip this entire phase and proceed to Phase 3. Always write `references/zotero_collection.json` in the project workspace: ```json { "schema_version": 1, "status": "synced", "collection": "RFA-Meta", "collection_key": "TZQEP4NH", "added": 8, "skipped": 3, "failed": 1 } ``` If Zotero is unavailable, write the same file with `status: "skipped"` and a human-readable `reason`. --- ## Phase 2.5: refs.bib snapshot refresh Better BibTeX "Keep updated" auto-export normally refreshes `manuscript/_src/refs.bib` within seconds of a Zotero change. This phase **verifies** the snapshot actually updated before downstream skills consume it. ### Step 2.5.1: Resolve path Read `SSOT.yaml` → `truth.refs_bib`. Default: `manuscript/_src/refs.bib`. If absent (legacy project), fall back to `manuscript/_src/refs.bib` and emit a WARN recommending SSOT migration. ### Step 2.5.1b: Precondition assertion (early-exit, do NOT poll) Before entering the 10s polling loop in Step 2.5.2, verify both preconditions. If **either** fails, abort Phase 2.5 with setup instructions instead of waiting for a timeout that will never resolve. 1. **Better BibTeX is answering.** Probe the running plugin, not a file on disk: ```bash curl -s -m 5 -o /dev/null -w "%{http_code}" \ http://127.0.0.1:23119/better-bibtex/json-rpc # expect 200 ``` A non-200 means Zotero is closed or BBT has not finished starting. Retry once after Zotero's window is up; BBT registers its endpoint a few seconds after the app does. ⚠️ **Do not gate on `~/Zotero/better-bibtex/read-only.json`.** Current BBT releases keep auto-export registrations in their own store, so that file is routinely `[]` on a perfectly healthy install. Treating an empty list as "not configured" skips this phase on working setups — and a skipped Phase 2.5 is how a stale `refs.bib` and an invented citekey reach a manuscript. On failure print: > Phase 2.5 skipped: Better BibTeX did not answer on `127.0.0.1:23119` (HTTP `<code>`). Open Zotero, wait for it to finish loading, then re-run `/lit-sync`. 2. **Target refs.bib exists.** The resolved `truth.refs_bib` path from Step 2.5.1 must exist on disk (even empty is OK — BBT will overwrite). On failure print: > Phase 2.5 skipped: target snapshot `<path>` not found. Configure BBT auto-export with "On Change" to the SSOT path, then re-run. In either early-exit, set `refs_bib_refreshed: false` + `reason: "precondition:<which>"` in the Step 2.5.3 JSON and return control to the caller. Record it and tell the user; nothing downstream enforces it. `verify_refs.py` has never read this flag, and the sentence that said it did was the only thing standing between a stale `refs.bib` and a manuscript. ### Step 2.5.2: Verify refresh After Phase 2 adds items: 1. Capture `stat -f "%m" manuscript/_src/refs.bib` before Zotero writes. 2. Wait up to 10s (Better BibTeX debounce). Poll mtime. 3. If mtime unchanged after 10s: - Prompt user to check Zotero is running and BBT export is "Keep updated". - If BBT auto-export path is wrong, print the expected path (`<project>/manuscript/_src/refs.bib`) and refer to `docs/zotero_policy.md` §Setup. - As last resort, offer manual export: `File → Export Library → Better BibTeX → target path`. 4. Once mtime advances, grep for the newly added citekeys. All must be present; if any is missing, report as failure (do NOT fabricate entries). ### Step 2.5.3: Record in zotero_collection.json Append to the JSON written in Step 2.3: ```json { "refs_bib_path": "manuscript/_src/refs.bib", "refs_bib_mtime": "2026-04-24T14:32:11Z", "refs_bib_refreshed": true, "citekeys_verified": ["smithDeepLearningRadiology2024", "..."] } ``` If refresh failed, set `refs_bib_refreshed: false` and include `reason`. The flag records whether the export ran. It is a note to the reader, not a gate. --- ## Phase 2.7: Fulltext Retrieval (opt-in, owner-only) **Run only when the user asks for full text** (e.g. "download the PDFs", "fetch full text", or a worklist supplied with that intent). Default `/lit-sync` stays metadata-only and network-light — do not auto-run this phase. Runs after items are in Zotero (Phase 2) and the snapshot is verified (Phase 2.5), before Obsidian notes (Phase 3). There are two complementary retrieval routes; offer both and reconcile them in one report: ### Route A — disk OA PDFs (for downstream skills) Delegate to the `/fulltext-retrieval` engine (do **not** re-implement the OA cascade or import its code; invoke it by path). Resolve the engine as: ```bash ENGINE="${MEDSCI_SKILLS_ROOT:-$HOME/workspace/medsci-skills}/skills/fulltext-retrieval/fetch_oa.py" python3 "$ENGINE" <worklist> -o pdfs/ -e <contact-email> --report pdfs/retrieval_report.json ``` `<worklist>` is the DOI/PMID(/Title) list — the Phase-1 `.bib` DOIs, the worklist supplied in the standalone mode below, or the project collection's DOIs. Output: `pdfs/*.pdf` for `/meta-analysis` and `pdf_to_md.py`, plus `pdfs/retrieval_report.json` (schema 2: retrieval `status`/`source`, `source_identity`, and `file_sha256`). Keep the distinction between having a file and assessing its identity. ### Route B — in-library PDFs (Zotero-native, higher yield, proxy-aware) Emit `${MEDSCI_SKILLS_ROOT:-$HOME/workspace/medsci-skills}/skills/fulltext-retrieval/references/find_available_pdf.js` for the user to paste into Zotero (*Tools → Developer → Run JavaScript*) with the project collection selected. It triggers Zotero's own `addAvailablePDF`/`addAvailablePDFs`, which reuse the **user's** OpenURL resolver / institutional proxy — so it typically retrieves more than OA-only, while **no credentials or institutional identifiers enter this skill**. The no-code equivalent is right-click → "Find Available PDF". This route is user-initiated and session-dependent; record its `{attached, missing}` summary from the printed JSON. ### Report Merge Route A's `pdfs/retrieval_report.json` (and the user-reported Route B summary) into `references/fulltext_retrieval.json` (owner of this file is `/lit-sync`): ```json { "schema_version": 2, "retrieved_oa_disk": [{"doi": "...", "source": "unpaywall", "file": "...", "file_sha256": "...", "title_match": "match", "source_identity": {"status": "unresolved", "reason": "identifier_not_found"}}], "retrieved_zotero_native": [{"doi": "...", "via": "addAvailablePDF"}], "not_retrieved": [{"doi": "...", "journal": "..."}], "institutional_fallback": ["<DOIs needing institutional access / ILL / author contact>"], "title_mismatch_flagged": ["<DOIs whose downloaded PDF title did not match>"], "identity_review_needed": ["<DOIs with conflict, unresolved, unavailable, or absent identity evidence>"] } ``` Also append a short `fulltext` block (counts) to `references/zotero_collection.json`. Copy each Route A `source_identity` object in full (abbreviated above), its file hash, and the identity-status counts. `retrieved_oa_disk` counts file retrieval, not verified papers. A `consistent` status is advisory corroboration, not claim verification; a changed file hash needs re-assessment. Route B attachments and legacy reports without identity evidence remain unassessed and appear in `identity_review_needed` until reviewed. `not_retrieved` DOIs are candidates for institutional access, interlibrary loan, or author contact — never bypass paywalls or access controls from this skill. --- ## Phase 3: Obsidian Literature Notes ### Step 3.1: Check existing literature notes ```bash # Default English layout; substitute the vault's existing folder if one is present # (e.g. "02 연구/문헌/" for a Korean-structured vault — see references/locale/ko/note_templates.md). ls "$VAULT/Literature/" | grep -v "📊" | wc -l ``` ### Step 3.2: Create literature notes For each .bib entry, create `Literature/{citekey}.md` (or the vault's existing literature folder). **Skip if the file already exists** (never overwrite). #### Citekey provenance — the note filename is a claim about the library A literature note's filename and its `citekey:` field assert that an entry with that key exists in Zotero. Every downstream use depends on it: `[@key]` in a manuscript, `[[key]]` between notes, the Zotero Integration plugin writing `{{citekey}}.md` into the same folder. A key that resolves to nothing turns all three into dead ends at once — and the note still looks correct, which is why this goes unnoticed for months. So the key is **read, never composed**: 1. Take it from the `.bib` entry, or ask Better BibTeX (`item.search` over json-rpc — see `references/bbt_lookup.md`). 2. If the paper is not in Zotero, **add it first** (`zotero_add_by_doi`) and let BBT mint the key. Phase 2 owns that step for a reason: a note written ahead of its library entry has no key to be right about. 3. If it cannot be added (no DOI, offline), write the note with `citekey: ""` and the tag `_needs-citekey`. An empty field is recoverable; an invented one is not, because nothing downstream can tell it apart from a real key. Verify before finishing: ```bash python3 scripts/check_citekey_provenance.py --vault "$VAULT" --bib "$REFS_BIB" ``` Every reported `INVENTED` is a note whose key exists nowhere — fix it here rather than letting it reach a manuscript. #### Template ```markdown --- notetype: literature citekey: "{citekey}" title: "{title}" authors: "{authors}" journal: "{journal}" year: {year} doi: "{doi}" pmid: "{pmid}" created: "{today}" tags: - type/literature - _unread --- # {title} ## Bibliographic info - **Authors**: {authors} - **Journal**: {journal}{volume_issue_pages} - **Year**: {year} - **DOI**: [{doi}](https://doi.org/{doi}) {pmid_line} ## Key points (in my own words) ## My thoughts ## Related notes - [[Research Hub]] - [[Papers & Reviews]] - - ``` (For a Korean-structured vault, use the Korean-heading template in `references/locale/ko/note_templates.md` and the vault's own hub-note names.) **Rules:** - `notetype: literature` — compatible with the Zotero Integration template. - `_unread` tag — change to `_read` later after the user reads the PDF in Zotero and adds highlights. - Leave `## Key points` and `## My thoughts` blank — the user fills these in personally. - `## Related notes` contains 2 hub links + 2 empty slots (reserved for later concept-note linking). - If a PMID is available, add a PubMed link. ### Step 3.3: Result report ``` Obsidian Literature Notes: Created: 8 notes (new) Skipped: 3 notes (already exist) Location: Literature/ Total in vault: 12 literature notes ``` --- ## Phase 4: Concept Extraction (conditional) ### Trigger condition Run this phase only when there are **≥10** literature notes in the vault. If fewer exist, print a status message like "N literature notes — concept extraction unlocks at ≥10" and stop. ### Step 4.1: Cross-cutting concept scan Read all files under `Literature/*.md` (or the vault's existing literature folder): 1. Extract keywords from each paper's title, journal, and tags. 2. Extract major concepts from the .bib entry titles. 3. Identify **concepts that co-occur across ≥3 literature notes**. ### Step 4.2: Filtering (5 exclusion rules) Exclude from concept candidates: - Model names (GPT-4, Claude, etc.). - Dataset names (MedQA, ImageNet, etc.). - Journal names. - Institution names. - Generic technique names (too unspecific). Whatever remains becomes a concept-note candidate. ### Step 4.3: Draft concept note Create `Concepts/{concept name}.md` (or the vault's existing concept-note folder): ```markdown --- title: "{concept name}" type: concept tags: - concept - {domain tag} aliases: - {alternative name} related_papers: - "[[{lit-note-1}]]" - "[[{lit-note-2}]]" - "[[{lit-note-3}]]" status: 🌱Seedling --- # {concept name} ## Definition (My Understanding) > TODO: write in your own words ## Why it matters {why the concept matters in this domain — AI supplies a draft} ## Per-paper perspectives - **[[{lit-note-1}]]**: {this paper's angle} - **[[{lit-note-2}]]**: {a different angle} - **[[{lit-note-3}]]**: {comparison / complement} ## Related concepts - [[{another concept}]] ## Open questions - {open question 1} - {open question 2} ## Related notes - [[Research Hub]] - [[{related project hub}]] - [[{lit-note-1}]] - [[{lit-note-2}]] ``` (For a Korean-structured vault, use the Korean-heading concept template in `references/locale/ko/note_templates.md`.) **Key rules:** - Keep the `## Definition` section as a `> TODO` marker — the 2nd-layer note only becomes meaningful once the user writes the definition in their own words. - `status` always starts at `🌱Seedling`. - At least 4 wikilinks under `## Related notes` (vault convention). ### Step 4.4: Propose to the user ``` Concept-note candidates (≥3 papers cross-referenced): 1. {Concept A} (4 papers) 2. {Concept B} (3 papers) 3. {Concept C} (5 papers) Create? (all / selected / skip) ``` Create only after user confirmation. **Auto-draft but always confirm.** --- ## Standalone Modes This skill can run without a fresh .bib file. ### Concept extraction only On an explicit concept-extraction request, scan existing `Literature/*.md` (or the vault's existing literature folder) and run only Phase 4. ### References tidy On a "tidy this project's references" request, locate `.bib` files inside the workspace and run Phase 1–3. ### Zotero sync only On a "sync Zotero" request, diff the Zotero collection against the `.bib` file and add whatever is missing. ### PMID-list ingestion (no .bib) When the user supplies a list of PMIDs (e.g., from a HANDOFF or a colleague), resolve PMIDs to DOIs via PubMed esummary first, then enter Phase 2 with the DOIs: ```bash PMIDS="12345,67890,..." curl -s "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed&id=${PMIDS}&retmode=json" \ | jq -r '.result | to_entries[] | select(.key != "uids") | "\(.value.uid)\t\(.value.elocationid)\t\(.value.title)"' ``` For each resolved DOI, search-first with `zotero_search_items`, then call `zotero_add_by_doi` — the search is what dedupes (add-by-doi alone does not). For items already in the library (detected via `zotero_search_items` by DOI), use `zotero_manage_collections` to attach them to the project collection **without re-adding** — re-adding by URL/PubMed-URL would bypass the search dedup and create duplicates. Record both `added` and `existing` items in `references/zotero_collection.json`. If a PMID has no DOI in PubMed (rare; older papers, non-indexed), fall back to `zotero_add_by_url` with the PubMed URL and mark the entry as `no_doi: true`. ### Worklist ingestion (DOI/PMID/Title; no .bib) When the user supplies a worklist file (a `.tsv`/`.csv`/`.md` table with a `DOI` column, optional `PMID`/`Title`, or a plain DOI-per-line list — e.g. an SR include set), enter Phase 2 directly from it: resolve any PMID-only rows to DOIs (esummary above), then run the search-first dedupe + add loop. The same worklist file feeds Phase 2.7 Route A (`fetch_oa.py` reads `.tsv`/`.csv`/`.md`/plain natively), so no reformatting is needed. --- ## Safety Rules 1. **Never overwrite literature notes** — the user may have added highlights or personal notes. 2. **Never auto-fill `## Definition` of a concept note** — keep the TODO marker; the essence of the 2nd-layer note is the user's own wording. 3. **Skip Zotero for entries without a DOI** — ask the user to add those manually. 4. **Gracefully skip Zotero when the MCP is not connected** — Obsidian notes are created independently; but do NOT hand-edit `refs.bib` to compensate (violates artifact contract). 5. **Always record the collection key** — report the key to the user when a new collection is created. 6. **Never write `refs.bib` directly.** Only Better BibTeX auto-export may write that file. If auto-export is broken, fix the Zotero setup rather than writing the file from this skill. 7. **Owner-only execution.** If the current user is a collaborator (no Zotero access per `SSOT.yaml` `reference_manager.required_for`), abort with instructions to flag `[@NEW:topic]` placeholders in the manuscript and notify the owner. 8. **Fulltext boundary (Phase 2.7).** Retrieve full text only via OA APIs (the `/fulltext-retrieval` engine) and the user-run Zotero "Find Available PDF" snippet (which uses the user's own proxy config). Never automate authenticated browser sessions, never bypass paywalls/access controls, and never hard-code institutional proxies, credentials, or hosts into this skill. `not_retrieved` items are routed to institutional access / ILL / author contact, not worked around. ## Anti-Hallucination - **Never fabricate DOIs, PMIDs, or citation metadata.** All bibliographic data must come from the .bib file or API responses. - **Never auto-fill the "Definition (My Understanding)" section** of concept notes. This must be written by the user. - **Never overwrite existing literature notes.** User highlights and annotations may be present. - If a DOI lookup fails, report the failure rather than guessing the metadata. -
skill.yml 2.9 KB
schema_version: 2 name: lit-sync layer: A owner_domain: zotero_sync maturity: official when_to_use: - User wants to add PMIDs / DOIs / Zotero collection items to the project library - After /search-lit when verified candidates need to flow into Zotero + refs.bib - Cross-cutting concept-note extraction from accumulated literature into Obsidian - Refreshing manuscript/_src/refs.bib via Better BibTeX auto-export before a build - Opt-in full-text retrieval (Phase 2.7) — disk OA PDFs via /fulltext-retrieval + in-library "Find Available PDF" when_NOT_to_use: - Pure literature search without sync (use /search-lit) - Hand-editing refs.bib (BBT is the sole writer; never touch the file directly) - Reference verification audit (use /verify-refs) inputs: - references/library.bib # search-lit candidate pool (optional) - PMID list, DOI list, or Zotero collection name outputs: - references/zotero_collection.json - references/fulltext_retrieval.json # SOLE WRITER; opt-in Phase 2.7 retrieval report - manuscript/_src/refs.bib # SOLE WRITER via Better BibTeX auto-export "Keep updated" - obsidian_literature_notes deterministic_scripts: - none_required # leverages Zotero MCP + Better BibTeX auto-export GUI; Phase 2.7 invokes /fulltext-retrieval fetch_oa.py side_effects: - may_update_zotero - may_write_obsidian_notes - refreshes_manuscript_refs_bib # via BBT auto-export downstream_consumers: - write-paper - verify-refs - manage-refs # consumes refreshed refs.bib for citekey validation + CSL render ssot_boundary: - SOLE writer of manuscript/_src/refs.bib. write-paper, manage-refs, verify-refs are read-only consumers. quality_gates: - refs_bib_refreshed: refs.bib mtime newer than collection snapshot in zotero_collection.json - bbt_auto_export_active: Better BibTeX "Keep updated" must be ON for the project collection (verified per ~/.claude/rules/zotero-workflow.md) forbidden_actions: - fabricate_bibliographic_metadata - overwrite_existing_literature_notes - hand_edit_manuscript_refs_bib # only Better BibTeX may write # v2.1 quality card purpose: "Sync verified references from .bib into Zotero and Obsidian literature notes, extracting cross-cutting concept notes when enough literature accumulates." safety_boundaries: - "Bibliographic metadata is never fabricated; only Better BibTeX may write refs.bib." - "Existing literature notes are not overwritten." known_limitations: - "Depends on a connected Zotero MCP + Better BibTeX auto-export; degrades to manual without them." - "No standalone demo; effects are in the user's Zotero/Obsidian." validation_commands: - "confirm refs.bib mtime refreshed via Better BibTeX" - "zotero_find_duplicates after sync" - "bash skills/lit-sync/tests/citekey_provenance_challenge.sh" - "bash skills/lit-sync/tests/citekey_provenance_challenge_selftest.sh" evidence_surface: manual_workflow
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.