interface:
  display_name: "DeepPaperNote"
  short_description: "Generate a high-quality deep-reading note for one paper with a raw-source manifest workflow and Obsidian-oriented save semantics."
  default_prompt: "Use $deeppapernote to create the deep-reading note for this paper."

# Architecture

This skill should be implemented as:
- a reusable core workflow
- thin platform adapters

That separation keeps the project useful even if the host environment changes later.

## Layer 1: Reusable Core

The reusable core includes:
- paper resolution logic
- metadata aggregation
- PDF acquisition strategy
- evidence extraction
- figure planning
- synthesis-bundle assembly
- note-quality checks
- Markdown note rendering constraints
- JSON contracts between stages

These pieces should live primarily in:
- `scripts/`
- `references/`

The core should be agent-agnostic wherever possible.

## Layer 2: Platform Adapter Layer

A platform adapter can include:
- `SKILL.md`
- `agents/openai.yaml`
- repository-level plugin manifests such as `.claude-plugin/plugin.json` and `.codex-plugin/plugin.json`
- trigger phrasing
- tool-selection instructions for the host environment
- interaction-style notes
- host-specific Obsidian or local-library calling conventions

This layer should stay thin.
Do not bury core business logic only inside prompt text or distribution metadata.

## Design Rule

When adding a new behavior, ask:

1. Would another agent framework also need this behavior?
   - If yes, put it in the core.
2. Is this only about how one host environment discovers or invokes the workflow?
   - If yes, put it in the adapter layer.

## What Belongs in Scripts

Put deterministic or repeated logic in scripts:
- normalization
- parsing
- metadata merge
- PDF discovery
- evidence-pack assembly
- synthesis-bundle assembly
- contract validation
- linting
- file writing

Do not put paper understanding into scripts:
- deciding the paper's real contribution
- choosing which result matters most
- reconstructing the true method chain
- writing the final deep-reading note prose

Those tasks belong to the language model after the deterministic bundle is prepared.

## What Belongs in References

Put durable reasoning guidance in references:
- what counts as a high-quality note
- how to adapt to paper types
- figure placement heuristics
- formatting rules
- source-priority rules

## What Belongs in SKILL.md

Keep only:
- when the skill should trigger
- the high-level workflow
- which scripts to use
- which references to read
- the non-negotiable workflow constraints that must remain visible at the entrypoint

## Portability Goals

A future adapter on another platform should be able to reuse:
- the same scripts
- the same contracts
- the same note-format rules
- the same evidence-first workflow

It should only need a different outer adapter.

## Anti-Patterns

Avoid:
- embedding essential contracts only in prompt text
- mixing platform-specific phrasing into script outputs
- writing natural-language-only intermediate artifacts when structured JSON is possible
- allowing note quality to depend on undocumented one-off prompt behavior

# Deep Analysis

Language note: the analytical standard is language-independent. Draft in the configured output language; for `en`, use the exact headings and labels in `output-language.md`. Chinese-specific examples below apply only to `zh-CN`.

Use this guide when the user wants a note that feels like a real research note rather than a cleaned-up summary.

## Goal

Produce a paper note in the configured language that helps future rereading answer:
- this paper is really solving what problem
- the core route or method chain is what
- which evidence actually supports the claim
- where the paper is weak, bounded, or easy to misread
- whether it is worth follow-up, comparison, implementation, or citation

## Key Principle

Do not treat deterministic script output as the final note.

Scripts in `DeepPaperNote` are for:
- resolving the paper
- fetching metadata and PDF
- extracting evidence and PDF assets
- planning figure/table candidates
- linting and writing files

The real value comes from the model reading the available material and writing the note in its own words.

## Source Priority

Use sources in this order:

1. synthesis bundle metadata
2. evidence extracted from the full PDF
3. figure/table captions and candidate assets
4. abstract metadata only for identity and context, not as a substitute for a finished deep note

For a finished long-term note, require a usable PDF-backed evidence path.

If you only have the abstract after the supported PDF acquisition paths have failed, stop and ask for a usable PDF, OCR, or source material rather than writing a provisional or abstract-only note.

## Analysis Checklist

Use the synthesis bundle's paper-type contract and decide:
- which sections deserve the most weight
- which details need `###` subheadings
- which 3 to 6 numbers matter most
- which central claims are supported by which source sections or pages
- what each central claim actually proves and does not prove
- which negative, weak, missing, or limiting results constrain the conclusion
- which research or engineering takeaways are specific enough to reuse
- which figure/table placeholders are essential
- whether the paper needs explicit formulas, objective functions, or complexity expressions

## Writing Rules

- Write for future rereading, not for one-time display.
- Prefer interpretation over translation.
- Prefer “这篇论文真正有价值的点是...” over “本文提出了...” style filler.
- Avoid pasting long English sentences into Chinese sections.
- Do not fabricate metrics, ablations, or claims not supported by evidence.
- When an individual claim has weak evidence within an otherwise sufficient Source Corpus, narrow the wording and state the claim boundary explicitly.
- For method papers, write like a replication-minded researcher rather than a summary assistant.

## Section Guide

### 核心信息

Must include:
- title
- authors
- affiliations or institutions when available
- published date
- venue or journal when available
- DOI
- source URL
- code repo or project page when available
- domain

### 一句话总结

Do not paraphrase the abstract.

Answer:
- what the paper's real contribution is
- what the title may overstate

### 研究问题

Answer:
- the concrete pain point
- why existing methods are not enough
- whether this is a new problem, a new angle on an old problem, or a more realistic reformulation

### 数据与任务定义

Must separate:
- where the data comes from
- what labels or supervision exist
- what the actual task is
- what the paper is not predicting

For clinical or social-science papers, spell out:
- collection setting
- weak supervision risks
- annotation or rating assumptions
- whether the task is realistic or simplified

### 方法主线

This is usually where a shallow note fails.

Explain:
- the information flow
- what each stage consumes and produces
- what the model is actually doing
- what is standard versus paper-specific
- what the training target or optimization target really is
- how inference or sampling actually proceeds
- which implementation details matter for reproducing the claimed gain
- even if an extracted Algorithm block is broken, reconstruct the mechanism in plain engineering language rather than giving up
- make the reader feel the Input -> key transformation -> Output flow, not just the paper's terminology

For method, framework, or system papers:
- default to an explicit `### 机制流程` subsection inside `方法主线`
- write it as a 3 to 4 step numbered list rather than a long paragraph
- each step should say what goes in, what operation happens, and where the output goes next
- if the paper has both training and inference details, use `### 机制流程` for the dominant execution chain and leave training recipe details to neighboring subsections

For complex papers, use `###` subheadings such as:
- `### 机制流程`
- `### 数据构建`
- `### 中间表征抽取`
- `### 模型结构`
- `### 训练与推理`

### 关键结果

Do not dump all metrics.

Include:
- the most important comparison
- the most important numbers
- at least one result that looks strong
- at least one result that limits the claim

For method papers, also ask:
- does the result support the claimed mechanism
- is the gain internal-only or external too
- if the paper reports ablations or removed-module comparisons, include at least one setting that hurt performance, made training unstable, or revealed a trade-off
- if the evidence bundle contains no such negative ablation signal, say explicitly that the paper did not clearly report failed or unstable settings

### 深度分析

This is the most important part.

Include:
- research value
- practical value
- why the method may work
- where the evidence is still thin
- hidden assumptions
- what the paper does not prove

Use the plan's `central_claims` as the spine of this section:
- connect each major claim to the evidence that supports it
- say exactly what the evidence proves
- say what remains unproven, untested, or only indirectly supported
- use `mechanism_result_map` to explain why the paper's mechanism, protocol, construct, or data decision should produce the observed result pattern
- use `comparative_positioning` to say what changes relative to strong baselines or obvious alternatives, not only that the paper is better
- when the paper has Discussion or Limitations, explain the mechanism behind those caveats rather than copying them as a list

Good subsections often include:
- `### 真正贡献是什么`
- `### 为什么结果成立`
- `### 哪些地方容易被误读`
- `### 训练目标`
- `### 推理与采样链路`
- `### 复杂度与扩展性`

### 局限

Write real limitations, not polite filler.

Prefer:
- dataset or sampling boundaries
- label leakage or weak-supervision risks
- evaluation mismatch
- deployment gap
- missing baselines
- unrealistic task framing

### 我的笔记

Seed future follow-up with prompts such as:
- one reusable idea
- one questionable assumption
- one experiment worth replicating
- one related paper to compare next

## Figures And Tables

When the paper has useful visuals:
- preserve placeholders for the important ones
- prioritize one method figure, one data/task figure, and one result figure or table
- if a high-confidence pipeline or architecture figure clearly matches the core execution chain, place it in `### 机制流程` first
- explain why each figure matters
- keep original paper numbering such as `Fig. 1` or `Table 2`

Do not dump every extracted image into the note body.

## Formula Guidance

If a formula is central to understanding the method, do not leave it out just because the rest of the prose reads smoothly.

Typical cases where a formula should appear:
- probability factorization
- optimization objective
- loss definition
- complexity comparison
- scaling-law fit

Prefer a few stable, well-explained formulas over many noisy ones.
- after each retained formula, add one short engineering explanation of what it means in implementation terms
- do not stop at naming variables; explain what operation, objective term, or state update the formula corresponds to

## Minimum Honesty Standard

A finished note requires sufficient PDF-backed source coverage. Within that coverage, match each claim's strength to its evidence and state uncertainty explicitly.

# User-editable domain routing taxonomy.
# Application domains are tried before fallback method domains.
domains:
  - label: 医疗健康
    aliases:
      - healthcare
      - medical
      - clinical medicine
    specialized_folders:
      - 心理健康
    keywords:
      - clinical
      - patient
      - patients
      - depression
      - anxiety
      - mental health
      - psychiatric
      - psychology
      - therapy
      - counseling
      - symptom
      - diagnosis
      - screening
      - hospital
      - healthcare
      - medical
    methods: []

  - label: 法律
    aliases:
      - legal
      - law
    keywords:
      - legal
      - law
      - court
      - judge
      - contract
      - statute
      - regulation
      - litigation
      - case law
    methods: []

  - label: 教育
    aliases:
      - education
      - educational
    keywords:
      - education
      - student
      - teacher
      - classroom
      - curriculum
      - tutoring
      - learning analytics
      - pedagogy
    methods: []

  - label: 金融
    aliases:
      - finance
      - financial
    keywords:
      - finance
      - financial
      - stock
      - market
      - trading
      - portfolio
      - risk
      - credit
      - banking
      - investment
    methods: []

  - label: 机器人
    aliases:
      - robotics
      - robotic
    keywords:
      - robot
      - robotics
      - robotic
      - manipulation
      - navigation
      - control policy
      - locomotion
      - autonomous driving
      - embodied
    methods:
      - diffusion policy

  - label: 软件工程
    aliases:
      - software engineering
    keywords:
      - software engineering
      - code generation
      - program repair
      - bug
      - repository
      - developer
      - code review
      - test generation
      - compiler
    methods: []

  - label: 生物医学
    aliases:
      - biomedical
      - bioinformatics
    keywords:
      - biomedical
      - genomics
      - protein
      - drug discovery
      - molecular
      - cell
      - gene
      - bioinformatics
    methods: []

  - label: 心理健康
    route_to: 医疗健康
    aliases:
      - mental health
      - psychology
      - psychiatry
    keywords:
      - depression
      - anxiety
      - mental health
      - psychiatric
      - psychology
      - therapy
      - counseling
      - symptom
    methods: []

  - label: 推荐系统
    aliases:
      - recommender systems
      - recommendation
    keywords:
      - recommendation
      - recommender
      - ctr prediction
      - ranking system
      - personalization
    methods: []

fallback_domains:
  - label: 大模型
    aliases:
      - llm
      - large language model
      - language model
      - foundation model
    keywords:
      - large language model
      - llm
      - foundation model
      - gpt
      - transformer
      - instruction tuning
      - pretrain
      - pre-training
      - language model
      - agent
      - multi-agent
      - multi agent
      - reasoning
      - multimodal
      - retrieval-augmented generation
      - rag
      - in-context learning
      - long-context
      - long context
      - mixture-of-experts
      - mixture of experts
      - moe
      - alignment
      - rlhf
    methods: []

  - label: 机器学习
    aliases:
      - machine learning
      - ml
    keywords:
      - machine learning
      - deep learning
      - neural network
      - representation learning
      - reinforcement learning
      - computer vision
      - graph neural network
      - speech recognition
    methods: []

# Evidence-First Note Writing

Use this guide when the goal is to approach the quality of a hand-written research note rather than a template-filled summary.

## Core Rule

Do not write the finished note directly from:
- the title
- the abstract
- one or two extracted snippets
- fixed headings alone

## Source Manifest Bundle

The source manifest and bundle should answer:
- what type of paper this is
- which parts of the PDF were actually found
- where the raw source text lives
- which numbers matter
- which datasets, metrics, baselines, or cohorts matter
- which figures are method figures, data figures, and result figures
- which conclusions are about scale, transfer, cost, limitations, or practical value
- which central claims have source evidence and which boundaries the evidence cannot cross

In `DeepPaperNote`, use:
- `scripts/run_pipeline.py`
- `scripts/extract_source_text.py`
- `scripts/build_synthesis_bundle.py`

## Explicit Note Plan

Before drafting the final note, the agent should create an explicit short planning artifact rather than silently "thinking it through" and jumping straight to the final Markdown.

Require a compact and inspectable JSON plan that satisfies the generated bundle contract. The bundle is authoritative for required fields, allowed paper types, and grounding expectations; do not expose a long free-form chain-of-thought block.

The plan should state:
- which sections this paper actually deserves
- which sections need more technical depth
- which subsections deserve `###` headings
- which evidence feeds each section
- which 3 to 6 numbers matter most
- which comparisons are the real ones
- which central claims are supported by which source sections or pages
- what each central claim actually proves and does not prove
- which weak, negative, limiting, or explicitly unreported results constrain the conclusion
- which research or engineering takeaways are reusable beyond the current paper
- whether this is mostly a method note, system note, dataset note, benchmark note, or empirical/clinical note
- whether key formulas or complexity expressions need to appear in the final note

Good note plans often add paper-specific sections such as:
- `### 数据构建`
- `### 量表代理特征抽取`
- `### 训练细节`
- `### 关键洞察`
- `### 为什么结果不等于临床可用`

Recommended shape:

```json
{
  "output_language": "zh-CN",
  "paper_type": "AI_method",
  "paper_type_rationale": "The paper proposes a model mechanism and evaluates it against baselines; the script suggestion was treated only as a hint.",
  "dominant_domain": "machine learning",
  "must_cover": ["数据构建", "方法主线", "关键消融"],
  "key_numbers": ["主结果提升 3.2 points", "训练成本降低 40%"],
  "real_comparisons": ["against the strongest reported baseline"],
  "central_claims": [
    {
      "claim": "The proposed mechanism improves multi-step tool use reliability.",
      "supporting_evidence": [{"section_id": "sec:experiments"}, {"pages": [7, 8]}],
      "what_it_actually_proves": "The reported benchmark settings show fewer unrecoverable tool errors than the named baseline.",
      "what_it_does_not_prove": "It does not prove robustness to arbitrary tools or production latency failures."
    }
  ],
  "claim_boundaries": ["The result is tied to the paper's tool set and benchmark distribution."],
  "negative_or_limiting_results": ["The paper does not clearly report a failed external-tool setting."],
  "mechanism_result_map": ["The state transition design explains the lower unrecoverable-error rate by preserving failed tool-call state for later repair."],
  "comparative_positioning": ["Compared with answer-only baselines, the paper evaluates a mechanism that keeps intermediate failure states inspectable."],
  "reuse_takeaways": ["Track tool failures as first-class state rather than hiding them in the final answer."],
  "followup_questions": ["Does the same state logging still help when external tools are slow, missing, or adversarially noisy?"],
  "section_plan": [
    {
      "section": "方法主线",
      "weight": "high",
      "subsections": ["机制流程", "训练目标"],
      "evidence_sources": [{"section_id": "sec:method"}, {"pages": [4, 6]}]
    }
  ]
}
```

Before drafting from this plan, run `scripts/lint_grounding.py --note-plan ... --source-manifest ... --bundle-json ... --figure-decisions ...`.

The plan should be short, structured, and directly useful for the final draft.

## Writing Layer

Only after the evidence bundle and explicit JSON `note_plan` exist should the model draft the final note.

Good final notes should:
- prioritize numbers and comparisons over generic summary sentences
- add paper-specific subsections when the evidence supports them
- avoid abstract-only rewriting
- explain why a figure or table matters, not just attach it
- separate “作者声称了什么” from “论文真正证明了什么”
- carry the plan's claim boundaries into `深度分析` and `局限`
- explain the mechanism deeply enough that an engineer could re-explain or re-implement the main flow

## Minimum Quality Bar

If the note does not clearly contain:
- the most important numbers
- the most important comparison
- one paper-specific insight
- one honest limitation
- one technically detailed subsection
- and, when necessary, one key formula or formal expression

then the note is still too close to a template summary.

# Figure Placement

Language note: target sections and figure-callout labels must follow the generated writing contract. For English output, use the labels in `output-language.md`.

In MVP, the skill must plan figure placement even when it cannot extract image files.

## Goal

Plan placeholders for every high-value figure or table that materially helps the note.
Do not collapse the paper down to only 1 to 3 items if the paper clearly has more important visuals.

## What to Prefer

Priority order:
1. study overview or method overview figure
2. data or task-definition figure
3. key result figure or table
4. other supporting figures that clarify a major argument

## Placement Logic

- Put method overview figures in `### 机制流程` when they directly explain the core execution chain
- If the match is weaker or the note does not need that micro-structure, keep them in `方法主线`
- Put data or task figures in `数据与任务定义`
- Put main result figures or tables in `关键结果`
- Put conceptual diagrams in `研究问题` or `深度分析` if they clarify the argument

## What to Read

Use:
- figure captions
- nearby正文对 figure 的引用
- section context
- candidate pages and candidate images from deterministic PDF asset extraction

Do not place figures by paper order alone.
Do not let scripts make the final semantic choice; scripts should only prepare candidates.

## Placeholder-First Rule

- The final note should first have the right placeholder structure.
- If a usable image is extracted and semantically matched with high confidence, replace that placeholder with the real image.
- If a reliable image is not available, keep the placeholder.
- Never silently remove a figure just because extraction failed.
- Text correctness is more important than image completeness.

## Machine Contract Boundary

The generated `writing_contract.figure_table_contract` owns figure/table fields, decision values, candidate statuses, and inspection requirements. Use that contract instead of treating this visual-judgment guide as a second schema.

When the contract requires manual inspection, open the actual candidate image before claiming that a visual review found a defect. Keep missing candidates separate from real materialization failures: `asset_candidate_missing`, an empty source image path, or no independent matching crop means no usable candidate was available; materialization failure means a selected real asset later failed to copy or write.

## Integrated Placement Rule

Every kept placeholder must be placed directly under the most relevant substantive section named by its `建议位置`.
Do not collect unresolved placeholders into a catch-all section such as `剩余图表占位`, `未放置图表`, `Remaining figures`, or `Leftover figures`.

Rejecting an extraction candidate does not by itself require a final-note placeholder. The final placeholder set should come from semantic importance to the note, not from the number of failed extraction candidates.

For survey papers with many representative project figures, appendix tables, or repetitive supplemental visuals:
- keep a callout only when the visual materially helps the reader understand the argument
- otherwise summarize the pattern in prose or point the reader back to the appendix/source paper
- do not stack low-value callouts just to demonstrate that the pipeline saw them

## Visual Quality Gate

Figure/table insertion has two separate gates:
- identity match: the candidate label, caption, and local context match the planned figure/table
- visual usability: the crop actually contains the visual body needed by the reader

A label or caption match is not insertion approval.
Deterministic signals may reject an obvious defect, but only an actual image inspection may approve insertion.
Fail closed when visual usability is weak: keep the placeholder instead of inserting the candidate.

Every selected candidate begins as `review_pending`. Review it after the synthesis bundle exists and before `note_plan`:

1. Open the exact candidate crop and its complete source page preview.
2. Compare them with the matched Figure/Table identity and external caption.
3. Confirm that the crop is a Visual-Body Crop: the complete scientific visual is present, all internal labels remain readable, and the external caption and surrounding paper prose are absent.
4. Record the review in that candidate's existing decision entry using the generated `writing_contract.figure_table_contract.visual_review` fields and values.

A passing review is valid only for the decision entry's current asset SHA-256. Any asset change returns the item to `review_pending`. Complete the Figure/Table Decision Freeze before planning or drafting; grounding must reject an unresolved or stale review.

## Visual-Body Crop Boundary

Keep axes, tick labels, legends, color bars, scale bars, panel letters, table headers, method labels, and other text that belongs inside the scientific visual. Keep a small safety margin so edge labels are not clipped. Exclude the source caption and unrelated running prose from the image pixels while retaining the caption as metadata and later Markdown explanation.

If the external caption cannot be separated without damaging the scientific visual, fail closed and keep the placeholder. The first implementation reviews complete Figure/Table bodies only; do not split panels into separate assets.

## Bounded Crop Repair

Use a Bounded Crop Repair only for a geometry-only failure allowed by the generated contract. Put the page-relative normalized bbox and repair request into the same decision entry, then run:

```bash
python3 scripts/plan_figure_table_decisions.py \
  --review-decisions <figure_table_decisions.json> \
  --output <figure_table_decisions.json>
```

The script validates the bbox, rerenders once from the source PDF at the contract's 300 dpi, refreshes the asset SHA-256, and returns the item to `review_pending` for a full fresh review. A terminal defect or a failed repaired crop remains a placeholder; never request a second recrop.

Reject candidates that are:
- caption-only crops
- tables with no visible table body
- table crops contaminated by running prose outside the table body or another Figure/Table caption
- figure crops contaminated by another Figure/Table caption or by a second figure body
- large text, title-page, or abstract crops masquerading as figures
- crops where the visual body is tiny relative to the crop

## Rendering Boundary

Use `obsidian-format.md` for final placeholder, image embed, caption, and numbering rules. This guide owns placement and visual judgment, not Markdown rendering.

## When to Skip

If the paper has no informative figures or tables:
- do not force one
- state that no high-value figure placeholder was added

# Final Writing

Language note: draft in `synthesis_bundle.writing_contract.language`. For `en`, use the exact schema in `output-language.md`; Chinese-only wording and typography rules below apply only to `zh-CN`.

The final note should not read like raw extracted evidence.

Use the structured artifacts as inputs:
- `metadata.json`
- `source_manifest.json`
- `raw_sections.jsonl`
- `figure_table_decisions.json`
- `synthesis_bundle.json`

Then let the model draft the final note in natural language.

## Front-Matter Structure

Every final note must start with an Obsidian YAML properties block above the `#` title heading.
Include at least:
- `tags`: a `papers/<domain>` hierarchy tag
- `aliases`: a short English name, acronym, or stable title alias useful for wikilinks
- `date`: when publication date or year is known
- `doi` or `arxiv_id`: when available

Do not invent placeholder metadata values. Omit unavailable fields instead.

Near the beginning of the note, include:
- `## 核心信息`
- `## 原文摘要翻译`
- `## 创新点`
- `## 一句话总结`

`## 核心信息` is a fixed metadata block, not an analysis block.
Rules for this section:
- Core info field schema: use only the following fields, in this order, and no free prose:
  `标题`, `标题翻译`, `作者`, `机构`, `发表时间`, `发表渠道`, `DOI`, `arXiv`, `论文链接`, `代码 / 项目`, `数据 / 资源`, `论文类型`
- keep each line in `- 字段名: 值` form
- omit fields that are unavailable or not applicable; do not add `未知`, `无`, or placeholder rows just to fill the schema
- do not add ad hoc fields such as judgments, takeaways, or mini-summaries
- do not move explanatory prose, evaluation, or "my view" sentences into this section
- move any paper-positioning or guide sentence to `一句话总结` or an analysis section, not under `核心信息`

The `原文摘要翻译` section should be a Chinese translation of the paper's original abstract:
- if the abstract is available, translate the original abstract into Chinese before the one-sentence summary
- do not let the summary replace the abstract
- do not treat `原文摘要翻译` as your own summary of the full paper; it is the original abstract translated into Chinese
- do not split this section into `### 英文原文` and `### 中文翻译`
- keep the section title exactly as `原文摘要翻译`
- the `原文摘要翻译` section itself must be written in Chinese; do not output English abstract sentences or English-original paragraphs here
- the Chinese abstract should be fluent and faithful, not a second `一句话总结`
- do not turn `原文摘要翻译` into a selective excerpt or a compressed highlight list
- do not add judgments, hindsight, or details learned from later sections of the paper into `原文摘要翻译`; only translate what the original abstract says

The `创新点` section should be a dedicated top-level section after `原文摘要翻译` rather than a hidden bullet buried later.
It should usually:
- enumerate 3 to 5 paper-specific innovations
- explain what problem each innovation addresses
- explain what new capability, mechanism, or evaluation angle it enables
- avoid generic praise such as `the paper is novel` without locating the novelty

## Writer Persona

Default to a high-bar technical reader and writer persona:
- you are a top-tier AI researcher and algorithm engineer
- you are preparing an internal replication-oriented reading note for your lab
- you are not writing a science-pop summary
- you should assume the reader is comfortable with Python, PyTorch, training loops, evaluation protocols, and ablation logic

For technical or method papers, write as if the note may later be used for:
- implementation planning
- reproduction
- comparison against later papers
- deciding whether the method is actually novel or just well-packaged

## Writing Priorities

1. explain the paper rather than quote it
2. distinguish research problem from task definition
3. explain the method or analysis flow in your own words
4. choose the most meaningful results rather than repeating every number
5. say what the paper does not prove
6. keep the note readable weeks later
7. make the technical core understandable enough for an engineer to re-explain it

## What Scripts Should Not Try To Fully Replace

Scripts are good at:
- resolution
- extraction
- formatting
- linting
- placeholder planning

Scripts are not enough on their own for:
- nuanced judgment
- identifying what is easy to misread
- deciding what the paper's real contribution is
- writing strong, natural Chinese analytical prose

The language model should do all of the following:
- use the grounded plan's selected paper type and section emphasis
- carry its evidence-backed claims, boundaries, limiting results, mechanism-result links, comparisons, reusable takeaways, and follow-up questions into the prose
- decide which sections need more weight
- decide where `###` subheadings are needed
- select the truly central results
- reconstruct the method or analysis flow
- decide whether the paper needs explicit LaTeX formulas for the core objective, factorization, or complexity
- write the final note in clean Chinese

## Final-Draft Standard

The note should feel like:
- a careful reading note
- not an abstract rewrite
- not a raw evidence dump
- not a benchmark table converted into bullets

For quantitative results, preserve the central numbers instead of replacing them with only qualitative claims.
When the source comparison is naturally tabular, especially with three or more compared systems, settings, tasks, datasets, metrics, ablations, or experimental conditions, use a compact Markdown table for the central comparison rather than prose-only or a loose bullet list.
Keep only the rows and metrics that matter for understanding the paper, and follow the table with interpretation of what the numbers mean.
If a paper is short, do not make the final note shallow; use the saved space to explain protocol details, ablations, limitations, and deployment or replication implications.

The final Chinese note must also pass a language-cleanliness check:
- no half-English half-Chinese prose lines
- English is allowed only for stable proper nouns or citation metadata
- if the style gate fails, do not write the note into Obsidian yet
- do not write for the linter; lint is only a minimum floor, not the writing objective
- after script lint passes, `final_quality_review` and then `final_readability_review` are still required before the note should be treated as polished and ready to save

正文术语策略:
- default to natural Chinese prose in正文分析
- keep English only when it is a stable proper noun or source-faithful technical label
- stable English that may remain:
  - model names
  - dataset names
  - metric names
  - method names
  - math symbols
  - code tokens
  - original paper figure/table ids
- when any of the above retained English terms or standalone key numbers appear inline within Chinese prose, wrap them in backticks for visual separation
- English that should usually be rewritten into natural Chinese:
  - ordinary English phrases
  - abstract descriptive phrases in analytical prose
  - leftover English wording that has no clear reason to remain
- when a first mention benefits from both forms, prefer Chinese-first wording with an English gloss in parentheses
- do not leave phrases such as `reasoning dataset`, `distillation risk`, or `reward model quality` directly inside Chinese prose when a natural Chinese rendering is available

For non-trivial papers, the note should usually not stop at only broad `##` sections.
It should use meaningful `###` subheadings where they improve technical clarity.

Draft only from a note plan that has already passed grounding. Use its paper type, evidence-backed claims, boundaries, limiting results, mechanism-result links, comparisons, reusable takeaways, and follow-up questions as writing commitments rather than reopening the planning contract here.

Examples:
- `### 数据来源`
- `### 任务定义`
- `### 中间特征抽取`
- `### 训练细节`
- `### 哪些结果最重要`
- `### 哪些地方容易被误读`

For technical papers, also strongly consider subsections such as:
- `### 机制流程`
- `### 训练目标`
- `### 推理与采样链路`
- `### 关键实现细节`
- `### 复杂度与扩展性`
- `### 消融到底说明了什么`

For method, framework, and system papers, prefer an explicit `### 机制流程` subsection instead of hiding the execution chain inside generic prose.
That subsection should usually be a 3 to 4 step numbered list covering:
- what the Input is
- what the main intermediate transformations are
- what the Output is
- what the training or inference loop is actually doing
- do not rely on a damaged Algorithm block to carry this explanation for you
- do not let the steps collapse into module-name listing; each step should describe an operation
- if a high-confidence pipeline or architecture figure matches this execution chain, place it in `### 机制流程`

## Formula Rule

Do not avoid formulas by default.
When the paper's method or claim depends on:
- a training objective
- a probability factorization
- a complexity expression
- a scaling-law fit
- a key update rule or optimization target

the note should usually include 1 to 3 essential LaTeX formulas in the relevant section.

Use formulas sparingly and purposefully:
- each formula should help explain the method
- do not dump many formulas just to look technical
- if the source extraction is noisy, prefer reconstructing a small, stable core formula rather than copying broken math verbatim
- after each retained formula, add one sentence explaining what it corresponds to in engineering or code terms
- do not only translate variable names; explain the concrete operation, loss term, update rule, or control effect
- formulas in the final Markdown should be written as directly renderable Obsidian/MathJax math, not as JSON-style escaped strings
- do not double-escape TeX commands such as `\\tau`, `\\frac`, `\\bar`, `\\begin`, or `\\end` when the final note should contain `\tau`, `\frac`, `\bar`, `\begin`, or `\end`
- use real math delimiters:
  - inline math: `$...$`
  - display math: `$$ ... $$`
- do not format formulas as inline code with backticks
- do not put formulas inside fenced code blocks unless you are literally discussing source code or pseudocode

## Prose Cleanliness

Chinese paragraphs should read like natural prose, not like PDF fragments.

Do not leave:
- mid-sentence line breaks after commas or semicolons
- one sentence broken into many short physical lines
- raw PDF folding artifacts inside normal paragraphs

Allowed line breaks:
- between paragraphs
- bullet lists
- block quotes
- figure callouts
- fenced code or formula blocks

## Figures and Tables

Place a high-value visual near the analysis it directly supports, and explain the argument in prose rather than using the visual as a substitute for reasoning. Missing or partial image extraction must not erase textual coverage of the paper.

Use `figure-placement.md` for semantic placement, identity matching, and visual-usability judgment. Use `obsidian-format.md` for final placeholder and inserted-image rendering.

## Final Self-Review

Before outputting the final Markdown, first run `final_quality_review` and explicitly check:
- does the note reconstruct the central evidence chain rather than only restating claims?
- does it separate what the evidence actually proves from what the paper has not proven?
- does it map mechanisms, protocols, constructs, data decisions, or study design choices to the result pattern they explain?
- does it position the paper against strong baselines, prior routes, human references, or obvious alternatives?
- does it explain the paper's own Discussion/Limitations claims mechanistically when those sections exist?
- are the planned `claim_boundaries`, `negative_or_limiting_results`, `mechanism_result_map`, `comparative_positioning`, `reuse_takeaways`, and `followup_questions` reflected in the final prose?
- does the note contain concrete numbers, dimensions, complexity terms, or formulas when the paper clearly depends on them?
- can a reader familiar with Python and deep learning frameworks follow the core method from this note alone?
- does the method section explain the mechanism rather than only summarize the claim?
- if this is a method/system/framework paper, does `方法主线` explicitly contain `### 机制流程` with a 3 to 4 step numbered list?
- if the raw source reports negative or unstable ablation settings, did the note include at least one of them?
- if the raw source does not clearly report such settings, did the note avoid inventing failed or unstable cases?
- does the note contain at least one honest limitation and one paper-specific insight?
- are there any suspicious mid-sentence line breaks left in the prose?
- if the note includes LaTeX formulas, did you quickly check that the final Markdown uses directly renderable TeX rather than double-escaped commands or broken math delimiters?

If `final_quality_review` finds missing evidence-chain coverage, missing mechanism-to-result explanation, missing comparative positioning, missing boundary judgment, missing negative/limiting result discussion, or generic reusable takeaways or follow-up questions, return to the source artifacts and revise the note before saving.

After `final_quality_review`, run `final_readability_review`.
This review is a language-and-expression pass, not a second evidence-judgment pass:
- improve fluency and readability
- remove stiff translations
- convert ordinary English phrase leftovers into natural Chinese
- remove mechanical term-replacement artifacts such as `KV缓存 of`, `批量ing`, `In相关 Researcher`, or `Single 序列 generation`; figure/table callout titles and captions count too
- keep stable proper nouns when forcing a translation would sound worse
- do not invent new facts, numbers, comparisons, or failure cases during this pass
- do not use polish as an excuse to flatten the note into a safer but shallower summary

If the answer to the first four quality-review questions is `no`, the draft is still too shallow and should be revised before save.

# Metadata Sources

Use the strongest available source first, but backfill aggressively.

## Preferred Order

1. user-provided exact source
2. local Zotero metadata and attachments
3. DOI resolution and publisher metadata
4. Semantic Scholar
5. OpenAlex
6. arXiv metadata

## Required Fields to Attempt

- title
- authors
- affiliations
- year
- venue
- DOI
- source URL

## Optional Fields

- abstract
- code URL
- project URL
- citation counts
- arXiv ID
- Zotero key

## Rules

- Use the bundled read-only Zotero Local API client first when the desktop API is enabled; compatible runtime or MCP integrations may provide the same trusted local metadata as an optional alternative.
- If the paper is already in the local Zotero library, treat a unique Zotero match as the identity anchor before doing title-based web resolution.
- If Zotero resolves the paper but does not expose a local attachment path, still use the Zotero metadata to avoid title ambiguity.
- Distinguish an ambiguous local identity from an unavailable attachment: never guess between equally plausible items, but keep a uniquely resolved parent item's metadata even when no local PDF is accessible.
- Do not let a weaker internet title match override a confident Zotero hit.
- Do not invent missing metadata.
- If a Chinese title is assistant-generated, mark it as a translation.
- Distinguish:
  - `not found`
  - `not provided by source`
  - `ambiguous`

# Note Quality

Language note: evaluate headings and language cleanliness against `synthesis_bundle.writing_contract`. For `en`, use `output-language.md`; Chinese-specific examples below apply only to `zh-CN`.

The note is high quality only if it satisfies most of the checks below.

## Minimum Bar

- It is not a paraphrase of the abstract.
- It distinguishes `research problem` from `task definition`.
- It explains how the method or analysis actually works.
- It reports the most meaningful results, not only the prettiest numbers.
- It includes at least one real limitation.
- It includes an explicit judgment about the paper's actual contribution.
- It includes at least one paper-specific technical subsection rather than only broad top-level sections.
- For method-heavy papers, it explains enough mechanism detail that an engineer could re-explain the pipeline without reopening the PDF.

## Structural Checks

The note should usually include:
- `核心信息`
- `原文摘要翻译`
- `创新点`
- `一句话总结`
- `研究问题`
- `数据与任务定义`
- `方法主线`
- `关键结果`
- `深度分析`
- `局限`
- `我的笔记`
- `引用`

For non-trivial papers, it should usually also include multiple `###` subheadings inside:
- `数据与任务定义`
- `方法主线`
- `关键结果`
- `深度分析`

Before the final note is written, the run should already have an inspectable, grounded plan whose analytical commitments are paper-specific.

Bad sign:
- the model jumps directly to a polished final note with no grounded plan at all

## Depth Checks

### Good signs

- The note explains the flow of information in the method.
- The note explains technical details with section-specific subheadings rather than one flat block.
- The note points out what the paper does not prove.
- The note identifies where labels, supervision, or evaluation may be weak.
- The note explains why the paper matters to later reading or research reuse.
- The note surfaces one paper-specific insight, not just generic praise.

### Bad signs

- It only repeats the introduction and abstract.
- It lists model names without explaining the pipeline.
- It copies metrics without noting the evaluation setting.
- It says the paper is innovative without locating the innovation.
- It has no dedicated `创新点` section and leaves the paper's novelty scattered across the note.
- It uses generic limitations such as "future work can use more data" and nothing more specific.
- It flattens a technically rich paper into only `##` headings with no internal structure.

## Quality Gate

Fail closed if any of these are missing:
- method evidence
- result evidence
- a clear paper identity
- enough metadata to label the note responsibly

Also fail closed if:
- the final Chinese note still contains mixed-language prose lines
- English remains in full clauses rather than only stable proper nouns, model names, venues, URLs, or DOIs
- figure placeholders include untranslated caption sentences that read like raw extraction rather than note prose

Strong notes should also clearly contain:
- the most important numbers
- the most important comparison
- the central evidence chain behind the paper's main claim
- a clear distinction between what the paper proves and what it does not prove
- at least one limiting, weak, negative, or explicitly unreported result that constrains the conclusion
- one paper-specific insight
- one honest limitation

For technical papers, strong notes should usually also contain:
- at least one method subsection that goes beyond summary into mechanism explanation
- at least one concrete training / inference / complexity detail
- at least one key formula or formal expression when the paper's contribution depends on it
- formulas rendered as math rather than code formatting

When abstract metadata exists, strong notes should also make `原文摘要翻译` a faithful Chinese translation of the abstract:
- translate the original abstract into Chinese rather than rewriting it as your own summary
- avoid reducing it to a shorter interpretation-only summary
- keep this section as `原文摘要翻译`, not a bilingual original-plus-translation block
- do not mix innovation takeaways, evaluation, or post-hoc interpretation into this section

# Obsidian Format

Language note: section names, metadata labels, figure callouts, and the mechanism-flow heading must follow `synthesis_bundle.writing_contract`. For `en`, use the exact schema in `output-language.md`; Chinese-specific examples below apply only to `zh-CN`.

## Heading Rules

- Use `#` for the note title only.
- Use `##` for major sections.
- Use `###` only when a section genuinely needs internal structure.
- Do not flatten everything into bullet points.
- For method, system, benchmark, or clinical empirical papers, prefer meaningful `###` subheadings in technical sections instead of one long undifferentiated block.
- For method, framework, or system papers, default to `### 机制流程` inside `方法主线` and write it as a numbered 3 to 4 step flow.

## File Naming

Default file name:
- sanitized English title with underscores
- default note layout is folder-per-paper:
  - `<领域>/<paper_slug>/<paper_slug>.zh-CN.md`
  - `<领域>/<paper_slug>/<paper_slug>.en.md`
  - `<领域>/<paper_slug>/images/...`
  - `<领域>/<paper_slug>/.deeppapernote.json`
- when deciding `<领域>`, prefer matching an existing first-level domain folder under the user's papers directory
- domain routing uses the editable taxonomy in `references/domain_rules.yaml`: application domains are checked before fallback method domains
- reuse existing first-level folders conservatively; method-only evidence should not force reuse of an unrelated application folder
- only create a new domain folder when no existing domain is a reasonable fit
- do not save new papers directly into the bare papers root
- always create the paper-local `images/` directory during final save, even if no real image is inserted
- the paper-local `images/` directory is part of the required note layout, not an optional optimization

The hidden `.deeppapernote.json` sidecar is the program-owned directory identity record. It groups verified source PDFs for one work and binds each saved note to exact source bytes and language. Keep the dot-prefixed name on every platform; on Windows the save script also applies and verifies the native Hidden attribute.

Before drafting, run the save script's preflight and use its directory, note path, and `asset_subdir`. Verified PDF-only directories can be reused automatically. Multiple source versions share the paper directory with separate notes and assets. Follow [the shared archive contract](paper-archive.md) for migration, candidate selection, and hash-bound overwrite rules.

If the user already has a vault convention, preserve it.

## Markdown Style

- Prefer short paragraphs over long bullet lists.
- Use bullets for metadata and sharply list-shaped content.
- Keep code or metric identifiers in backticks.
- When English proper nouns (model names, dataset names, method names, metric names, venue abbreviations) or standalone key numeric values appear inline within Chinese prose, wrap them in backticks for visual separation — e.g. `GPT-4`、`SQuAD`、`BLEU`、`87.3%`.
- Preserve stable internal links where useful.
- Use normal LaTeX delimiters for math:
  - inline math: `$...$`
  - display math:
    `$$`
    `...`
    `$$`
- Do not wrap formulas in backticks or fenced code blocks unless you are literally showing source code.

## Core Info Block

`## 核心信息` is a fixed metadata zone.

Formatting and scope rules:
- Core info field schema: use only the following fields, in this order, and no free prose:
  `标题`, `标题翻译`, `作者`, `机构`, `发表时间`, `发表渠道`, `DOI`, `arXiv`, `论文链接`, `代码 / 项目`, `数据 / 资源`, `论文类型`
- keep each entry in `- 字段名: 值` form
- omit fields that are unavailable or not applicable; do not add placeholder rows just to fill the schema
- do not add interpretation, commentary, judgment, or takeaway lines inside `核心信息`
- do not use the last metadata bullet as a place to append extra analysis
- move explanatory content to `一句话总结`、`深度分析`、`我的笔记` or another true analysis section

## YAML Frontmatter

Every note must start with an Obsidian properties block **above** the `#` title heading.

Required fields:
- `tags`: use `papers/<domain>` hierarchy, e.g. `papers/NLP`, `papers/CV`, `papers/multimodal`
- `aliases`: English short name or common abbreviation for wikilink resolution
- `date`: ISO publication date; use `YYYY` if only the year is known
- `doi`: DOI string without the `https://doi.org/` prefix; omit the field entirely if unavailable

Example:

```yaml
---
tags:
  - papers/NLP
aliases:
  - "Paper Short Name"
date: 2024-05-01
doi: 10.18653/v1/2024.acl-long.1
---
```

Rules:
- Do not invent placeholder values for missing fields; omit them instead.
- The `tags` field must always be present with at least one `papers/<domain>` tag.
- `aliases` should be the paper's short name or acronym (e.g. "GPT-4", "LoRA"), not a paraphrase.

## Figure Placeholder Style

Use this callout format only for placeholders that remain unresolved in the final note:

```md
> [!figure] Fig. 3 数据分布与质量评估
> 建议位置：数据与任务定义
> 放置原因：这张图同时展示样本构成、对话长度统计和专家质检结果，是理解 `PsyInterview` 数据边界最重要的图之一。
> 当前状态：保留占位；当前提取结果只拿到局部子图，无法稳定恢复成可独立解释的完整原图。
```

Formatting rules:
- keep the original paper numbering, for example `Fig. 3` or `Table 2`
- keep a short human-readable label on the first line
- always include `建议位置`
- always include `放置原因`
- always include `当前状态`

`当前状态` should be explicit, for example:
- `保留占位；未找到高置信度整图。`
- `保留占位；当前只匹配到疑似局部子图，不足以稳定替换。`

The structured `[FIGURE_PLACEHOLDER] ... [/FIGURE_PLACEHOLDER]` block is legacy/internal only.
Do not use it in the final user-facing note unless you are debugging the pipeline.

If a real image has been selected and materialized into the vault, do not keep the `[!figure]` callout for that same figure.
Prefer an Obsidian embed, or use a Markdown image embed when that is the available path.
The embed must be followed immediately by exactly one italic caption line:

```md
![[Research/Papers/DeepPaperNote/paper_slug/images/page_003_img_01.png]]
*论文原图编号：Fig. 2。数据生成流程图。这里插入是因为它最能帮助理解方法主线。*
```

## Default Section Order

1. `核心信息`
2. `原文摘要翻译`
3. `创新点`
4. `一句话总结`
5. `研究问题`
6. `数据与任务定义`
7. `方法主线`
8. `关键结果`
9. `深度分析`
10. `局限`
11. `我的笔记`
12. `引用`

When abstract metadata exists, `原文摘要翻译` should be a single Chinese translation block for the original abstract rather than a bilingual subsection pair.

This order is the stable backbone, not a full outline.
When the paper is complex, add `###` subsections such as:
- `### 数据来源`
- `### 任务定义`
- `### 机制流程`
- `### 为什么结果成立`
- `### 哪些地方容易被误读`

## 引用 Section Format

Entries in `## 引用` should link to existing notes in the vault where possible.
If the synthesis bundle includes `references.candidates`, use confirmed candidate `wikilink` values when present. When `wikilink` is empty, treat `display_text` as the plain-text fallback.
Follow this priority order for each reference:

1. **Vault lookup first**: check whether the cited paper already has a note in the vault.
   - Match by note basename (the `<paper_slug>` part of the folder name).
   - Match by the `aliases` field in the note's YAML frontmatter.
2. **If a match is found**: write a wikilink that separates the target from the display text:
   ```
   - [[paper_slug_or_alias|Human Readable Title]]
   ```
3. **If no match is found**: do not invent a wikilink target. Write the reference as plain text instead:
   ```
   - Vaswani et al. (2017). Attention Is All You Need.
   ```
   Use the candidate `display_text` as the plain fallback when available.

Rules:
- Never use a raw English paper title as the wikilink target; it will not match vault filenames.
- To derive a likely slug from a title: lowercase the title and replace spaces and special characters with underscores — but only use the result as the target if you have confirmed the file exists.
- List only papers cited or directly relevant to this note.
- Do not add extra DOIs or author metadata when using wikilink format; the display text is enough.

# Output Language

DeepPaperNote supports two output schemas:

| Setting | Language |
|---|---|
| `zh-CN` | Simplified Chinese |
| `en` | English |

Resolve the profile through `user-configuration.md`. For a single command, use `--language en` with `run_pipeline.py`, `build_synthesis_bundle.py`, `lint_note.py`, or `write_obsidian_note.py` where applicable. A Run Override never changes User Configuration.

## Simplified Chinese note schema

Use these top-level sections in this order:

1. `核心信息`
2. `原文摘要翻译`
3. `创新点`
4. `一句话总结`
5. `研究问题`
6. `数据与任务定义`
7. `方法主线`
8. `关键结果`
9. `深度分析`
10. `局限`
11. `我的笔记`
12. `引用`

Use `### 机制流程` for the mechanism-flow subsection. Chinese figure placeholders use the labels `建议位置：`, `放置原因：`, and `当前状态：`; a materialized image caption begins with `论文原图编号：`.

`原文摘要翻译` is a faithful Chinese translation of the source abstract. Preserve its meaning and scope; contribution claims, result interpretation, and hindsight judgment belong later unless the source abstract itself contains them.

## English note schema

Use these top-level sections in this order:

1. `Core Information`
2. `Abstract`
3. `Contributions`
4. `One-Sentence Summary`
5. `Research Question`
6. `Data and Task Definition`
7. `Method`
8. `Key Results`
9. `Deep Analysis`
10. `Limitations`
11. `Research Notes`
12. `References`

The allowed Core Information fields, in order, are:

`Title`, `Translated title`, `Authors`, `Institutions`, `Publication date`, `Venue`, `DOI`, `arXiv`, `Paper link`, `Code / Project`, `Data / Resources`, `Paper type`.

Use `### Mechanism Flow` for the mechanism-flow subsection. Each figure placeholder uses:

```md
> [!figure] Figure 2 Human-readable label
> Suggested location: Method
> Why it matters: This figure clarifies the execution path.
> Current status: Placeholder retained; the recovered crop is incomplete.
```

For a materialized image, use the normal image embed followed immediately by one italic caption beginning with `Original paper item:`.

`Abstract` is a faithful rendering of the source abstract in English. Preserve its meaning and scope; contribution claims, result interpretation, and hindsight judgment belong later unless the source abstract itself contains them.

The English style gate checks headings, Core Information labels, figure callouts, inserted-image captions, and prose. Mark original non-English metadata with inline code inside `Core Information`, or with inline code or a Markdown link inside `References`; closed fenced code blocks and URLs keep their normal source text. In prose, a CJK identifier or stable proper noun must be an HTTP Markdown link or Obsidian wikilink. A source formula may retain only these CJK `\operatorname{...}` labels: `输入`, `输出`, `损失`, `状态`, `动作`, `奖励`, `标签`, `样本`, `预测`, and `目标`. The surrounding text remains subject to the English gate; inline code and free-form math text do not exempt Chinese prose.

## Contract ownership

`SKILL.md` owns the cross-stage Language Integrity Contract. This reference owns only the profile-specific schema and labels above; apply them under the resolved language carried by that contract.

# Shared paper archive

Use this reference for Obsidian directory reuse, Connector handoff, and multiple PDF sources of one paper. `scripts/paper_archive.py` owns the deterministic archive rules; Connector vendors that module unchanged. The canonical Skill workflow still owns all reading, review, and Formal Save gates.

## Acquisition and selection

Pass the original user reference to `fetch_pdf.py --reference <reference>` alongside the accepted identity artifact. Before remote download, it searches the resolved Obsidian Vault for a verified local PDF. An explicit local input remains authoritative. An explicit arXiv revision wins; if absent locally, acquisition requests that revision of the accepted work and verifies the returned PDF. Without an explicit revision, select the highest local arXiv revision only when all available sources belong to the same comparable revision sequence. Never order sources by download time. Same-revision different-byte PDFs and incomparable sources require a choice.

For standalone read-only discovery, use `find_archived_source.py --input <metadata.json-or-reference> --vault <vault> --reference <original-reference>`. A `not_found` result permits normal acquisition. A `blocked` result is an ambiguity or verification failure, not permission to create a duplicate archive.

When multiple directories match, show their paths and content counts and ask the user to select the current destination. Carry `--target-directory <absolute-path>` through discovery, acquisition, preflight, and Formal Save. For ambiguous PDF sources, use `--source-sha256 <chosen-hash>` in discovery/acquisition. The returned `archive_source` also carries the selected directory through the Source Manifest. Selection does not waive identity or path checks.

## Admission and writing

Run the save script's preflight before drafting. Prefer verified matches over title-only candidates. Reuse a unique verified work directory, including a legacy PDF-only directory, without asking for registration. Work identity or exact PDF bytes must verify the match; a folder title alone is only a candidate. Mixed-work PDFs, malformed records, and paths outside the Vault fail closed. Existing directories anywhere inside the Vault are eligible; new directories remain under the configured papers root.

Read `asset_subdir` from preflight and pass it to `plan_figure_table_decisions.py --asset-subdir <value>`; rerun planning if earlier decisions used the default path. Bounded repairs retain this path. Use that paper-relative path for real figure embeds in the draft and figure/table decisions. The first source retains `images/`; additional sources use `images/<full-source-sha256>/`. Every figure still requires the normal source/identity/readability checks. Save does not reinterpret an old figure as belonging to another version.

The note path is bound to exact source bytes plus `output_language`. New source variants get separate version-labelled, hash-disambiguated note paths; existing filenames and links remain unchanged. An existing note for the same source and language still requires `--overwrite-existing-note --expected-existing-note-sha256 <reported-hash>` after user approval. Unbound notes are not adopted as completed notes.

## Directory record and migration

`.deeppapernote.json` schema 2 groups sources under one `work` identity. `work.provenance` retains evidence per identifier, including actual PDF-first-page evidence bound to its SHA-256; legacy records without evidence are marked `legacy_record`, not relabelled as verified provenance. `sources` is keyed by the actual PDF SHA-256. Each entry keeps its relative `pdf_path` when archived, known `arxiv_id`, source URLs, frozen `note_stem`, `asset_subdir`, and `notes` keyed by language. Each saved note binds its filename and final note SHA-256 to that source. A note-only source may omit `pdf_path`; Connector can attach its PDF later without rewriting the note.

Schema 1 records are read without mutation and upgraded on a successful write. Preserve original note/source bindings, filenames, and unrelated fields. Unknown versions remain unknown. A PDF collection adds no completed-note record. Both writers lock the Vault, reread the record, recheck source/destination, and replace the record atomically. A Connector registration failure reports the retained PDF as incomplete rather than claiming success.

## Shared location

Connector reads only `obsidian_vault` and `papers_dir` from the Skill's persistent User Configuration at `~/.deeppapernote/config.json`. Its extension ID and native-host installation settings remain separate. A missing or invalid shared location requires configuration repair; never use a conflicting legacy Connector destination. Reading or saving does not change language, save mode, unknown preferences, or persistent settings through a temporary Skill Run Override.

# Paper Types

Language note: use the localized `contracts_by_paper_type` and section names from the generated synthesis bundle. The examples below use the backward-compatible Chinese schema; `output-language.md` defines the English schema.

Every note keeps the same 12 top-level sections from `NOTE_REQUIRED_SECTIONS`.
Paper type only changes the typed semantics of shared sections and the recommended `###` subsections used in `note_plan.section_plan`.

Use `contracts_by_paper_type[note_plan.paper_type]` as the canonical structured source:
- `section_semantics`: how each fixed top-level section should be interpreted for this paper type.
- `recommended_subsections`: paper-type-specific `###` candidates for technical or analytical sections.
- `boundary_questions`: paper-type-specific questions that should shape `central_claims`, `claim_boundaries`, `negative_or_limiting_results`, `mechanism_result_map`, `comparative_positioning`, and `followup_questions`.

## `AI_method`

section_semantics:
- 研究问题: 方法要解决的具体技术问题和现有方法短板。
- 数据与任务定义: 数据集、输入输出、评测任务和实验设置。
- 方法主线: 模型、算法、训练或推理机制。
- 关键结果: 主结果、强基线、消融和关键数字。
- 深度分析: 方法为什么有效、何处脆弱、复现和扩展代价。

recommended_subsections:
- 方法主线: `机制流程`, `模型结构`, `训练目标`, `推理与采样链路`, `关键实现细节`
- 关键结果: `主结果与强基线`, `消融到底说明了什么`, `失败或不稳定设置`
- 深度分析: `为什么有效`, `复杂度与扩展性`, `复现注意点`

boundary_questions:
- 核心机制的收益由哪个实验或消融支撑，而不是只由主结果暗示？
- 哪些比较只能证明在当前数据、基线、算力或协议下有效，不能外推到通用场景？
- 论文是否给出失败、退化、不稳定或成本上升的证据；如果没有，结论边界是什么？

## `benchmark_or_dataset`

section_semantics:
- 研究问题: 这个 benchmark/dataset 想补足的评测或数据缺口。
- 数据与任务定义: 数据来源、任务拆分、标签/题目定义、样本范围。
- 方法主线: 数据构建、筛选、标注和评测协议，不写成模型 pipeline。
- 关键结果: 基线表现、难度分布、覆盖范围和偏差。
- 深度分析: 它真正测到了什么，以及不能代表什么。

recommended_subsections:
- 数据与任务定义: `数据来源`, `任务拆分`, `标注/筛选协议`
- 方法主线: `构建流程`, `评测协议`, `Baseline 设置`
- 关键结果: `基线表现`, `难度分布`, `覆盖与偏差`
- 深度分析: `benchmark 真正测到了什么`, `适用边界`

boundary_questions:
- 这个 benchmark/dataset 实际测量的构念是什么，哪些能力只是间接近似？
- 任务、标签、采样、过滤或评测协议会引入哪些覆盖缺口或偏差？
- 基线结果证明了评测集有区分度，还是只证明某类模型适应该协议？
- 样本时长、语料长度、人口统计、类别分布、数据可访问性或隐私限制如何影响复现和外推？

## `clinical_or_psychology_empirical`

section_semantics:
- 研究问题: 临床、心理学或行为科学中的研究问题、假设或变量关系。
- 数据与任务定义: 样本来源、纳排标准、变量/量表、测量方式。
- 方法主线: 研究设计、分组、测量流程和统计分析路径。
- 关键结果: 主要效应、相关性、组间差异、不确定性或显著性。
- 深度分析: 结果解释、因果边界、临床/心理学意义和外推限制。

recommended_subsections:
- 数据与任务定义: `样本与纳排标准`, `变量与量表`, `测量流程`
- 方法主线: `研究设计`, `分析模型`, `主要比较`
- 关键结果: `主要效应`, `不确定性与显著性`, `临床或心理学解释`
- 深度分析: `因果解释边界`, `外推限制`

boundary_questions:
- 样本来源、纳排标准、测量工具和标注流程如何限制外推？
- 结果支持相关、预测、组间差异还是因果解释；不要越过论文设计能证明的范围。
- 临床或心理学意义是否依赖未观测混杂、量表阈值、文本/语音缺失或场景约束？
- 样本构成、数据缺失、隐私限制或材料不可公开会怎样限制复现与再分析？

## `humanities_or_social_science`

section_semantics:
- 研究问题: 作者要解释的社会、文化、历史、制度或理论问题。
- 数据与任务定义: 材料、案例、文本、访谈、档案或语料范围，不写成 ML task。
- 方法主线: 理论框架、概念区分和论证路径。
- 关键结果: 核心解释性发现、概念贡献或对既有观点的修正。
- 深度分析: 论证强度、材料边界、解释替代性和可迁移性。

recommended_subsections:
- 数据与任务定义: `材料范围`, `选择标准`, `案例或语料边界`
- 方法主线: `理论框架`, `概念区分`, `论证路径`
- 关键结果: `核心解释性发现`, `概念贡献`
- 深度分析: `论证强度`, `替代解释`, `材料边界`

boundary_questions:
- 作者的解释依赖哪些材料、案例或理论前提？
- 是否存在同样能解释材料的替代解释，论文如何排除或没有排除？
- 哪些结论是概念贡献或规范性判断，而不是可直接当作经验事实？

## `survey_or_review`

section_semantics:
- 研究问题: 综述试图整理的领域问题、争议或知识缺口。
- 数据与任务定义: 纳入文献范围、检索/筛选标准和综述对象。
- 方法主线: 分类体系、综述组织方式和证据综合逻辑，不写成单篇方法架构。
- 关键结果: 领域共识、分歧、趋势、代表性方向和开放问题。
- 深度分析: 综述覆盖的盲区、分类体系的解释力和未来研究机会。

recommended_subsections:
- 数据与任务定义: `综述范围`, `纳入/排除标准`, `文献覆盖`
- 方法主线: `分类体系`, `方法谱系`, `证据组织方式`
- 关键结果: `代表性方向`, `共识与分歧`, `开放问题`
- 深度分析: `分类体系的局限`, `未覆盖区域`, `后续研究机会`

boundary_questions:
- 检索范围、纳入排除标准或分类轴会遗漏哪些研究路线？
- 综述给出的是领域共识、作者分类，还是尚未解决的分歧？
- 哪些趋势结论来自覆盖范围内的文献分布，不能直接当作技术成熟度判断？

## Selection Rule

Choose one primary `note_plan.paper_type` from the synthesis bundle's allowed values first.
Then keep the fixed top-level sections and use that paper type's `section_semantics` plus `recommended_subsections` to write `note_plan.section_plan`.

# User Configuration

DeepPaperNote can resolve a run entirely from Run Overrides. It also supports one optional device-local User Configuration at `~/.deeppapernote/config.json` for fallback values and explicit future preferences:

For isolated validation only, `DEEPPAPERNOTE_CONFIG_PATH` may relocate this one file for the current process. It is not a preference, does not create a second configuration layer, and must not be persisted into the configuration itself.

- `output_language`: `zh-CN` or `en`
- `save_mode`: `workspace` or `obsidian`
- `obsidian_vault`: existing absolute directory, required only in Obsidian mode
- `papers_dir`: safe relative path inside the Vault, required only in Obsidian mode

There is no implicit language or save-mode default. Workspace mode ignores stored Obsidian fields for the current run but preserves them for a later Obsidian run. Destination writability is a Formal Save concern; configuration inspection never creates a probe file in the workspace or Vault.

## Configuration admission

Complete Configuration Readiness before paper identity resolution:

1. Resolve the explicit request, CLI arguments, and current process environment in precedence order.
2. When those Run Overrides contain every active field and pass validation, complete Configuration Readiness without reading User Configuration.
3. Otherwise run `scripts/user_configuration.py` without setters. The inspector returns exactly one structured User Configuration state: `ready`, `needs_input`, `invalid`, or `blocked`.
4. For `needs_input` on first use, ask one Configuration Prompt Batch for the unresolved active fields. Require `obsidian_vault` and `papers_dir` when the resolved Save Mode is Obsidian. For later repair, ask only for `prompt_fields`.
5. For migration candidates, show the candidates and obtain confirmation before persisting future preferences. Candidates may still act as current-process Run Overrides when they are actually present in the process environment.
6. Persist confirmed preferences with the relevant `--set-output-language`, `--set-save-mode`, `--set-vault`, and `--set-papers-dir` options. Use `--replace-invalid` only after the user explicitly confirms replacement of malformed or non-object JSON.
7. Run the inspector again after a Preference Change. Persistence completes only when it returns `ready` after atomic write and readback validation; then resolve the run again.

Treat `invalid` as repairable input. Treat `blocked` as an I/O boundary: report its issue, preserve the current file, and stop before paper work. Never claim a preference was saved unless readback returned `ready`.

## Resolution and persistence

Resolve each preference using this exact precedence; an explicit request is an explicit current-run parameter, including a natural-language request:

`explicit request > CLI > current process environment > User Configuration`

An explicit request about the current paper is a Run Override. Translate it to the matching runtime override and leave `config.json` byte-for-byte unchanged. Persist only explicit future-default wording as a Preference Change.

Current process environment values are first-class Run Overrides and may satisfy the entire run without a configuration-file read. Shell startup files are not read when the inherited process environment is complete. If fallback is required while `config.json` is absent, supported shell values may be shown as migration candidates; they become persistent preferences only after confirmation.

Preference Changes preserve unknown JSON fields and report a warning. Malformed or non-object JSON receives a unique invalid backup before a confirmed replacement. Writes use a same-directory temporary file, atomic replacement, and exact reread comparison.

## Advanced Run Overrides

Normal Agent use does not require these options. For direct CLI or environment-based runs, use the following mappings:

| Preference | Current-process environment | CLI option |
|---|---|---|
| `output_language` | `DEEPPAPERNOTE_OUTPUT_LANGUAGE` | `--language` |
| `save_mode` | `DEEPPAPERNOTE_SAVE_MODE` | `--save-mode` |
| `obsidian_vault` | `DEEPPAPERNOTE_OBSIDIAN_VAULT` | `--vault` |
| `papers_dir` | `DEEPPAPERNOTE_PAPERS_DIR` | `--papers-dir` |

These values are Run Overrides for the current process or command. They do not become saved preferences unless the user separately confirms a Preference Change.

## Completion criteria

Configuration is ready only when every active field is present and valid, the resolved values contain no missing or invalid field, and the workflow has not begun identity resolution. A User Configuration file is not required when Run Overrides already meet that condition. Obsidian mode requires an existing absolute Vault and a traversal-safe relative paper directory. Workspace mode requires neither Obsidian field and cannot be redirected by stale values.

## Connector archive location

DeepPaperNote Connector reads `obsidian_vault` and `papers_dir` from this same persistent User Configuration. It keeps extension/native-host settings separately and does not change `output_language`, `save_mode`, or unknown preferences. Missing or invalid location fields require explicit configuration repair; old Connector destinations are not a fallback. Skill Run Overrides remain temporary and do not redirect future Connector saves. See [the shared archive contract](paper-archive.md).

#!/usr/bin/env python3
"""Emit the accepted acquisition Paper Identity contract for trusted inputs."""

from __future__ import annotations

import argparse

from common import (
    ACCEPTED_IDENTITY_VERDICTS,
    build_canonical_identity_artifact,
    build_identity_repair_trace,
    emit,
    maybe_load_json_record,
    require_ok_input_artifact,
)


def parser() -> argparse.ArgumentParser:
    p = argparse.ArgumentParser(description=__doc__ or "build identity contract")
    p.add_argument("--input", required=True, help="Metadata JSON path or JSON string.")
    p.add_argument("--resolve", required=True, help="Resolve artifact path for provenance.")
    p.add_argument("--trace-output", required=True, help="Identity repair trace JSON output path.")
    p.add_argument("--output", required=True, help="Canonical Identity Artifact JSON output path.")
    return p


def load_required_record(value: str, consumer: str, *, producer: str) -> dict:
    record = maybe_load_json_record(value)
    if record is None:
        raise SystemExit(f"{consumer} requires a JSON acquisition artifact input.")
    accepted = dict(require_ok_input_artifact(record, consumer))
    if str(accepted.get("script", "")) != producer:
        raise SystemExit(f"{consumer} requires a {producer} producer artifact.")
    return accepted


def main() -> None:
    args = parser().parse_args()
    metadata = load_required_record(
        args.input,
        "build_identity_contract.py",
        producer="collect_metadata.py",
    )
    if not isinstance(metadata.get("identity_observations"), list):
        raise SystemExit(
            "build_identity_contract.py requires metadata identity_observations."
        )
    source_record = load_required_record(
        args.resolve,
        "build_identity_contract.py",
        producer="resolve_paper.py",
    )

    trace = build_identity_repair_trace(
        metadata,
        source_record=source_record,
        resolve_artifact_path=args.resolve,
        metadata_artifact_path=args.input,
    )
    emit(trace, args.trace_output)

    identity = build_canonical_identity_artifact(
        metadata,
        source_record=source_record,
        repair_trace_path=args.trace_output,
        resolve_artifact_path=args.resolve,
        metadata_artifact_path=args.input,
    )
    emit(identity, args.output)
    verdict = str(identity.get("identity_verdict", "")).lower().replace("-", "_")
    if verdict not in ACCEPTED_IDENTITY_VERDICTS:
        raise SystemExit(1)


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""Assemble a model-facing synthesis bundle from deterministic DeepPaperNote artifacts."""

from __future__ import annotations

import argparse
from copy import deepcopy
from pathlib import Path

from citation_links import resolve_reference_links
from common import (
    canonical_identity_acceptance_error,
    maybe_load_json_record,
    normalize_whitespace,
    runtime_config,
)
from contracts import (
    NOTE_PLAN_REQUIRED_FIELDS,
    PAPER_TYPE_VALUES,
    paper_type_contracts,
    writing_contract_rules,
)
from localization import normalize_output_language, require_artifact_output_language


def parser() -> argparse.ArgumentParser:
    p = argparse.ArgumentParser(description=__doc__ or "build synthesis bundle")
    p.add_argument("--metadata", required=True, help="Metadata JSON path or string.")
    p.add_argument("--evidence", required=True, help="Evidence JSON path or string.")
    p.add_argument("--figures", default="", help="Figure plan JSON path or string.")
    p.add_argument("--assets", default="", help="PDF assets JSON path or string.")
    p.add_argument("--source-manifest", required=True, help="Source manifest JSON path or string.")
    p.add_argument(
        "--figure-decisions",
        required=True,
        help="Figure/table decision JSON path or string.",
    )
    p.add_argument("--language", default="", help="Run Override for output language: en or zh-CN.")
    p.add_argument("--output", default="", help="Output JSON path.")
    return p


def load_record(value: str) -> dict:
    return maybe_load_json_record(value) or {}


def sanitize_reference_candidates(
    evidence_pack: dict, config: dict, *, limit: int = 20
) -> list[dict]:
    candidates = evidence_pack.get("reference_candidates", []) or []
    if not isinstance(candidates, list):
        return []
    try:
        matched_candidates = resolve_reference_links(candidates[:limit], config)
    except Exception:
        matched_candidates = [
            {
                **candidate,
                "wikilink": "",
                "vault_target": "",
                "match_status": "vault_unavailable",
                "match_reason": "none",
            }
            for candidate in candidates[:limit]
            if isinstance(candidate, dict)
        ]

    sanitized: list[dict] = []
    for item in matched_candidates:
        if not isinstance(item, dict):
            continue
        raw_text = normalize_whitespace(str(item.get("raw_text", "")))
        display_text = normalize_whitespace(str(item.get("display_text", "")))
        if not raw_text and not display_text:
            continue
        sanitized.append(
            {
                "raw_text": raw_text,
                "display_text": display_text,
                "page_hint": normalize_whitespace(str(item.get("page_hint", ""))),
                "doi": normalize_whitespace(str(item.get("doi", ""))),
                "arxiv_id": normalize_whitespace(str(item.get("arxiv_id", ""))),
                "wikilink": normalize_whitespace(str(item.get("wikilink", ""))),
                "vault_target": normalize_whitespace(str(item.get("vault_target", ""))),
                "match_status": normalize_whitespace(
                    str(item.get("match_status", "no_vault_match"))
                ),
                "match_reason": normalize_whitespace(str(item.get("match_reason", "none"))),
            }
        )
    return sanitized


def figure_quality_summary(assets_wrapper: dict) -> dict[str, int]:
    summary = {"usable": 0, "review": 0, "reject": 0, "unknown": 0}
    for item in assets_wrapper.get("figure_assets", []) or []:
        if not isinstance(item, dict):
            summary["unknown"] += 1
            continue
        quality_signals = item.get("quality_signals", {})
        if not isinstance(quality_signals, dict):
            summary["unknown"] += 1
            continue
        status = normalize_whitespace(str(quality_signals.get("visual_quality_status", ""))).lower()
        if status == "needs_review":
            status = "review"
        if status not in summary:
            status = "unknown"
        summary[status] += 1
    return summary


def truncation_warnings(source_coverage: dict, asset_coverage: dict) -> list[str]:
    warnings: list[str] = []
    if source_coverage.get("text_truncated") or source_coverage.get("truncated_due_to_page_limit"):
        warnings.append("source_text_truncated")
    if asset_coverage.get("truncated_due_to_asset_page_limit"):
        warnings.append("asset_page_limit")
    return warnings


def coverage_summary(
    evidence_pack: dict,
    metadata: dict | None = None,
    assets_wrapper: dict | None = None,
    source_manifest: dict | None = None,
) -> dict:
    metadata = metadata or {}
    assets_wrapper = assets_wrapper or {}
    source_manifest = source_manifest or {}
    pdf_coverage = evidence_pack.get("pdf_coverage", {}) or {}
    asset_coverage = assets_wrapper.get("asset_coverage", {}) or {}
    source_coverage = (
        source_manifest.get("coverage", {})
        if isinstance(source_manifest.get("coverage"), dict)
        else {}
    )
    if not source_coverage and isinstance(source_manifest.get("pdf"), dict):
        pdf = source_manifest.get("pdf", {})
        source_coverage = {
            "total_pages": pdf.get("total_pages"),
            "text_max_pages": pdf.get("text_max_pages"),
            "text_pages_extracted": pdf.get("text_pages_extracted"),
            "text_truncated": pdf.get("text_truncated"),
        }
    return {
        "language_hint": source_manifest.get("language_hint")
        or evidence_pack.get("language_hint", "unknown"),
        "section_extraction_coverage": evidence_pack.get("section_extraction_coverage", {}) or {},
        "pdf_coverage": pdf_coverage,
        "source_coverage": source_coverage,
        "source_manifest": {
            "raw_sections_path": source_manifest.get("raw_sections_path", ""),
            "full_text_md_path": source_manifest.get("full_text_md_path", ""),
            "section_count": len(source_manifest.get("sections", []) or []),
            "page_count": len(source_manifest.get("pages", []) or []),
            "text_hash_sha256": source_manifest.get("text_hash_sha256", ""),
        },
        "appendix_evidence_counts": appendix_evidence_counts(evidence_pack),
        "extraction_failures": evidence_pack.get("extraction_failures", []) or [],
        "asset_coverage": asset_coverage if isinstance(asset_coverage, dict) else {},
        "figure_quality_summary": figure_quality_summary(assets_wrapper),
        "truncation_warnings": truncation_warnings(
            source_coverage if isinstance(source_coverage, dict) else {},
            asset_coverage if isinstance(asset_coverage, dict) else {},
        ),
        "identity_confidence": metadata.get("identity_confidence", ""),
        "identity_confidence_reasons": metadata.get("identity_confidence_reasons", []) or [],
    }


def appendix_evidence_counts(evidence_pack: dict) -> dict[str, int]:
    appendix_evidence = evidence_pack.get("appendix_evidence", {}) or {}
    if not isinstance(appendix_evidence, dict):
        return {}
    return {
        normalize_whitespace(str(category)): len(items)
        for category, items in appendix_evidence.items()
        if isinstance(items, list)
    }


def sanitize_page_assets(assets_wrapper: dict, *, limit: int = 24) -> list[dict]:
    sanitized: list[dict] = []
    for item in (assets_wrapper.get("page_assets", []) or [])[:limit]:
        if not isinstance(item, dict):
            continue
        sanitized.append(
            {
                "page_number": item.get("page_number", 0),
                "searchable_text_chars": item.get("searchable_text_chars", 0),
                "text_extraction_method": item.get("text_extraction_method", ""),
                "ocr_used": item.get("ocr_used", False),
                "image_count": item.get("image_count", 0),
                "text_preview": item.get("text_preview", ""),
            }
        )
    return sanitized


def sanitize_figure_assets(assets_wrapper: dict, *, limit: int = 48) -> list[dict]:
    sanitized: list[dict] = []
    for item in (assets_wrapper.get("figure_assets", []) or [])[:limit]:
        if not isinstance(item, dict):
            continue
        record = {
            "filename": item.get("filename", ""),
            "path": item.get("path", ""),
            "page_number": item.get("page_number", 0),
            "label": item.get("label", ""),
            "kind": item.get("kind", ""),
            "caption_text": normalize_whitespace(str(item.get("caption_text", ""))),
            "width": item.get("width", 0),
            "height": item.get("height", 0),
            "size_bytes": item.get("size_bytes", 0),
            "extraction_level": item.get("extraction_level", ""),
        }
        if isinstance(item.get("quality_signals"), dict):
            record["quality_signals"] = item.get("quality_signals")
        sanitized.append(record)
    return sanitized


def source_index(source_manifest: dict) -> dict:
    if not isinstance(source_manifest, dict):
        return {}
    return {
        "raw_sections_path": source_manifest.get("raw_sections_path", ""),
        "full_text_md_path": source_manifest.get("full_text_md_path", ""),
        "sections": source_manifest.get("sections", []) or [],
        "pages": source_manifest.get("pages", []) or [],
        "captions": source_manifest.get("captions", {}) or {},
        "math_index": source_manifest.get("math_index", []) or [],
        "appendix_index": source_manifest.get("appendix_index", {}) or {},
        "language_hint": source_manifest.get("language_hint", "unknown"),
        "text_hash_sha256": source_manifest.get("text_hash_sha256", ""),
    }


def identity_contract_summary(metadata: dict, source_manifest: dict) -> dict:
    contract = {}
    if isinstance(source_manifest.get("identity_contract"), dict):
        contract = source_manifest.get("identity_contract", {})
    elif isinstance(metadata.get("identity_contract"), dict):
        contract = metadata.get("identity_contract", {})
    if not isinstance(contract, dict) or not contract:
        return {}
    return {
        "artifact_type": contract.get("artifact_type", ""),
        "schema_version": contract.get("schema_version", 1),
        "paper_id": contract.get("paper_id", ""),
        "identity_verdict": contract.get("identity_verdict", ""),
        "work_level_identity": deepcopy(contract.get("work_level_identity", {}) or {}),
        "source_manifestation": deepcopy(contract.get("source_manifestation", {}) or {}),
        "accepted_metadata": deepcopy(contract.get("accepted_metadata", {}) or {}),
        "bound_sources": deepcopy(contract.get("bound_sources", []) or []),
        "field_provenance": deepcopy(contract.get("field_provenance", {}) or {}),
        "selected_identity_evidence": deepcopy(
            contract.get("selected_identity_evidence", []) or []
        ),
        "equivalence_decision": deepcopy(contract.get("equivalence_decision", {}) or {}),
        "warnings": deepcopy(contract.get("warnings", []) or []),
        "repair_trace_path": contract.get("repair_trace_path", ""),
        "provenance": deepcopy(contract.get("provenance", {}) or {}),
    }


def accepted_bundle_metadata(metadata: dict, identity_contract: dict) -> dict:
    if not identity_contract:
        raise ValueError("synthesis requires an accepted canonical identity contract")
    error = canonical_identity_acceptance_error(identity_contract)
    if error:
        raise ValueError(f"synthesis requires an accepted canonical identity: {error}")
    accepted = deepcopy(identity_contract.get("accepted_metadata", {}) or {})
    work_identity = identity_contract.get("work_level_identity", {}) or {}
    if not accepted:
        accepted = deepcopy(work_identity)
    for key, value in work_identity.items():
        if value not in ("", None, [], {}):
            accepted[key] = deepcopy(value)
    return accepted


def figure_table_manifest(
    figure_decisions_wrapper: dict,
    source_manifest: dict,
    figure_plan: dict,
) -> dict:
    decisions = (
        figure_decisions_wrapper.get("decisions", [])
        if isinstance(figure_decisions_wrapper, dict)
        else []
    )
    if not isinstance(decisions, list):
        decisions = []
    captions = (
        source_manifest.get("captions", {})
        if isinstance(source_manifest.get("captions"), dict)
        else {}
    )
    decisions_path = (
        figure_decisions_wrapper.get("decisions_path", "")
        if isinstance(figure_decisions_wrapper, dict)
        else ""
    )
    planned_items = (
        len(figure_plan.get("figures", []) or []) if isinstance(figure_plan, dict) else 0
    )
    return {
        "decisions_path": decisions_path,
        "decisions": decisions,
        "caption_counts": {
            "figures": len(captions.get("figures", []) or []),
            "tables": len(captions.get("tables", []) or []),
        },
        "planned_items": planned_items,
        "requires_full_decision_table": True,
    }


def compact_writing_contract(language: str | None = None) -> dict:
    rules = writing_contract_rules(language)
    depth_requirements = dict(rules["note_plan_depth_requirements"])
    depth_requirements["required_section_focus_fields"] = list(
        depth_requirements["required_section_focus_fields"]
    )
    depth_requirements["generic_focus_phrases"] = list(
        depth_requirements["generic_focus_phrases"]
    )
    usable_insert_candidate = dict(rules["usable_insert_candidate"])
    usable_insert_candidate["kinds"] = list(usable_insert_candidate["kinds"])
    visual_review_contract = deepcopy(rules["visual_review_contract"])
    for field in (
        "review_fields",
        "review_status_values",
        "review_evidence_fields",
        "repairable_failure_reasons",
        "terminal_failure_reasons",
    ):
        visual_review_contract[field] = list(visual_review_contract[field])
    analysis_coverage = deepcopy(rules["analysis_coverage_contract"])
    analysis_coverage["central_claim_fields"] = list(
        analysis_coverage["central_claim_fields"]
    )
    analysis_coverage["required_plan_fields"] = list(
        analysis_coverage["required_plan_fields"]
    )
    analysis_coverage["final_quality_review_checks"] = list(
        analysis_coverage["final_quality_review_checks"]
    )
    contract = {
        "language": rules["language"],
        "contract_role": "manifest_quality_contract",
        "canonical_source": (
            "SKILL.md defines the workflow; scripts/contracts.py defines "
            "machine-checkable contract data."
        ),
        "must_include_sections": list(rules["required_sections"]),
        "core_info_fields": list(rules["core_info_fields"]),
        "figure_labels": dict(rules["figure_labels"]),
        "mechanism_flow_heading": rules["mechanism_flow_heading"],
        "note_plan_contract": {
            "required_fields": ["output_language", *NOTE_PLAN_REQUIRED_FIELDS],
            "field_types": {
                "output_language": "string",
                **dict(rules["note_plan_field_types"]),
            },
            "required_field_checks": deepcopy(
                rules["note_plan_required_field_checks"]
            ),
            "artifact_preference": "short_json_planning_file",
            "grounding_field": "section_plan[*].evidence_sources",
            "analysis_coverage_field": "central_claims[*]",
        },
        "paper_type_selection": {
            "source_of_truth": "note_plan.paper_type",
            "suggested_paper_type_role": "none",
            "allowed_paper_types": list(PAPER_TYPE_VALUES),
        },
        "contracts_by_paper_type": paper_type_contracts(rules["language"]),
        "grounding_contract": {
            "source_of_truth": "source_manifest",
            "source_index_source_of_truth": "source_manifest",
            "truncation_source_of_truth": "source_manifest.coverage_or_pdf",
            "partial_reading_acceptance_owner": "note_plan_or_grounding",
            "accepted_reference_forms": list(
                rules["allowed_grounding_reference_forms"]
            ),
            "required_sections": list(rules["grounding_required_sections"]),
            "note_plan_depth_requirements": depth_requirements,
            "excluded_model_input_fields": list(
                rules["excluded_model_input_fields"]
            ),
            "reject_old_references": list(rules["old_bundle_reference_prefixes"]),
            "lint_command": (
                "scripts/lint_grounding.py --note-plan ... "
                "--source-manifest ... --bundle-json ... --figure-decisions ..."
            ),
        },
        "figure_table_contract": {
            "placeholder_first": True,
            "visual_quality_gate": "fail_closed",
            "decision_table_required": True,
            "decision_values": list(rules["figure_decision_values"]),
            "usable_insert_candidate": usable_insert_candidate,
            "allowed_usable_placeholder_reasons": list(
                rules["allowed_usable_placeholder_reasons"]
            ),
            "manual_visual_review_required_statuses": list(
                rules["manual_visual_review_required_statuses"]
            ),
            "automatic_fail_closed_visual_statuses": list(
                rules["automatic_fail_closed_visual_statuses"]
            ),
            "manual_review_claim_requires_image_inspection": True,
            "visual_review": visual_review_contract,
        },
        "analysis_coverage_contract": analysis_coverage,
    }
    abstract_contract = rules.get("abstract_contract")
    if abstract_contract:
        contract["abstract_contract"] = deepcopy(abstract_contract)
    return contract


def bundle(
    metadata: dict,
    evidence_wrapper: dict,
    figures_wrapper: dict,
    assets_wrapper: dict,
    source_manifest: dict | None = None,
    figure_decisions_wrapper: dict | None = None,
    output_language: str | None = None,
) -> dict:
    evidence_pack = (
        evidence_wrapper.get("evidence_pack", {})
        if isinstance(evidence_wrapper.get("evidence_pack"), dict)
        else {}
    )
    figure_plan = (
        figures_wrapper.get("figure_plan", {})
        if isinstance(figures_wrapper.get("figure_plan"), dict)
        else {}
    )
    source_manifest = source_manifest or {}
    figure_decisions_wrapper = figure_decisions_wrapper or {}
    config = runtime_config(cli_overrides={"output_language": output_language or ""})
    resolved_output_language = normalize_output_language(config["output_language"])
    require_artifact_output_language(
        figures_wrapper,
        "Figure Plan",
        resolved_output_language,
    )
    require_artifact_output_language(
        figure_decisions_wrapper,
        "Figure/Table Decisions",
        resolved_output_language,
    )
    identity_contract = identity_contract_summary(metadata, source_manifest)
    canonical_metadata = accepted_bundle_metadata(metadata, identity_contract)

    return {
        "status": "ok",
        "script": "build_synthesis_bundle.py",
        "output_language": resolved_output_language,
        "paper_id": identity_contract.get("paper_id")
        or canonical_metadata.get("paper_id")
        or evidence_wrapper.get("paper_id", ""),
        "title": canonical_metadata.get("title") or evidence_wrapper.get("title", ""),
        "identity_contract": identity_contract,
        "metadata": {
            "title": canonical_metadata.get("title", ""),
            "translated_title": canonical_metadata.get("translated_title", ""),
            "authors": canonical_metadata.get("authors", []),
            "affiliations": canonical_metadata.get("affiliations", []),
            "year": canonical_metadata.get("year", ""),
            "venue": canonical_metadata.get("venue", ""),
            "doi": canonical_metadata.get("doi", ""),
            "source_url": canonical_metadata.get("source_url", ""),
            "abstract": canonical_metadata.get("abstract", ""),
            "arxiv_id": canonical_metadata.get("arxiv_id", ""),
            "zotero_key": canonical_metadata.get("zotero_key", ""),
            "metadata_sources": canonical_metadata.get("metadata_sources", []),
            "identity_confidence": canonical_metadata.get("identity_confidence", ""),
            "identity_confidence_reasons": canonical_metadata.get(
                "identity_confidence_reasons", []
            )
            or [],
        },
        "evidence_quality": evidence_pack.get("evidence_quality", "unknown"),
        "coverage": coverage_summary(
            evidence_pack,
            canonical_metadata,
            assets_wrapper,
            source_manifest,
        ),
        "source_manifest": {
            "paper_id": source_manifest.get("paper_id", ""),
            "title": source_manifest.get("title", ""),
            "source_kind": source_manifest.get("source_kind", ""),
            "raw_sections_path": source_manifest.get("raw_sections_path", ""),
            "full_text_md_path": source_manifest.get("full_text_md_path", ""),
            "pdf": source_manifest.get("pdf", {}) or {},
            "coverage": source_manifest.get("coverage", {}) or {},
        },
        "source_index": source_index(source_manifest),
        "references": {"candidates": sanitize_reference_candidates(evidence_pack, config)},
        "figure_plan": figure_plan,
        "figure_table_manifest": figure_table_manifest(
            figure_decisions_wrapper,
            source_manifest,
            figure_plan,
        ),
        "pdf_assets": {
            "asset_root": assets_wrapper.get("asset_root", ""),
            "images_dir": assets_wrapper.get("images_dir", ""),
            "page_assets": sanitize_page_assets(assets_wrapper),
            "image_assets": assets_wrapper.get("image_assets", []),
            "figure_assets": sanitize_figure_assets(assets_wrapper),
            "ocr_available": assets_wrapper.get("ocr_available", False),
        },
        "writing_contract": compact_writing_contract(resolved_output_language),
    }


def main() -> None:
    from common import emit

    args = parser().parse_args()
    metadata = load_record(args.metadata)
    evidence = load_record(args.evidence)
    figures = load_record(args.figures) if args.figures else {}
    assets = load_record(args.assets) if args.assets else {}
    source_manifest = load_record(args.source_manifest) if args.source_manifest else {}
    figure_decisions = load_record(args.figure_decisions) if args.figure_decisions else {}
    if args.figure_decisions and isinstance(figure_decisions, dict):
        decision_path = Path(args.figure_decisions).expanduser()
        if decision_path.exists():
            figure_decisions.setdefault("decisions_path", str(decision_path.resolve()))
    emit(
        bundle(
            metadata,
            evidence,
            figures,
            assets,
            source_manifest,
            figure_decisions,
            args.language,
        ),
        args.output,
    )


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""Inspect the local DeepPaperNote environment for maintenance and troubleshooting."""

from __future__ import annotations

import argparse
import importlib.util
import shutil
import sys
from pathlib import Path

from _zotero_local import probe_zotero_local_api
from common import emit, env_config_value, runtime_config
from user_configuration import inspect_configuration


def parser() -> argparse.ArgumentParser:
    p = argparse.ArgumentParser(description=__doc__ or "check environment")
    p.add_argument("--output", default="", help="Optional JSON output path.")
    return p


def import_available(module_name: str) -> bool:
    return importlib.util.find_spec(module_name) is not None


def find_obsidian_candidates() -> list[str]:
    roots = [
        Path.home() / "Documents",
        Path.home() / "Desktop",
    ]
    results: list[str] = []
    seen: set[str] = set()
    for root in roots:
        if not root.exists():
            continue
        try:
            for path in root.rglob("*"):
                if not path.is_dir():
                    continue
                if path == root:
                    continue
                name = path.name.lower()
                if "obsidian" not in name and "vault" not in name:
                    continue
                resolved = str(path.resolve())
                if resolved in seen:
                    continue
                seen.add(resolved)
                results.append(resolved)
                if len(results) >= 8:
                    return results
        except Exception:
            continue
    return results


def find_local_zotero_hints() -> list[str]:
    candidates = [
        Path.home() / "Zotero",
        Path.home() / "Library" / "Application Support" / "Zotero",
    ]
    hits: list[str] = []
    for path in candidates:
        if path.exists():
            hits.append(str(path.resolve()))
    return hits


def main() -> None:
    args = parser().parse_args()
    configuration = inspect_configuration()
    config = runtime_config() if configuration["state"] == "ready" else {}
    zotero_local_api = probe_zotero_local_api()

    obsidian_vault = str(config.get("obsidian_vault", "")).strip()
    obsidian_vault_exists = bool(obsidian_vault) and Path(obsidian_vault).expanduser().exists()

    tesseract_path = shutil.which("tesseract") or ""
    pdftoppm_path = shutil.which("pdftoppm") or ""

    payload = {
        "status": "ok",
        "script": "check_environment.py",
        "tool_role": "maintenance",
        "user_configuration": configuration,
        "python": {
            "executable": sys.executable,
            "version": sys.version.split()[0],
            "fitz_installed": import_available("fitz"),
            "pytesseract_installed": import_available("pytesseract"),
            "pillow_installed": import_available("PIL"),
        },
        "obsidian": {
            "configured": bool(obsidian_vault),
            "vault_path": obsidian_vault,
            "vault_exists": obsidian_vault_exists,
            "papers_dir": str(config.get("papers_dir", "")),
            "output_dir": str(config.get("output_dir", "")),
            "candidate_vaults": find_obsidian_candidates(),
        },
        "workspace_fallback": {
            "available": True,
            "current_working_directory": str(Path.cwd().resolve()),
            "workspace_output_dir": str(config.get("workspace_output_dir", "DeepPaperNote_output")),
            "note": (
                "With save_mode=workspace, DeepPaperNote saves under the current working directory."
            ),
        },
        "zotero": {
            "local_hints": find_local_zotero_hints(),
            "local_api": zotero_local_api,
            "local_api_available": bool(zotero_local_api.get("ready")),
            "local_api_status": str(zotero_local_api.get("status", "error")),
            "local_api_version": str(zotero_local_api.get("api_version", "")),
            "local_api_schema_version": str(zotero_local_api.get("schema_version", "")),
            "mcp_available_from_script": False,
            "session_integration_checked_by_script": False,
            "note": (
                "The built-in read-only Local API check is reported here. Session-scoped "
                "library integrations must still be checked by the active agent at runtime."
            ),
        },
        "ocr": {
            "tesseract_installed": bool(tesseract_path),
            "tesseract_path": tesseract_path,
            "pytesseract_installed": import_available("pytesseract"),
            "pillow_installed": import_available("PIL"),
            "pdftoppm_installed": bool(pdftoppm_path),
            "pdftoppm_path": pdftoppm_path,
        },
        "metadata": {
            "maintenance_utility": True,
            "semantic_scholar_api_key_configured": bool(
                env_config_value(
                    "DEEPPAPERNOTE_SEMANTIC_SCHOLAR_API_KEY", "SEMANTIC_SCHOLAR_API_KEY"
                )
            ),
        },
    }
    emit(payload, args.output)


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""Best-effort reference extraction and Obsidian note matching helpers."""

from __future__ import annotations

import re
from pathlib import Path
from typing import Any

from common import configured_obsidian_vault, extract_arxiv_id, extract_doi, fitz, normalize_whitespace


NUMBERED_REF_RE = re.compile(r"^(?:\[(\d{1,4})\]|(\d{1,4})\.)\s+(.+)$")
YEAR_RE = re.compile(r"\b(?:19|20)\d{2}[a-z]?\b", flags=re.IGNORECASE)
FRONTMATTER_RE = re.compile(r"\A---\s*\n(.*?)\n---\s*", flags=re.DOTALL)
FRONTMATTER_KEY_RE = re.compile(r"^([A-Za-z_][\w-]*)\s*:\s*(.*)$")


def _empty_candidate(raw_text: str, page_hint: str = "") -> dict[str, Any]:
    display_text = _display_text(raw_text)
    return {
        "raw_text": normalize_whitespace(raw_text),
        "display_text": display_text,
        "page_hint": page_hint,
        "doi": extract_doi(raw_text) or "",
        "arxiv_id": extract_arxiv_id(raw_text) or "",
        "wikilink": "",
        "vault_target": "",
        "match_status": "no_vault_match",
        "match_reason": "none",
    }


def _display_text(raw_text: str, *, max_chars: int = 320) -> str:
    text = normalize_whitespace(raw_text)
    text = re.sub(r"^(?:\[\d{1,4}\]|\d{1,4}\.)\s+", "", text)
    if len(text) <= max_chars:
        return text
    return text[: max_chars - 3].rstrip() + "..."


def _reference_lines_from_pdf(pdf_path: Path, references_start_page: int, max_pages: int) -> list[tuple[str, int]]:
    if fitz is None or references_start_page < 1 or max_pages < 1:
        return []
    doc = fitz.open(pdf_path)
    try:
        start_index = max(0, references_start_page - 1)
        stop_index = min(len(doc), start_index + max_pages)
        lines: list[tuple[str, int]] = []
        for page_index in range(start_index, stop_index):
            page_number = page_index + 1
            for raw_line in doc[page_index].get_text("text").splitlines():
                line = normalize_whitespace(raw_line)
                if not line:
                    continue
                if line.lower() in {"references", "bibliography", "参考文献"}:
                    continue
                lines.append((line, page_number))
        return lines
    finally:
        doc.close()


def _group_numbered_references(lines: list[tuple[str, int]]) -> list[tuple[str, int]]:
    grouped: list[tuple[list[str], int]] = []
    current: list[str] = []
    current_page = 0

    for line, page_number in lines:
        if NUMBERED_REF_RE.match(line):
            if current:
                grouped.append((current, current_page))
            current = [line]
            current_page = page_number
            continue
        if current:
            previous = current[-1]
            starts_like_continuation = bool(
                re.match(r"^(?:https?://|doi\b|10\.|[a-z,;:)])", line.strip())
            )
            previous_looks_complete = previous.rstrip().endswith((".", "。", "!", "?", ";"))
            if not previous_looks_complete or starts_like_continuation:
                current.append(line)

    if current:
        grouped.append((current, current_page))

    return [(normalize_whitespace(" ".join(parts)), page_number) for parts, page_number in grouped]


def _fallback_year_lines(lines: list[tuple[str, int]]) -> list[tuple[str, int]]:
    return [(line, page_number) for line, page_number in lines if YEAR_RE.search(line)]


def extract_reference_candidates_from_pdf(
    pdf_path: str | Path,
    references_start_page: int | None,
    *,
    max_pages: int = 3,
    max_items: int = 20,
) -> list[dict[str, Any]]:
    """Extract best-effort reference candidates from the PDF reference section."""
    if references_start_page is None:
        return []
    path = Path(pdf_path).expanduser()
    if not path.is_file():
        return []

    try:
        lines = _reference_lines_from_pdf(path, int(references_start_page), max_pages)
    except Exception:
        return []

    references = _group_numbered_references(lines)
    if not references:
        references = _fallback_year_lines(lines)

    candidates: list[dict[str, Any]] = []
    seen: set[str] = set()
    for raw_text, page_number in references:
        raw_text = normalize_whitespace(raw_text)
        if not raw_text or raw_text in seen:
            continue
        seen.add(raw_text)
        candidates.append(_empty_candidate(raw_text, page_hint=f"p. {page_number}"))
        if len(candidates) >= max_items:
            break
    return candidates


def _strip_wrapping_quotes(value: str) -> str:
    value = value.strip()
    if len(value) >= 2 and value[0] == value[-1] and value[0] in {"'", '"'}:
        return value[1:-1].strip()
    return value


def _parse_inline_aliases(value: str) -> list[str]:
    value = value.strip()
    if not value:
        return []
    if value.startswith("[") and value.endswith("]"):
        value = value[1:-1]
        return [_strip_wrapping_quotes(part) for part in value.split(",") if _strip_wrapping_quotes(part)]
    alias = _strip_wrapping_quotes(value)
    return [alias] if alias else []


def _frontmatter_fields(text: str) -> dict[str, Any] | None:
    match = FRONTMATTER_RE.match(text)
    if not match:
        return None

    fields: dict[str, Any] = {"aliases": []}
    lines = match.group(1).splitlines()
    index = 0
    while index < len(lines):
        line = lines[index]
        stripped = line.strip()
        key_match = FRONTMATTER_KEY_RE.match(stripped)
        if not key_match:
            index += 1
            continue

        key = key_match.group(1).strip().lower().replace("-", "_")
        inline_value = key_match.group(2).strip()
        if key == "aliases":
            if inline_value:
                fields["aliases"] = _parse_inline_aliases(inline_value)
                index += 1
                continue

            aliases: list[str] = []
            index += 1
            while index < len(lines):
                item = lines[index].strip()
                if not item.startswith("- "):
                    break
                alias = _strip_wrapping_quotes(item[2:])
                if alias:
                    aliases.append(alias)
                index += 1
            fields["aliases"] = aliases
            continue

        if key in {"doi", "arxiv", "arxiv_id", "title"} and inline_value:
            fields[key] = _strip_wrapping_quotes(inline_value)
        index += 1

    return fields


def _h1_title(text: str) -> str:
    match = FRONTMATTER_RE.match(text)
    body = text[match.end() :] if match else text
    for line in body.splitlines():
        stripped = line.strip()
        if stripped.startswith("# ") and not stripped.startswith("## "):
            return normalize_whitespace(stripped[2:].strip().strip("#"))
    return ""


def _normalize_doi(value: str) -> str:
    doi = extract_doi(value) or ""
    doi = re.sub(r"^https?://(?:dx\.)?doi\.org/", "", doi, flags=re.IGNORECASE)
    return doi.strip().lower()


def _normalize_arxiv_id(value: str) -> str:
    arxiv_id = extract_arxiv_id(value) or ""
    return re.sub(r"^arxiv:\s*", "", arxiv_id, flags=re.IGNORECASE).strip().lower()


def _short_acronym_like(value: str) -> bool:
    compact = re.sub(r"[^A-Za-z0-9]", "", value)
    if len(compact) > 6:
        return False
    key = _match_key(value)
    return bool(compact and " " not in key)


def _safe_text_key(value: str) -> str:
    key = normalize_whitespace(value)
    if not key or _short_acronym_like(key):
        return ""
    if len(re.sub(r"\s+", "", _match_key(key))) < 8:
        return ""
    return key


def _dedupe_values(values: list[str]) -> list[str]:
    seen: set[str] = set()
    deduped: list[str] = []
    for value in values:
        cleaned = normalize_whitespace(value)
        if not cleaned:
            continue
        folded = cleaned.casefold()
        if folded in seen:
            continue
        seen.add(folded)
        deduped.append(cleaned)
    return deduped


def _note_text_keys(note: dict[str, Any]) -> list[str]:
    values = [
        str(note.get("stem", "")),
        str(note.get("frontmatter_title", "")),
        str(note.get("h1_title", "")),
    ]
    values.extend(str(alias) for alias in note.get("aliases", []) or [])
    keys = [_safe_text_key(value) for value in values]
    return _dedupe_values([key for key in keys if key])


def _candidate_doi(candidate: dict[str, Any], candidate_text: str) -> str:
    return _normalize_doi(str(candidate.get("doi", "")) or candidate_text)


def _candidate_arxiv_ids(candidate: dict[str, Any], candidate_text: str) -> set[str]:
    values = [
        str(candidate.get("arxiv_id", "")),
        str(candidate.get("doi", "")),
        candidate_text,
    ]
    return {_normalize_arxiv_id(value) for value in values if _normalize_arxiv_id(value)}


def _match_key(value: str) -> str:
    text = normalize_whitespace(value).casefold()
    text = re.sub(r"[_-]+", " ", text)
    text = re.sub(r"[^\w\s\u3400-\u9fff]", " ", text, flags=re.UNICODE)
    return normalize_whitespace(text)


def _contains_key(raw_text: str, key: str) -> bool:
    raw_key = _match_key(raw_text)
    needle = _match_key(key)
    if not raw_key or not needle:
        return False
    if re.search(r"[\w]", needle):
        return bool(re.search(rf"(?<!\w){re.escape(needle)}(?!\w)", raw_key))
    return needle in raw_key


def _note_wikilink(target: str, display_text: str) -> str:
    display = normalize_whitespace(display_text)
    if display:
        return f"[[{target}|{display}]]"
    return f"[[{target}]]"


def build_vault_note_index(config: dict[str, Any]) -> dict[str, Any]:
    """Build a limited note index under the configured papers directory."""
    try:
        vault_path = configured_obsidian_vault(config)
    except Exception:
        return {"status": "vault_unavailable", "notes": []}
    if vault_path is None:
        return {"status": "vault_unavailable", "notes": []}

    papers_dir = str(config.get("papers_dir", "Research/Papers")).strip() or "Research/Papers"
    base_dir = (vault_path / Path(papers_dir)).resolve()
    try:
        base_dir.relative_to(vault_path)
    except ValueError:
        return {"status": "vault_unavailable", "notes": []}
    if not base_dir.exists() or not base_dir.is_dir():
        return {"status": "vault_unavailable", "notes": []}

    notes: list[dict[str, Any]] = []
    for path in sorted(base_dir.glob("**/*.md")):
        try:
            relative_path = path.relative_to(vault_path)
        except ValueError:
            continue
        try:
            # utf-8-sig strips a leading BOM; without it a BOM-prefixed note
            # fails the frontmatter check and is silently dropped from the
            # index, so wiki-links to it never resolve.
            text = path.read_text(encoding="utf-8-sig", errors="ignore")[:4096]
        except OSError:
            text = ""
        frontmatter = _frontmatter_fields(text)
        if frontmatter is None:
            continue
        aliases = frontmatter.get("aliases", [])
        doi = normalize_whitespace(str(frontmatter.get("doi", "")))
        arxiv_id = normalize_whitespace(
            str(frontmatter.get("arxiv_id", "") or frontmatter.get("arxiv", ""))
        )
        normalized_arxiv_ids = {
            _normalize_arxiv_id(value)
            for value in [arxiv_id, doi]
            if _normalize_arxiv_id(value)
        }
        notes.append(
            {
                "stem": path.stem,
                "aliases": aliases if isinstance(aliases, list) else [],
                "frontmatter_title": normalize_whitespace(str(frontmatter.get("title", ""))),
                "h1_title": _h1_title(text),
                "doi": doi,
                "doi_norm": _normalize_doi(doi) if doi else "",
                "arxiv_id": arxiv_id,
                "arxiv_ids": sorted(normalized_arxiv_ids),
                "text_keys": [],
                "vault_target": path.stem,
                "vault_relative_path": str(relative_path),
            }
        )
        notes[-1]["text_keys"] = _note_text_keys(notes[-1])
    return {"status": "ok", "notes": notes}


def resolve_reference_links(candidates: list[dict[str, Any]], config: dict[str, Any]) -> list[dict[str, Any]]:
    """Attach exact Obsidian wikilink matches to extracted reference candidates."""
    index = build_vault_note_index(config)
    if index.get("status") != "ok":
        return [
            {
                **candidate,
                "wikilink": "",
                "vault_target": "",
                "match_status": "vault_unavailable",
                "match_reason": "none",
            }
            for candidate in candidates
            if isinstance(candidate, dict)
        ]

    notes = [note for note in index.get("notes", []) if isinstance(note, dict)]

    matched: list[dict[str, Any]] = []
    for candidate in candidates:
        if not isinstance(candidate, dict):
            continue
        raw_text = str(candidate.get("raw_text", "") or candidate.get("display_text", ""))
        display_text = normalize_whitespace(str(candidate.get("display_text", ""))) or raw_text
        candidate_text = normalize_whitespace(f"{raw_text} {display_text}")
        resolved = dict(candidate)
        resolved.setdefault("match_status", "no_vault_match")
        resolved.setdefault("match_reason", "none")
        resolved.setdefault("wikilink", "")
        resolved.setdefault("vault_target", "")

        priority_matches: list[tuple[str, list[dict[str, Any]]]] = []
        doi = _candidate_doi(candidate, candidate_text)
        if doi:
            priority_matches.append(
                ("doi", [note for note in notes if doi and doi == str(note.get("doi_norm", ""))])
            )
        arxiv_ids = _candidate_arxiv_ids(candidate, candidate_text)
        if arxiv_ids:
            priority_matches.append(
                (
                    "arxiv_id",
                    [
                        note
                        for note in notes
                        if arxiv_ids.intersection(set(note.get("arxiv_ids", []) or []))
                    ],
                )
            )
        priority_matches.append(
            (
                "basename_or_title_or_alias",
                [
                    note
                    for note in notes
                    if any(
                        _contains_key(candidate_text, key)
                        for key in note.get("text_keys", []) or []
                    )
                ],
            )
        )

        for reason, matches in priority_matches:
            if not matches:
                continue
            if len(matches) == 1:
                target = normalize_whitespace(str(matches[0].get("vault_target", "")))
                resolved["wikilink"] = _note_wikilink(target, display_text)
                resolved["vault_target"] = target
                resolved["match_status"] = "vault_match"
                resolved["match_reason"] = reason
            else:
                resolved["wikilink"] = ""
                resolved["vault_target"] = ""
                resolved["match_status"] = "ambiguous_match"
                resolved["match_reason"] = reason
                resolved["match_candidates"] = [
                    {
                        "wikilink": _note_wikilink(
                            normalize_whitespace(str(note.get("vault_target", ""))),
                            display_text,
                        ),
                        "vault_target": normalize_whitespace(str(note.get("vault_target", ""))),
                        "match_status": "vault_match",
                        "match_reason": reason,
                    }
                    for note in matches
                ]
            break

        matched.append(resolved)
    return matched

#!/usr/bin/env python3
"""Collect and merge metadata from Zotero, DOI, Semantic Scholar, OpenAlex, arXiv, and publisher pages."""

from __future__ import annotations

from common import (
    base_parser,
    collect_metadata_observations,
    emit,
    maybe_load_json_record,
    paper_id_for_record,
    require_ok_input_artifact,
    resolve_reference,
)


def main() -> None:
    parser = base_parser(__doc__ or "collect metadata")
    args = parser.parse_args()

    if not args.input:
        raise SystemExit("collect_metadata.py requires --input.")

    input_record = maybe_load_json_record(args.input)
    if input_record is not None:
        record = dict(require_ok_input_artifact(input_record, "collect_metadata.py"))
    else:
        record = resolve_reference(args.input)

    metadata = dict(record)
    metadata["identity_observations"] = collect_metadata_observations(record)
    metadata["paper_id"] = args.paper_id or metadata.get("paper_id") or paper_id_for_record(metadata)
    metadata["status"] = "ok"
    metadata["script"] = "collect_metadata.py"
    emit(metadata, args.output)


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""Shared helpers for DeepPaperNote scripts."""

from __future__ import annotations

import argparse
import hashlib
import json
import os
import re
import ssl
import unicodedata
import urllib.parse
import urllib.request
import xml.etree.ElementTree as ET
from copy import deepcopy
from pathlib import Path, PureWindowsPath
from typing import Any

from user_configuration import (
    inspect_configuration,
    resolve_preferences,
    resolve_run_overrides,
)

ARXIV_NS = {
    "atom": "http://www.w3.org/2005/Atom",
    "arxiv": "http://arxiv.org/schemas/atom",
}

SEMANTIC_SCHOLAR_SEARCH_URL = "https://api.semanticscholar.org/graph/v1/paper/search"
OPENALEX_WORKS_URL = "https://api.openalex.org/works"
CROSSREF_WORKS_URL = "https://api.crossref.org/works"
DEFAULT_USER_AGENT = "DeepPaperNote/0.1"
PROVENANCE_FIELDS = (
    "title",
    "translated_title",
    "authors",
    "affiliations",
    "year",
    "venue",
    "doi",
    "arxiv_id",
    "zotero_key",
    "abstract",
    "source_url",
    "pdf_url",
    "local_pdf_path",
)
SHELL_CONFIG_FILES = [
    Path.home() / ".zshenv",
    Path.home() / ".zprofile",
    Path.home() / ".zshrc",
    Path.home() / ".bash_profile",
    Path.home() / ".bashrc",
]

try:
    import pymupdf as fitz  # type: ignore
except ImportError:  # pragma: no cover
    try:
        import fitz  # type: ignore
    except ImportError:
        fitz = None

try:
    import certifi  # type: ignore
except ImportError:  # pragma: no cover
    certifi = None


def base_parser(description: str) -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(description=description)
    parser.add_argument("--input", help="Primary input path, JSON artifact, or identifier.")
    parser.add_argument("--output", help="Output path for JSON or Markdown.")
    parser.add_argument("--paper-id", help="Canonical paper id if already known.")
    return parser


def ensure_parent(path: str | Path) -> None:
    Path(path).expanduser().resolve().parent.mkdir(parents=True, exist_ok=True)


def file_sha256(path: str | Path) -> str:
    candidate = Path(path).expanduser()
    if not candidate.is_file():
        return ""
    digest = hashlib.sha256()
    with candidate.open("rb") as handle:
        for chunk in iter(lambda: handle.read(1024 * 1024), b""):
            digest.update(chunk)
    return digest.hexdigest()


def emit(payload: dict[str, Any], output_path: str | None = None) -> None:
    text = json.dumps(payload, ensure_ascii=False, indent=2)
    if output_path:
        ensure_parent(output_path)
        Path(output_path).write_text(text + "\n", encoding="utf-8")
    else:
        print(text)


def stub_payload(script: str, description: str, outputs: list[str]) -> dict[str, Any]:
    return {
        "status": "scaffold",
        "script": script,
        "description": description,
        "next_step": "Implement this contract incrementally.",
        "outputs": outputs,
    }


def load_json_file(path: str | Path) -> dict[str, Any]:
    # utf-8-sig transparently strips a UTF-8 BOM (e.g. files written by
    # PowerShell Out-File / Notepad on Windows) that would otherwise make
    # json.loads raise "Unexpected UTF-8 BOM". It is a no-op for BOM-less files.
    data = json.loads(Path(path).expanduser().resolve().read_text(encoding="utf-8-sig"))
    if not isinstance(data, dict):
        raise RuntimeError("Expected a JSON object.")
    return data


def maybe_load_json_record(value: str | None) -> dict[str, Any] | None:
    if not value:
        return None
    stripped = value.strip()
    if not stripped:
        return None
    path = Path(stripped).expanduser()
    if path.exists() and path.is_file() and path.suffix.lower() == ".json":
        return load_json_file(path)
    if stripped.startswith("{"):
        data = json.loads(stripped)
        if isinstance(data, dict):
            return data
    return None


def require_ok_input_artifact(record: dict[str, Any], consumer: str) -> dict[str, Any]:
    status = normalize_whitespace(str(record.get("status", ""))).lower()
    if status and status != "ok":
        producer = normalize_whitespace(str(record.get("script", ""))) or "unknown producer"
        message = f"{consumer} refuses non-ok input artifact from {producer}: status={status}"
        raise SystemExit(message)
    return record


def normalize_whitespace(text: str) -> str:
    return re.sub(r"\s+", " ", (text or "")).strip()


def strip_tags(text: str) -> str:
    return normalize_whitespace(re.sub(r"<[^>]+>", " ", text or ""))


def normalize_title(text: str) -> str:
    return re.sub(r"[^a-z0-9\s]", "", normalize_whitespace(text).lower()).strip()


LOCAL_PDF_PREFIX_PATTERN = re.compile(r"^(?:[^-]{1,120})\s+-\s+(?:19|20)\d{2}\s+-\s+")
LOCAL_PDF_SUFFIX_ID_PATTERN = re.compile(r"\s*-\s*\d{4,}\s*$")
PREPRINT_HINTS = ("medrxiv", "biorxiv", "preprint", "arxiv", "10.1101/", "10.21203/rs.", "preprints.org")
PDF_LIGATURE_MAP = {
    "\u00df": "ss",
    "\ufb00": "ff",
    "\ufb01": "fi",
    "\ufb02": "fl",
    "\ufb03": "ffi",
    "\ufb04": "ffl",
}


def clean_local_pdf_stem(stem: str) -> str:
    raw = normalize_whitespace((stem or "").replace("_", " "))
    if not raw:
        return ""
    cleaned = LOCAL_PDF_PREFIX_PATTERN.sub("", raw)
    cleaned = LOCAL_PDF_SUFFIX_ID_PATTERN.sub("", cleaned)
    cleaned = normalize_whitespace(cleaned)
    return cleaned or raw


def is_probable_local_pdf_artifact_title(title: str) -> bool:
    normalized = normalize_whitespace(title)
    if not normalized:
        return False
    if LOCAL_PDF_PREFIX_PATTERN.match(normalized):
        return True
    if LOCAL_PDF_SUFFIX_ID_PATTERN.search(normalized):
        return True
    return bool(re.search(r"\b(?:et al\.?|等)\b", normalized, flags=re.IGNORECASE) and re.search(r"\b(?:19|20)\d{2}\b", normalized))


def _dedupe_string_list(value: Any) -> list[str]:
    if value in ("", None, [], {}):
        return []
    values = value if isinstance(value, list) else [value]
    deduped: list[str] = []
    for item in values:
        cleaned = normalize_whitespace(str(item))
        if cleaned and cleaned not in deduped:
            deduped.append(cleaned)
    return deduped


def _append_reason(reasons: list[str], reason: str) -> None:
    cleaned = normalize_whitespace(reason)
    if cleaned and cleaned not in reasons:
        reasons.append(cleaned)


def apply_identity_confidence(record: dict[str, Any]) -> dict[str, Any]:
    reasons = _dedupe_string_list(record.get("identity_confidence_reasons", []))
    confidence = normalize_whitespace(str(record.get("identity_confidence", ""))).lower()
    if confidence not in {"low", "medium", "high"}:
        confidence = ""

    if record.get("doi"):
        _append_reason(reasons, "doi_present")
        confidence = "high"
    if record.get("arxiv_id"):
        _append_reason(reasons, "arxiv_id_present")
        confidence = "high"
    if record.get("zotero_key"):
        _append_reason(reasons, "zotero_key_present")
        confidence = "high"

    source_type = normalize_whitespace(str(record.get("source_type", "")))
    metadata_sources = set(_dedupe_string_list(record.get("metadata_sources", [])))
    has_local_pdf = source_type == "local_pdf" or "local_pdf" in metadata_sources
    has_title_query = source_type == "title_query" or "title_query" in metadata_sources
    external_sources = metadata_sources - {"local_pdf", "title_query"}

    title_source_reason = normalize_whitespace(str(record.get("local_pdf_title_source", "")))
    if has_local_pdf and title_source_reason:
        _append_reason(reasons, title_source_reason)
    if has_local_pdf and record.get("local_pdf_artifact_title"):
        _append_reason(reasons, "local_pdf_artifact_title")

    if confidence != "high":
        if has_local_pdf and record.get("title_corrected_from_external_metadata"):
            _append_reason(reasons, "external_metadata_title_match")
            confidence = "medium"
        elif has_title_query:
            if external_sources:
                _append_reason(reasons, "external_metadata_title_match")
                confidence = "medium"
            else:
                _append_reason(reasons, "title_query_unmatched")
                confidence = "low"
        elif has_local_pdf:
            if is_probable_local_pdf_artifact_title(str(record.get("title", ""))):
                _append_reason(reasons, "local_pdf_artifact_title")
            confidence = "low"

    if confidence:
        record["identity_confidence"] = confidence
        record["identity_confidence_reasons"] = reasons
    return record


def normalize_pdf_text_artifacts(text: str) -> str:
    normalized = text or ""
    for original, replacement in PDF_LIGATURE_MAP.items():
        normalized = normalized.replace(original, replacement)
    return normalized


def slugify_filename(text: str) -> str:
    text = normalize_whitespace(text)
    text = re.sub(r"[^\w\s-]", "", text, flags=re.UNICODE)
    text = re.sub(r"[-\s]+", "_", text).strip("_")
    return text or "paper_note"


def shell_config_value(name: str) -> str:
    pattern = re.compile(rf"^\s*(?:export\s+)?{re.escape(name)}=(.*)$")
    for path in SHELL_CONFIG_FILES:
        if not path.exists() or not path.is_file():
            continue
        try:
            lines = path.read_text(encoding="utf-8-sig").splitlines()
        except Exception:
            continue
        for raw_line in reversed(lines):
            line = raw_line.strip()
            if not line or line.startswith("#"):
                continue
            match = pattern.match(line)
            if not match:
                continue
            value = match.group(1).strip()
            if len(value) >= 2 and value[0] == value[-1] and value[0] in {'"', "'"}:
                value = value[1:-1]
            return value.strip()
    return ""


def env_config_value(*names: str, default: str = "") -> str:
    for name in names:
        value = os.environ.get(name, "").strip()
        if value:
            return value
    disable_shell_fallback = os.environ.get("DEEPPAPERNOTE_DISABLE_SHELL_CONFIG", "").strip().lower()
    if disable_shell_fallback in {"1", "true", "yes", "on"}:
        return default
    for name in names:
        value = shell_config_value(name)
        if value:
            return value
    return default


def title_similarity(a: str, b: str) -> float:
    a_identity = normalize_identity_title(a)
    b_identity = normalize_identity_title(b)
    if a_identity and a_identity == b_identity:
        return 1.0
    a_norm = normalize_title(a)
    b_norm = normalize_title(b)
    if not a_norm or not b_norm:
        return 0.0
    if a_norm == b_norm:
        return 1.0
    words_a = set(a_norm.split())
    words_b = set(b_norm.split())
    if not words_a or not words_b:
        return 0.0
    return len(words_a & words_b) / len(words_a | words_b)


def publication_quality_score(record: dict[str, Any]) -> int:
    venue = normalize_whitespace(str(record.get("venue", ""))).lower()
    source_url = normalize_whitespace(str(record.get("source_url", ""))).lower()
    source = normalize_whitespace(str(record.get("source", ""))).lower()
    doi = normalize_whitespace(str(record.get("doi", ""))).lower()
    joined = " ".join([venue, source_url, source, doi])
    if any(token in joined for token in PREPRINT_HINTS):
        return 0
    if venue or source == "crossref":
        return 2
    return 1


def candidate_priority_score(record: dict[str, Any]) -> int:
    source = normalize_whitespace(str(record.get("source", ""))).lower()
    source_url = normalize_whitespace(str(record.get("source_url", ""))).lower()
    doi = normalize_whitespace(str(record.get("doi", ""))).lower()
    joined = " ".join([source, source_url, doi])

    if "10.20944/preprints" in joined or any(token in joined for token in PREPRINT_HINTS):
        return 0

    if record.get("doi") and publication_quality_score(record) >= 2:
        return 4

    if record.get("arxiv_id") or source == "arxiv" or "arxiv.org" in source_url:
        return 3

    if record.get("pdf_url"):
        return 2

    return 1


def extract_arxiv_id(paper_ref: str) -> str | None:
    paper_ref = (paper_ref or "").strip()
    patterns = [
        r"arxiv:(\d{4}\.\d{4,5})(?:v\d+)?",
        r"arxiv[./]\s*(\d{4}\.\d{4,5})(?:v\d+)?",
        r"abs/(\d{4}\.\d{4,5})(?:v\d+)?",
        r"pdf/(\d{4}\.\d{4,5})(?:v\d+)?(?:\.pdf)?",
        r"(?<![A-Za-z0-9./-])(\d{4}\.\d{4,5})(?:v\d+)?(?![A-Za-z0-9./-])",
    ]
    for pattern in patterns:
        match = re.search(pattern, paper_ref, flags=re.IGNORECASE)
        if match:
            return match.group(1)
    return None


def extract_doi(text: str) -> str | None:
    if not text:
        return None
    match = re.search(r"(10\.\d{4,9}/[-._;()/:A-Z0-9]+)", text, flags=re.IGNORECASE)
    if not match:
        return None
    return match.group(1).rstrip(").,;]")


def is_probable_url(text: str) -> bool:
    return bool(re.match(r"^https?://", (text or "").strip(), flags=re.IGNORECASE))


def is_probable_zotero_key(text: str) -> bool:
    return bool(re.fullmatch(r"[A-Z0-9]{8}", (text or "").strip()))


def infer_source_type(value: str) -> str:
    stripped = (value or "").strip()
    if not stripped:
        return "unknown"
    path = Path(stripped).expanduser()
    if path.exists() and path.is_file() and path.suffix.lower() == ".pdf":
        return "local_pdf"
    if is_probable_url(stripped):
        if "arxiv.org" in stripped.lower() and extract_arxiv_id(stripped):
            return "arxiv_url"
        if extract_doi(stripped):
            return "doi_url"
        if stripped.lower().endswith(".pdf"):
            return "pdf_url"
        return "url"
    if extract_doi(stripped):
        return "doi"
    if extract_arxiv_id(stripped):
        return "arxiv_id"
    if is_probable_zotero_key(stripped):
        return "zotero_key"
    return "title"


def paper_id_for_record(record: dict[str, Any]) -> str:
    if record.get("paper_id"):
        return str(record["paper_id"])
    if record.get("doi"):
        return f"doi:{str(record['doi']).lower()}"
    if record.get("arxiv_id"):
        return f"arxiv:{record['arxiv_id']}"
    if record.get("zotero_key"):
        return f"zotero:{record['zotero_key']}"
    if record.get("title"):
        digest = hashlib.sha1(normalize_title(str(record["title"])).encode("utf-8")).hexdigest()[:12]
        return f"title:{digest}"
    source = str(record.get("source_url") or record.get("local_pdf_path") or "unknown")
    digest = hashlib.sha1(source.encode("utf-8")).hexdigest()[:12]
    return f"paper:{digest}"


def _string_field(record: dict[str, Any], key: str) -> str:
    return normalize_whitespace(str(record.get(key, "")))


def _path_string(path_value: str) -> str:
    value = normalize_whitespace(path_value)
    if not value:
        return ""
    return str(Path(value).expanduser().resolve())


def _artifact_path(path_value: str) -> str:
    value = normalize_whitespace(path_value)
    if not value:
        return ""
    path = Path(value).expanduser()
    return str(path.resolve()) if path.exists() else str(path)


def selected_identity_evidence(
    record: dict[str, Any],
    source_record: dict[str, Any] | None = None,
) -> list[dict[str, str]]:
    evidence: list[dict[str, str]] = []
    seen: set[tuple[str, str, str]] = set()

    def append(kind: str, value: str, trust: str, source: str) -> None:
        cleaned = normalize_whitespace(value)
        marker = (kind, cleaned, source)
        if cleaned and marker not in seen:
            seen.add(marker)
            evidence.append(
                {
                    "kind": kind,
                    "value": cleaned,
                    "trust": trust,
                    "source": source,
                }
            )

    append("doi", _string_field(record, "doi"), "strong", "metadata")
    append("arxiv_id", _string_field(record, "arxiv_id"), "strong", "metadata")
    append("zotero_key", _string_field(record, "zotero_key"), "strong", "metadata")
    append("title", _string_field(record, "title"), "metadata", "metadata")
    local_pdf = _string_field(record, "local_pdf_path")
    if local_pdf:
        append("trusted_source_path", _path_string(local_pdf), "strong", "source_manifestation")
    pdf_url = _string_field(record, "pdf_url")
    if pdf_url:
        append("pdf_url", pdf_url, "source_manifestation", "metadata")
    source_url = _string_field(record, "source_url")
    if source_url and source_url != pdf_url:
        append("source_url", source_url, "source_manifestation", "metadata")
    if source_record and source_record is not record:
        local_pdf = _string_field(source_record, "local_pdf_path")
        if local_pdf:
            append("trusted_source_path", _path_string(local_pdf), "strong", "source_manifestation")
        pdf_url = _string_field(source_record, "pdf_url")
        if pdf_url:
            append("pdf_url", pdf_url, "source_manifestation", "source_manifestation")
        source_url = _string_field(source_record, "source_url")
        if source_url and source_url != pdf_url:
            append("source_url", source_url, "source_manifestation", "source_manifestation")
    return evidence


def work_level_identity_from_record(record: dict[str, Any]) -> dict[str, Any]:
    return {
        "title": _string_field(record, "title"),
        "translated_title": _string_field(record, "translated_title"),
        "authors": _dedupe_string_list(record.get("authors", [])),
        "year": _string_field(record, "year"),
        "venue": _string_field(record, "venue"),
        "doi": _string_field(record, "doi"),
        "arxiv_id": _string_field(record, "arxiv_id"),
        "zotero_key": _string_field(record, "zotero_key"),
    }


def source_manifestation_from_record(
    record: dict[str, Any],
    work_record: dict[str, Any] | None = None,
) -> dict[str, str]:
    work_record = work_record or {}
    local_pdf = _string_field(record, "local_pdf_path")
    if not local_pdf:
        local_pdf = _string_field(work_record, "local_pdf_path")
    source_kind = _string_field(record, "source_type")
    if not source_kind:
        source_kind = _string_field(work_record, "source_type")
    if local_pdf:
        source_kind = "local_pdf"
    elif _string_field(record, "pdf_url") or _string_field(work_record, "pdf_url"):
        source_kind = source_kind or "pdf_url"
    elif _string_field(record, "source_url") or _string_field(work_record, "source_url"):
        source_kind = source_kind or "source_url"
    source_url = _string_field(record, "source_url")
    if not source_url and local_pdf:
        source_url = _path_string(local_pdf)
    return {
        "source_kind": source_kind or "unknown",
        "title": _string_field(record, "title") or _string_field(work_record, "title"),
        "year": _string_field(record, "year"),
        "venue": _string_field(record, "venue"),
        "source_url": source_url or _string_field(work_record, "source_url"),
        "pdf_url": _string_field(record, "pdf_url") or _string_field(work_record, "pdf_url"),
        "local_pdf_path": _path_string(local_pdf),
        "doi": _string_field(record, "doi"),
        "arxiv_id": _string_field(record, "arxiv_id"),
        "zotero_key": _string_field(record, "zotero_key"),
    }


def _normalized_identifier(record: dict[str, Any], key: str) -> str:
    value = _string_field(record, key)
    if key == "doi":
        value = extract_doi(value) or value
    if key == "arxiv_id":
        value = extract_arxiv_id(value) or value
    return value.lower()


def _record_arxiv_id(record: dict[str, Any]) -> str:
    for key in ("arxiv_id", "doi", "source_url", "pdf_url"):
        arxiv_id = extract_arxiv_id(_string_field(record, key))
        if arxiv_id:
            return arxiv_id.lower()
    return ""


def _author_key(name: str) -> str:
    raw = normalize_whitespace(name)
    if not raw:
        return ""
    if "," in raw:
        raw = raw.split(",", 1)[0]
    parts = normalize_title(raw).split()
    if not parts:
        return ""
    return parts[-1]


def normalize_identity_title(title: str) -> str:
    return normalize_whitespace(unicodedata.normalize("NFKC", title).casefold())


def _author_identity_parts(name: str) -> list[str]:
    normalized = unicodedata.normalize("NFKC", normalize_whitespace(name)).casefold()
    if "," in normalized:
        family, given = normalized.split(",", 1)
        normalized = f"{given} {family}"
    return re.findall(r"[^\W_]+", normalized, flags=re.UNICODE)


def _full_leading_author_matches(
    source_record: dict[str, Any],
    work_record: dict[str, Any],
) -> bool:
    source_authors = _dedupe_string_list(source_record.get("authors", []))
    work_authors = _dedupe_string_list(work_record.get("authors", []))
    if not source_authors or not work_authors:
        return False
    source_normalized = normalize_identity_title(source_authors[0])
    work_normalized = normalize_identity_title(work_authors[0])
    if source_normalized == work_normalized:
        return True
    source_parts = _author_identity_parts(source_authors[0])
    work_parts = _author_identity_parts(work_authors[0])
    if len(source_parts) < 2 or len(work_parts) < 2:
        return False
    if source_parts[-1] != work_parts[-1]:
        return False
    source_given = source_parts[:-1]
    work_given = work_parts[:-1]
    return len(source_given) == len(work_given) and all(
        left == right
        for left, right in zip(source_given, work_given)
    )


def _leading_author_status(
    source_record: dict[str, Any],
    work_record: dict[str, Any],
) -> dict[str, Any] | None:
    source_authors = _dedupe_string_list(source_record.get("authors", []))
    work_authors = _dedupe_string_list(work_record.get("authors", []))
    if not source_authors or not work_authors:
        return None
    source_normalized = normalize_identity_title(source_authors[0])
    work_normalized = normalize_identity_title(work_authors[0])
    source_key = _author_key(source_authors[0])
    work_key = _author_key(work_authors[0])
    status = (
        "match"
        if source_normalized == work_normalized
        or (source_key and source_key == work_key)
        else "conflict"
    )
    return {
        "kind": "leading_author",
        "status": status,
        "source_value": source_authors[0],
        "work_value": work_authors[0],
    }


def _record_abstract(record: dict[str, Any]) -> str:
    for key in ("abstract", "first_paragraph", "first_page_text", "summary"):
        value = _string_field(record, key)
        if value:
            return value
    return ""


def _shared_identifier_evidence(
    source_record: dict[str, Any],
    work_record: dict[str, Any],
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
    shared: list[dict[str, Any]] = []
    conflicts: list[dict[str, Any]] = []
    source_doi = _normalized_identifier(source_record, "doi")
    work_doi = _normalized_identifier(work_record, "doi")
    source_arxiv = _record_arxiv_id(source_record)
    work_arxiv = _record_arxiv_id(work_record)
    source_zotero = _normalized_identifier(source_record, "zotero_key")
    work_zotero = _normalized_identifier(work_record, "zotero_key")

    if source_doi and work_doi and source_doi == work_doi:
        shared.append({"kind": "shared_identifier", "value": f"doi:{work_doi}"})
    elif source_doi and work_doi:
        conflicts.append(
            {
                "kind": "identifier",
                "status": "conflict",
                "source_value": f"doi:{source_doi}",
                "work_value": f"doi:{work_doi}",
            }
        )

    if source_arxiv and work_arxiv and source_arxiv == work_arxiv:
        shared.append({"kind": "shared_identifier", "value": f"arxiv_id:{work_arxiv}"})
    elif source_arxiv and work_arxiv:
        conflicts.append(
            {
                "kind": "identifier",
                "status": "conflict",
                "source_value": f"arxiv_id:{source_arxiv}",
                "work_value": f"arxiv_id:{work_arxiv}",
            }
        )

    if source_zotero and work_zotero and source_zotero == work_zotero:
        shared.append({"kind": "shared_identifier", "value": f"zotero_key:{work_zotero}"})
    elif source_zotero and work_zotero:
        conflicts.append(
            {
                "kind": "identifier",
                "status": "conflict",
                "source_value": f"zotero_key:{source_zotero}",
                "work_value": f"zotero_key:{work_zotero}",
            }
        )
    return shared, conflicts


def manifestation_equivalence_decision(
    work_record: dict[str, Any],
    source_record: dict[str, Any] | None = None,
) -> dict[str, Any]:
    source_record = source_record or work_record
    source_title = _string_field(source_record, "title")
    work_title = _string_field(work_record, "title")
    title_score = title_similarity(source_title, work_title)
    evidence: list[dict[str, Any]] = []

    shared_identifiers, identifier_conflicts = _shared_identifier_evidence(
        source_record,
        work_record,
    )
    evidence.extend(shared_identifiers)
    evidence.extend(identifier_conflicts)
    if source_title and work_title:
        title_status = "match" if title_score >= 0.82 else "weak_match"
        evidence.append(
            {
                "kind": "title_similarity",
                "status": title_status,
                "score": round(title_score, 3),
                "source_value": source_title,
                "work_value": work_title,
            }
        )

    author_evidence = _leading_author_status(source_record, work_record)
    author_conflict = False
    author_match = False
    if author_evidence:
        evidence.append(author_evidence)
        author_match = author_evidence["status"] == "match"
        author_conflict = author_evidence["status"] == "conflict"

    source_abstract = _record_abstract(source_record)
    work_abstract = _record_abstract(work_record)
    abstract_score = title_similarity(source_abstract, work_abstract)
    abstract_conflict = bool(source_abstract and work_abstract and abstract_score < 0.25)
    abstract_match = bool(source_abstract and work_abstract and abstract_score >= 0.45)
    if source_abstract and work_abstract:
        evidence.append(
            {
                "kind": "abstract_similarity",
                "status": "match" if abstract_match else "weak_match",
                "score": round(abstract_score, 3),
            }
        )

    if identifier_conflicts:
        status = "ambiguous"
        reason = "competing_identity_evidence"
    elif title_score < 0.55 and (author_conflict or abstract_conflict):
        status = "ambiguous"
        reason = "competing_identity_evidence"
    elif shared_identifiers:
        status = "equivalent"
        reason = "shared_work_identifier"
    elif title_score >= 0.82 and (author_match or abstract_match):
        status = "equivalent"
        reason = "title_author_or_abstract_supports_equivalence"
    elif title_score >= 0.9:
        status = "equivalent"
        reason = "minor_title_variation_only"
    else:
        status = "equivalent"
        reason = "uncontested_manifestation"

    return {
        "status": status,
        "reason": reason,
        "location_binding": "source_manifestation",
        "evidence": evidence,
    }


def identity_verdict_from_equivalence(equivalence_decision: dict[str, Any]) -> str:
    if _string_field(equivalence_decision, "status") == "ambiguous":
        return "ambiguous"
    return "accepted"


def identity_provenance(
    *,
    resolve_artifact_path: str = "",
    metadata_artifact_path: str = "",
) -> dict[str, str]:
    return {
        "resolve_artifact_path": _artifact_path(resolve_artifact_path),
        "metadata_artifact_path": _artifact_path(metadata_artifact_path),
    }


PROVIDER_IDENTITY_SOURCES = {
    "arxiv",
    "crossref",
    "doi",
    "openalex",
    "semantic_scholar",
    "zotero",
    "zotero_key",
}
CHALLENGEABLE_IDENTITY_SOURCES = {"local_pdf", "pdf_url", "title_query"}
STRONG_ANCHOR_SOURCE_TYPES = {
    "doi": {"doi", "doi_url"},
    "arxiv_id": {"arxiv_id", "arxiv_url"},
    "zotero_key": {"zotero_key"},
}
ACCEPTED_IDENTITY_VERDICTS = {"accepted", "accepted_with_warnings"}
IDENTITY_FAILURE_SUMMARIES = {
    "ambiguous_competing_identities": (
        "Identity repair exhausted: multiple plausible papers remain. Provide a "
        "stronger identifier such as a DOI, arXiv ID, Zotero key, or trusted source PDF."
    ),
    "insufficient_evidence": (
        "Identity repair exhausted: there is not enough trusted evidence to choose the "
        "paper. Provide a stronger identifier such as a DOI, arXiv ID, Zotero key, or "
        "trusted source PDF."
    ),
    "provider_unavailable": (
        "Identity repair exhausted: metadata providers were unavailable, so the paper "
        "could not be verified. Retry later or provide a stronger identifier."
    ),
    "metadata_contradiction": (
        "Identity repair exhausted: trusted identity metadata contradicts other "
        "acquisition evidence. Provide a stronger identifier or a trusted source PDF."
    ),
    "source_pdf_mismatch": (
        "Identity repair exhausted: the selected PDF appears to describe a different "
        "paper. Provide the correct PDF or a stronger identifier."
    ),
}


def _metadata_source_set(record: dict[str, Any]) -> set[str]:
    sources = set()
    source_type = _string_field(record, "source_type").lower()
    if source_type:
        sources.add(source_type)
    for source in _dedupe_string_list(record.get("metadata_sources", [])):
        sources.add(source.lower())
    return sources


def _boolish_field(record: dict[str, Any], key: str) -> bool:
    value = record.get(key)
    if isinstance(value, bool):
        return value
    if isinstance(value, str):
        return value.strip().lower() in {"1", "true", "yes", "on", "unavailable"}
    return bool(value)


def _has_provider_unavailable(record: dict[str, Any]) -> bool:
    if _boolish_field(record, "provider_unavailable") or _boolish_field(
        record, "metadata_provider_unavailable"
    ):
        return True
    if record.get("provider_errors") or record.get("provider_failures"):
        return True
    status = _string_field(record, "provider_status").lower()
    return status in {"unavailable", "failed", "timeout"}


def _has_metadata_contradiction(record: dict[str, Any]) -> bool:
    contradiction_keys = (
        "identity_contradictions",
        "metadata_contradictions",
        "unresolved_metadata_contradictions",
    )
    return any(bool(record.get(key)) for key in contradiction_keys)


def _has_trusted_source_manifestation(record: dict[str, Any]) -> bool:
    if _string_field(record, "local_pdf_path"):
        return True
    if _string_field(record, "pdf_url"):
        return True
    source_url = _string_field(record, "source_url")
    source_type = _string_field(record, "source_type").lower()
    return bool(source_url and source_type not in {"", "title", "title_query"})


def _has_minimum_identity_evidence(
    record: dict[str, Any],
    source_record: dict[str, Any] | None,
) -> bool:
    if _has_stronger_identity_evidence(record):
        return True
    if _has_trusted_source_manifestation(record):
        return True
    if source_record and source_record is not record:
        return _has_stronger_identity_evidence(source_record) or _has_trusted_source_manifestation(
            source_record
        )
    return False


def _normalized_anchor_value(key: str, value: Any) -> str:
    cleaned = normalize_whitespace(str(value or ""))
    if not cleaned:
        return ""
    if key == "doi":
        return (extract_doi(cleaned) or cleaned).lower()
    if key == "arxiv_id":
        return (extract_arxiv_id(cleaned) or cleaned).lower()
    if key == "zotero_key":
        return cleaned.upper()
    return cleaned.lower()


def _paper_id_for_effective_identity(record: dict[str, Any]) -> str:
    without_stale_id = dict(record)
    without_stale_id.pop("paper_id", None)
    return paper_id_for_record(without_stale_id)


def _has_stronger_identity_evidence(record: dict[str, Any]) -> bool:
    if any(_string_field(record, key) for key in ("doi", "arxiv_id", "zotero_key")):
        return True
    return bool(_metadata_source_set(record) & PROVIDER_IDENTITY_SOURCES)


def _repair_evidence_used(record: dict[str, Any]) -> list[dict[str, str]]:
    evidence = selected_identity_evidence(record)
    metadata_sources = _dedupe_string_list(record.get("metadata_sources", []))
    if metadata_sources:
        trust = (
            "provider"
            if _metadata_source_set(record) & PROVIDER_IDENTITY_SOURCES
            else "metadata"
        )
        evidence.append(
            {
                "kind": "metadata_sources",
                "value": ", ".join(metadata_sources),
                "trust": trust,
                "source": "metadata",
            }
        )
    return evidence


def _challengeable_title_replacement_reason(record: dict[str, Any]) -> str:
    title = _string_field(record, "title")
    sources = _metadata_source_set(record)
    title_source = _string_field(record, "local_pdf_title_source").lower()
    reasons = {
        reason.lower()
        for reason in _dedupe_string_list(record.get("identity_confidence_reasons", []))
    }

    if not title and sources & CHALLENGEABLE_IDENTITY_SOURCES:
        return "blank_challengeable_title_filled_by_stronger_identity_evidence"
    if title_source == "first_page_title_used" or "first_page_title_used" in reasons:
        return "challengeable_first_page_title_replaced_by_stronger_identity_evidence"
    if (
        title_source == "local_pdf_stem_used"
        or record.get("local_pdf_artifact_title")
        or "local_pdf_stem_used" in reasons
        or "local_pdf_artifact_title" in reasons
    ):
        return "challengeable_filename_title_replaced_by_stronger_identity_evidence"
    if title_source == "pdf_metadata_title_used" or "pdf_metadata_title_used" in reasons:
        return "challengeable_pdf_metadata_title_replaced_by_stronger_identity_evidence"
    if "title_query" in sources:
        return "challengeable_title_query_replaced_by_stronger_identity_evidence"
    if "pdf_url" in sources:
        return "challengeable_pdf_url_title_replaced_by_stronger_identity_evidence"
    return ""


def _title_was_replaced(original: dict[str, Any], accepted: dict[str, Any]) -> bool:
    original_title = _string_field(original, "title")
    accepted_title = _string_field(accepted, "title")
    if not accepted_title:
        return False
    if not original_title:
        return True
    return normalize_title(original_title) != normalize_title(accepted_title)


def _resolve_anchor_is_strong(resolve_record: dict[str, Any], key: str) -> bool:
    if not _string_field(resolve_record, key):
        return False
    source_type = _string_field(resolve_record, "source_type").lower()
    if source_type in STRONG_ANCHOR_SOURCE_TYPES.get(key, set()):
        return True
    sources = _metadata_source_set(resolve_record)
    if key == "doi":
        return bool(sources & {"doi", "crossref"})
    if key == "arxiv_id":
        return "arxiv" in sources
    if key == "zotero_key":
        return bool(sources & {"zotero", "zotero_key"})
    return False


def _strong_anchor_conflicts(
    metadata: dict[str, Any],
    resolve_record: dict[str, Any],
) -> list[dict[str, str]]:
    conflicts: list[dict[str, str]] = []
    for key in ("doi", "arxiv_id", "zotero_key"):
        if not _resolve_anchor_is_strong(resolve_record, key):
            continue
        resolve_value = _normalized_anchor_value(key, resolve_record.get(key))
        metadata_value = _normalized_anchor_value(key, metadata.get(key))
        if metadata_value and resolve_value and metadata_value != resolve_value:
            conflicts.append(
                {
                    "kind": key,
                    "accepted_value": _string_field(resolve_record, key),
                    "rejected_value": _string_field(metadata, key),
                }
            )
    return conflicts


def _apply_strong_anchor_protection(
    effective: dict[str, Any],
    resolve_record: dict[str, Any],
    conflicts: list[dict[str, str]],
) -> dict[str, Any]:
    rejected_candidate = work_level_identity_from_record(effective)
    protected_fields = (
        "title",
        "translated_title",
        "authors",
        "year",
        "venue",
        "doi",
        "arxiv_id",
        "zotero_key",
        "source_type",
        "source_url",
        "pdf_url",
        "local_pdf_path",
    )
    removable_fields = {
        "title",
        "translated_title",
        "authors",
        "year",
        "venue",
        "source_url",
        "pdf_url",
    }
    for key in protected_fields:
        if resolve_record.get(key) not in ("", None, [], {}):
            effective[key] = deepcopy(resolve_record[key])
        elif key in removable_fields:
            effective.pop(key, None)
    effective["paper_id"] = _paper_id_for_effective_identity(effective)
    return {
        "attempt": 1,
        "action": "protect_strong_identity_anchor",
        "status": "accepted",
        "evidence_used": _repair_evidence_used(resolve_record),
        "rejected_candidate_identity": rejected_candidate,
        "accepted_correction": work_level_identity_from_record(effective),
        "replacement_reason": "strong_identity_anchor_protected_from_unrelated_provider_evidence",
        "conflicting_anchors": conflicts,
    }


WARNING_SCOPED_METADATA_FIELDS = {
    "year": {
        "reason": "source_manifestation_year_differs_from_work_identity",
        "impact": "avoid_over_specific_year_claims",
    },
    "venue": {
        "reason": "source_manifestation_venue_differs_from_work_identity",
        "impact": "avoid_over_specific_venue_claims",
    },
}


def _metadata_values_differ(left: str, right: str) -> bool:
    if not left or not right:
        return False
    return normalize_title(left) != normalize_title(right)


def _warning_scoped_metadata_uncertainty(
    work_record: dict[str, Any],
    source_record: dict[str, Any],
) -> list[dict[str, str]]:
    warnings: list[dict[str, str]] = []
    for field, config in WARNING_SCOPED_METADATA_FIELDS.items():
        work_value = _string_field(work_record, field)
        source_value = _string_field(source_record, field)
        if not _metadata_values_differ(work_value, source_value):
            continue
        warnings.append(
            {
                "reason": config["reason"],
                "scope": "metadata",
                "impact": config["impact"],
                "field": field,
                "work_value": work_value,
                "source_value": source_value,
            }
        )
    return warnings


def _identity_observation_record(observation: dict[str, Any]) -> dict[str, Any]:
    record = observation.get("record")
    if isinstance(record, dict):
        return deepcopy(record)
    return {
        key: deepcopy(value)
        for key, value in observation.items()
        if key not in {"provider", "retrieved_by", "diagnostics", "relation"}
    }


def _identity_observation_summary(
    observation: dict[str, Any],
    *,
    status: str,
    reason: str,
) -> dict[str, Any]:
    record = _identity_observation_record(observation)
    summary = {
        "provider": _string_field(observation, "provider")
        or _string_field(record, "source_type")
        or "unknown",
        "retrieved_by": deepcopy(observation.get("retrieved_by", {}) or {}),
        "status": status,
        "reason": reason,
        "record": record,
    }
    for key in ("relation", "diagnostics"):
        value = observation.get(key)
        if value not in (None, "", [], {}):
            summary[key] = deepcopy(value)
    return summary


def _append_bound_source(
    sources: list[dict[str, str]],
    *,
    kind: str,
    value: str,
    provider: str,
    binding_reason: str,
) -> None:
    cleaned = normalize_whitespace(value)
    if not cleaned or any(item["kind"] == kind and item["value"] == cleaned for item in sources):
        return
    sources.append(
        {
            "kind": kind,
            "value": cleaned,
            "provider": provider or "unknown",
            "binding_reason": binding_reason,
        }
    )


def _bound_sources_from_record(
    record: dict[str, Any],
    *,
    provider: str,
    binding_reason: str,
) -> list[dict[str, str]]:
    sources: list[dict[str, str]] = []
    local_pdf = _string_field(record, "local_pdf_path")
    if local_pdf:
        _append_bound_source(
            sources,
            kind="local_pdf",
            value=_path_string(local_pdf),
            provider=provider,
            binding_reason=binding_reason,
        )
    pdf_url = _string_field(record, "pdf_url")
    if pdf_url:
        _append_bound_source(
            sources,
            kind="pdf_url",
            value=pdf_url,
            provider=provider,
            binding_reason=binding_reason,
        )
    source_url = _string_field(record, "source_url")
    if source_url.lower().endswith(".pdf"):
        _append_bound_source(
            sources,
            kind="pdf_url",
            value=source_url,
            provider=provider,
            binding_reason=binding_reason,
        )
    arxiv_id = _record_arxiv_id(record)
    if arxiv_id:
        _append_bound_source(
            sources,
            kind="pdf_url",
            value=f"https://arxiv.org/pdf/{arxiv_id}.pdf",
            provider=provider,
            binding_reason="accepted_identifier_derived",
        )
    doi = extract_doi(_string_field(record, "doi")) or ""
    if doi.lower().startswith("10.3389/"):
        _append_bound_source(
            sources,
            kind="pdf_url",
            value=f"https://www.frontiersin.org/articles/{doi}/pdf",
            provider=provider,
            binding_reason="accepted_identifier_derived",
        )
    return sources


def _merge_admitted_observation(
    accepted: dict[str, Any],
    observation: dict[str, Any],
    provider: str,
) -> dict[str, Any]:
    merged = deepcopy(accepted)
    for key, value in observation.items():
        if value in ("", None, [], {}):
            continue
        if key == "metadata_sources":
            sources = _dedupe_string_list(merged.get(key, []))
            for source in _dedupe_string_list(value):
                if source not in sources:
                    sources.append(source)
            merged[key] = sources
            continue
        if merged.get(key) in ("", None, [], {}):
            merged[key] = deepcopy(value)
    sources = _dedupe_string_list(merged.get("metadata_sources", []))
    if provider != "unknown" and provider not in sources:
        sources.append(provider)
    if sources:
        merged["metadata_sources"] = sources
    merged["paper_id"] = _paper_id_for_effective_identity(merged)
    return merged


def _is_unique_exact_zotero_title_observation(
    anchor: dict[str, Any],
    item: dict[str, Any],
    observation: dict[str, Any],
) -> bool:
    relation = item.get("relation")
    if not isinstance(relation, dict):
        return False
    anchor_title = normalize_identity_title(_string_field(anchor, "title"))
    observation_title = normalize_identity_title(_string_field(observation, "title"))
    return bool(
        _string_field(item, "provider").lower() == "zotero"
        and _string_field(relation, "kind") == "zotero_lookup"
        and _string_field(relation, "match_kind") == "title"
        and _string_field(relation, "match_resolution") == "unique_exact"
        and _string_field(observation, "zotero_key")
        and anchor_title
        and anchor_title == observation_title
    )


def _is_unique_exact_arxiv_title_observation(
    anchor: dict[str, Any],
    item: dict[str, Any],
    observation: dict[str, Any],
) -> bool:
    relation = item.get("relation")
    if not isinstance(relation, dict):
        return False
    return bool(
        _string_field(item, "provider").lower() == "arxiv"
        and _string_field(relation, "kind") == "arxiv_lookup"
        and _string_field(relation, "match_kind") == "title"
        and _string_field(relation, "match_resolution") == "unique_exact"
        and _record_arxiv_id(observation)
        and normalize_identity_title(_string_field(anchor, "title"))
        == normalize_identity_title(_string_field(observation, "title"))
    )


def adjudicate_identity_observations(
    anchor: dict[str, Any],
    observations: list[Any],
) -> dict[str, Any]:
    accepted_metadata = deepcopy(anchor)
    accepted_metadata.pop("identity_observations", None)
    accepted_observations: list[dict[str, Any]] = []
    rejected_observations: list[dict[str, Any]] = []
    repair_attempts: list[dict[str, Any]] = []
    anchor_provider = _string_field(anchor, "source_type") or "input"
    field_provenance = {
        field: {
            "provider": anchor_provider,
            "retrieved_by": {"kind": "input", "value": _string_field(anchor, "paper_id")},
        }
        for field in PROVENANCE_FIELDS
        if anchor.get(field) not in ("", None, [], {})
    }
    bound_sources = _bound_sources_from_record(
        anchor,
        provider=_string_field(anchor, "source_type") or "input",
        binding_reason="trusted_input",
    )

    for item in observations:
        if not isinstance(item, dict):
            continue
        observation = _identity_observation_record(item)
        adjudication_anchor = accepted_metadata
        shared_identifiers, identifier_conflicts = _shared_identifier_evidence(
            adjudication_anchor,
            observation,
        )
        if identifier_conflicts:
            summary = _identity_observation_summary(
                item,
                status="rejected",
                reason="conflicting_strong_identifier",
            )
            rejected_observations.append(summary)
            repair_attempts.append(
                {
                    "attempt": len(repair_attempts) + 1,
                    "action": "reject_conflicting_provider_observation",
                    "status": "rejected",
                    "evidence_used": _repair_evidence_used(adjudication_anchor),
                    "rejected_candidate_identity": work_level_identity_from_record(
                        observation
                    ),
                    "accepted_correction": work_level_identity_from_record(
                        adjudication_anchor
                    ),
                    "conflicting_anchors": identifier_conflicts,
                }
            )
            continue
        anchor_year = _record_publication_year(adjudication_anchor)
        observation_year = _record_publication_year(observation)
        title_author_year_match = bool(
            normalize_identity_title(_string_field(adjudication_anchor, "title"))
            and normalize_identity_title(_string_field(adjudication_anchor, "title"))
            == normalize_identity_title(_string_field(observation, "title"))
            and _full_leading_author_matches(adjudication_anchor, observation)
            and anchor_year
            and anchor_year == observation_year
        )
        unique_exact_zotero_title_match = _is_unique_exact_zotero_title_observation(
            anchor,
            item,
            observation,
        )
        unique_exact_arxiv_title_match = _is_unique_exact_arxiv_title_observation(
            anchor,
            item,
            observation,
        )
        if not title_author_year_match and not shared_identifiers:
            observation_provider = (
                _string_field(item, "provider")
                or _string_field(observation, "source_type")
            ).lower()
            for other_item in observations:
                if other_item is item or not isinstance(other_item, dict):
                    continue
                other = _identity_observation_record(other_item)
                other_provider = (
                    _string_field(other_item, "provider")
                    or _string_field(other, "source_type")
                ).lower()
                anchor_title = normalize_identity_title(
                    _string_field(anchor, "title")
                )
                observation_title = normalize_identity_title(
                    _string_field(observation, "title")
                )
                other_title = normalize_identity_title(
                    _string_field(other, "title")
                )
                _, provider_conflicts = _shared_identifier_evidence(
                    observation,
                    other,
                )
                if (
                    observation_provider
                    and other_provider
                    and observation_provider != other_provider
                    and not provider_conflicts
                    and anchor_title
                    and anchor_title == observation_title == other_title
                    and _full_leading_author_matches(observation, other)
                    and observation_year
                    and observation_year == _record_publication_year(other)
                ):
                    title_author_year_match = True
                    break
        if (
            not shared_identifiers
            and not title_author_year_match
            and not unique_exact_zotero_title_match
            and not unique_exact_arxiv_title_match
        ):
            rejected_observations.append(
                _identity_observation_summary(
                    item,
                    status="rejected",
                    reason="insufficient_equivalence_evidence",
                )
            )
            continue

        if shared_identifiers:
            acceptance_reason = "shared_identifier"
        elif unique_exact_zotero_title_match:
            acceptance_reason = "unique_exact_zotero_title"
        elif unique_exact_arxiv_title_match:
            acceptance_reason = "unique_exact_arxiv_title"
        else:
            acceptance_reason = "title_author_year"
        summary = _identity_observation_summary(
            item,
            status="accepted",
            reason=acceptance_reason,
        )
        accepted_observations.append(summary)
        provider = summary["provider"]
        observation.setdefault("metadata_sources", [])
        if provider != "unknown" and provider not in observation["metadata_sources"]:
            observation["metadata_sources"].append(provider)
        previous_metadata = deepcopy(accepted_metadata)
        accepted_metadata = _merge_admitted_observation(
            accepted_metadata,
            observation,
            provider,
        )
        retrieved_by = deepcopy(item.get("retrieved_by", {}) or {})
        for field in PROVENANCE_FIELDS:
            value = observation.get(field)
            if value in ("", None, [], {}):
                continue
            if accepted_metadata.get(field) != value:
                continue
            if previous_metadata.get(field) not in ("", None, [], {}):
                continue
            field_provenance[field] = {
                "provider": provider,
                "retrieved_by": retrieved_by,
            }
        for source in _bound_sources_from_record(
            observation,
            provider=provider,
            binding_reason=acceptance_reason,
        ):
            _append_bound_source(sources=bound_sources, **source)
        promoted_identifiers = {
            key: _string_field(observation, key)
            for key in ("doi", "arxiv_id", "zotero_key")
            if not _string_field(previous_metadata, key)
            and _string_field(observation, key)
        }
        if promoted_identifiers:
            repair_attempts.append(
                {
                    "attempt": len(repair_attempts) + 1,
                    "action": "accept_identity_promotion",
                    "status": "accepted",
                    "evidence_used": deepcopy(shared_identifiers)
                    or [
                        {
                            "kind": "title_author_year",
                            "value": _string_field(observation, "title"),
                            "source": provider,
                        }
                    ],
                    "rejected_candidate_identity": work_level_identity_from_record(
                        previous_metadata
                    ),
                    "accepted_correction": work_level_identity_from_record(
                        accepted_metadata
                    ),
                    "promoted_identifiers": promoted_identifiers,
                    "previous_paper_id": _string_field(previous_metadata, "paper_id"),
                    "accepted_paper_id": _string_field(accepted_metadata, "paper_id"),
                }
            )

    return {
        "accepted_metadata": apply_identity_confidence(accepted_metadata),
        "accepted_observations": accepted_observations,
        "rejected_observations": rejected_observations,
        "bound_sources": bound_sources,
        "repair_attempts": repair_attempts,
        "field_provenance": field_provenance,
    }


def identity_repair_decision(
    metadata: dict[str, Any],
    *,
    resolve_record: dict[str, Any] | None = None,
) -> dict[str, Any]:
    observations = metadata.get("identity_observations", []) or []
    observation_decision = adjudicate_identity_observations(
        resolve_record or metadata,
        observations if isinstance(observations, list) else [],
    )
    effective = observation_decision["accepted_metadata"]
    for diagnostic_key in ("provider_unavailable", "provider_status"):
        if metadata.get(diagnostic_key) not in (None, "", False):
            effective[diagnostic_key] = deepcopy(metadata[diagnostic_key])
    repair_attempts = list(observation_decision["repair_attempts"])
    identity_verdict = "accepted"
    failure_class = ""
    unresolved_risks: list[str] = []

    if resolve_record:
        conflicts = _strong_anchor_conflicts(effective, resolve_record)
        if conflicts:
            repair_attempts.append(
                _apply_strong_anchor_protection(effective, resolve_record, conflicts)
            )
        else:
            replacement_reason = _challengeable_title_replacement_reason(resolve_record)
            if (
                replacement_reason
                and _title_was_replaced(resolve_record, effective)
                and _has_stronger_identity_evidence(effective)
            ):
                repair_attempts.append(
                    {
                        "attempt": 1,
                        "action": "replace_challengeable_identity_anchor",
                        "status": "accepted",
                        "evidence_used": _repair_evidence_used(effective),
                        "rejected_candidate_identity": work_level_identity_from_record(
                            resolve_record
                        ),
                        "accepted_correction": work_level_identity_from_record(effective),
                        "replacement_reason": replacement_reason,
                    }
                )
                effective["paper_id"] = _paper_id_for_effective_identity(effective)

    if _has_provider_unavailable(effective) and not _has_minimum_identity_evidence(
        effective,
        resolve_record,
    ):
        identity_verdict = "failed"
        failure_class = "provider_unavailable"
    elif _has_metadata_contradiction(effective):
        identity_verdict = "failed"
        failure_class = "metadata_contradiction"
    elif not _has_minimum_identity_evidence(effective, resolve_record):
        identity_verdict = "failed"
        failure_class = "insufficient_evidence"

    if failure_class:
        unresolved_risks.append(failure_class)

    bound_sources = list(observation_decision["bound_sources"])
    return {
        "record": effective,
        "identity_verdict": identity_verdict,
        "identity_failure_class": failure_class,
        "repair_attempts": repair_attempts,
        "warnings": [],
        "unresolved_risks": unresolved_risks,
        "accepted_metadata": deepcopy(effective),
        "accepted_observations": observation_decision["accepted_observations"],
        "rejected_observations": observation_decision["rejected_observations"],
        "bound_sources": bound_sources,
        "field_provenance": observation_decision["field_provenance"],
    }


def _equivalence_has_identifier_conflict(equivalence_decision: dict[str, Any]) -> bool:
    return any(
        item.get("kind") == "identifier" and item.get("status") == "conflict"
        for item in equivalence_decision.get("evidence", []) or []
        if isinstance(item, dict)
    )


def _is_source_pdf_record(record: dict[str, Any] | None) -> bool:
    if not record:
        return False
    return bool(
        _string_field(record, "local_pdf_path")
        or _string_field(record, "source_type").lower() == "local_pdf"
    )


def identity_failure_class_for_state(
    decision: dict[str, Any],
    equivalence_decision: dict[str, Any],
    source_record: dict[str, Any] | None,
) -> str:
    failure_class = _string_field(decision, "identity_failure_class")
    if _string_field(equivalence_decision, "status") == "ambiguous":
        if _is_source_pdf_record(source_record):
            return "source_pdf_mismatch"
        if _equivalence_has_identifier_conflict(equivalence_decision):
            return "metadata_contradiction"
        if failure_class not in {"provider_unavailable", "metadata_contradiction"}:
            return "ambiguous_competing_identities"
    return failure_class


def identity_failure_summary(failure_class: str) -> str:
    return IDENTITY_FAILURE_SUMMARIES.get(
        failure_class,
        "Identity repair exhausted: acquisition could not safely choose the paper identity.",
    )


def _repair_attempts_with_failure(
    attempts: list[dict[str, Any]],
    *,
    failure_class: str,
    evidence_used: list[dict[str, str]],
    equivalence_decision: dict[str, Any],
) -> list[dict[str, Any]]:
    if not failure_class:
        return attempts
    repaired = deepcopy(attempts)
    repaired.append(
        {
            "attempt": len(repaired) + 1,
            "action": "repair_exhausted_fail_closed",
            "status": "failed",
            "failure_class": failure_class,
            "evidence_used": evidence_used,
            "unresolved_reason": _string_field(equivalence_decision, "reason") or failure_class,
        }
    )
    return repaired


def identity_contract_state(
    metadata: dict[str, Any],
    *,
    source_record: dict[str, Any] | None = None,
) -> dict[str, Any]:
    decision = identity_repair_decision(metadata, resolve_record=source_record)
    effective = decision["record"]
    source_record = source_record or effective
    paper_id = _paper_id_for_effective_identity(effective)
    effective["paper_id"] = paper_id
    accepted_metadata = deepcopy(decision["accepted_metadata"])
    accepted_metadata["paper_id"] = paper_id
    equivalence_decision = manifestation_equivalence_decision(effective, source_record)
    identity_verdict = identity_verdict_from_equivalence(equivalence_decision)
    warnings = []
    if identity_verdict == "accepted":
        identity_verdict = decision["identity_verdict"]
        if identity_verdict == "accepted":
            warnings = deepcopy(decision.get("warnings", []) or [])
            warnings.extend(_warnin

#!/usr/bin/env python3
"""Scaffolded JSON contracts for the paper-deep-notes core workflow."""

from __future__ import annotations

from copy import deepcopy
from typing import Any, TypedDict

from localization import note_schema, normalize_output_language, required_sections

NOTE_REQUIRED_SECTIONS: tuple[str, ...] = (
    "核心信息",
    "原文摘要翻译",
    "创新点",
    "一句话总结",
    "研究问题",
    "数据与任务定义",
    "方法主线",
    "关键结果",
    "深度分析",
    "局限",
    "我的笔记",
    "引用",
)

def note_required_sections(language: str | None = None) -> tuple[str, ...]:
    return required_sections(language)

PAPER_TYPE_VALUES: tuple[str, ...] = (
    "AI_method",
    "benchmark_or_dataset",
    "clinical_or_psychology_empirical",
    "humanities_or_social_science",
    "survey_or_review",
)

NOTE_PLAN_STRING_FIELDS: tuple[str, ...] = (
    "paper_type",
    "paper_type_rationale",
    "dominant_domain",
)

NOTE_PLAN_LIST_FIELDS: tuple[str, ...] = (
    "must_cover",
    "key_numbers",
    "real_comparisons",
    "central_claims",
    "claim_boundaries",
    "negative_or_limiting_results",
    "mechanism_result_map",
    "comparative_positioning",
    "reuse_takeaways",
    "followup_questions",
    "section_plan",
)

NOTE_PLAN_REQUIRED_FIELDS: tuple[str, ...] = NOTE_PLAN_STRING_FIELDS + NOTE_PLAN_LIST_FIELDS
NOTE_PLAN_FIELD_TYPES: dict[str, str] = {
    **dict.fromkeys(NOTE_PLAN_STRING_FIELDS, "string"),
    **dict.fromkeys(NOTE_PLAN_LIST_FIELDS, "array"),
}
REQUIRED_FIELD_CHECKS: dict[str, dict[str, bool]] = {
    "string": {"non_empty": True},
    "array": {"non_empty": True},
}
CENTRAL_CLAIM_FIELD_TYPES: dict[str, str] = {
    "claim": "string",
    "supporting_evidence": "array",
    "what_it_actually_proves": "string",
    "what_it_does_not_prove": "string",
}


def required_field_value_error(
    value: Any,
    field_type: str,
    checks: dict[str, dict[str, bool]],
) -> str:
    if field_type == "string":
        if not isinstance(value, str):
            return "invalid"
        return "empty" if checks["string"]["non_empty"] and not value.strip() else ""
    if field_type == "array":
        if not isinstance(value, list):
            return "invalid"
        return "empty" if checks["array"]["non_empty"] and not value else ""
    return "invalid"

PAPER_TYPE_SECTION_PROFILES: dict[str, dict[str, dict[str, Any]]] = {
    "AI_method": {
        "section_semantics": {
            "研究问题": "方法要解决的具体技术问题和现有方法短板。",
            "数据与任务定义": "数据集、输入输出、评测任务和实验设置。",
            "方法主线": "模型、算法、训练或推理机制。",
            "关键结果": "主结果、强基线、消融和关键数字。",
            "深度分析": "方法为什么有效、何处脆弱、复现和扩展代价。",
        },
        "recommended_subsections": {
            "方法主线": ["机制流程", "模型结构", "训练目标", "推理与采样链路", "关键实现细节"],
            "关键结果": ["主结果与强基线", "消融到底说明了什么", "失败或不稳定设置"],
            "深度分析": ["为什么有效", "复杂度与扩展性", "复现注意点"],
        },
    },
    "benchmark_or_dataset": {
        "section_semantics": {
            "研究问题": "这个 benchmark/dataset 想补足的评测或数据缺口。",
            "数据与任务定义": "数据来源、任务拆分、标签/题目定义、样本范围。",
            "方法主线": "数据构建、筛选、标注和评测协议，不写成模型 pipeline。",
            "关键结果": "基线表现、难度分布、覆盖范围和偏差。",
            "深度分析": "它真正测到了什么，以及不能代表什么。",
        },
        "recommended_subsections": {
            "数据与任务定义": ["数据来源", "任务拆分", "标注/筛选协议"],
            "方法主线": ["构建流程", "评测协议", "Baseline 设置"],
            "关键结果": ["基线表现", "难度分布", "覆盖与偏差"],
            "深度分析": ["benchmark 真正测到了什么", "适用边界"],
        },
    },
    "clinical_or_psychology_empirical": {
        "section_semantics": {
            "研究问题": "临床、心理学或行为科学中的研究问题、假设或变量关系。",
            "数据与任务定义": "样本来源、纳排标准、变量/量表、测量方式。",
            "方法主线": "研究设计、分组、测量流程和统计分析路径。",
            "关键结果": "主要效应、相关性、组间差异、不确定性或显著性。",
            "深度分析": "结果解释、因果边界、临床/心理学意义和外推限制。",
        },
        "recommended_subsections": {
            "数据与任务定义": ["样本与纳排标准", "变量与量表", "测量流程"],
            "方法主线": ["研究设计", "分析模型", "主要比较"],
            "关键结果": ["主要效应", "不确定性与显著性", "临床或心理学解释"],
            "深度分析": ["因果解释边界", "外推限制"],
        },
    },
    "humanities_or_social_science": {
        "section_semantics": {
            "研究问题": "作者要解释的社会、文化、历史、制度或理论问题。",
            "数据与任务定义": "材料、案例、文本、访谈、档案或语料范围，不写成 ML task。",
            "方法主线": "理论框架、概念区分和论证路径。",
            "关键结果": "核心解释性发现、概念贡献或对既有观点的修正。",
            "深度分析": "论证强度、材料边界、解释替代性和可迁移性。",
        },
        "recommended_subsections": {
            "数据与任务定义": ["材料范围", "选择标准", "案例或语料边界"],
            "方法主线": ["理论框架", "概念区分", "论证路径"],
            "关键结果": ["核心解释性发现", "概念贡献"],
            "深度分析": ["论证强度", "替代解释", "材料边界"],
        },
    },
    "survey_or_review": {
        "section_semantics": {
            "研究问题": "综述试图整理的领域问题、争议或知识缺口。",
            "数据与任务定义": "纳入文献范围、检索/筛选标准和综述对象。",
            "方法主线": "分类体系、综述组织方式和证据综合逻辑，不写成单篇方法架构。",
            "关键结果": "领域共识、分歧、趋势、代表性方向和开放问题。",
            "深度分析": "综述覆盖的盲区、分类体系的解释力和未来研究机会。",
        },
        "recommended_subsections": {
            "数据与任务定义": ["综述范围", "纳入/排除标准", "文献覆盖"],
            "方法主线": ["分类体系", "方法谱系", "证据组织方式"],
            "关键结果": ["代表性方向", "共识与分歧", "开放问题"],
            "深度分析": ["分类体系的局限", "未覆盖区域", "后续研究机会"],
        },
    },
}

PAPER_TYPE_CONTRACTS: dict[str, dict[str, Any]] = {
    "AI_method": {
        "paper_type": "AI_method",
        "reader_lens": "面向能复现方法机制的技术读者",
        "section_focus": [
            "问题设置",
            "方法机制",
            "训练/推理流程",
            "关键公式",
            "比较基线",
            "消融与失败边界",
        ],
        "required_checks": ["需要说明机制流程、关键公式、实验设计、消融含义和失败边界。"],
        "formula_rules": ["仅保留理解方法必需的 1 到 3 个关键公式，并解释其工程含义。"],
        "avoid_rules": ["不要把非 AI_method 论文强行改写成模型架构。"],
        "boundary_questions": [
            "核心机制的收益由哪个实验或消融支撑，而不是只由主结果暗示？",
            "哪些比较只能证明在当前数据、基线、算力或协议下有效，不能外推到通用场景？",
            "论文是否给出失败、退化、不稳定或成本上升的证据；如果没有，结论边界是什么？",
        ],
        **PAPER_TYPE_SECTION_PROFILES["AI_method"],
        "mechanism_flow_contract": {
            "apply_when_paper_type_in": ["AI_method"],
            "required_step_count": "3_to_4",
            "required_step_fields": ["input", "operation", "output_destination"],
        },
    },
    "benchmark_or_dataset": {
        "paper_type": "benchmark_or_dataset",
        "reader_lens": "面向要判断 benchmark/dataset 可用性和偏差边界的研究者",
        "section_focus": [
            "任务拆分",
            "数据来源与构建流程",
            "标注协议",
            "评测指标",
            "覆盖范围与偏差",
            "样本统计与数据开放限制",
        ],
        "required_checks": [
            "需要说明数据来源、构建/标注流程、评测指标、基线表现、样本统计、数据开放或隐私限制和适用边界。"
        ],
        "formula_rules": ["仅保留核心评测指标、采样规则或划分定义。"],
        "avoid_rules": ["不要把数据构建流程写成模型 pipeline。"],
        "boundary_questions": [
            "这个 benchmark/dataset 实际测量的构念是什么，哪些能力只是间接近似？",
            "任务、标签、采样、过滤或评测协议会引入哪些覆盖缺口或偏差？",
            "基线结果证明了评测集有区分度，还是只证明某类模型适应该协议？",
            "样本时长、语料长度、人口统计、类别分布、数据可访问性或隐私限制如何影响复现和外推？",
        ],
        **PAPER_TYPE_SECTION_PROFILES["benchmark_or_dataset"],
    },
    "clinical_or_psychology_empirical": {
        "paper_type": "clinical_or_psychology_empirical",
        "reader_lens": "面向关注临床/心理学样本、变量关系和外推边界的研究读者",
        "section_focus": [
            "样本来源",
            "纳排标准",
            "变量或量表",
            "分析管线",
            "效应量与不确定性",
            "样本统计、伦理和数据可访问性",
        ],
        "required_checks": [
            "需要区分相关、预测、组间差异和因果解释，说明样本统计、伦理/隐私约束与外推边界。"
        ],
        "formula_rules": ["仅保留核心统计模型、效应量、置信区间或量表定义。"],
        "avoid_rules": ["不要把相关性、预测性能或组间差异写成未经证明的因果结论。"],
        "boundary_questions": [
            "样本来源、纳排标准、测量工具和标注流程如何限制外推？",
            "结果支持相关、预测、组间差异还是因果解释；不要越过论文设计能证明的范围。",
            "临床或心理学意义是否依赖未观测混杂、量表阈值、文本/语音缺失或场景约束？",
            "样本构成、数据缺失、隐私限制或材料不可公开会怎样限制复现与再分析？",
        ],
        **PAPER_TYPE_SECTION_PROFILES["clinical_or_psychology_empirical"],
    },
    "humanities_or_social_science": {
        "paper_type": "humanities_or_social_science",
        "reader_lens": "面向关注理论框架、材料解释和论证结构的研究读者",
        "section_focus": ["研究对象", "材料来源", "理论框架", "论证路径", "概念贡献", "解释边界"],
        "required_checks": ["需要区分作者论证、材料证据、规范性判断和实验事实。"],
        "formula_rules": ["通常不强行保留公式；仅保留核心形式化定义或编码规则。"],
        "avoid_rules": ["不要把规范性判断、文本解释或案例分析写成实验事实。"],
        "boundary_questions": [
            "作者的解释依赖哪些材料、案例或理论前提？",
            "是否存在同样能解释材料的替代解释，论文如何排除或没有排除？",
            "哪些结论是概念贡献或规范性判断，而不是可直接当作经验事实？",
        ],
        **PAPER_TYPE_SECTION_PROFILES["humanities_or_social_science"],
    },
    "survey_or_review": {
        "paper_type": "survey_or_review",
        "reader_lens": "面向需要梳理综述脉络、分类体系和证据边界的研究读者",
        "section_focus": [
            "综述范围",
            "纳入排除标准",
            "主题分类",
            "方法谱系",
            "共识与分歧",
            "开放问题",
        ],
        "required_checks": ["需要说明综述范围、文献选择、分类体系、共识分歧和开放问题。"],
        "formula_rules": ["仅保留分类轴、纳入排除准则、证据汇总规则或 meta-analysis 统计量。"],
        "avoid_rules": ["不要把综述中的代表性结论写成作者自己完成的单项实验结果。"],
        "boundary_questions": [
            "检索范围、纳入排除标准或分类轴会遗漏哪些研究路线？",
            "综述给出的是领域共识、作者分类，还是尚未解决的分歧？",
            "哪些趋势结论来自覆盖范围内的文献分布，不能直接当作技术成熟度判断？",
        ],
        **PAPER_TYPE_SECTION_PROFILES["survey_or_review"],
    },
}

PAPER_TYPE_CONTRACTS_EN: dict[str, dict[str, Any]] = {
    "AI_method": {
        "paper_type": "AI_method",
        "reader_lens": "A technical reader who may need to reproduce the method and its mechanism.",
        "section_focus": ["problem setting", "method mechanism", "training or inference flow", "key equations", "strong baselines", "ablations and failure boundaries"],
        "required_checks": ["Explain the mechanism flow, essential equations, experimental design, what the ablations establish, and the failure boundary."],
        "formula_rules": ["Keep only the one to three equations needed to understand the method and explain their engineering meaning."],
        "avoid_rules": ["Do not force a non-method paper into a model-architecture narrative."],
        "boundary_questions": [
            "Which experiment or ablation supports the claimed benefit of the core mechanism?",
            "Which comparisons apply only under the reported data, baselines, compute, or protocol?",
            "What evidence shows failure, degradation, instability, or rising cost; if none is reported, what remains unproven?",
        ],
        "section_semantics": {
            "Research Question": "The specific technical problem and the shortcomings of existing methods.",
            "Data and Task Definition": "Datasets, inputs and outputs, evaluation tasks, and experimental settings.",
            "Method": "Model, algorithm, training, and inference mechanisms.",
            "Key Results": "Main results, strong baselines, ablations, and decisive numbers.",
            "Deep Analysis": "Why the method works, where it is fragile, and the cost of reproduction or extension.",
        },
        "recommended_subsections": {
            "Method": ["Mechanism Flow", "Model Architecture", "Training Objective", "Inference and Sampling", "Implementation Details"],
            "Key Results": ["Main Results and Strong Baselines", "What the Ablations Establish", "Failure or Unstable Settings"],
            "Deep Analysis": ["Why It Works", "Complexity and Scalability", "Reproduction Notes"],
        },
        "mechanism_flow_contract": {"apply_when_paper_type_in": ["AI_method"], "required_step_count": "3_to_4", "required_step_fields": ["input", "operation", "output_destination"]},
    },
    "benchmark_or_dataset": {
        "paper_type": "benchmark_or_dataset",
        "reader_lens": "A researcher assessing whether a benchmark or dataset is useful and where it is biased.",
        "section_focus": ["task decomposition", "data sources and construction", "annotation protocol", "evaluation metrics", "coverage and bias", "sample statistics and access limits"],
        "required_checks": ["Explain sources, construction or annotation, metrics, baselines, sample statistics, access or privacy constraints, and applicability."],
        "formula_rules": ["Keep only essential metrics, sampling rules, or split definitions."],
        "avoid_rules": ["Do not describe data construction as a model pipeline."],
        "boundary_questions": [
            "What construct does the resource actually measure, and which capabilities are only proxies?",
            "Which coverage gaps or biases follow from its tasks, labels, sampling, filtering, or evaluation protocol?",
            "Do baseline results demonstrate discrimination, or only adaptation to this protocol?",
            "How do sample composition, access, and privacy limits affect reproduction and generalization?",
        ],
        "section_semantics": {
            "Research Question": "The evaluation or data gap the resource is designed to address.",
            "Data and Task Definition": "Sources, task splits, labels, and sample scope.",
            "Method": "Construction, filtering, annotation, and evaluation protocol—not a model pipeline.",
            "Key Results": "Baseline performance, difficulty, coverage, and bias.",
            "Deep Analysis": "What the resource measures and what it cannot represent.",
        },
        "recommended_subsections": {
            "Data and Task Definition": ["Data Sources", "Task Splits", "Annotation and Filtering"],
            "Method": ["Construction Process", "Evaluation Protocol", "Baseline Setup"],
            "Key Results": ["Baseline Performance", "Difficulty Distribution", "Coverage and Bias"],
            "Deep Analysis": ["What It Actually Measures", "Applicability Boundary"],
        },
    },
    "clinical_or_psychology_empirical": {
        "paper_type": "clinical_or_psychology_empirical",
        "reader_lens": "A research reader focused on samples, variable relationships, uncertainty, and generalization.",
        "section_focus": ["sample source", "inclusion and exclusion", "variables and instruments", "analysis pipeline", "effect sizes and uncertainty", "ethics, access, and generalization"],
        "required_checks": ["Distinguish association, prediction, group difference, and causal interpretation; report sample, ethics, privacy, and generalization limits."],
        "formula_rules": ["Keep only essential statistical models, effect sizes, intervals, or instrument definitions."],
        "avoid_rules": ["Do not turn association, prediction, or group differences into unsupported causal claims."],
        "boundary_questions": [
            "How do recruitment, eligibility, measurement, and annotation constrain generalization?",
            "Does the design support association, prediction, group difference, or causality?",
            "Does the interpretation depend on unobserved confounding, thresholds, missingness, or setting?",
            "How do sample composition, missing data, privacy, and unavailable materials constrain reproduction?",
        ],
        "section_semantics": {
            "Research Question": "The clinical, psychological, or behavioral question, hypothesis, or variable relationship.",
            "Data and Task Definition": "Recruitment, eligibility, variables, instruments, and measurement.",
            "Method": "Study design, grouping, measurement flow, and statistical analysis.",
            "Key Results": "Effects, associations, group differences, uncertainty, and significance.",
            "Deep Analysis": "Interpretation, causal boundary, substantive meaning, and generalization limits.",
        },
        "recommended_subsections": {
            "Data and Task Definition": ["Sample and Eligibility", "Variables and Instruments", "Measurement Process"],
            "Method": ["Study Design", "Analysis Model", "Primary Comparisons"],
            "Key Results": ["Primary Effects", "Uncertainty and Significance", "Clinical or Psychological Interpretation"],
            "Deep Analysis": ["Causal Interpretation Boundary", "Generalization Limits"],
        },
    },
    "humanities_or_social_science": {
        "paper_type": "humanities_or_social_science",
        "reader_lens": "A reader evaluating theoretical framing, material interpretation, and argument structure.",
        "section_focus": ["object of study", "materials", "theoretical framework", "argument path", "conceptual contribution", "interpretive boundary"],
        "required_checks": ["Distinguish the author's argument, material evidence, normative judgment, and empirical fact."],
        "formula_rules": ["Do not force equations; retain only essential formal definitions or coding rules."],
        "avoid_rules": ["Do not present normative judgment, textual interpretation, or case analysis as experimental fact."],
        "boundary_questions": [
            "Which materials, cases, or theoretical premises support the interpretation?",
            "What alternative explanations fit the same material, and how are they addressed?",
            "Which conclusions are conceptual or normative rather than directly empirical?",
        ],
        "section_semantics": {
            "Research Question": "The social, cultural, historical, institutional, or theoretical problem.",
            "Data and Task Definition": "Materials, cases, texts, interviews, archives, or corpus scope—not an ML task.",
            "Method": "Theoretical framework, conceptual distinctions, and argument path.",
            "Key Results": "Interpretive findings, conceptual contribution, or revision of prior views.",
            "Deep Analysis": "Argument strength, material limits, alternative explanations, and transferability.",
        },
        "recommended_subsections": {
            "Data and Task Definition": ["Material Scope", "Selection Criteria", "Case or Corpus Boundary"],
            "Method": ["Theoretical Framework", "Conceptual Distinctions", "Argument Path"],
            "Key Results": ["Core Interpretive Findings", "Conceptual Contribution"],
            "Deep Analysis": ["Argument Strength", "Alternative Explanations", "Material Boundary"],
        },
    },
    "survey_or_review": {
        "paper_type": "survey_or_review",
        "reader_lens": "A reader mapping a literature, taxonomy, evidence boundary, and open questions.",
        "section_focus": ["review scope", "inclusion and exclusion", "taxonomy", "method families", "consensus and disagreement", "open questions"],
        "required_checks": ["Explain scope, study selection, taxonomy, consensus, disagreement, and open questions."],
        "formula_rules": ["Keep only classification axes, eligibility rules, evidence-synthesis rules, or meta-analytic statistics."],
        "avoid_rules": ["Do not present findings summarized from the literature as a new experiment by the review authors."],
        "boundary_questions": [
            "Which research routes may be missed by the search scope, eligibility criteria, or taxonomy?",
            "Which statements reflect consensus, author-defined categories, or unresolved disagreement?",
            "Which trends are artifacts of the covered literature and cannot establish technical maturity?",
        ],
        "section_semantics": {
            "Research Question": "The field problem, controversy, or knowledge gap organized by the review.",
            "Data and Task Definition": "Literature scope, search and screening criteria, and review objects.",
            "Method": "Taxonomy, review organization, and evidence-synthesis logic—not a single method architecture.",
            "Key Results": "Consensus, disagreement, trends, representative directions, and open questions.",
            "Deep Analysis": "Coverage blind spots, explanatory power of the taxonomy, and future opportunities.",
        },
        "recommended_subsections": {
            "Data and Task Definition": ["Review Scope", "Inclusion and Exclusion", "Literature Coverage"],
            "Method": ["Taxonomy", "Method Families", "Evidence Organization"],
            "Key Results": ["Representative Directions", "Consensus and Disagreement", "Open Questions"],
            "Deep Analysis": ["Taxonomy Limits", "Uncovered Areas", "Future Research Opportunities"],
        },
    },
}


def paper_type_contracts(language: str | None = None) -> dict[str, dict[str, Any]]:
    return deepcopy(PAPER_TYPE_CONTRACTS_EN if normalize_output_language(language) == "en" else PAPER_TYPE_CONTRACTS)

WRITING_CONTRACT_RULES: dict[str, Any] = {
    "required_sections": NOTE_REQUIRED_SECTIONS,
    "paper_type_values": PAPER_TYPE_VALUES,
    "note_plan_required_fields": NOTE_PLAN_REQUIRED_FIELDS,
    "note_plan_field_types": NOTE_PLAN_FIELD_TYPES,
    "note_plan_required_field_checks": REQUIRED_FIELD_CHECKS,
    "grounding_required_sections": (
        "研究问题",
        "数据与任务定义",
        "方法主线",
        "关键结果",
        "深度分析",
        "局限",
    ),
    "allowed_grounding_reference_forms": ("section_id", "pages"),
    "excluded_model_input_fields": (
        "evidence",
        "evidence_pack",
        "candidate_chunks",
        "section_texts",
        "summary",
        "summary_hints",
    ),
    "old_bundle_reference_prefixes": (
        "synthesis_bundle.evidence",
        "bundle.evidence",
        "synthesis_bundle.candidate_chunks",
        "synthesis_bundle.section_texts",
        "synthesis_bundle.summary",
        "bundle.candidate_chunks",
        "bundle.section_texts",
        "bundle.summary",
    ),
    "old_evidence_reference_tokens": (
        "evidence_pack",
        "summary.paper_type",
        "problem_evidence",
        "task_evidence",
        "data_evidence",
        "method_evidence",
        "mechanism_evidence",
        "results_evidence",
        "ablation_evidence",
        "limitations_evidence",
        "candidate_chunks",
        "section_texts",
    ),
    "figure_decision_values": (
        "review_pending",
        "insert",
        "placeholder",
        "low_priority",
        "visual_defect",
        "skip",
    ),
    "usable_insert_candidate": {
        "kinds": ("figure", "table"),
        "visual_quality_status": "usable_candidate",
        "requires_source_image_path": True,
    },
    "allowed_usable_placeholder_reasons": (
        "visual_defect",
        "materialization_blocked",
    ),
    "manual_visual_review_required_statuses": (
        "usable_candidate",
        "needs_visual_quality_check",
        "review",
    ),
    "automatic_fail_closed_visual_statuses": (
        "reject_visual_quality",
        "asset_candidate_missing",
    ),
    "visual_review_contract": {
        "selected_render_dpi": 300,
        "page_preview_dpi": 96,
        "review_fields": (
            "status",
            "reviewed_asset_sha256",
            "preserved_scientific_elements",
            "omitted_scientific_elements",
            "notes",
            "failure_reason",
            "repair_attempts",
            "revised_bbox",
        ),
        "review_status_values": ("pending", "pass", "fail", "repair_requested"),
        "repair_limit": 1,
        "asset_sha256_bound": True,
        "caption_free_visual_body_required": True,
        "decision_freeze_before": "note_plan",
        "review_evidence_fields": (
            "candidate_path",
            "page_preview_path",
            "source_pdf_path",
            "source_page",
            "caption",
            "bbox_pt",
            "normalized_bbox",
            "render_dpi",
        ),
        "repairable_failure_reasons": (
            "caption_contamination",
            "surrounding_prose_contamination",
            "scientific_content_clipped",
            "insufficient_safety_margin",
        ),
        "terminal_failure_reasons": (
            "identity_mismatch",
            "caption_inseparable",
            "ambiguous_visual_body",
            "unreadable_source",
            "scientific_content_missing",
            "repair_limit_exhausted",
        ),
    },
    "note_plan_depth_requirements": {
        "required_section_focus_min_chars": 20,
        "required_section_focus_fields": ("focus", "reading_goal", "purpose"),
        "generic_focus_phrases": (
            "use the raw source to explain",
            "paper-specific role of",
            "explain the paper-specific role",
            "explain this section",
            "summarize this section",
        ),
    },
    "analysis_coverage_contract": {
        "central_claim_fields": tuple(CENTRAL_CLAIM_FIELD_TYPES),
        "central_claim_field_types": CENTRAL_CLAIM_FIELD_TYPES,
        "central_claim_required_field_checks": REQUIRED_FIELD_CHECKS,
        "required_plan_fields": (
            "central_claims",
            "claim_boundaries",
            "negative_or_limiting_results",
            "mechanism_result_map",
            "comparative_positioning",
            "reuse_takeaways",
            "followup_questions",
        ),
        "final_quality_review_checks": (
            "central_claims_are_supported_by_raw_sections_or_pages",
            "key_experimental_settings_and_numbers_are_present",
            "mechanisms_or_protocol_choices_are_mapped_to_results",
            "comparisons_explain_positioning_against_alternatives",
            "discussion_or_limitation_claims_are_explained_mechanistically",
            "proven_claims_are_separated_from_unproven_or_unvalidated_claims",
            "research_or_engineering_takeaways_are_specific_and_reusable",
            "followup_questions_are_specific_to_replication_or_extension",
        ),
    },
}

def writing_contract_rules(language: str | None = None) -> dict[str, Any]:
    resolved = normalize_output_language(language)
    schema = note_schema(resolved)
    rules = deepcopy(WRITING_CONTRACT_RULES)
    rules["language"] = resolved
    rules["required_sections"] = tuple(schema["sections"].values())
    rules["grounding_required_sections"] = tuple(schema["sections"][key] for key in ("research_questions", "data_and_task", "method", "key_results", "deep_analysis", "limitations"))
    rules["core_info_fields"] = tuple(schema["core_info_fields"])
    rules["figure_labels"] = dict(schema["figure_labels"])
    rules["mechanism_flow_heading"] = schema["mechanism_flow"]
    if schema.get("abstract_contract"):
        rules["abstract_contract"] = deepcopy(schema["abstract_contract"])
    return rules


class MetadataRecord(TypedDict, total=False):
    title: str
    translated_title: str
    paper_id: str
    source_type: str
    source_url: str
    year: str
    authors: list[str]
    affiliations: list[str]
    venue: str
    doi: str
    abstract: str
    code_url: str
    project_url: str
    zotero_key: str
    arxiv_id: str
    metadata_sources: list[str]
    identity_confidence: str
    identity_confidence_reasons: list[str]


class EvidenceItem(TypedDict, total=False):
    claim: str
    evidence: str
    source_section: str
    page_hint: str


class CandidateChunk(TypedDict, total=False):
    text: str
    source_section: str
    actual_source_section: str
    is_abstract_fallback: bool
    page_hint: str
    kind_hint: str


class EquationCandidate(TypedDict, total=False):
    equation: str
    source_section: str
    kind_hint: str


class ReferenceCandidate(TypedDict, total=False):
    raw_text: str
    display_text: str
    page_hint: str
    doi: str
    arxiv_id: str
    wikilink: str
    vault_target: str
    match_status: str
    match_reason: str


class FigureQualitySignals(TypedDict, total=False):
    visual_quality_status: str
    quality_reason_codes: list[str]
    page_coverage_ratio: float
    visual_rect_count: int
    visual_body_ratio: float
    paragraph_text_chars: int
    table_body_rows: int
    caption_text_chars: int


class FigureAssetCandidate(TypedDict, total=False):
    filename: str
    path: str
    width: int
    height: int
    size_bytes: int
    label: str
    extraction_level: str
    quality_signals: FigureQualitySignals
    candidate_status: str


class SectionExtractionCoverage(TypedDict, total=False):
    coverage_status: str
    recognized_sections: list[str]
    core_sections_found: list[str]
    missing_core_sections: list[str]
    section_text_chars: dict[str, int]
    fallback_sections: list[str]


class PdfCoverage(TypedDict, total=False):
    total_pages: int | None
    text_max_pages: int | None
    text_pages_scanned: int
    truncated_due_to_page_limit: bool
    appendix_detected: bool
    appendix_start_page: int | None
    references_start_page: int | None
    section_stop_reason: str
    section_stop_page: int | None


class AppendixIndex(TypedDict, total=False):
    appendix_detected: bool
    start_page: int | None
    sections: list[dict[str, Any]]
    figure_captions: list[dict[str, Any]]
    table_captions: list[dict[str, Any]]


class AppendixEvidenceItem(TypedDict, total=False):
    evidence: str
    source_section: str
    page_hint: str
    kind_hint: str


class EvidencePack(TypedDict, total=False):
    paper_id: str
    problem_evidence: list[EvidenceItem]
    task_evidence: list[EvidenceItem]
    data_evidence: list[EvidenceItem]
    method_evidence: list[EvidenceItem]
    mechanism_evidence: list[EvidenceItem]
    results_evidence: list[EvidenceItem]
    ablation_evidence: list[EvidenceItem]
    limitations_evidence: list[EvidenceItem]
    equation_candidates: list[EquationCandidate]
    reference_candidates: list[ReferenceCandidate]
    figure_captions: list[dict[str, Any]]
    table_captions: list[dict[str, Any]]
    sections: list[dict[str, Any]]
    section_texts: dict[str, str]
    candidate_chunks: dict[str, list[CandidateChunk]]
    language_hint: str
    section_sources: dict[str, str]
    section_extraction_coverage: SectionExtractionCoverage
    pdf_coverage: PdfCoverage
    appendix_index: AppendixIndex
    appendix_evidence: dict[str, list[AppendixEvidenceItem]]
    quotes: list[dict[str, Any]]
    evidence_quality: str
    extraction_failures: list[str]


class FigurePlanItem(TypedDict, total=False):
    id: str
    caption: str
    kind: str
    section: str
    reason: str
    priority: int
    anchor_text: str
    insert_mode: str
    figure_asset_candidate: FigureAssetCandidate
    candidate_pages: list[dict[str, Any]]
    candidate_status: str
    matching_strategy: str


class FigurePlan(TypedDict, total=False):
    paper_id: str
    figures: list[FigurePlanItem]


class SynthesisBundle(TypedDict, total=False):
    paper_id: str
    title: str
    metadata: dict[str, Any]
    evidence_quality: str
    coverage: dict[str, Any]
    source_manifest: dict[str, Any]
    source_index: dict[str, Any]
    references: dict[str, Any]
    figure_plan: dict[str, Any]
    figure_table_manifest: dict[str, Any]
    pdf_assets: dict[str, Any]
    writing_contract: dict[str, Any]


def empty_metadata() -> MetadataRecord:
    return MetadataRecord(
        title="",
        paper_id="",
        source_type="",
        source_url="",
        year="",
        authors=[],
        affiliations=[],
        metadata_sources=[],
        identity_confidence="",
        identity_confidence_reasons=[],
    )


def empty_evidence_pack() -> EvidencePack:
    return EvidencePack(
        paper_id="",
        problem_evidence=[],
        task_evidence=[],
        data_evidence=[],
        method_evidence=[],
        mechanism_evidence=[],
        results_evidence=[],
        ablation_evidence=[],
        limitations_evidence=[],
        equation_candidates=[],
        reference_candidates=[],
        figure_captions=[],
        table_captions=[],
        sections=[],
        section_texts={},
        candidate_chunks={},
        language_hint="unknown",
        section_sources={},
        section_extraction_coverage={},
        pdf_coverage={},
        appendix_index={},
        appendix_evidence={},
        quotes=[],
        extraction_failures=[],
        evidence_quality="unknown",
    )


def empty_figure_plan() -> FigurePlan:
    return FigurePlan(paper_id="", figures=[])


def empty_synthesis_bundle() -> SynthesisBundle:
    return SynthesisBundle(
        paper_id="",
        title="",
        metadata={},
        evidence_quality="unknown",
        coverage={},
        source_manifest={},
        source_index={},
        references={},
        figure_plan={},
        figure_table_manifest={},
        pdf_assets={},
        writing_contract={},
    )

#!/usr/bin/env python3
"""Create a deterministic paper input record from trusted metadata such as Zotero results."""

from __future__ import annotations

import argparse
import json

from common import emit, normalize_whitespace, paper_id_for_record


def parser() -> argparse.ArgumentParser:
    p = argparse.ArgumentParser(description=__doc__ or "create input record")
    p.add_argument("--title", required=True, help="Canonical paper title.")
    p.add_argument("--translated-title", default="", help="Optional translated title.")
    p.add_argument("--authors-json", default="[]", help="JSON array of author names.")
    p.add_argument("--affiliations-json", default="[]", help="JSON array of affiliations.")
    p.add_argument("--year", default="", help="Publication year.")
    p.add_argument("--venue", default="", help="Venue or journal.")
    p.add_argument("--doi", default="", help="DOI if known.")
    p.add_argument("--arxiv-id", default="", help="arXiv id if known.")
    p.add_argument("--source-url", default="", help="Canonical source URL.")
    p.add_argument("--pdf-url", default="", help="Preferred PDF URL if known.")
    p.add_argument("--local-pdf-path", default="", help="Local PDF path if already available.")
    p.add_argument("--zotero-key", default="", help="Local Zotero item key.")
    p.add_argument("--abstract", default="", help="Abstract text when available.")
    p.add_argument("--source-type", default="", help="Optional explicit source type.")
    p.add_argument("--output", default="", help="Output JSON path.")
    return p


def parse_json_list(raw: str) -> list[str]:
    try:
        data = json.loads(raw)
    except Exception:
        return []
    if not isinstance(data, list):
        return []
    values: list[str] = []
    for item in data:
        cleaned = normalize_whitespace(str(item))
        if cleaned:
            values.append(cleaned)
    return values


def main() -> None:
    args = parser().parse_args()
    record = {
        "status": "ok",
        "script": "create_input_record.py",
        "title": normalize_whitespace(args.title),
        "translated_title": normalize_whitespace(args.translated_title),
        "authors": parse_json_list(args.authors_json),
        "affiliations": parse_json_list(args.affiliations_json),
        "year": normalize_whitespace(args.year),
        "venue": normalize_whitespace(args.venue),
        "doi": normalize_whitespace(args.doi),
        "arxiv_id": normalize_whitespace(args.arxiv_id),
        "source_url": normalize_whitespace(args.source_url),
        "pdf_url": normalize_whitespace(args.pdf_url),
        "local_pdf_path": normalize_whitespace(args.local_pdf_path),
        "zotero_key": normalize_whitespace(args.zotero_key),
        "abstract": normalize_whitespace(args.abstract),
        "source_type": normalize_whitespace(args.source_type)
        or ("zotero_seed" if normalize_whitespace(args.zotero_key) else "seed_record"),
        "metadata_sources": ["zotero_seed"] if normalize_whitespace(args.zotero_key) else ["seed_record"],
    }
    record["paper_id"] = paper_id_for_record(record)
    emit(record, args.output)


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""Extract a richer evidence pack from PDF or full text, including candidate chunks and captions."""

from __future__ import annotations

import argparse
import re
from pathlib import Path
from typing import Any

from common import (
    SECTION_ALIASES,
    emit,
    extract_appendix_index,
    extract_appendix_page_texts,
    extract_caption_lines,
    extract_dataset_candidates,
    extract_mechanism_flow_sentences,
    extract_metric_claims,
    extract_negative_claims,
    extract_pdf_sections,
    extract_pdf_text,
    infer_paper_type,
    maybe_load_json_record,
    normalize_heading,
    normalize_whitespace,
    paper_id_for_record,
    pick_sentences_by_keywords,
    pdf_coverage_summary,
    require_accepted_fetch_artifact,
    split_sentences,
)
from contracts import empty_evidence_pack
from citation_links import extract_reference_candidates_from_pdf
from source_corpus import SourceCorpus, SourceCorpusLoadError, load_source_corpus, validate_source_corpus


CORE_SECTIONS = ("introduction", "method", "experiment")
CAPTION_LIST_LIMIT = 48
APPENDIX_EVIDENCE_CATEGORIES = (
    "ablation",
    "implementation_details",
    "dataset_details",
    "extra_results",
    "qualitative_examples",
)


def parser() -> argparse.ArgumentParser:
    p = argparse.ArgumentParser(description=__doc__ or "extract evidence")
    p.add_argument("--input", default="", help="Successful fetch JSON path or JSON artifact.")
    p.add_argument("--source-manifest", default="", help="Source Corpus source_manifest.json path.")
    p.add_argument("--output", default="", help="Output JSON path.")
    p.add_argument("--max-pages", type=int, default=32, help="Maximum number of PDF pages to scan.")
    p.add_argument("--max-chunks-per-section", type=int, default=12, help="Maximum number of candidate chunks to keep per section.")
    return p


def ensure_record(input_value: str) -> dict:
    record = maybe_load_json_record(input_value)
    if record is None:
        raise SystemExit("extract_evidence.py requires a JSON acquisition artifact.")
    return dict(require_accepted_fetch_artifact(record, "extract_evidence.py"))


def build_items(sentences: list[str], section: str) -> list[dict]:
    items = []
    for sentence in sentences:
        cleaned = normalize_whitespace(sentence)
        if not cleaned:
            continue
        items.append(
            {
                "claim": cleaned,
                "evidence": cleaned,
                "source_section": section,
                "page_hint": "",
            }
        )
    return items


def language_hint_for_text(text: str) -> str:
    cjk_chars = len(re.findall(r"[\u3400-\u9fff]", text or ""))
    latin_chars = len(re.findall(r"[A-Za-z]", text or ""))
    total = cjk_chars + latin_chars
    if total == 0:
        return "unknown"
    cjk_ratio = cjk_chars / total
    latin_ratio = latin_chars / total
    if cjk_ratio >= 0.6:
        return "zh"
    if latin_ratio >= 0.6:
        return "en"
    return "mixed"


def section_text_with_source(
    section_map: dict[str, str],
    section: str,
    fallback: str = "",
) -> tuple[str, str]:
    text = section_map.get(section, "")
    if text:
        return text, section
    if fallback:
        return fallback, "abstract"
    return "", ""


def build_section_extraction_coverage(
    section_map: dict[str, str],
    section_sources: dict[str, str],
) -> dict:
    core_sections_found = [section for section in CORE_SECTIONS if section_map.get(section)]
    missing_core_sections = [section for section in CORE_SECTIONS if section not in core_sections_found]
    fallback_sections = [
        section
        for section, source in section_sources.items()
        if section != "abstract" and source == "abstract"
    ]
    if len(core_sections_found) >= len(CORE_SECTIONS):
        coverage_status = "good"
    elif core_sections_found:
        coverage_status = "partial"
    else:
        coverage_status = "poor"
    return {
        "coverage_status": coverage_status,
        "recognized_sections": list(section_map.keys()),
        "core_sections_found": core_sections_found,
        "missing_core_sections": missing_core_sections,
        "section_text_chars": {
            section: len(normalize_whitespace(text))
            for section, text in section_map.items()
            if normalize_whitespace(text)
        },
        "fallback_sections": fallback_sections,
    }


def source_corpus_error_payload(issues: list[dict[str, Any]]) -> dict[str, Any]:
    return {
        "status": "error",
        "script": "extract_evidence.py",
        "error": "source_corpus_invalid",
        "source_corpus_issues": issues,
    }


def ensure_source_corpus(source_manifest_path: str, output_path: str = "") -> SourceCorpus:
    try:
        corpus = load_source_corpus(source_manifest_path)
    except SourceCorpusLoadError as exc:
        emit(
            source_corpus_error_payload(
                [
                    {
                        "code": "source_corpus_load_error",
                        "severity": "error",
                        "message": str(exc),
                        "path": source_manifest_path,
                    }
                ]
            ),
            output_path,
        )
        raise SystemExit(1) from exc

    issues = validate_source_corpus(corpus)
    if issues:
        emit(source_corpus_error_payload(issues), output_path)
        raise SystemExit(1)
    return corpus


def record_from_source_corpus(corpus: SourceCorpus, input_value: str) -> dict:
    manifest = corpus.manifest
    record = {
        key: manifest.get(key, "")
        for key in ("paper_id", "title", "abstract", "doi", "arxiv_id", "url")
        if manifest.get(key)
    }
    if input_value:
        input_record = ensure_record(input_value)
        for key, value in input_record.items():
            if value not in ("", None):
                record[key] = value
    return record


def source_section_key(record: dict[str, Any]) -> str:
    for key in ("kind", "title", "section_id"):
        normalized = normalize_heading(str(record.get(key, "")).replace("_", " "))
        if not normalized:
            continue
        for section, aliases in SECTION_ALIASES.items():
            if normalized == section or normalized in aliases:
                return section
        if normalized in {"appendix", "appendices", "supplementary material", "附录", "补充材料"}:
            return "appendix"
    kind = normalize_whitespace(str(record.get("kind", ""))).lower()
    return kind if kind else ""


def section_texts_from_source_corpus(corpus: SourceCorpus) -> dict[str, str]:
    sections: dict[str, list[str]] = {}
    for record in corpus.raw_sections:
        text = normalize_whitespace(str(record.get("text", "")))
        section = source_section_key(record)
        if not text or not section:
            continue
        sections.setdefault(section, []).append(text)
    return {
        section: normalize_whitespace(" ".join(texts))
        for section, texts in sections.items()
        if normalize_whitespace(" ".join(texts))
    }


def page_hint_for_caption(item: dict[str, Any]) -> str:
    pages = item.get("pages")
    page = pages[0] if isinstance(pages, list) and pages else item.get("page")
    if page in ("", None):
        return ""
    return f"p.{page}"


def captions_from_source_corpus(corpus: SourceCorpus, kind: str) -> list[dict]:
    captions: list[dict] = []
    for item in corpus.caption_items():
        if item.get("kind") != kind:
            continue
        caption = {
            "id": normalize_whitespace(str(item.get("id", ""))),
            "caption": normalize_whitespace(str(item.get("caption", ""))),
        }
        page_hint = normalize_whitespace(str(item.get("page_hint", ""))) or page_hint_for_caption(item)
        section_id = normalize_whitespace(str(item.get("section_id", "")))
        if page_hint:
            caption["page_hint"] = page_hint
        if section_id:
            caption["section_id"] = section_id
        if caption["id"] and caption["caption"]:
            captions.append(caption)
    return captions[:CAPTION_LIST_LIMIT]


def pdf_coverage_from_source_corpus(corpus: SourceCorpus) -> dict:
    coverage = corpus.coverage()
    truncation = corpus.truncation()
    appendix_index = corpus.manifest.get("appendix_index", {})
    appendix_start_page = coverage.get("appendix_start_page")
    if not appendix_start_page and isinstance(appendix_index, dict):
        appendix_start_page = appendix_index.get("start_page")
    return {
        "total_pages": coverage.get("total_pages"),
        "text_max_pages": coverage.get("text_max_pages"),
        "text_pages_extracted": coverage.get("text_pages_extracted"),
        "text_pages_scanned": coverage.get("text_pages_scanned")
        or coverage.get("text_pages_extracted"),
        "text_truncated": truncation["text_truncated"],
        "truncated_due_to_page_limit": truncation["truncated_due_to_page_limit"],
        "appendix_detected": bool(coverage.get("appendix_detected") or appendix_start_page),
        "appendix_start_page": appendix_start_page,
        "references_start_page": coverage.get("references_start_page"),
        "section_stop_reason": coverage.get("section_stop_reason", ""),
        "section_stop_page": coverage.get("section_stop_page"),
    }


def appendix_index_from_source_corpus(corpus: SourceCorpus) -> dict:
    appendix_index = corpus.manifest.get("appendix_index", {})
    if isinstance(appendix_index, dict):
        return dict(appendix_index)
    coverage = corpus.coverage()
    return {
        "appendix_detected": bool(coverage.get("appendix_detected")),
        "start_page": coverage.get("appendix_start_page"),
        "sections": [],
        "figure_captions": [],
        "table_captions": [],
    }


def empty_appendix_evidence() -> dict[str, list[dict]]:
    return {category: [] for category in APPENDIX_EVIDENCE_CATEGORIES}


def appendix_evidence_category(sentence: str) -> str:
    lower = sentence.lower()
    if any(token in lower for token in ["ablation", "without", "w/o", "remove", "removing"]):
        if any(token in lower for token in ["drop", "decrease", "worse", "unstable", "fail", "%"]):
            return "ablation"
    if any(
        token in lower
        for token in [
            "learning rate",
            "batch size",
            "optimizer",
            "epoch",
            "hardware",
            "gpu",
            "hyperparameter",
            "implementation",
            "prompt",
            "temperature",
        ]
    ):
        return "implementation_details"
    if re.search(r"\d", sentence) and any(
        token in lower
        for token in ["accuracy", "f1", "auc", "auprc", "score", "result", "outperform", "improv"]
    ):
        return "extra_results"
    if any(
        token in lower
        for token in [
            "dataset",
            "split",
            "training set",
            "validation",
            "test set",
            "annotation",
            "annotator",
            "corpus",
            "participants",
            "samples",
        ]
    ):
        return "dataset_details"
    if any(token in lower for token in ["case study", "qualitative", "example", "failure case"]):
        return "qualitative_examples"
    return ""


def appendix_section_for_page(appendix_index: dict, page_number: int) -> str:
    sections = appendix_index.get("sections", []) if isinstance(appendix_index, dict) else []
    current = "appendix"
    for section in sections:
        if not isinstance(section, dict):
            continue
        section_page = int(section.get("page", 0) or 0)
        if section_page and section_page <= page_number:
            current = normalize_whitespace(str(section.get("title", ""))) or current
    return current


def build_appendix_evidence(
    appendix_pages: list[dict],
    appendix_index: dict,
    *,
    limit_per_category: int = 8,
) -> dict[str, list[dict]]:
    evidence = empty_appendix_evidence()
    seen = set()
    for page in appendix_pages:
        page_number = int(page.get("page", 0) or 0)
        source_section = appendix_section_for_page(appendix_index, page_number)
        for sentence in split_sentences(str(page.get("text", ""))):
            cleaned = normalize_whitespace(sentence)
            if not cleaned:
                continue
            category = appendix_evidence_category(cleaned)
            if not category or len(evidence[category]) >= limit_per_category:
                continue
            marker = f"{category}::{cleaned.lower()}"
            if marker in seen:
                continue
            seen.add(marker)
            evidence[category].append(
                {
                    "evidence": cleaned,
                    "source_section": source_section,
                    "page_hint": f"p.{page_number}" if page_number else "",
                    "kind_hint": category,
                }
            )
    return evidence


def text_chunks(
    text: str,
    *,
    section: str,
    kind_hint: str = "",
    actual_source_section: str = "",
    is_abstract_fallback: bool = False,
    max_chunks: int = 12,
    sentences_per_chunk: int = 2,
    max_chars: int = 520,
) -> list[dict]:
    sentences = split_sentences(text)
    chunks: list[dict] = []
    seen = set()
    for idx in range(0, len(sentences), sentences_per_chunk):
        group = sentences[idx : idx + sentences_per_chunk]
        if not group:
            continue
        chunk = normalize_whitespace(" ".join(group))
        if not chunk:
            continue
        if len(chunk) > max_chars:
            chunk = chunk[: max_chars - 3].rstrip(" ,;:") + "..."
        marker = chunk.lower()
        if marker in seen:
            continue
        seen.add(marker)
        item = {
            "text": chunk,
            "source_section": section,
            "page_hint": "",
            "kind_hint": kind_hint,
            "actual_source_section": actual_source_section or section,
        }
        if is_abstract_fallback:
            item["is_abstract_fallback"] = True
        chunks.append(item)
        if len(chunks) >= max_chunks:
            break
    return chunks


def first_chunks(chunks: list[dict], limit: int) -> list[dict]:
    return [
        {
            "claim": normalize_whitespace(str(item.get("text", ""))),
            "evidence": normalize_whitespace(str(item.get("text", ""))),
            "source_section": normalize_whitespace(str(item.get("source_section", ""))),
            "page_hint": normalize_whitespace(str(item.get("page_hint", ""))),
        }
        for item in chunks[:limit]
        if normalize_whitespace(str(item.get("text", "")))
    ]


def keyword_chunks(
    text: str,
    keywords: list[str],
    *,
    section: str,
    max_chunks: int = 6,
) -> list[dict]:
    picked = pick_sentences_by_keywords(text, keywords, limit=max_chunks)
    return text_chunks(
        " ".join(picked),
        section=section,
        kind_hint=section,
        max_chunks=max_chunks,
        sentences_per_chunk=1,
    )


def candidate_map(
    *,
    abstract: str,
    intro_text: str,
    method_text: str,
    experiment_text: str,
    conclusion_text: str,
    data_text: str,
    section_sources: dict[str, str],
    max_chunks_per_section: int,
) -> dict[str, list[dict]]:
    combined_general = " ".join(
        part
        for part in [abstract, intro_text, method_text, experiment_text, conclusion_text]
        if part
    )
    return {
        "abstract": text_chunks(
            abstract,
            section="abstract",
            kind_hint="abstract",
            actual_source_section="abstract",
            max_chunks=max_chunks_per_section,
        ),
        "introduction": text_chunks(
            intro_text,
            section="introduction",
            kind_hint="problem",
            actual_source_section=section_sources.get("introduction", "introduction"),
            is_abstract_fallback=section_sources.get("introduction") == "abstract",
            max_chunks=max_chunks_per_section,
        ),
        "method": text_chunks(
            method_text,
            section="method",
            kind_hint="method",
            actual_source_section=section_sources.get("method", "method"),
            is_abstract_fallback=section_sources.get("method") == "abstract",
            max_chunks=max_chunks_per_section,
        ),
        "experiment": text_chunks(
            experiment_text,
            section="experiment",
            kind_hint="results",
            actual_source_section=section_sources.get("experiment", "experiment"),
            is_abstract_fallback=section_sources.get("experiment") == "abstract",
            max_chunks=max_chunks_per_section,
        ),
        "conclusion": text_chunks(
            conclusion_text,
            section="conclusion",
            kind_hint="limitations",
            actual_source_section=section_sources.get("conclusion", "conclusion"),
            is_abstract_fallback=section_sources.get("conclusion") == "abstract",
            max_chunks=max_chunks_per_section,
        ),
        "data": text_chunks(
            data_text,
            section="data",
            kind_hint="data",
            actual_source_section=section_sources.get("data", "mixed"),
            max_chunks=max_chunks_per_section,
        ),
        "general": text_chunks(
            combined_general,
            section="general",
            kind_hint="general",
            actual_source_section="mixed",
            max_chunks=max_chunks_per_section,
        ),
    }


EXPLICIT_COMPLEXITY_RE = re.compile(r"\bO\([^)]{1,80}\)")
TEX_OR_MATH_SIGNAL_RE = re.compile(
    r"\\(?:sum|prod|int|frac|sqrt|log|argmax|argmin)\b|(?:>=|<=)|[\^∑∏∫≤≥≈≠]"
)
SUBSCRIPT_SIGNAL_RE = re.compile(
    r"(?<![A-Za-z0-9])\\?(?P<base>[A-Za-z]+)\s*_\s*(?:\{[^}]{1,30}\}|[A-Za-z0-9])"
)
GREEK_MATH_NAMES = {
    "alpha",
    "beta",
    "gamma",
    "delta",
    "theta",
    "lambda",
    "mu",
    "sigma",
    "phi",
    "psi",
    "omega",
}
CONFIG_ASSIGNMENT_RE = re.compile(
    r"^\s*(?P<lhs>[A-Za-z][A-Za-z0-9_ -]{1,40})\s*=\s*(?P<rhs>[^=]{1,80})\s*$"
)
CONFIG_LHS_RE = re.compile(
    r"\b(?:model|dataset|temperature|lr|learning_rate|batch_size|optimizer|epoch|seed)\b",
    re.IGNORECASE,
)


def looks_like_math_candidate(text: str, kind_hint: str = "") -> bool:
    cleaned = normalize_whitespace(text)
    if not cleaned:
        return False
    if EXPLICIT_COMPLEXITY_RE.search(cleaned):
        return True
    has_signal = bool(TEX_OR_MATH_SIGNAL_RE.search(cleaned))
    if not has_signal:
        for match in SUBSCRIPT_SIGNAL_RE.finditer(cleaned):
            base = match.group("base").lower()
            if len(base) <= 3 or base in GREEK_MATH_NAMES:
                has_signal = True
                break
    if not has_signal:
        return False
    assignment_match = CONFIG_ASSIGNMENT_RE.match(cleaned.rstrip(" .;:"))
    if assignment_match and CONFIG_LHS_RE.search(assignment_match.group("lhs")):
        return False
    return True


def extract_equation_candidates(*, full_text: str, method_text: str, experiment_text: str, conclusion_text: str, limit: int = 8) -> list[dict]:
    candidates: list[dict] = []
    seen = set()

    def add_candidate(text: str, section: str, kind_hint: str) -> None:
        cleaned = normalize_whitespace(text)
        if not looks_like_math_candidate(cleaned, kind_hint):
            return
        marker = cleaned.lower()
        if marker in seen:
            return
        seen.add(marker)
        candidates.append(
            {
                "equation": cleaned,
                "source_section": section,
                "kind_hint": kind_hint,
            }
        )

    math_like_patterns = [
        (r"O\([^)]*\)", "method", "complexity"),
        (r"\bp\([^)]*\)\s*=\s*[^.]{1,120}", "method", "objective"),
        (r"\b(?:L|Loss|Err|ELBO|FID|IS|NLL)[A-Za-z0-9_]*\s*=\s*[^.]{1,120}", "experiment", "metric_equation"),
        (r"[A-Za-z][A-Za-z0-9_]*\s*=\s*\([^)]*\)\s*\^[^\s,.;]+", "experiment", "scaling_law"),
        (r"[A-Za-z][A-Za-z0-9_{}()\\^\s]{0,60}\s*(?:>=|<=|≤|≥|≈|≠)\s*[A-Za-z][A-Za-z0-9_{}()\\^\s]{0,60}", "method", "comparison_equation"),
        (r"[A-Za-z][A-Za-z0-9_]*\s*=\s*[^.]{1,100}", "method", "equation"),
    ]

    for pattern, section, kind_hint in math_like_patterns:
        for match in re.finditer(pattern, full_text or ""):
            add_candidate(match.group(0), section, kind_hint)
            if len(candidates) >= limit:
                return candidates

    equation_sentences = pick_sentences_by_keywords(
        " ".join(part for part in [method_text, experiment_text, conclusion_text] if part),
        [
            "objective",
            "loss",
            "likelihood",
            "probability",
            "optimiz",
            "maximize",
            "minimize",
            "complexity",
            "scaling law",
            "equation",
        ],
        limit=limit,
    )
    for sentence in equation_sentences:
        section = "method"
        lower = sentence.lower()
        if "scaling" in lower or "loss" in lower or "err" in lower:
            section = "experiment"
        add_candidate(sentence, section, "formula_context")
        if len(candidates) >= limit:
            break

    return candidates[:limit]


def evidence_quality(pack: dict) -> str:
    score = 0
    core_score = 0
    candidate_chunks = pack.get("candidate_chunks", {}) or {}
    coverage = pack.get("section_extraction_coverage", {}) or {}
    core_sections_found = coverage.get("core_sections_found", []) if isinstance(coverage, dict) else []
    if core_sections_found:
        core_score = len([section for section in CORE_SECTIONS if section in core_sections_found])
    else:
        for section in CORE_SECTIONS:
            chunks = candidate_chunks.get(section, []) or []
            if any(
                isinstance(chunk, dict)
                and not chunk.get("is_abstract_fallback")
                and chunk.get("actual_source_section", section) != "abstract"
                for chunk in chunks
            ):
                core_score += 1
    score += core_score
    if pack.get("equation_candidates"):
        score += 1
    if pack.get("figure_captions"):
        score += 1
    if pack.get("table_captions"):
        score += 1
    if core_score == 0:
        return "low"
    if score >= 6:
        return "high"
    if score >= 3:
        return "medium"
    return "low"


def main() -> None:
    args = parser().parse_args()
    section_map: dict[str, str] = {}
    full_text = ""
    extraction_failures: list[str] = []
    language_hint = ""
    source_corpus_used = bool(args.source_manifest)
    if not args.input and not args.source_manifest:
        raise SystemExit("--input or --source-manifest is required.")

    if source_corpus_used:
        corpus = ensure_source_corpus(args.source_manifest, args.output)
        record = record_from_source_corpus(corpus, args.input)
        has_pdf = False
        pdf_path = None
        section_map = section_texts_from_source_corpus(corpus)
        full_text = corpus.full_text()
        pdf_coverage = pdf_coverage_from_source_corpus(corpus)
        appendix_index = appendix_index_from_source_corpus(corpus)
        appendix_pages = corpus.appendix_text_pages()
        appendix_evidence = build_appendix_evidence(appendix_pages, appendix_index)
        figure_captions = captions_from_source_corpus(corpus, "figure")
        table_captions = captions_from_source_corpus(corpus, "table")
        language_hint = corpus.language_hint()
    else:
        record = ensure_record(args.input)
        pdf_value = str(record.get("pdf_path", "")).strip()
        pdf_path = Path(pdf_value).expanduser() if pdf_value else None

        if pdf_path is None or not pdf_path.is_file():
            from_fetch = maybe_load_json_record(args.input) or {}
            pdf_candidate = str(from_fetch.get("pdf_path", "")).strip()
            if pdf_candidate:
                candidate_path = Path(pdf_candidate).expanduser()
                if candidate_path.is_file():
                    pdf_path = candidate_path

        has_pdf = pdf_path is not None and pdf_path.is_file()
        pdf_coverage = (
            pdf_coverage_summary(pdf_path.resolve(), max_pages=args.max_pages)
            if has_pdf
            else pdf_coverage_summary(Path(""), max_pages=args.max_pages)
        )
        appendix_index = (
            extract_appendix_index(pdf_path.resolve(), pdf_coverage)
            if has_pdf and pdf_coverage.get("appendix_detected")
            else extract_appendix_index(Path(""), pdf_coverage)
        )
        appendix_pages = (
            extract_appendix_page_texts(pdf_path.resolve(), pdf_coverage.get("appendix_start_page"))
            if has_pdf and pdf_coverage.get("appendix_detected")
            else []
        )
        appendix_evidence = build_appendix_evidence(appendix_pages, appendix_index)
        if has_pdf:
            try:
                section_map = extract_pdf_sections(pdf_path.resolve(), max_pages=args.max_pages)
                full_text = extract_pdf_text(pdf_path.resolve(), max_pages=args.max_pages)
            except Exception as exc:
                extraction_failures.append(f"pdf_parse_failed: {exc}")
        else:
            extraction_failures.append("pdf_missing")
        figure_captions = extract_caption_lines(full_text, "figure")[:CAPTION_LIST_LIMIT] if full_text else []
        table_captions = extract_caption_lines(full_text, "table")[:CAPTION_LIST_LIMIT] if full_text else []

    paper_type, paper_type_rationale = infer_paper_type(record.get("title", ""), record.get("abstract", ""))

    metadata_abstract = normalize_whitespace(str(record.get("abstract", "")).strip())
    abstract = metadata_abstract or normalize_whitespace(section_map.get("abstract", ""))
    intro_text, intro_source = section_text_with_source(section_map, "introduction", abstract)
    method_text, method_source = section_text_with_source(section_map, "method", abstract)
    if section_map.get("experiment"):
        experiment_text = section_map["experiment"]
        experiment_source = "experiment"
    elif section_map.get("conclusion"):
        experiment_text = section_map["conclusion"]
        experiment_source = "conclusion"
    else:
        experiment_text = abstract
        experiment_source = "abstract" if abstract else ""
    conclusion_text, conclusion_source = section_text_with_source(section_map, "conclusion", abstract)
    mechanism_caption_text = " ".join(
        item.get("caption", "")
        for item in figure_captions
        if isinstance(item, dict)
        and any(token in str(item.get("caption", "")).lower() for token in ["pipeline", "framework", "overview", "architecture", "system", "workflow", "stage"])
    )
    data_text = " ".join(
        part
        for part in [
            section_map.get("abstract", ""),
            section_map.get("introduction", ""),
            section_map.get("method", ""),
            section_map.get("data", ""),
            section_map.get("experiment", ""),
        ]
        if part
    )
    section_sources = {
        "abstract": "abstract",
        "introduction": intro_source,
        "method": method_source,
        "experiment": experiment_source,
        "conclusion": conclusion_source,
        "data": "data" if section_map.get("data") else "mixed",
    }
    section_extraction_coverage = build_section_extraction_coverage(section_map, section_sources)
    if (
        section_extraction_coverage["coverage_status"] == "poor"
        and "section_coverage_poor" not in extraction_failures
    ):
        extraction_failures.append("section_coverage_poor")

    candidates = candidate_map(
        abstract=abstract,
        intro_text=intro_text,
        method_text=method_text,
        experiment_text=experiment_text,
        conclusion_text=conclusion_text,
        data_text=data_text,
        section_sources=section_sources,
        max_chunks_per_section=args.max_chunks_per_section,
    )

    problem_sentences = pick_sentences_by_keywords(
        intro_text or abstract,
        ["we address", "we investigate", "we study", "challenge", "problem", "aim", "objective", "however"],
        limit=4,
    ) or split_sentences(intro_text or abstract)[:3]
    task_sentences = pick_sentences_by_keywords(
        " ".join([abstract, intro_text, method_text]),
        ["task", "predict", "classification", "identify", "detect", "estimate", "evaluate", "diagnos", "screen"],
        limit=5,
    ) or [chunk["text"] for chunk in candidates.get("introduction", [])[:3]]
    data_sentences = pick_sentences_by_keywords(
        data_text,
        ["dataset", "datasets", "participants", "patients", "outpatients", "interviews", "corpus", "recordings", "collected"],
        limit=5,
    ) or [chunk["text"] for chunk in candidates.get("data", [])[:3]]
    method_sentences = pick_sentences_by_keywords(
        method_text,
        ["we propose", "we present", "we introduce", "framework", "pipeline", "model", "method", "feature", "classifier", "fine-tun", "zero-shot"],
        limit=6,
    ) or [chunk["text"] for chunk in candidates.get("method", [])[:4]]
    mechanism_sentences = extract_mechanism_flow_sentences(
        " ".join(part for part in [method_text, mechanism_caption_text] if part),
        limit=6,
    ) or method_sentences[:4]
    result_sentences = extract_metric_claims(experiment_text) or pick_sentences_by_keywords(
        experiment_text,
        ["outperform", "improve", "accuracy", "f1", "auc", "auprc", "score", "results show", "achieved"],
        limit=6,
    ) or [chunk["text"] for chunk in candidates.get("experiment", [])[:4]]
    ablation_sentences = extract_negative_claims(" ".join(part for part in [experiment_text, conclusion_text] if part), limit=6)
    limitation_sentences = pick_sentences_by_keywords(
        conclusion_text,
        ["limitation", "future work", "however", "remain", "generaliz", "need", "further"],
        limit=4,
    ) or [chunk["text"] for chunk in candidates.get("conclusion", [])[:3]]

    pack = empty_evidence_pack()
    pack["paper_id"] = record.get("paper_id") or paper_id_for_record(record)
    pack["problem_evidence"] = build_items(problem_sentences, "introduction")
    pack["task_evidence"] = build_items(task_sentences, "task")
    pack["data_evidence"] = build_items(data_sentences, "data")
    pack["method_evidence"] = build_items(method_sentences, "method")
    pack["mechanism_evidence"] = build_items(mechanism_sentences, "method")
    pack["results_evidence"] = build_items(result_sentences, "experiment")
    pack["ablation_evidence"] = build_items(ablation_sentences, "experiment")
    pack["limitations_evidence"] = build_items(limitation_sentences, "conclusion")
    pack["equation_candidates"] = extract_equation_candidates(
        full_text=full_text,
        method_text=method_text,
        experiment_text=experiment_text,
        conclusion_text=conclusion_text,
    )
    pack["reference_candidates"] = []
    if has_pdf and pdf_coverage.get("references_start_page"):
        pack["reference_candidates"] = extract_reference_candidates_from_pdf(
            pdf_path.resolve(),
            pdf_coverage.get("references_start_page"),
        )
    pack["candidate_chunks"] = candidates
    pack["language_hint"] = language_hint or language_hint_for_text(
        " ".join(part for part in [full_text, abstract] if part)
    )
    pack["section_sources"] = section_sources
    pack["section_extraction_coverage"] = section_extraction_coverage
    pack["pdf_coverage"] = pdf_coverage
    pack["appendix_index"] = appendix_index
    pack["appendix_evidence"] = appendix_evidence
    pack["section_texts"] = {
        key: normalize_whitespace(value)
        for key, value in section_map.items()
        if normalize_whitespace(value)
    }
    pack["figure_captions"] = figure_captions
    pack["table_captions"] = table_captions
    pack["sections"] = [
        {"name": key, "length": len(value), "preview": value[:240]}
        for key, value in section_map.items()
    ]
    pack["quotes"] = []
    pack["extraction_failures"] = extraction_failures
    pack["evidence_quality"] = evidence_quality(pack)

    payload = {
        "status": "ok",
        "script": "extract_evidence.py",
        "paper_id": pack["paper_id"],
        "title": record.get("title", ""),
        "evidence_pack": pack,
        "summary": {
            "paper_type": paper_type,
            "paper_type_rationale": paper_type_rationale,
            "datasets": extract_dataset_candidates(data_text)[:8],
            "metrics": extract_metric_claims(experiment_text)[:8],
            "mechanism_signals": mechanism_sentences[:6],
            "ablation_signals": ablation_sentences[:6],
            "equation_candidates": pack["equation_candidates"][:6],
            "section_keys": list(section_map.keys()),
            "language_hint": pack["language_hint"],
            "section_coverage_status": section_extraction_coverage["coverage_status"],
            "fallback_sections": section_extraction_coverage["fallback_sections"],
            "pdf_coverage": pdf_coverage,
            "appendix_evidence_counts": {
                category: len(items)
                for category, items in appendix_evidence.items()
            },
            "pdf_used": has_pdf,
            "source_corpus_used": source_corpus_used,
            "candidate_chunk_sections": sorted([key for key, value in candidates.items() if value]),
        },
    }
    emit(payload, args.output)


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""Extract page-level PDF assets for later model-side semantic figure matching.

Two extraction strategies run in parallel:
1. xref-level: extract raw embedded image objects (legacy behaviour).
2. figure-level: locate Figure/Table captions on each page, compute a bounding
   box that covers the visual content above the caption, and render that region
   from the page pixmap at high DPI.  This produces complete, human-readable
   figures even when the PDF stores them as many small xref fragments or as
   pure vector art.

Downstream consumers (plan_figures.py, materialize_figure_asset.py) should
prefer figure-level assets when available.
"""

from __future__ import annotations

import argparse
import io
import re
from pathlib import Path

from common import (
    default_assets_dir,
    emit,
    fitz,
    maybe_load_json_record,
    normalize_whitespace,
    require_accepted_fetch_artifact,
)

try:
    from PIL import Image  # type: ignore
except ImportError:  # pragma: no cover
    Image = None

try:
    import pytesseract  # type: ignore
except ImportError:  # pragma: no cover
    pytesseract = None

FIGURE_RENDER_DPI = 96
PAGE_PREVIEW_DPI = 96
MIN_FIGURE_HEIGHT_PT = 60
MIN_FIGURE_WIDTH_PT = 100

CAPTION_RE = re.compile(
    r"^((?:"
    r"supplementary\s+(?:fig(?:ure)?|table)\.?\s*\d+[a-z]?"
    r"|extended\s+data\s+(?:fig(?:ure)?|table)\.?\s*\d+[a-z]?"
    r"|scheme\.?\s*\d+[a-z]?"
    r"|algorithm\.?\s*\d+[a-z]?"
    r"|(?:fig(?:ure)?|table)\.?\s*[AS]?\d+[a-z]?"
    r"))(?!\.\d)(?=$|[\s:：.。,\、|—–-])",
    re.IGNORECASE,
)

# Used to decide whether a continuation line still belongs to the caption text
# or has already entered the data body of a table.  A row of pure tabular data
# usually contains many short numeric tokens separated by spaces, e.g.
# "0.283  0.321  0.236  0.282".  When such a row appears immediately after the
# caption start, we must NOT merge it into the caption bbox; otherwise the
# downstream "table body lives below caption" cropping logic will mistake the
# numeric row for caption text and shrink the table bbox accordingly.
_NUMERIC_TOKEN_RE = re.compile(r"^[+-]?(?:\d+\.\d+|\.\d+|\d+)(?:[eE][+-]?\d+)?$")
_TEXT_TABLE_SEPARATOR_RE = re.compile(r"(?:\t+|\s{2,}|[|;])")
_TEXT_TABLE_PROSE_STARTERS = {
    "a",
    "an",
    "it",
    "our",
    "that",
    "the",
    "these",
    "this",
    "those",
    "we",
}
_TEXT_TABLE_PROSE_CONNECTORS = {
    "although",
    "because",
    "therefore",
    "whereas",
    "which",
}


def _looks_like_data_row(text: str) -> bool:
    """Heuristic: a data row from a tabular layout, not part of a caption."""
    tokens = text.split()
    if len(tokens) < 3:
        return False
    numeric_tokens = sum(1 for tok in tokens if _NUMERIC_TOKEN_RE.match(tok))
    return numeric_tokens >= max(2, len(tokens) // 2)


def _looks_like_text_table_row(text: str) -> bool:
    """Heuristic: a short textual row from a comparison/categorization table."""
    raw = text.strip()
    cleaned = normalize_whitespace(raw)
    tokens = cleaned.split()
    if len(tokens) < 3 or len(cleaned) > 180:
        return False

    cells = [cell.strip() for cell in _TEXT_TABLE_SEPARATOR_RE.split(raw) if cell.strip()]
    if len(cells) >= 3 and all(len(cell.split()) <= 8 for cell in cells):
        return True

    normalized_tokens = [tok.strip(".,:()[]{}").lower() for tok in tokens]
    if cleaned.endswith((".", "?", "!")):
        return False
    if normalized_tokens[0] in _TEXT_TABLE_PROSE_STARTERS:
        return False
    if any(tok in _TEXT_TABLE_PROSE_CONNECTORS for tok in normalized_tokens):
        return False

    average_token_length = sum(len(tok.strip(".,:()[]{}")) for tok in tokens) / len(tokens)
    long_tokens = sum(1 for tok in tokens if len(tok.strip(".,:()[]{}")) > 18)
    return len(tokens) <= 8 and average_token_length <= 9.0 and long_tokens <= 1


def _rect_area(bbox: tuple[float, float, float, float]) -> float:
    return max(0.0, bbox[2] - bbox[0]) * max(0.0, bbox[3] - bbox[1])


def _intersection_area(
    a: tuple[float, float, float, float],
    b: tuple[float, float, float, float],
) -> float:
    x0 = max(a[0], b[0])
    y0 = max(a[1], b[1])
    x1 = min(a[2], b[2])
    y1 = min(a[3], b[3])
    return _rect_area((x0, y0, x1, y1))


def _rects_intersect(
    a: tuple[float, float, float, float],
    b: tuple[float, float, float, float],
) -> bool:
    return _intersection_area(a, b) > 0


def _classify_visual_quality(
    *,
    kind: str,
    page_coverage_ratio: float,
    visual_rect_count: int,
    visual_body_ratio: float,
    paragraph_text_chars: int,
    table_body_rows: int,
    caption_text_chars: int,
    other_caption_labels: list[str] | None = None,
) -> dict:
    """Classify whether a caption-matched crop is visually usable.

    This is intentionally conservative. A label/caption match proves identity,
    but not that the rendered crop contains the figure or table body.
    """
    normalized_kind = kind.strip().lower()
    other_caption_labels = list(other_caption_labels or [])
    reasons: list[str] = []

    if normalized_kind == "table":
        text_per_table_row = paragraph_text_chars / max(1, table_body_rows)
        if table_body_rows <= 0:
            reasons.append("table_body_missing")
        if table_body_rows <= 1 and visual_body_ratio < 0.03 and caption_text_chars >= 40:
            reasons.append("caption_only_suspected")
        # Dense tables naturally contain many text spans. Treat prose-like text
        # as contamination when table structure is weak or the text density is
        # far higher than the detected table body can explain.
        if paragraph_text_chars >= 450 and (
            table_body_rows <= 2
            or (paragraph_text_chars >= 900 and visual_body_ratio <= 0.03 and text_per_table_row > 90)
            or text_per_table_row > 140
        ):
            reasons.append("table_text_contamination_suspected")
        if other_caption_labels:
            reasons.append("multiple_caption_regions_suspected")
        status = "reject" if reasons else "usable"
    else:
        if other_caption_labels:
            reasons.append("multiple_caption_regions_suspected")
        visual_dominant = visual_rect_count >= 3 and visual_body_ratio >= 0.18
        if paragraph_text_chars >= 450 and not visual_dominant:
            reasons.append("large_text_block_suspected")
        if page_coverage_ratio >= 0.70 and paragraph_text_chars >= 250:
            reasons.append("oversized_page_crop")
        if visual_rect_count <= 1 and visual_body_ratio < 0.03:
            reasons.append("low_visual_body_ratio")
        if any(
            code in reasons
            for code in (
                "multiple_caption_regions_suspected",
                "large_text_block_suspected",
                "oversized_page_crop",
                "low_visual_body_ratio",
            )
        ):
            status = "reject"
        elif visual_rect_count == 0 or visual_body_ratio < 0.08:
            if "low_visual_body_ratio" not in reasons:
                reasons.append("low_visual_body_ratio")
            status = "review"
        else:
            status = "usable"

    return {
        "visual_quality_status": status,
        "quality_reason_codes": reasons,
        "page_coverage_ratio": round(page_coverage_ratio, 6),
        "visual_rect_count": int(visual_rect_count),
        "visual_body_ratio": round(visual_body_ratio, 6),
        "paragraph_text_chars": int(paragraph_text_chars),
        "table_body_rows": int(table_body_rows),
        "caption_text_chars": int(caption_text_chars),
        "other_caption_count": len(other_caption_labels),
        "other_caption_labels": other_caption_labels,
    }


def _classify_caption_kind(label: str) -> str:
    """Return 'table' if the caption label starts with 'Table', else 'figure'."""
    return (
        "table"
        if re.match(r"^(?:(?:supplementary|extended\s+data)\s+)?table\b", label.strip(), re.IGNORECASE)
        else "figure"
    )


def parser() -> argparse.ArgumentParser:
    p = argparse.ArgumentParser(description=__doc__ or "extract pdf assets")
    p.add_argument("--input", required=True, help="Successful fetch JSON path or JSON artifact.")
    p.add_argument("--output", default="", help="Output JSON path.")
    p.add_argument("--assets-dir", default="", help="Optional explicit assets directory.")
    p.add_argument("--max-pages", type=int, default=40, help="Maximum pages to scan.")
    p.add_argument("--min-searchable-chars", type=int, default=100, help="Minimum characters for a page to count as searchable text.")
    p.add_argument("--ocr-dpi", type=int, default=300, help="DPI used when OCR fallback is needed.")
    p.add_argument("--figure-dpi", type=int, default=FIGURE_RENDER_DPI, help="DPI for figure-level page rendering.")
    return p


def ensure_record(input_value: str) -> dict:
    record = maybe_load_json_record(input_value)
    if record is None:
        raise SystemExit("extract_pdf_assets.py requires a JSON acquisition artifact.")
    return dict(require_accepted_fetch_artifact(record, "extract_pdf_assets.py"))


def save_image_bytes(path: Path, data: bytes) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    path.write_bytes(data)


def ocr_page(page, dpi: int) -> str:
    if fitz is None or pytesseract is None or Image is None:
        return ""
    scale = dpi / 72.0
    matrix = fitz.Matrix(scale, scale)
    pix = page.get_pixmap(matrix=matrix, alpha=False)
    image = Image.open(io.BytesIO(pix.tobytes("png")))
    return normalize_whitespace(pytesseract.image_to_string(image))


def extract_page_images(doc, page, page_number: int, images_dir: Path) -> list[dict]:
    """Legacy xref-level extraction."""
    assets: list[dict] = []
    seen_xrefs = set()
    for image_index, image_info in enumerate(page.get_images(full=True), start=1):
        if not image_info:
            continue
        xref = int(image_info[0])
        if xref in seen_xrefs:
            continue
        seen_xrefs.add(xref)
        extracted = doc.extract_image(xref)
        image_bytes = extracted.get("image")
        if not image_bytes:
            continue
        ext = normalize_whitespace(str(extracted.get("ext", "png"))).lower() or "png"
        filename = f"page_{page_number:03d}_img_{image_index:02d}.{ext}"
        output_path = images_dir / filename
        save_image_bytes(output_path, image_bytes)
        assets.append(
            {
                "page_number": page_number,
                "image_index": image_index,
                "xref": xref,
                "filename": filename,
                "path": str(output_path),
                "ext": ext,
                "width": extracted.get("width", 0),
                "height": extracted.get("height", 0),
                "colorspace": extracted.get("colorspace", 0),
                "size_bytes": len(image_bytes),
                "extraction_level": "xref",
            }
        )
    return assets


# ---------------------------------------------------------------------------
# Figure-level extraction: caption-anchored page-render cropping
# ---------------------------------------------------------------------------

def _find_caption_blocks(page) -> list[dict]:
    """Return caption anchors sorted top-to-bottom by their y0 coordinate.

    Each anchor contains the full multi-line caption bbox so that the
    downstream crop includes the entire caption text, not just the first line.

    Each anchor::

        {
            "label": "Figure 3",
            "kind": "figure" | "table",
            "bbox": (x0, y0, x1, y1),
            "line_text": ...,
        }
    """
    anchors: list[dict] = []
    blocks = page.get_text("dict", flags=fitz.TEXT_PRESERVE_WHITESPACE)["blocks"]
    for block in blocks:
        if block.get("type") != 0:
            continue
        lines = block.get("lines", [])
        for line_idx, line in enumerate(lines):
            spans = line.get("spans", [])
            if not spans:
                continue
            line_text = "".join(s.get("text", "") for s in spans).strip()
            match = CAPTION_RE.match(line_text)
            if not match or _caption_match_is_inline_reference(line_text, match):
                continue
            label = normalize_whitespace(match.group(1))
            kind = _classify_caption_kind(label)

            caption_lines_text = [line_text]
            first_bbox = line["bbox"]
            x0, y0, x1, y1 = first_bbox
            prev_line_bottom = first_bbox[3]
            line_height = max(first_bbox[3] - first_bbox[1], 6.0)

            for cont_line in lines[line_idx + 1:]:
                cont_spans = cont_line.get("spans", [])
                if not cont_spans:
                    break
                cont_text = "".join(s.get("text", "") for s in cont_spans).strip()
                if not cont_text:
                    break
                if CAPTION_RE.match(cont_text):
                    break
                if _looks_like_data_row(cont_text):
                    break
                cb = cont_line["bbox"]
                # Stop merging if the next line is too far below the previous one
                # (it is then a separate paragraph, not a caption continuation).
                if cb[1] - prev_line_bottom > line_height * 1.6:
                    break
                x0 = min(x0, cb[0])
                y1 = max(y1, cb[3])
                x1 = max(x1, cb[2])
                prev_line_bottom = cb[3]
                caption_lines_text.append(cont_text)

            full_caption = " ".join(caption_lines_text)
            anchors.append({
                "label": label,
                "kind": kind,
                "bbox": (x0, y0, x1, y1),
                "label_bbox": tuple(first_bbox),
                "line_text": full_caption,
            })
    anchors.sort(key=lambda a: a["bbox"][1])
    return anchors


def _caption_match_is_inline_reference(line_text: str, match: re.Match[str]) -> bool:
    """Reject prose references such as "Table 8 summarizes ..."."""
    tail = line_text[match.end():]
    if not tail or not tail[0].isspace():
        return False
    rest = tail.strip()
    if not rest:
        return False
    first_word = re.match(r"[A-Za-z]+", rest)
    return bool(first_word and first_word.group(0)[0].islower())


def _collect_xref_rects(page) -> list[tuple[float, float, float, float]]:
    """Gather the page-level bounding boxes of all embedded images."""
    rects: list[tuple[float, float, float, float]] = []
    for img_info in page.get_images(full=True):
        xref = int(img_info[0])
        try:
            img_rects = page.get_image_rects(xref)
        except Exception:
            continue
        for r in img_rects:
            if r.is_empty or r.is_infinite:
                continue
            rects.append((r.x0, r.y0, r.x1, r.y1))
    return rects


def _collect_drawing_rects(page) -> list[tuple[float, float, float, float]]:
    """Gather bounding boxes of vector drawings on the page."""
    rects: list[tuple[float, float, float, float]] = []
    try:
        for drawing in page.get_drawings():
            r = drawing.get("rect")
            if r is None:
                continue
            rect = fitz.Rect(r)
            if rect.is_empty or rect.is_infinite:
                continue
            if rect.width < 10 or rect.height < 10:
                continue
            rects.append((rect.x0, rect.y0, rect.x1, rect.y1))
    except Exception:
        pass
    return rects


def _visual_signal_for_bbox(page, bbox: tuple[float, float, float, float]) -> tuple[int, float]:
    """Return visual rect count and visual-area ratio inside a crop."""
    crop_area = _rect_area(bbox)
    if crop_area <= 0:
        return 0, 0.0
    rects = _collect_xref_rects(page) + _collect_drawing_rects(page)
    count = 0
    visual_area = 0.0
    for rect in rects:
        area = _intersection_area(rect, bbox)
        if area <= 0:
            continue
        count += 1
        visual_area += area
    return count, min(1.0, visual_area / crop_area)


def _find_body_text_blocks(page) -> list[tuple[float, float, float, float, str]]:
    """Return bounding boxes of body-text blocks (non-caption) sorted top-to-bottom."""
    results: list[tuple[float, float, float, float, str]] = []
    blocks = page.get_text("dict", flags=fitz.TEXT_PRESERVE_WHITESPACE)["blocks"]
    for block in blocks:
        if block.get("type") != 0:
            continue
        full_text = ""
        for line in block.get("lines", []):
            for span in line.get("spans", []):
                full_text += span.get("text", "")
        full_text = full_text.strip()
        if len(full_text) < 40:
            continue
        if CAPTION_RE.match(full_text):
            continue
        bb = block["bbox"]
        results.append((bb[0], bb[1], bb[2], bb[3], full_text))
    results.sort(key=lambda b: b[1])
    return results


def _find_paragraph_blocks(page, *, min_chars: int = 200) -> list[tuple[float, float, float, float, str]]:
    """Return only large prose blocks that look like running paragraphs.

    PyMuPDF often groups an entire tabular column ("DS-Ulysses 629.9 418.3 ...")
    into a single text block, so the legacy ``_find_body_text_blocks`` filter
    catches table cells too aggressively.  For deciding whether we have walked
    out of a table region we want a stricter notion: only blocks whose total
    text mass and line count look like real prose count as paragraph blocks.
    """
    results: list[tuple[float, float, float, float, str]] = []
    blocks = page.get_text("dict", flags=fitz.TEXT_PRESERVE_WHITESPACE)["blocks"]
    for block in blocks:
        if block.get("type") != 0:
            continue
        lines = block.get("lines", [])
        full_text = ""
        for line in lines:
            for span in line.get("spans", []):
                full_text += span.get("text", "")
        full_text = full_text.strip()
        if len(full_text) < min_chars:
            continue
        if CAPTION_RE.match(full_text):
            continue
        # Real prose paragraphs have many lines and few numeric-heavy lines.
        if len(lines) < 3:
            continue
        numeric_line_share = 0
        for line in lines:
            line_text = "".join(s.get("text", "") for s in line.get("spans", [])).strip()
            if _looks_like_data_row(line_text):
                numeric_line_share += 1
        if numeric_line_share > len(lines) * 0.4:
            continue
        bb = block["bbox"]
        results.append((bb[0], bb[1], bb[2], bb[3], full_text))
    results.sort(key=lambda b: b[1])
    return results


def _count_paragraph_text_chars_in_bbox(
    page,
    bbox: tuple[float, float, float, float],
    caption_bbox: tuple[float, float, float, float],
) -> int:
    """Count prose-like text intersecting a crop, excluding the caption area."""
    chars = 0
    blocks = page.get_text("dict", flags=fitz.TEXT_PRESERVE_WHITESPACE)["blocks"]
    for block in blocks:
        if block.get("type") != 0:
            continue
        bb = tuple(block["bbox"])
        if not _rects_intersect(bb, bbox):
            continue
        if _intersection_area(bb, caption_bbox) / max(_rect_area(bb), 1.0) > 0.6:
            continue

        lines = block.get("lines", [])
        line_texts = [
            "".join(s.get("text", "") for s in line.get("spans", [])).strip()
            for line in lines
        ]
        line_texts = [text for text in line_texts if text]
        full_text = normalize_whitespace(" ".join(line_texts))
        if len(full_text) < 80:
            continue
        if CAPTION_RE.match(full_text):
            continue
        numeric_rows = sum(1 for text in line_texts if _looks_like_data_row(text))
        if line_texts and numeric_rows > len(line_texts) * 0.4:
            continue
        chars += len(full_text)
    return chars


def _other_caption_labels_for_crop(
    caption_anchors: list[dict],
    current_anchor: dict,
    bbox: tuple[float, float, float, float],
) -> list[str]:
    """Return other caption labels substantially covered by this crop."""
    labels: list[str] = []
    current_label = normalize_whitespace(str(current_anchor.get("label", "")))
    current_bbox = tuple(current_anchor.get("bbox", ()))

    for anchor in caption_anchors:
        label = normalize_whitespace(str(anchor.get("label", "")))
        anchor_bbox = tuple(anchor.get("bbox", ()))
        label_bbox = tuple(anchor.get("label_bbox", anchor_bbox))
        if not label or len(anchor_bbox) != 4:
            continue
        if label == current_label and anchor_bbox == current_bbox:
            continue

        overlap = _intersection_area(anchor_bbox, bbox)
        label_overlap = _intersection_area(label_bbox, bbox) if len(label_bbox) == 4 else 0.0
        if overlap <= 0 and label_overlap <= 0:
            continue
        caption_overlap_ratio = overlap / max(_rect_area(anchor_bbox), 1.0)
        label_overlap_ratio = label_overlap / max(_rect_area(label_bbox), 1.0) if len(label_bbox) == 4 else 0.0
        if caption_overlap_ratio < 0.5 and label_overlap_ratio < 0.15:
            continue
        labels.append(label)

    return sorted(set(labels))


def _quality_signals_for_crop(
    page,
    kind: str,
    bbox: tuple[float, float, float, float],
    caption_anchor: dict,
    page_rect,
    *,
    table_body_rows: int,
    caption_anchors: list[dict] | None = None,
) -> dict:
    page_area = _rect_area((page_rect.x0, page_rect.y0, page_rect.x1, page_rect.y1))
    page_coverage_ratio = _rect_area(bbox) / page_area if page_area > 0 else 0.0
    visual_rect_count, visual_body_ratio = _visual_signal_for_bbox(page, bbox)
    caption_bbox = tuple(caption_anchor["bbox"])
    paragraph_text_chars = _count_paragraph_text_chars_in_bbox(page, bbox, caption_bbox)
    caption_text_chars = len(normalize_whitespace(str(caption_anchor.get("line_text", ""))))
    other_caption_labels = _other_caption_labels_for_crop(caption_anchors or [], caption_anchor, bbox)
    return _classify_visual_quality(
        kind=kind,
        page_coverage_ratio=page_coverage_ratio,
        visual_rect_count=visual_rect_count,
        visual_body_ratio=visual_body_ratio,
        paragraph_text_chars=paragraph_text_chars,
        table_body_rows=table_body_rows,
        caption_text_chars=caption_text_chars,
        other_caption_labels=other_caption_labels,
    )


def _clip_to_page(
    bbox: tuple[float, float, float, float],
    page_rect,
    *,
    padding: float = 4.0,
) -> tuple[float, float, float, float]:
    x0, y0, x1, y1 = bbox
    x0 = max(page_rect.x0, x0 - padding)
    y0 = max(page_rect.y0, y0 - padding)
    x1 = min(page_rect.x1, x1 + padding)
    y1 = min(page_rect.y1, y1 + padding)
    return (x0, y0, x1, y1)


def _estimate_figure_bbox_above_caption(
    page,
    caption_anchor: dict,
    prev_anchor: dict | None,
    page_rect,
) -> tuple[float, float, float, float] | None:
    """Estimate the bounding box of the figure that lives ABOVE its caption.

    Strategy:
    1. Collect all xref image rects and vector drawing rects on the page.
    2. Keep only those rects whose vertical centre is between the previous
       boundary (top of page or previous caption) and the current caption.
    3. Union them and expand slightly for padding.
    4. If no rects are found (pure-text or OCR page), use the region between
       the nearest body-text block above and the caption.
    """
    caption_y_top = caption_anchor["bbox"][1]
    upper_bound = 0.0
    if prev_anchor is not None:
        upper_bound = prev_anchor["bbox"][3] + 2.0

    img_rects = _collect_xref_rects(page)
    draw_rects = _collect_drawing_rects(page)
    all_rects = img_rects + draw_rects

    relevant: list[tuple[float, float, float, float]] = []
    for r in all_rects:
        ry_mid = (r[1] + r[3]) / 2.0
        if upper_bound <= ry_mid <= caption_y_top + 5:
            clipped_y1 = min(r[3], caption_y_top - 2.0)
            if clipped_y1 > r[1]:
                relevant.append((r[0], r[1], r[2], clipped_y1))

    if relevant:
        active_bbox = (
            min(r[0] for r in relevant),
            min(r[1] for r in relevant),
            max(r[2] for r in relevant),
            max(r[3] for r in relevant),
        )
        for line in _collect_text_lines(page):
            text = normalize_whitespace(str(line.get("text", "")))
            bb = tuple(line.get("bbox", ()))
            if (
                len(bb) != 4
                or len(text) > 80
                or CAPTION_RE.match(text)
                or bb[1] >= caption_y_top - 1.0
                or bb[3] <= upper_bound
            ):
                continue
            horizontal_gap = max(active_bbox[0] - bb[2], bb[0] - active_bbox[2], 0.0)
            vertical_gap = max(active_bbox[1] - bb[3], bb[1] - active_bbox[3], 0.0)
            if (vertical_gap == 0.0 and horizontal_gap <= 36.0) or (
                horizontal_gap == 0.0 and vertical_gap <= 12.0
            ):
                accepted_bbox = (bb[0], bb[1], bb[2], min(bb[3], caption_y_top - 2.0))
                relevant.append(accepted_bbox)
                active_bbox = (
                    min(active_bbox[0], accepted_bbox[0]),
                    min(active_bbox[1], accepted_bbox[1]),
                    max(active_bbox[2], accepted_bbox[2]),
                    max(active_bbox[3], accepted_bbox[3]),
                )
        x0 = min(r[0] for r in relevant)
        y0 = min(r[1] for r in relevant)
        x1 = max(r[2] for r in relevant)
        y1 = max(r[3] for r in relevant)
    else:
        body_blocks = _find_body_text_blocks(page)
        nearest_above_y = upper_bound
        for bb in body_blocks:
            if bb[3] < caption_y_top - 5 and bb[3] > nearest_above_y:
                nearest_above_y = bb[3]
        y0 = nearest_above_y + 2.0
        x0 = page_rect.x0
        x1 = page_rect.x1
        y1 = caption_y_top - 2.0

    bbox = _clip_to_page((x0, y0, x1, y1), page_rect)
    bbox = (bbox[0], bbox[1], bbox[2], min(bbox[3], caption_y_top - 1.0))
    width = bbox[2] - bbox[0]
    height = bbox[3] - bbox[1]
    if width < MIN_FIGURE_WIDTH_PT or height < MIN_FIGURE_HEIGHT_PT:
        return None
    return bbox


def _collect_text_lines(page) -> list[dict]:
    """Return per-line records sorted top-to-bottom.

    Each record::

        {"bbox": (x0, y0, x1, y1), "text": str}
    """
    lines_out: list[dict] = []
    blocks = page.get_text("dict", flags=fitz.TEXT_PRESERVE_WHITESPACE)["blocks"]
    for block in blocks:
        if block.get("type") != 0:
            continue
        for line in block.get("lines", []):
            spans = line.get("spans", [])
            if not spans:
                continue
            text = "".join(s.get("text", "") for s in spans).strip()
            if not text:
                continue
            lines_out.append({"bbox": tuple(line["bbox"]), "text": text})
    lines_out.sort(key=lambda r: r["bbox"][1])
    return lines_out


def _cluster_lines_into_rows(
    lines: list[dict], *, y_tolerance: float = 2.0
) -> list[dict]:
    """Cluster sibling text lines that share roughly the same vertical band.

    PDFs created by LaTeX often emit one PyMuPDF "line" per cell, so a single
    visual row of a table is split into many independent line records.  We
    merge lines whose ``y0`` falls within ``y_tolerance`` points of the row
    seed so that downstream heuristics can reason about a true logical row.

    Each output record::

        {
            "bbox": (x0, y0, x1, y1),  # union of all member bboxes
            "tokens": [str, ...],       # text content of each member, left-to-right
            "text": str,                # tokens joined by single spaces
            "members": [dict, ...],     # original line records inside the row
        }
    """
    rows: list[dict] = []
    sorted_lines = sorted(lines, key=lambda r: (r["bbox"][1], r["bbox"][0]))
    for line in sorted_lines:
        bx0, by0, bx1, by1 = line["bbox"]
        placed = False
        for row in rows:
            rx0, ry0, rx1, ry1 = row["bbox"]
            row_mid = (ry0 + ry1) / 2.0
            line_mid = (by0 + by1) / 2.0
            if abs(line_mid - row_mid) <= y_tolerance:
                row["bbox"] = (
                    min(rx0, bx0),
                    min(ry0, by0),
                    max(rx1, bx1),
                    max(ry1, by1),
                )
                row["members"].append(line)
                placed = True
                break
        if not placed:
            rows.append({
                "bbox": (bx0, by0, bx1, by1),
                "members": [line],
            })

    for row in rows:
        row["members"].sort(key=lambda m: m["bbox"][0])
        row["tokens"] = [m["text"] for m in row["members"]]
        row["text"] = " ".join(row["tokens"])
    rows.sort(key=lambda r: r["bbox"][1])
    return rows


def _row_is_table_like(row: dict) -> bool:
    """A logical row that looks like part of a data table.

    The row qualifies if it has many short tokens (typical for tabular cells).
    Either:
    - many independent cells (≥ 3 separate line members), or
    - a single text whose tokens look like numeric or textual table body cells.
    """
    members = row.get("members", [])
    text = row.get("text", "")
    if len(members) >= 3:
        # Many separated cells: the typical case for LaTeX-rendered tables
        # where every cell becomes its own PyMuPDF line.
        return True
    return _looks_like_data_row(text) or _looks_like_text_table_row(text)


def _restrict_row_to_caption_column(row: dict, caption_bbox: tuple[float, float, float, float], page_rect) -> dict | None:
    """Keep only row cells in the same page column as a narrow caption.

    In two-column papers, unrelated left/right column tables often share the
    same y bands. Row clustering can merge them unless we trim by caption side.
    """
    members = list(row.get("members", []) or [])
    if not members:
        return row

    page_x0 = float(getattr(page_rect, "x0", 0.0))
    page_x1 = float(getattr(page_rect, "x1", 0.0))
    page_width = max(1.0, page_x1 - page_x0)
    page_mid = (page_x0 + page_x1) / 2.0
    cx0, _, cx1, _ = caption_bbox
    caption_width = cx1 - cx0
    if caption_width >= page_width * 0.45:
        return row

    caption_mid = (cx0 + cx1) / 2.0
    caption_is_left = caption_mid < page_mid
    expanded_caption_x0 = cx0 - 24.0
    expanded_caption_x1 = cx1 + 24.0

    def overlaps_caption_band(x0: float, x1: float) -> bool:
        overlap = max(0.0, min(x1, expanded_caption_x1) - max(x0, expanded_caption_x0))
        width = max(1.0, x1 - x0)
        return overlap >= min(16.0, width * 0.25)

    rx0, _, rx1, _ = row["bbox"]
    row_mid = (rx0 + rx1) / 2.0
    row_same_side = (row_mid < page_mid) == caption_is_left
    row_overlaps_caption_band = overlaps_caption_band(rx0, rx1)
    if not (rx0 < page_mid - 10.0 and rx1 > page_mid + 10.0):
        return row if row_same_side or row_overlaps_caption_band else None

    filtered = []
    for member in members:
        mx0, _, mx1, _ = member["bbox"]
        member_mid = (mx0 + mx1) / 2.0
        same_side = (member_mid < page_mid) == caption_is_left
        if same_side or overlaps_caption_band(mx0, mx1):
            filtered.append(member)

    if not filtered:
        return None
    if len(filtered) == len(members):
        return row

    x0 = min(member["bbox"][0] for member in filtered)
    y0 = min(member["bbox"][1] for member in filtered)
    x1 = max(member["bbox"][2] for member in filtered)
    y1 = max(member["bbox"][3] for member in filtered)
    tokens = [member["text"] for member in sorted(filtered, key=lambda m: m["bbox"][0])]
    return {
        "bbox": (x0, y0, x1, y1),
        "members": sorted(filtered, key=lambda m: m["bbox"][0]),
        "tokens": tokens,
        "text": " ".join(tokens),
    }


def _line_is_inside_any_block(
    line_bbox: tuple[float, float, float, float],
    blocks: list[tuple[float, float, float, float, str]],
) -> bool:
    for bb in blocks:
        if (
            line_bbox[0] >= bb[0] - 0.5
            and line_bbox[1] >= bb[1] - 0.5
            and line_bbox[2] <= bb[2] + 0.5
            and line_bbox[3] <= bb[3] + 0.5
        ):
            return True
    return False


def _grow_table_region(
    page,
    caption_anchor: dict,
    rows: list[dict],
    paragraph_blocks: list[tuple[float, float, float, float, str]],
    *,
    direction: str,
    upper_bound: float,
    lower_bound: float,
) -> tuple[list[tuple[float, float, float, float]], int]:
    """Walk away from the caption in ``direction`` ('up' or 'down') and collect
    logical rows that look like part of a tabular layout.

    Returns the list of accepted row bboxes (caption excluded) and the number
    of rows confirmed as table body rows.  The caller decides which direction
    wins.
    """
    caption_y0 = caption_anchor["bbox"][1]
    caption_y1 = caption_anchor["bbox"][3]

    accepted: list[tuple[float, float, float, float]] = []
    data_row_count = 0
    consecutive_non_data = 0
    seen_data = False
    last_accepted_edge = caption_y1 if direction == "down" else caption_y0

    if direction == "down":
        candidates = [r for r in rows if r["bbox"][1] > caption_y1 + 0.5]
        candidates.sort(key=lambda r: r["bbox"][1])
        boundary_check = lambda ly0, ly1: ly1 >= lower_bound
    else:
        candidates = [r for r in rows if r["bbox"][3] < caption_y0 - 0.5]
        candidates.sort(key=lambda r: r["bbox"][3], reverse=True)
        boundary_check = lambda ly0, ly1: ly0 <= upper_bound

    for row in candidates:
        restricted_row = _restrict_row_to_caption_column(row, caption_anchor["bbox"], page.rect)
        if restricted_row is None:
            continue
        row = restricted_row
        rx0, ry0, rx1, ry1 = row["bbox"]
        if boundary_check(ry0, ry1):
            break
        text = row["text"]
        is_table_row = _row_is_table_like(row)
        in_paragraph_block = _line_is_inside_any_block(row["bbox"], paragraph_blocks)
        row_gap = ry0 - last_accepted_edge if direction == "down" else last_accepted_edge - ry1
        row_height = max(ry1 - ry0, 1.0)
        if seen_data and row_gap > max(12.0, row_height * 1.8) and not is_table_row:
            break
        if (
            seen_data
            and len(row.get("members", []) or []) <= 1
            and in_paragraph_block
            and not _looks_like_data_row(text)
        ):
            break
        # If this row sits entirely inside a real prose paragraph and does not
        # look table-shaped, we have walked out of the table.
        if not is_table_row and (in_paragraph_block or len(text) > 200):
            break
        if is_table_row:
            consecutive_non_data = 0
            data_row_count += 1
            seen_data = True
        else:
            consecutive_non_data += 1
            # Header / footnote rows are allowed but we should not collect an
            # unbounded run of them when no real data has been seen yet.
            if consecutive_non_data > 4 and not seen_data:
                break
            if consecutive_non_data > 8:
                break
        accepted.append(row["bbox"])
        last_accepted_edge = ry1 if direction == "down" else ry0

    return accepted, data_row_count


def _finalize_table_bbox(
    page,
    caption_anchor: dict,
    extra_rects: list[tuple[float, float, float, float]],
    page_rect,
    *,
    direction: str,
) -> tuple[float, float, float, float] | None:
    if not extra_rects:
        return None
    _, caption_y0, _, caption_y1 = caption_anchor["bbox"]
    accepted: list[tuple[float, float, float, float]] = list(extra_rects)

    y0 = min(b[1] for b in accepted)
    y1 = max(b[3] for b in accepted)
    initial_x0 = min(b[0] for b in accepted)
    initial_x1 = max(b[2] for b in accepted)
    for r in _collect_drawing_rects(page):
        ry_mid = (r[1] + r[3]) / 2.0
        if y0 - 4.0 <= ry_mid <= y1 + 4.0 and r[2] >= initial_x0 - 12.0 and r[0] <= initial_x1 + 12.0:
            accepted.append(r)
    for r in _collect_xref_rects(page):
        ry_mid = (r[1] + r[3]) / 2.0
        if y0 - 4.0 <= ry_mid <= y1 + 4.0 and r[2] >= initial_x0 - 12.0 and r[0] <= initial_x1 + 12.0:
            accepted.append(r)

    x0 = min(b[0] for b in accepted)
    y0 = min(b[1] for b in accepted)
    x1 = max(b[2] for b in accepted)
    y1 = max(b[3] for b in accepted)

    bbox = _clip_to_page((x0, y0, x1, y1), page_rect, padding=6.0)
    if direction == "down":
        bbox = (bbox[0], max(bbox[1], caption_y1 + 1.0), bbox[2], bbox[3])
    else:
        bbox = (bbox[0], bbox[1], bbox[2], min(bbox[3], caption_y0 - 1.0))
    width = bbox[2] - bbox[0]
    height = bbox[3] - bbox[1]
    if width < MIN_FIGURE_WIDTH_PT or height < MIN_FIGURE_HEIGHT_PT:
        return None
    return bbox


def _estimate_table_bbox(
    page,
    caption_anchor: dict,
    prev_anchor: dict | None,
    next_anchor: dict | None,
    page_rect,
) -> tuple[float, float, float, float] | None:
    result = _estimate_table_bbox_with_rows(page, caption_anchor, prev_anchor, next_anchor, page_rect)
    return result[0] if result is not None else None


def _estimate_table_bbox_with_rows(
    page,
    caption_anchor: dict,
    prev_anchor: dict | None,
    next_anchor: dict | None,
    page_rect,
) -> tuple[tuple[float, float, float, float], int] | None:
    r"""Estimate the bounding box of a table.

    Tables in academic papers come in two layouts:

    - caption-on-top: ``\caption`` precedes ``\begin{tabular}``;
    - caption-on-bottom: tabular body precedes ``\caption``.

    LaTeX makes both common, and within a single paper both forms can mix
    (e.g. wide tables placed with ``[t]`` vs. ``[b]``).  We therefore probe
    both directions and pick the side with strictly more confirmed table body
    rows.  Ties go to the downward side, matching the most common ACM / IEEE
    template defaults.

    Tables are usually pure text + thin separator lines, so the page rendering
    of just the union of text-line bboxes is sufficient.  We additionally
    union any drawing rects (``\hline``, frames) and image rects that fall in
    the same y-range, in case the paper places company-logo plots inside a
    table cell.
    """
    caption_y1 = caption_anchor["bbox"][3]

    upper_bound = page_rect.y0
    if prev_anchor is not None:
        upper_bound = max(page_rect.y0, prev_anchor["bbox"][3] + 2.0)

    lower_bound = page_rect.y1
    if next_anchor is not None:
        lower_bound = max(caption_y1 + 1.0, next_anchor["bbox"][1] - 2.0)

    text_lines = _collect_text_lines(page)
    rows = _cluster_lines_into_rows(text_lines)
    paragraph_blocks = _find_paragraph_blocks(page)

    down_lines, down_data = _grow_table_region(
        page,
        caption_anchor,
        rows,
        paragraph_blocks,
        direction="down",
        upper_bound=upper_bound,
        lower_bound=lower_bound,
    )
    up_lines, up_data = _grow_table_region(
        page,
        caption_anchor,
        rows,
        paragraph_blocks,
        direction="up",
        upper_bound=upper_bound,
        lower_bound=lower_bound,
    )

    if down_data == 0 and up_data == 0:
        return None

    chosen: list[tuple[float, float, float, float]]
    chosen_data_rows: int
    if up_data > down_data:
        chosen = up_lines
        chosen_data_rows = up_data
        direction = "up"
    else:
        chosen = down_lines
        chosen_data_rows = down_data
        direction = "down"

    bbox = _finalize_table_bbox(
        page,
        caption_anchor,
        chosen,
        page_rect,
        direction=direction,
    )
    if bbox is None:
        return None
    return bbox, chosen_data_rows


def _render_crop(page, bbox: tuple[float, float, float, float], dpi: int) -> bytes:
    """Render a page region to PNG bytes at the given DPI."""
    clip = fitz.Rect(*bbox)
    scale = dpi / 72.0
    matrix = fitz.Matrix(scale, scale)
    pix = page.get_pixmap(matrix=matrix, clip=clip, alpha=False)
    return pix.tobytes("png")


def _unique_figure_asset_filename(page_number: int, label: str, used_filenames: set[str]) -> str:
    safe_label = re.sub(r"[^a-zA-Z0-9]+", "_", label.lower()).strip("_") or "unlabeled"
    base = f"page_{page_number:03d}_fig_{safe_label}.png"
    if base not in used_filenames:
        used_filenames.add(base)
        return base

    stem = base.removesuffix(".png")
    suffix = 2
    while True:
        candidate = f"{stem}_{suffix}.png"
        if candidate not in used_filenames:
            used_filenames.add(candidate)
            return candidate
        suffix += 1


def extract_figure_regions(
    page, page_number: int, images_dir: Path, *, dpi: int = FIGURE_RENDER_DPI
) -> list[dict]:
    """Detect figure/table captions and crop the corresponding visual region."""
    if fitz is None:
        return []

    anchors = _find_caption_blocks(page)
    if not anchors:
        return []

    page_rect = page.rect
    assets: list[dict] = []
    used_filenames: set[str] = set()

    for idx, anchor in enumerate(anchors):
        prev_anchor = anchors[idx - 1] if idx > 0 else None
        next_anchor = anchors[idx + 1] if idx + 1 < len(anchors) else None
        kind = anchor.get("kind", "figure")

        bbox: tuple[float, float, float, float] | None
        table_body_rows = 0
        if kind == "table":
            table_result = _estimate_table_bbox_with_rows(page, anchor, prev_anchor, next_anchor, page_rect)
            if table_result is not None:
                bbox, table_body_rows = table_result
            else:
                bbox = None
            if bbox is None:
                # Fall back to the figure-shape estimator in case the table is
                # actually rendered as an embedded image.
                bbox = _estimate_figure_bbox_above_caption(page, anchor, prev_anchor, page_rect)
        else:
            bbox = _estimate_figure_bbox_above_caption(page, anchor, prev_anchor, page_rect)
            if bbox is None:
                bbox = _estimate_table_bbox(page, anchor, prev_anchor, next_anchor, page_rect)

        if bbox is None:
            continue

        label = anchor["label"]
        filename = _unique_figure_asset_filename(page_number, label, used_filenames)
        output_path = images_dir / filename

        try:
            png_bytes = _render_crop(page, bbox, dpi)
        except Exception:
            continue

        save_image_bytes(output_path, png_bytes)

        width_px = int((bbox[2] - bbox[0]) * dpi / 72.0)
        height_px = int((bbox[3] - bbox[1]) * dpi / 72.0)
        quality_signals = _quality_signals_for_crop(
            page,
            kind,
            bbox,
            anchor,
            page_rect,
            table_body_rows=table_body_rows,
            caption_anchors=anchors,
        )

        assets.append(
            {
                "page_number": page_number,
                "label": label,
                "kind": kind,
                "caption_text": normalize_whitespace(anchor["line_text"]),
                "filename": filename,
                "path": str(output_path),
                "ext": "png",
                "width": width_px,
                "height": height_px,
                "bbox_pt": list(bbox),
                "size_bytes": len(png_bytes),
                "extraction_level": "figure",
                "quality_signals": quality_signals,
            }
        )

    return assets


def main() -> None:
    args = parser().parse_args()
    record = ensure_record(args.input)
    pdf_path = Path(str(record.get("pdf_path", "")).strip()).expanduser()
    if not pdf_path.exists():
        from_fetch = maybe_load_json_record(args.input) or {}
        pdf_candidate = str(from_fetch.get("pdf_path", "")).strip()
        if pdf_candidate:
            pdf_path = Path(pdf_candidate).expanduser()
    if not pdf_path.exists():
        raise SystemExit("extract_pdf_assets.py requires a resolvable local PDF path.")
    if fitz is None:
        raise SystemExit("extract_pdf_assets.py requires PyMuPDF (`fitz`).")

    asset_root = Path(args.assets_dir).expanduser().resolve() if args.assets_dir else default_assets_dir(record)
    images_dir = asset_root / "images"
    previews_dir = asset_root / "page_previews"
    images_dir.mkdir(parents=True, exist_ok=True)

    figure_dpi = args.figure_dpi

    doc = fitz.open(pdf_path.resolve())
    page_records: list[dict] = []
    image_assets: list[dict] = []
    figure_assets: list[dict] = []
    asset_coverage: dict = {}
    try:
        total_pages = len(doc)
        page_limit = min(total_pages, args.max_pages)
        asset_coverage = {
            "total_pages": total_pages,
            "asset_max_pages": args.max_pages,
            "asset_pages_scanned": page_limit,
            "truncated_due_to_asset_page_limit": total_pages > args.max_pages,
        }
        for idx in range(page_limit):
            page = doc[idx]
            page_number = idx + 1
            text = normalize_whitespace(page.get_text("text"))
            searchable_chars = len(text)
            extraction_method = "text" if searchable_chars >= args.min_searchable_chars else "none"
            ocr_text = ""
            if extraction_method == "none":
                ocr_text = ocr_page(page, args.ocr_dpi)
                if ocr_text:
                    extraction_method = "ocr"
            page_images = extract_page_images(doc, page, page_number, images_dir)
            image_assets.extend(page_images)

            page_figures = extract_figure_regions(page, page_number, images_dir, dpi=figure_dpi)
            page_preview_path = ""
            if page_figures:
                preview_path = previews_dir / f"page_{page_number:03d}.png"
                save_image_bytes(
                    preview_path,
                    _render_crop(
                        page,
                        (page.rect.x0, page.rect.y0, page.rect.x1, page.rect.y1),
                        PAGE_PREVIEW_DPI,
                    ),
                )
                page_preview_path = str(preview_path)
                for figure in page_figures:
                    figure["page_preview_path"] = page_preview_path
            figure_assets.extend(page_figures)

            page_records.append(
                {
                    "page_number": page_number,
                    "searchable_text_chars": searchable_chars,
                    "text_extraction_method": extraction_method,
                    "ocr_used": extraction_method == "ocr",
                    "image_count": len(page_images),
                    "figure_count": len(page_figures),
                    "page_preview_path": page_preview_path,
                    "page_text": text or ocr_text,
                    "text_preview": (text or ocr_text)[:240],
                }
            )
    finally:
        doc.close()

    payload = {
        "status": "ok",
        "script": "extract_pdf_assets.py",
        "paper_id": record.get("paper_id", ""),
        "pdf_path": str(pdf_path.resolve()),
        "asset_root": str(asset_root),
        "images_dir": str(images_dir),
        "page_assets": page_records,
        "image_assets": image_assets,
        "figure_assets": figure_assets,
        "asset_coverage": asset_coverage,
        "ocr_available": bool(pytesseract and Image),
    }
    emit(payload, args.output)


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""Extract canonical raw source text and a compact source manifest from a PDF."""

from __future__ import annotations

import argparse
import hashlib
import json
import re
from pathlib import Path
from typing import Any

from common import (
    clean_pdf_line,
    emit,
    ensure_parent,
    extract_appendix_index,
    extract_caption_lines,
    file_sha256,
    fitz,
    match_section_heading,
    maybe_load_json_record,
    normalize_heading,
    normalize_whitespace,
    paper_id_for_record,
    pdf_coverage_summary,
    require_accepted_fetch_artifact,
    stop_section_reason,
)

MATH_SIGNAL_RE = re.compile(
    r"(?:\\(?:frac|sum|log|exp|argmax|argmin|mathbb|mathbf)|[$=]|[<>]=?|[∑∏≤≥≈∈]|O\([^)]+\))"
)


def parser() -> argparse.ArgumentParser:
    p = argparse.ArgumentParser(description=__doc__ or "extract source text")
    p.add_argument(
        "--input",
        required=True,
        help="Successful fetch JSON path or JSON artifact.",
    )
    p.add_argument("--output", default="", help="Source manifest JSON output path.")
    p.add_argument(
        "--raw-sections-output",
        default="",
        help="Canonical raw sections JSONL output path.",
    )
    p.add_argument("--full-text-output", default="", help="Optional derived Markdown output path.")
    p.add_argument(
        "--max-pages",
        type=int,
        default=None,
        help="Optional page limit. Omit for all pages.",
    )
    return p


def ensure_record(input_value: str) -> dict[str, Any]:
    record = maybe_load_json_record(input_value)
    if record is None:
        raise SystemExit("extract_source_text.py requires a JSON acquisition artifact.")
    return dict(require_accepted_fetch_artifact(record, "extract_source_text.py"))


def resolve_pdf_path(record: dict[str, Any]) -> Path | None:
    for key in ("pdf_path", "local_pdf_path"):
        value = normalize_whitespace(str(record.get(key, "")))
        if not value:
            continue
        path = Path(value).expanduser()
        if path.exists() and path.is_file():
            return path.resolve()
    return None


def section_id(base: str, seen: dict[str, int]) -> str:
    safe = re.sub(r"[^a-z0-9]+", "-", normalize_heading(base)).strip("-") or "section"
    seen[safe] = seen.get(safe, 0) + 1
    suffix = "" if seen[safe] == 1 else f"-{seen[safe]}"
    return f"sec:{safe}{suffix}"


def text_hash(text: str) -> str:
    return hashlib.sha256((text or "").encode("utf-8")).hexdigest()


def language_hint_for_text(text: str) -> str:
    cjk_chars = len(re.findall(r"[\u3400-\u9fff]", text or ""))
    latin_chars = len(re.findall(r"[A-Za-z]", text or ""))
    total = cjk_chars + latin_chars
    if total == 0:
        return "unknown"
    if cjk_chars / total >= 0.6:
        return "zh"
    if latin_chars / total >= 0.6:
        return "en"
    return "mixed"


def new_record(kind: str, title: str, page_number: int, seen: dict[str, int]) -> dict[str, Any]:
    sid = section_id(title or kind, seen)
    return {
        "record_type": "section",
        "section_id": sid,
        "kind": kind,
        "title": normalize_whitespace(title) or kind,
        "page_start": page_number,
        "page_end": page_number,
        "_lines": [],
    }


def finalize_section(record: dict[str, Any]) -> dict[str, Any] | None:
    lines = [line for line in record.pop("_lines", []) if normalize_whitespace(str(line))]
    text = "\n".join(lines).strip()
    if not text:
        return None
    record["text"] = text
    record["char_count"] = len(text)
    record["text_hash_sha256"] = text_hash(text)
    return record


def extract_page_texts(pdf_path: Path, max_pages: int | None) -> list[dict[str, Any]]:
    if fitz is None:
        raise RuntimeError("PyMuPDF is required for source text extraction.")
    doc = fitz.open(pdf_path)
    try:
        page_limit = len(doc) if max_pages is None else min(len(doc), max_pages)
        return [
            {"page": page_index + 1, "text": doc[page_index].get_text("text")}
            for page_index in range(page_limit)
        ]
    finally:
        doc.close()


def extract_raw_sections(page_texts: list[dict[str, Any]]) -> list[dict[str, Any]]:
    seen: dict[str, int] = {}
    sections: list[dict[str, Any]] = []
    current: dict[str, Any] | None = None

    for page in page_texts:
        page_number = int(page["page"])
        if current is None:
            current = new_record("preamble", "preamble", page_number, seen)

        for raw_line in str(page.get("text", "")).splitlines():
            line = clean_pdf_line(raw_line)
            if not line:
                continue
            stop_reason = stop_section_reason(line, allow_prefix=True)
            heading = match_section_heading(line)
            if stop_reason or (heading and heading != "stop"):
                finalized = finalize_section(current)
                if finalized is not None:
                    sections.append(finalized)
                kind = stop_reason or str(heading)
                current = new_record(kind, line, page_number, seen)
                continue
            current["page_end"] = page_number
            current.setdefault("_lines", []).append(line)

    if current is not None:
        finalized = finalize_section(current)
        if finalized is not None:
            sections.append(finalized)
    return sections


def section_ids_for_page(sections: list[dict[str, Any]], page_number: int) -> list[str]:
    ids = [
        str(section.get("section_id", ""))
        for section in sections
        if int(section.get("page_start", 0) or 0)
        <= page_number
        <= int(section.get("page_end", 0) or 0)
    ]
    return [sid for sid in ids if sid]


def build_pages(
    page_texts: list[dict[str, Any]],
    sections: list[dict[str, Any]],
) -> list[dict[str, Any]]:
    pages: list[dict[str, Any]] = []
    for page in page_texts:
        page_number = int(page["page"])
        text = str(page.get("text", ""))
        pages.append(
            {
                "page": page_number,
                "char_count": len(normalize_whitespace(text)),
                "section_ids": section_ids_for_page(sections, page_number),
            }
        )
    return pages


def primary_section_for_page(sections: list[dict[str, Any]], page_number: int) -> str:
    ids = section_ids_for_page(sections, page_number)
    return ids[0] if ids else ""


def caption_manifest(
    page_texts: list[dict[str, Any]],
    sections: list[dict[str, Any]],
) -> dict[str, list[dict[str, Any]]]:
    captions = {"figures": [], "tables": []}
    for page in page_texts:
        page_number = int(page["page"])
        section = primary_section_for_page(sections, page_number)
        for key, kind in (("figures", "figure"), ("tables", "table")):
            for item in extract_caption_lines(str(page.get("text", "")), kind):
                captions[key].append(
                    {
                        **item,
                        "page": page_number,
                        "pages": [page_number],
                        "section_id": section,
                    }
                )
    return captions


def math_index(sections: list[dict[str, Any]], *, max_items: int = 200) -> list[dict[str, Any]]:
    items: list[dict[str, Any]] = []
    for section in sections:
        for line in str(section.get("text", "")).splitlines():
            cleaned = normalize_whitespace(line)
            if not cleaned or len(cleaned) > 240 or not MATH_SIGNAL_RE.search(cleaned):
                continue
            items.append(
                {
                    "text": cleaned,
                    "section_id": section.get("section_id", ""),
                    "page_start": section.get("page_start"),
                    "page_end": section.get("page_end"),
                }
            )
            if len(items) >= max_items:
                return items
    return items


def write_jsonl(records: list[dict[str, Any]], output_path: str) -> None:
    ensure_parent(output_path)
    path = Path(output_path).expanduser().resolve()
    path.write_text(
        "\n".join(json.dumps(record, ensure_ascii=False) for record in records) + "\n",
        encoding="utf-8",
    )


def write_full_text_markdown(records: list[dict[str, Any]], output_path: str, title: str) -> None:
    ensure_parent(output_path)
    lines = [f"# {title or 'Full Source Text'}", ""]
    for record in records:
        lines.extend(
            [
                f"## {record.get('section_id', '')} {record.get('title', '')}".strip(),
                f"_Pages {record.get('page_start')}-{record.get('page_end')}_",
                "",
                str(record.get("text", "")).strip(),
                "",
            ]
        )
    Path(output_path).expanduser().resolve().write_text(
        "\n".join(lines).rstrip() + "\n",
        encoding="utf-8",
    )


def build_manifest(
    *,
    record: dict[str, Any],
    pdf_path: Path,
    page_texts: list[dict[str, Any]],
    sections: list[dict[str, Any]],
    raw_sections_output: str,
    full_text_output: str,
    max_pages: int | None,
) -> dict[str, Any]:
    coverage = pdf_coverage_summary(pdf_path, max_pages=max_pages)
    total_pages = coverage.get("total_pages")
    extracted_pages = len(page_texts)
    full_text = "\n".join(str(page.get("text", "")) for page in page_texts)
    source_coverage = {
        "total_pages": total_pages,
        "text_max_pages": max_pages,
        "text_pages_extracted": extracted_pages,
        "text_pages_scanned": extracted_pages,
        "text_truncated": bool(coverage.get("truncated_due_to_page_limit")),
        "truncated_due_to_page_limit": bool(coverage.get("truncated_due_to_page_limit")),
        "appendix_detected": bool(coverage.get("appendix_detected")),
        "appendix_start_page": coverage.get("appendix_start_page"),
        "references_start_page": coverage.get("references_start_page"),
    }
    manifest = {
        "status": "ok",
        "script": "extract_source_text.py",
        "schema_version": 1,
        "paper_id": record.get("paper_id") or paper_id_for_record(record),
        "title": record.get("title", ""),
        "source_sha256": record.get("source_sha256", ""),
        "source_kind": "pdf_text",
        "raw_sections_path": (
            str(Path(raw_sections_output).expanduser().resolve()) if raw_sections_output else ""
        ),
        "full_text_md_path": (
            str(Path(full_text_output).expanduser().resolve()) if full_text_output else ""
        ),
        "pdf": {
            "path": str(pdf_path),
            "total_pages": total_pages,
            "text_pages_extracted": extracted_pages,
            "text_max_pages": max_pages,
            "text_truncated": bool(coverage.get("truncated_due_to_page_limit")),
        },
        "coverage": source_coverage,
        "sections": [
            {
                key: section.get(key)
                for key in (
                    "section_id",
                    "kind",
                    "title",
                    "page_start",
                    "page_end",
                    "char_count",
                    "text_hash_sha256",
                )
            }
            for section in sections
        ],
        "pages": build_pages(page_texts, sections),
        "captions": caption_manifest(page_texts, sections),
        "math_index": math_index(sections),
        "appendix_index": extract_appendix_index(pdf_path, coverage),
        "language_hint": language_hint_for_text(full_text),
        "text_hash_sha256": text_hash(full_text),
    }
    if isinstance(record.get("archive_source"), dict):
        manifest["archive_source"] = record["archive_source"]
    if isinstance(record.get("identity_contract"), dict):
        manifest["identity_contract"] = record.get("identity_contract")
    if isinstance(record.get("source_manifestation"), dict):
        manifest["source_manifestation"] = record.get("source_manifestation")
    return manifest


def main() -> None:
    args = parser().parse_args()
    record = ensure_record(args.input)
    record["paper_id"] = record.get("paper_id") or paper_id_for_record(record)
    pdf_path = resolve_pdf_path(record)
    if pdf_path is None:
        raise SystemExit("extract_source_text.py requires a resolvable local PDF path.")
    source_sha256 = file_sha256(pdf_path)
    reported_source_sha256 = normalize_whitespace(str(record.get("source_sha256", "")))
    if reported_source_sha256 and reported_source_sha256 != source_sha256:
        raise SystemExit("extract_source_text.py source_sha256 does not match acquired PDF.")
    record["source_sha256"] = source_sha256

    raw_sections_output = args.raw_sections_output
    if not raw_sections_output and args.output:
        raw_sections_output = str(
            Path(args.output).with_name(
                Path(args.output).stem.replace("_source_manifest", "") + "_raw_sections.jsonl"
            )
        )

    page_texts = extract_page_texts(pdf_path, args.max_pages)
    sections = extract_raw_sections(page_texts)
    if raw_sections_output:
        write_jsonl(sections, raw_sections_output)
    if args.full_text_output:
        write_full_text_markdown(sections, args.full_text_output, str(record.get("title", "")))

    manifest = build_manifest(
        record=record,
        pdf_path=pdf_path,
        page_texts=page_texts,
        sections=sections,
        raw_sections_output=raw_sections_output,
        full_text_output=args.full_text_output,
        max_pages=args.max_pages,
    )
    emit(manifest, args.output)


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""Acquire the best available PDF or equivalent full text for one paper."""

from __future__ import annotations

import argparse
import re
from pathlib import Path

from common import (
    canonical_identity_summary,
    default_pdf_path,
    emit,
    fetch_record_from_canonical_identity,
    file_sha256,
    http_get_bytes,
    maybe_load_json_record,
    paper_id_for_record,
    require_accepted_canonical_identity,
    require_ok_input_artifact,
    runtime_config,
)
from paper_archive import ARXIV, ArchiveError, find_source, verify_pdf, work_identity


def parser() -> argparse.ArgumentParser:
    p = argparse.ArgumentParser(description=__doc__ or "fetch pdf")
    p.add_argument(
        "--input", required=True, help="Metadata JSON path, JSON string, or raw paper reference."
    )
    p.add_argument("--output", default="", help="Output path for JSON status.")
    p.add_argument("--vault", default="", help="Obsidian Vault to search before downloading.")
    p.add_argument(
        "--reference", default="", help="Original user reference, including any explicit version."
    )
    p.add_argument(
        "--target-directory",
        default="",
        help="Selected existing directory when lookup is ambiguous.",
    )
    p.add_argument(
        "--source-sha256",
        default="",
        help="Selected archived source when versions are incomparable.",
    )
    p.add_argument("--dest-dir", default="", help="Directory for downloaded PDFs.")
    p.add_argument(
        "--identity",
        default="",
        help="Accepted Canonical Identity Artifact JSON path or JSON string.",
    )
    return p


def is_pdf_content(data: bytes) -> bool:
    return b"%PDF-" in data[:1024]


def main(argv: list[str] | None = None) -> None:
    args = parser().parse_args(argv)
    if not args.identity:
        raise SystemExit("fetch_pdf.py requires an accepted --identity artifact.")
    identity_summary: dict = {}
    source_manifestation: dict = {}
    input_record = maybe_load_json_record(args.input)
    if input_record is not None:
        require_ok_input_artifact(input_record, "fetch_pdf.py")
    identity_record = maybe_load_json_record(args.identity)
    if identity_record is None:
        raise SystemExit("fetch_pdf.py requires --identity to be a JSON artifact.")
    identity = require_accepted_canonical_identity(identity_record, "fetch_pdf.py")
    if not isinstance(identity.get("bound_sources"), list):
        raise SystemExit("fetch_pdf.py requires canonical identity bound_sources.")
    identity_summary = canonical_identity_summary(identity)
    source_manifestation = dict(identity_summary.get("source_manifestation", {}) or {})
    record = fetch_record_from_canonical_identity(identity)

    record["paper_id"] = record.get("paper_id") or paper_id_for_record(record)
    source_candidates = [
        (str(item.get("kind", "")), str(item.get("value", "")))
        for item in identity.get("bound_sources", []) or []
        if isinstance(item, dict)
        and item.get("kind") in {"local_pdf", "pdf_url"}
        and str(item.get("value", "")).strip()
    ]

    # Explicit local input stays authoritative; archive discovery applies to remote acquisition.
    archived_source = None
    if not any(kind == "local_pdf" for kind, _ in source_candidates):
        config = runtime_config(
            cli_overrides={
                "obsidian_vault": args.vault,
                "save_mode": "obsidian" if args.vault else "",
            }
        )
        vault = args.vault or (
            config.get("obsidian_vault", "") if config.get("save_mode") == "obsidian" else ""
        )
        if vault:
            try:
                archived_source = find_source(
                    Path(vault).expanduser().resolve(),
                    record,
                    args.reference or record.get("source_url", ""),
                    args.target_directory,
                    args.source_sha256,
                )
                if archived_source:
                    verify_pdf(Path(archived_source["pdf_path"]), record)
                    source_candidates = [("local_pdf", archived_source["pdf_path"])]
                    source_manifestation = {
                        **source_manifestation,
                        "source_kind": "local_pdf",
                        "local_pdf_path": archived_source["pdf_path"],
                        "source_url": archived_source.get("source_url", ""),
                        "pdf_url": archived_source.get("pdf_url", ""),
                        "arxiv_id": archived_source.get("arxiv_id", ""),
                    }
                    identity_summary["source_manifestation"] = source_manifestation
                    identity_summary["bound_sources"] = [
                        {
                            "kind": "local_pdf",
                            "value": archived_source["pdf_path"],
                            "source_sha256": archived_source["source_sha256"],
                            "provider": "verified_obsidian_archive",
                        }
                    ]
            except ArchiveError as exc:
                emit(
                    {
                        "status": "blocked",
                        "script": "fetch_pdf.py",
                        "conflict_code": exc.code,
                        "candidates": exc.paths,
                    },
                    args.output,
                )
                raise SystemExit(2) from exc

    reference = args.reference or str(record.get("source_url", ""))
    if re.fullmatch(r"\d{4}\.\d{4,5}v\d+", reference):
        reference = "arxiv:" + reference
    requested = ARXIV.search(reference)
    explicit_revision = requested[1] if requested and re.search(r"v\d+$", requested[1]) else ""
    if explicit_revision and not archived_source:
        # Refine an already-bound arXiv work to the revision explicitly requested by the user.
        base = re.sub(r"v\d+$", "", explicit_revision)
        bound_arxiv = [
            ARXIV.search(value) for kind, value in source_candidates if kind == "pdf_url"
        ]
        if work_identity(record)["identifiers"].get("arxiv") != base or not any(
            match and re.sub(r"v\d+$", "", match[1]) == base for match in bound_arxiv
        ):
            raise SystemExit("Requested revision is not bound to the accepted arXiv work.")
        url = f"https://arxiv.org/pdf/{explicit_revision}.pdf"
        source_candidates = [("pdf_url", url)]
        source_manifestation = {
            **source_manifestation,
            "arxiv_id": explicit_revision,
            "source_url": reference,
            "pdf_url": url,
        }
        identity_summary["source_manifestation"] = source_manifestation
        identity_summary["bound_sources"] = [
            {
                "kind": "pdf_url",
                "value": url,
                "provider": "arxiv",
                "binding_reason": "explicit_revision_of_accepted_work",
            }
        ]

    if not source_candidates:
        payload = {
            "status": "error",
            "script": "fetch_pdf.py",
            "paper_id": record["paper_id"],
            "title": record.get("title", ""),
            "error": "No accessible PDF source found.",
            "source_url": record.get("source_url", ""),
        }
        if identity_summary:
            payload["identity_contract"] = identity_summary
            payload["source_manifestation"] = source_manifestation
        emit(payload, args.output)
        raise SystemExit(1)

    attempted_sources: list[dict[str, str]] = []
    downloaded: tuple[str, str, bytes] | None = None
    for candidate_kind, candidate_value in source_candidates:
        if candidate_kind == "local_pdf":
            pdf_path = Path(candidate_value).expanduser()
            if not pdf_path.exists() or not pdf_path.is_file():
                attempted_sources.append(
                    {
                        "kind": "local_pdf",
                        "path": candidate_value,
                        "status": "missing_file",
                    }
                )
                continue
            pdf_path = pdf_path.resolve()
            payload = {
                "status": "ok",
                "script": "fetch_pdf.py",
                "paper_id": record["paper_id"],
                "title": record.get("title", ""),
                "pdf_path": str(pdf_path),
                "pdf_source": "local_pdf",
                "source_url": record.get("source_url", "") or str(pdf_path),
                "pdf_url": "",
                "source_sha256": file_sha256(pdf_path),
            }
            if archived_source:
                payload["archive_source"] = archived_source
                payload["pdf_source"] = "obsidian_archive"
                payload["source_url"] = archived_source.get("source_url", "")
            if identity_summary:
                payload["identity_contract"] = identity_summary
                payload["source_manifestation"] = source_manifestation
            if attempted_sources:
                payload["attempted_sources"] = attempted_sources
            emit(payload, args.output)
            return
        if candidate_kind != "pdf_url":
            continue
        try:
            data = http_get_bytes(candidate_value)
        except Exception as exc:
            attempted_sources.append(
                {"kind": candidate_kind, "url": candidate_value, "status": f"download_error:{exc}"}
            )
            continue
        if not is_pdf_content(data):
            attempted_sources.append(
                {"kind": candidate_kind, "url": candidate_value, "status": "not_pdf_content"}
            )
            continue
        downloaded = (candidate_kind, candidate_value, data)
        break

    if downloaded is None:
        payload = {
            "status": "error",
            "script": "fetch_pdf.py",
            "paper_id": record["paper_id"],
            "title": record.get("title", ""),
            "error": "No candidate URL returned PDF content.",
            "source_url": record.get("source_url", ""),
            "attempted_sources": attempted_sources,
        }
        if identity_summary:
            payload["identity_contract"] = identity_summary
            payload["source_manifestation"] = source_manifestation
        emit(payload, args.output)
        raise SystemExit(1)

    target_path = default_pdf_path(record, dest_dir=args.dest_dir)
    _, source_value, pdf_bytes = downloaded
    target_path.write_bytes(pdf_bytes)
    if explicit_revision:
        try:
            verify_pdf(
                target_path,
                {**record, "source_manifestation": source_manifestation},
                exact_version=True,
            )
        except ArchiveError as exc:
            target_path.unlink(missing_ok=True)
            emit(
                {"status": "blocked", "script": "fetch_pdf.py", "conflict_code": exc.code},
                args.output,
            )
            raise SystemExit(2) from exc
    payload = {
        "status": "ok",
        "script": "fetch_pdf.py",
        "paper_id": record["paper_id"],
        "title": record.get("title", ""),
        "pdf_path": str(target_path),
        "pdf_source": "downloaded",
        "source_url": record.get("source_url", ""),
        "pdf_url": source_value,
        "file_size": target_path.stat().st_size,
        "source_sha256": file_sha256(target_path),
    }
    if args.target_directory:
        payload["archive_source"] = {"target_directory": args.target_directory}
    if identity_summary:
        payload["identity_contract"] = identity_summary
        payload["source_manifestation"] = source_manifestation
    if attempted_sources:
        payload["attempted_sources"] = attempted_sources
    emit(payload, args.output)


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""Find a verified local source without changing the Vault."""

from __future__ import annotations

import argparse
import re
from pathlib import Path

from common import emit, maybe_load_json_record, runtime_config
from paper_archive import ArchiveError, find_source


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--input", required=True, help="Metadata JSON or original paper reference.")
    parser.add_argument("--vault", default="")
    parser.add_argument(
        "--reference", default="", help="Original user reference; preserves explicit versions."
    )
    parser.add_argument("--target-directory", default="")
    parser.add_argument("--source-sha256", default="")
    parser.add_argument("--output", default="")
    args = parser.parse_args()
    record = maybe_load_json_record(args.input) or {"source_url": args.input}
    if re.fullmatch(r"\d{4}\.\d{4,5}(?:v\d+)?", args.input):
        record["arxiv_id"] = args.input
    vault = args.vault or runtime_config().get("obsidian_vault", "")
    try:
        if not vault or not Path(vault).expanduser().is_dir():
            raise ArchiveError("invalid_vault", [vault])
        source = find_source(
            Path(vault).expanduser().resolve(),
            record,
            args.reference or args.input,
            args.target_directory,
            args.source_sha256,
        )
        emit({"status": "ok" if source else "not_found", **(source or {})}, args.output)
    except ArchiveError as exc:
        emit({"status": "blocked", "conflict_code": exc.code, "candidates": exc.paths}, args.output)
        raise SystemExit(2) from exc


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""Validate that a note plan is grounded in the canonical source manifest."""

from __future__ import annotations

import argparse
import json
from pathlib import Path
from typing import Any

from common import (
    caption_label_key,
    emit,
    file_sha256,
    maybe_load_json_record,
    normalize_whitespace,
)
from contracts import (
    NOTE_PLAN_FIELD_TYPES,
    NOTE_PLAN_REQUIRED_FIELDS,
    PAPER_TYPE_VALUES,
    WRITING_CONTRACT_RULES,
    required_field_value_error,
    writing_contract_rules,
)
from localization import normalize_output_language, require_artifact_output_language
from source_corpus import SourceCorpusLoadError, load_source_corpus


def visual_review_is_valid(review: Any) -> bool:
    contract = WRITING_CONTRACT_RULES["visual_review_contract"]
    if not isinstance(review, dict):
        return False
    if set(review) != set(contract["review_fields"]):
        return False
    status = normalize_whitespace(str(review.get("status", "")))
    if status not in set(contract["review_status_values"]):
        return False
    reviewed_sha256 = review.get("reviewed_asset_sha256")
    if not isinstance(reviewed_sha256, str):
        return False
    if not all(
        isinstance(review.get(field), list)
        and all(isinstance(value, str) for value in review[field])
        for field in ("preserved_scientific_elements", "omitted_scientific_elements")
    ):
        return False
    if not isinstance(review.get("notes"), str) or not isinstance(
        review.get("failure_reason"), str
    ):
        return False
    failure_reason = normalize_whitespace(review["failure_reason"])
    attempts = review.get("repair_attempts")
    if not isinstance(attempts, int) or isinstance(attempts, bool):
        return False
    if attempts < 0 or attempts > int(contract["repair_limit"]):
        return False
    bbox = review.get("revised_bbox")
    if not isinstance(bbox, list):
        return False
    if bbox:
        if len(bbox) != 4 or not all(
            isinstance(value, (int, float)) and not isinstance(value, bool)
            for value in bbox
        ):
            return False
        x0, y0, x1, y1 = (float(value) for value in bbox)
        if not (0.0 <= x0 < x1 <= 1.0 and 0.0 <= y0 < y1 <= 1.0):
            return False
    repairable = set(contract["repairable_failure_reasons"])
    terminal = set(contract["terminal_failure_reasons"])
    if status == "pending":
        return not reviewed_sha256 and not failure_reason
    if status == "pass":
        return bool(
            reviewed_sha256
            and not failure_reason
            and not review["omitted_scientific_elements"]
        )
    if status == "repair_requested":
        return bool(
            reviewed_sha256
            and failure_reason in repairable
            and bbox
            and attempts < int(contract["repair_limit"])
        )
    if status == "fail":
        return bool(
            reviewed_sha256
            and (
                failure_reason in terminal
                or (
                    attempts >= int(contract["repair_limit"])
                    and failure_reason in repairable
                )
            )
        )
    return True


def visual_review_evidence_is_valid(item: dict[str, Any]) -> bool:
    evidence = item.get("review_evidence", {})
    contract = WRITING_CONTRACT_RULES["visual_review_contract"]
    if not isinstance(evidence, dict):
        return False
    if set(evidence) != set(contract["review_evidence_fields"]):
        return False
    source_path = Path(str(item.get("source_image_path", ""))).expanduser()
    candidate_path = Path(str(evidence.get("candidate_path", ""))).expanduser()
    preview_path = Path(str(evidence.get("page_preview_path", ""))).expanduser()
    pdf_path = Path(str(evidence.get("source_pdf_path", ""))).expanduser()
    if not (
        source_path.is_file()
        and candidate_path.is_file()
        and source_path.resolve() == candidate_path.resolve()
        and preview_path.is_file()
        and pdf_path.is_file()
    ):
        return False
    if not isinstance(evidence.get("source_page"), int) or int(
        evidence.get("source_page", 0)
    ) <= 0:
        return False
    if not isinstance(evidence.get("caption"), str):
        return False
    bbox_pt = evidence.get("bbox_pt")
    if not isinstance(bbox_pt, list) or len(bbox_pt) != 4 or not all(
        isinstance(value, (int, float)) and not isinstance(value, bool)
        for value in bbox_pt
    ):
        return False
    x0, y0, x1, y1 = (float(value) for value in bbox_pt)
    if not (x0 < x1 and y0 < y1):
        return False
    normalized = evidence.get("normalized_bbox")
    if not isinstance(normalized, list) or len(normalized) != 4 or not all(
        isinstance(value, (int, float)) and not isinstance(value, bool)
        for value in normalized
    ):
        return False
    nx0, ny0, nx1, ny1 = (float(value) for value in normalized)
    if not (0.0 <= nx0 < nx1 <= 1.0 and 0.0 <= ny0 < ny1 <= 1.0):
        return False
    return evidence.get("render_dpi") == contract["selected_render_dpi"]


def final_visual_failure_is_valid(item: dict[str, Any]) -> bool:
    if normalize_whitespace(str(item.get("decision", ""))) != "visual_defect":
        return False
    review = item.get("visual_review", {})
    if not visual_review_is_valid(review) or not visual_review_evidence_is_valid(item):
        return False
    if normalize_whitespace(str(review.get("status", ""))) != "fail":
        return False
    failure_reason = normalize_whitespace(str(review.get("failure_reason", "")))
    contract = WRITING_CONTRACT_RULES["visual_review_contract"]
    allowed = set(contract["terminal_failure_reasons"])
    if int(review.get("repair_attempts", 0) or 0) >= int(contract["repair_limit"]):
        allowed.update(contract["repairable_failure_reasons"])
    source_path = normalize_whitespace(str(item.get("source_image_path", "")))
    current_sha256 = file_sha256(source_path)
    return bool(
        failure_reason in allowed
        and current_sha256
        and normalize_whitespace(str(item.get("source_image_sha256", "")))
        == current_sha256
        and normalize_whitespace(str(review.get("reviewed_asset_sha256", "")))
        == current_sha256
    )


def parser() -> argparse.ArgumentParser:
    p = argparse.ArgumentParser(description=__doc__ or "lint grounding")
    p.add_argument("--note-plan", required=True, help="note_plan JSON path or JSON string.")
    p.add_argument(
        "--source-manifest",
        required=True,
        help="Source manifest JSON path or JSON string.",
    )
    p.add_argument(
        "--bundle-json",
        default="",
        help="Optional synthesis bundle JSON path or JSON string.",
    )
    p.add_argument(
        "--figure-decisions",
        required=True,
        help="Figure/table decisions JSON path or JSON string.",
    )
    p.add_argument("--output", default="", help="Output JSON path.")
    p.add_argument(
        "--language",
        default="",
        help="Run Override for output language: en or zh-CN.",
    )
    return p


def load_record(value: str) -> dict[str, Any]:
    record = maybe_load_json_record(value)
    if record is not None:
        return record
    path = Path(value).expanduser()
    if path.exists() and path.is_file():
        # utf-8-sig strips a leading UTF-8 BOM (common on Windows-authored
        # JSON) that would otherwise crash json.loads; no-op without a BOM.
        data = json.loads(path.read_text(encoding="utf-8-sig"))
        if isinstance(data, dict):
            return data
    raise SystemExit(f"Expected JSON object for {value!r}.")


def issue(code: str, severity: str = "error", **details: Any) -> dict[str, Any]:
    payload = {"code": code, "severity": severity}
    payload.update(details)
    return payload


def input_is_file(value: str) -> bool:
    if not value or value.strip().startswith("{"):
        return False
    try:
        return Path(value).expanduser().is_file()
    except OSError:
        return False


def source_caption_items(
    source_manifest: dict[str, Any],
    source_manifest_input: str = "",
) -> list[dict[str, Any]]:
    if input_is_file(source_manifest_input):
        try:
            return load_source_corpus(source_manifest_input).caption_items()
        except SourceCorpusLoadError as exc:
            raise SystemExit(str(exc)) from exc
    captions = (
        source_manifest.get("captions", {})
        if isinstance(source_manifest.get("captions"), dict)
        else {}
    )
    items: list[dict[str, Any]] = []
    for manifest_key, kind in (("figures", "figure"), ("tables", "table")):
        for item in captions.get(manifest_key, []) or []:
            if isinstance(item, dict):
                items.append({**dict(item), "kind": kind})
    return items


def source_section_ids(source_manifest: dict[str, Any]) -> set[str]:
    ids: set[str] = set()
    for section in source_manifest.get("sections", []) or []:
        if isinstance(section, dict):
            sid = normalize_whitespace(str(section.get("section_id", "")))
            if sid:
                ids.add(sid)
    for page in source_manifest.get("pages", []) or []:
        if not isinstance(page, dict):
            continue
        for sid in page.get("section_ids", []) or []:
            cleaned = normalize_whitespace(str(sid))
            if cleaned:
                ids.add(cleaned)
    return ids


def total_pages(source_manifest: dict[str, Any]) -> int:
    for container in (source_manifest.get("coverage", {}), source_manifest.get("pdf", {})):
        if isinstance(container, dict):
            value = int(container.get("total_pages", 0) or 0)
            if value:
                return value
    return 0


def source_is_truncated(source_manifest: dict[str, Any]) -> bool:
    for container in (source_manifest.get("coverage", {}), source_manifest.get("pdf", {})):
        if isinstance(container, dict) and (
            container.get("text_truncated") or container.get("truncated_due_to_page_limit")
        ):
            return True
    return False


def accepts_partial_reading(note_plan: dict[str, Any]) -> bool:
    coverage = note_plan.get("source_coverage", {})
    if isinstance(coverage, dict) and coverage.get("partial_reading_accepted") is True:
        return True
    return normalize_whitespace(str(note_plan.get("reading_mode", ""))).lower() in {
        "partial",
        "accepted_partial",
        "partial_reading",
    }


def contains_old_reference(value: Any) -> bool:
    prefixes = WRITING_CONTRACT_RULES["old_bundle_reference_prefixes"]
    tokens = WRITING_CONTRACT_RULES.get("old_evidence_reference_tokens", ())
    if isinstance(value, str):
        return any(prefix in value for prefix in prefixes) or any(
            token in value for token in tokens
        )
    if isinstance(value, dict):
        return any(contains_old_reference(item) for item in value.values())
    if isinstance(value, list):
        return any(contains_old_reference(item) for item in value)
    return False


def valid_section_id(value: Any, valid_ids: set[str]) -> bool:
    section_id = normalize_whitespace(str(value))
    return bool(section_id and section_id in valid_ids)


def valid_pages(value: Any, max_page: int) -> bool:
    if max_page <= 0:
        return False
    if isinstance(value, dict):
        try:
            start = int(value.get("start", 0) or value.get("page_start", 0) or 0)
            end = int(value.get("end", 0) or value.get("page_end", 0) or start or 0)
        except (TypeError, ValueError):
            return False
        return 1 <= start <= end <= max_page
    if isinstance(value, int):
        return 1 <= value <= max_page
    if isinstance(value, list) and value:
        try:
            pages = [int(page) for page in value]
        except (TypeError, ValueError):
            return False
        return all(1 <= page <= max_page for page in pages)
    return False


def evidence_sources(item: dict[str, Any]) -> list[Any]:
    for key in ("evidence_sources", "source_refs", "sources", "grounding"):
        value = item.get(key)
        if isinstance(value, list):
            return value
        if value:
            return [value]
    return []


def section_focus_text(item: dict[str, Any]) -> str:
    fields = WRITING_CONTRACT_RULES["note_plan_depth_requirements"][
        "required_section_focus_fields"
    ]
    for field in fields:
        value = normalize_whitespace(str(item.get(field, "")))
        if value:
            return value
    return ""


def focus_is_substantive(text: str) -> bool:
    requirements = WRITING_CONTRACT_RULES["note_plan_depth_requirements"]
    normalized = normalize_whitespace(text)
    lowered = normalized.lower()
    if any(phrase in lowered for phrase in requirements.get("generic_focus_phrases", ())):
        return False
    min_chars = int(requirements["required_section_focus_min_chars"])
    compact = "".join(ch for ch in normalized if not ch.isspace())
    return len(compact) >= min_chars


def source_grounding_errors(source: Any, valid_ids: set[str], max_page: int) -> list[str]:
    if contains_old_reference(source):
        return ["old_bundle_reference"]
    if isinstance(source, str):
        return [] if valid_section_id(source, valid_ids) else ["source_reference_unresolved"]
    if not isinstance(source, dict):
        return ["source_reference_invalid"]

    has_valid_section = valid_section_id(source.get("section_id", ""), valid_ids)
    has_valid_pages = valid_pages(source.get("pages"), max_page) or valid_pages(
        source.get("page_range"),
        max_page,
    )
    if has_valid_section or has_valid_pages:
        return []
    return ["source_reference_missing_valid_section_or_pages"]


def validate_note_plan(
    note_plan: dict[str, Any],
    source_manifest: dict[str, Any],
    language: str | None = None,
) -> list[dict[str, Any]]:
    issues: list[dict[str, Any]] = []
    required_checks = WRITING_CONTRACT_RULES["note_plan_required_field_checks"]
    for field in NOTE_PLAN_REQUIRED_FIELDS:
        if field not in note_plan:
            issues.append(issue("note_plan_required_field_missing", field=field))
            continue
        if required_field_value_error(
            note_plan[field], NOTE_PLAN_FIELD_TYPES[field], required_checks
        ):
            issues.append(issue("note_plan_required_field_empty", field=field))

    paper_type = normalize_whitespace(str(note_plan.get("paper_type", "")))
    if paper_type not in PAPER_TYPE_VALUES:
        issues.append(issue("note_plan_paper_type_invalid", paper_type=paper_type))

    if source_is_truncated(source_manifest) and not accepts_partial_reading(note_plan):
        issues.append(issue("source_manifest_truncated_without_partial_acceptance"))

    valid_ids = source_section_ids(source_manifest)
    max_page = total_pages(source_manifest)
    required_sections = set(writing_contract_rules(language)["grounding_required_sections"])
    grounded_sections: set[str] = set()
    section_plan = note_plan.get("section_plan", [])
    if not isinstance(section_plan, list):
        issues.append(issue("note_plan_section_plan_invalid"))
        return issues

    for item in section_plan:
        if not isinstance(item, dict):
            issues.append(issue("note_plan_section_plan_item_invalid"))
            continue
        section_name = normalize_whitespace(
            str(item.get("section") or item.get("name") or item.get("heading") or "")
        )
        if section_name in required_sections:
            grounded_sections.add(section_name)
        sources = evidence_sources(item)
        if section_name in required_sections and not sources:
            issues.append(issue("section_plan_grounding_missing", section=section_name))
            continue
        if section_name in required_sections and not focus_is_substantive(
            section_focus_text(item)
        ):
            issues.append(issue("section_plan_focus_too_thin", section=section_name))
        for source in sources:
            for code in source_grounding_errors(source, valid_ids, max_page):
                issues.append(issue(code, section=section_name, source=source))

    for section_name in sorted(required_sections - grounded_sections):
        issues.append(issue("section_plan_required_section_missing", section=section_name))

    issues.extend(validate_central_claims(note_plan, valid_ids, max_page))

    if contains_old_reference(note_plan):
        issues.append(issue("note_plan_old_bundle_reference_present"))

    return issues


def validate_central_claims(
    note_plan: dict[str, Any],
    valid_ids: set[str],
    max_page: int,
) -> list[dict[str, Any]]:
    central_claims = note_plan.get("central_claims", [])
    if "central_claims" not in note_plan:
        return []
    if not isinstance(central_claims, list):
        return [issue("central_claims_invalid")]

    issues: list[dict[str, Any]] = []
    contract = WRITING_CONTRACT_RULES["analysis_coverage_contract"]
    required_fields = contract["central_claim_fields"]
    field_types = contract["central_claim_field_types"]
    required_checks = contract["central_claim_required_field_checks"]
    for index, item in enumerate(central_claims):
        if not isinstance(item, dict):
            issues.append(issue("central_claim_item_invalid", index=index))
            continue
        supporting_evidence_missing = False
        for field in required_fields:
            field_value = item.get(field)
            field_type = field_types[field]
            if not required_field_value_error(field_value, field_type, required_checks):
                continue
            if field == "supporting_evidence":
                supporting_evidence_missing = True
                issues.append(issue("central_claim_supporting_evidence_missing", index=index))
            else:
                issues.append(
                    issue("central_claim_required_field_missing", index=index, field=field)
                )
        if supporting_evidence_missing:
            continue
        sources = item["supporting_evidence"]
        for source in sources:
            for code in source_grounding_errors(source, valid_ids, max_page):
                issues.append(issue(code, section="central_claims", source=source, index=index))
    return issues


def validate_bundle_contract(
    note_plan: dict[str, Any],
    bundle: dict[str, Any],
) -> list[dict[str, Any]]:
    if not bundle:
        return []
    issues: list[dict[str, Any]] = []
    for old_key in WRITING_CONTRACT_RULES["excluded_model_input_fields"]:
        if old_key in bundle:
            issues.append(issue("bundle_old_model_input_field_present", field=old_key))
    writing_contract = bundle.get("writing_contract", {})
    if not isinstance(writing_contract, dict):
        issues.append(issue("bundle_writing_contract_missing"))
        return issues
    contracts = writing_contract.get("contracts_by_paper_type", {})
    if not isinstance(contracts, dict) or not contracts:
        issues.append(issue("bundle_contracts_by_paper_type_missing"))
        return issues
    missing = [paper_type for paper_type in PAPER_TYPE_VALUES if paper_type not in contracts]
    if missing:
        issues.append(issue("bundle_contracts_by_paper_type_incomplete", missing=missing))
    paper_type = normalize_whitespace(str(note_plan.get("paper_type", "")))
    if paper_type and paper_type in PAPER_TYPE_VALUES and paper_type not in contracts:
        issues.append(issue("bundle_note_plan_paper_type_contract_missing", paper_type=paper_type))
    return issues


def validate_figure_decisions(
    source_manifest: dict[str, Any],
    decisions_wrapper: dict[str, Any],
    source_manifest_input: str = "",
) -> list[dict[str, Any]]:
    if not isinstance(decisions_wrapper, dict) or "decisions" not in decisions_wrapper:
        return [issue("figure_table_decisions_missing")]
    issues: list[dict[str, Any]] = []
    decisions = decisions_wrapper.get("decisions", [])
    if not isinstance(decisions, list):
        return [issue("figure_table_decisions_invalid")]
    valid_decisions = set(WRITING_CONTRACT_RULES["figure_decision_values"])
    usable_insert = WRITING_CONTRACT_RULES["usable_insert_candidate"]
    insertable_kinds = set(usable_insert.get("kinds", ()))
    decision_ids = {
        normalize_whitespace(
            str(item.get("source_id") or item.get("label") or item.get("item_id") or "")
        )
        for item in decisions
        if isinstance(item, dict)
    }
    decision_label_keys = {
        caption_label_key(source_id)
        for source_id in decision_ids
        if caption_label_key(source_id)
    }
    for caption in source_caption_items(source_manifest, source_manifest_input):
        if not isinstance(caption, dict):
            continue
        caption_id = normalize_whitespace(str(caption.get("id") or caption.get("label") or ""))
        caption_key = caption_label_key(caption_id)
        if caption_id and caption_id not in decision_ids and caption_key not in decision_label_keys:
            issues.append(
                issue(
                    "figure_table_caption_missing_decision",
                    caption_id=caption_id,
                )
            )
    for item in decisions:
        if not isinstance(item, dict):
            issues.append(issue("figure_table_decision_item_invalid"))
            continue
        decision = normalize_whitespace(str(item.get("decision", "")))
        if decision not in valid_decisions:
            issues.append(
                issue(
                    "figure_table_decision_value_invalid",
                    decision=decision,
                    source_id=item.get("source_id") or item.get("label") or "",
                )
            )
        if decision == "review_pending":
            issues.append(
                issue(
                    "figure_visual_review_unresolved",
                    source_id=item.get("source_id") or item.get("label") or "",
                )
            )
        if decision == "insert" and not normalize_whitespace(
            str(item.get("source_image_path", ""))
        ):
            issues.append(
                issue(
                    "figure_insert_decision_missing_source_image",
                    source_id=item.get("source_id") or item.get("label") or "",
                )
            )
        if decision == "insert":
            source_path = normalize_whitespace(str(item.get("source_image_path", "")))
            current_sha256 = file_sha256(source_path)
            review = item.get("visual_review", {})
            if not visual_review_evidence_is_valid(item):
                issues.append(
                    issue(
                        "figure_visual_review_evidence_invalid",
                        source_id=item.get("source_id") or item.get("label") or "",
                    )
                )
            if not visual_review_is_valid(review):
                issues.append(
                    issue(
                        "figure_visual_review_invalid",
                        source_id=item.get("source_id") or item.get("label") or "",
                    )
                )
            reviewed_sha256 = (
                normalize_whitespace(str(review.get("reviewed_asset_sha256", "")))
                if isinstance(review, dict)
                else ""
            )
            recorded_sha256 = normalize_whitespace(
                str(item.get("source_image_sha256", ""))
            )
            if (
                not isinstance(review, dict)
                or not visual_review_is_valid(review)
                or normalize_whitespace(str(review.get("status", ""))) != "pass"
                or not current_sha256
                or reviewed_sha256 != current_sha256
                or recorded_sha256 != current_sha256
            ):
                issues.append(
                    issue(
                        "figure_visual_review_stale",
                        source_id=item.get("source_id") or item.get("label") or "",
                    )
                )
        is_required_insert_candidate = (
            normalize_whitespace(str(item.get("kind", ""))) in insertable_kinds
            and normalize_whitespace(str(item.get("visual_quality_status", "")))
            == usable_insert["visual_quality_status"]
            and normalize_whitespace(str(item.get("source_image_path", "")))
        )
        if (
            is_required_insert_candidate
            and decision not in {"insert", "review_pending"}
            and not final_visual_failure_is_valid(item)
        ):
            skip_reason = normalize_whitespace(str(item.get("skip_reason", "")))
            issues.append(
                issue(
                    "usable_insert_candidate_left_placeholder",
                    source_id=item.get("source_id") or item.get("label") or "",
                    skip_reason=skip_reason,
                )
            )
    return issues


def main() -> None:
    from common import runtime_config

    args = parser().parse_args()
    note_plan = load_record(args.note_plan)
    source_manifest = load_record(args.source_manifest)
    bundle = load_record(args.bundle_json) if args.bundle_json else {}
    decisions = load_record(args.figure_decisions)

    language = normalize_output_language(
        runtime_config(cli_overrides={"output_language": args.language})["output_language"]
    )
    writing_contract = bundle.get("writing_contract", {}) if isinstance(bundle, dict) else {}
    issues = []
    for artifact, name in (
        (bundle, "Synthesis Bundle"),
        (note_plan, "Note Plan"),
        (decisions, "Figure/Table Decisions"),
        (
            {"output_language": writing_contract.get("language")}
            if isinstance(writing_contract, dict)
            else {},
            "Synthesis Bundle writing_contract",
        ),
    ):
        try:
            require_artifact_output_language(artifact, name, language)
        except ValueError as exc:
            issues.append(issue("output_language_contract_failed", artifact=name, reason=str(exc)))
    issues.extend(validate_note_plan(note_plan, source_manifest, language))
    issues.extend(validate_bundle_contract(note_plan, bundle))
    issues.extend(validate_figure_decisions(source_manifest, decisions, args.source_manifest))
    error_issues = [item for item in issues if item.get("severity", "error") == "error"]
    payload = {
        "status": "ok",
        "script": "lint_grounding.py",
        "paper_id": source_manifest.get("paper_id", note_plan.get("paper_id", "")),
        "output_language": language,
        "issues": issues,
        "warnings": [item for item in issues if item.get("severity") == "warning"],
        "passes_grounding": not error_issues,
    }
    emit(payload, args.output)


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""Check whether a drafted note meets structure and quality expectations."""

from __future__ import annotations

import argparse
import hashlib
import json
import re
from pathlib import Path

from contracts import (
    NOTE_PLAN_FIELD_TYPES,
    NOTE_PLAN_REQUIRED_FIELDS,
    NOTE_REQUIRED_SECTIONS,
    PAPER_TYPE_VALUES,
    WRITING_CONTRACT_RULES,
    required_field_value_error,
)
from localization import (
    normalize_output_language,
    note_schema,
    require_artifact_output_language,
)

ACTIVE_LANGUAGE = "zh-CN"
SCHEMA: dict = {}
SECTIONS: dict[str, str] = {}
REQUIRED_SECTIONS: tuple[str, ...] = NOTE_REQUIRED_SECTIONS
CORE_INFO_FIELDS: list[str] = []
CORE_INFO_FIELD_INDEX: dict[str, int] = {}
CORE_INFO_FIELD_ALIASES: dict[str, str] = {}
FIGURE_TARGET_SECTIONS: set[str] = set()
FIGURE_LABELS: dict[str, str] = {}
MECHANISM_FLOW_HEADING = "机制流程"

def configure_output_language(language: str | None = None) -> str:
    global ACTIVE_LANGUAGE, SCHEMA, SECTIONS, REQUIRED_SECTIONS, CORE_INFO_FIELDS
    global CORE_INFO_FIELD_INDEX, CORE_INFO_FIELD_ALIASES, FIGURE_TARGET_SECTIONS, FIGURE_LABELS, MECHANISM_FLOW_HEADING
    ACTIVE_LANGUAGE = normalize_output_language(language)
    SCHEMA = note_schema(ACTIVE_LANGUAGE)
    SECTIONS = dict(SCHEMA["sections"])
    REQUIRED_SECTIONS = tuple(SECTIONS.values())
    CORE_INFO_FIELDS = list(SCHEMA["core_info_fields"])
    CORE_INFO_FIELD_INDEX = {field: idx for idx, field in enumerate(CORE_INFO_FIELDS)}
    CORE_INFO_FIELD_ALIASES = dict(SCHEMA.get("core_info_aliases", {}))
    FIGURE_TARGET_SECTIONS = {SECTIONS[key] for key in ("research_questions", "data_and_task", "method", "key_results", "deep_analysis", "limitations", "my_notes")}
    FIGURE_LABELS = dict(SCHEMA["figure_labels"])
    MECHANISM_FLOW_HEADING = str(SCHEMA["mechanism_flow"])
    return ACTIVE_LANGUAGE

def section(key: str) -> str:
    return SECTIONS[key]

def figure_prefix(key: str) -> str:
    return f"> {FIGURE_LABELS[key]}"

configure_output_language()

FIGURE_BUCKET_RESIDUE_TOKENS = {
    "剩余",
    "残余",
    "未放置",
    "未处理",
    "待补",
}

FIGURE_BUCKET_VISUAL_TOKENS = {
    "图",
    "表",
    "图片",
    "图表",
    "占位",
}

ENGLISH_FIGURE_BUCKET_RESIDUE_TOKENS = {
    "remaining",
    "leftover",
    "unplaced",
    "unresolved",
    "backlog",
}

ENGLISH_FIGURE_BUCKET_VISUAL_TOKENS = {
    "figure",
    "figures",
    "fig",
    "figs",
    "table",
    "tables",
    "placeholder",
    "placeholders",
}

NONSTANDARD_FIGURE_PLACEHOLDER_RE = re.compile(
    r"""(?ix)
    ^\s*
    (?:
        \[\s*(?:图表|图片|图|表)\s*占位\s*\|[^\]]+\]
        |
        (?:图表|图片|图|表)\s*占位\s*[:：]\s*\S+
        |
        \[\s*(?:figure|fig|table)\s+placeholder\s*(?:\||:|\]|-|\s+(?:fig(?:ure)?|table)\.?\s*\d)
        |
        (?:figure|fig|table)\s+placeholder\s*(?:\||:|-|\s+(?:fig(?:ure)?|table)\.?\s*\d)
    )
    """
)

REAL_IMAGE_STATUS_RE = re.compile(
    r"""
    (?:
        已\s*(?:替换|插入|复制|拷贝|物化|写入)
        |
        (?:替换|插入)\s*为\s*真实图片
        |
        \b(?:inserted|replaced|copied|materialized)\b
    )
    """,
    flags=re.IGNORECASE | re.VERBOSE,
)

USABLE_CANDIDATE_STATUS_RE = re.compile(
    r"""
    (?:
        候选[^。；，\n>]{0,24}(?<!不)(?:可用|可读|清晰)
        |
        (?<!不)可用[^。；，\n>]{0,12}候选
        |
        (?:图像|图片|表格|图|表)?\s*裁剪[^。；，\n>]{0,12}(?<!不)(?:可用|可读|清晰)
        |
        (?:图像|图片|表格|图|表)[^。；，\n>]{0,12}(?<!不)(?:可用|可读|清晰)
        |
        图号\s*匹配
        |
        匹配度\s*高
        |
        高\s*置信(?:度)?[^。；，\n>]{0,12}候选
        |
        usable\s+candidate
        |
        readable\s+crop
        |
        clear\s+crop
        |
        (?<!no\s)(?<!not\s)high[-\s]*(?:confidence|match)
    )
    """,
    flags=re.IGNORECASE | re.VERBOSE,
)

USABLE_CANDIDATE_VISUAL_DEFECT_RE = re.compile(
    r"""
    (?:
        混入|污染|相邻|裁切|截断|切断|缺失|缺少|表体不完整|表格主体缺失|正文污染
        |
        只拿到|局部(?:子图|面板|截图|区域)|部分(?:子图|裁剪)
        |
        无法稳定|不可独立解释|质量门|reject_visual_quality
        |
        partial|subpanel|contaminat|truncat|incomplete|missing
        |
        caption\s*(?:missing|cut|truncated)
    )
    """,
    flags=re.IGNORECASE | re.VERBOSE,
)

USABLE_CANDIDATE_MATERIALIZATION_BLOCKED_RE = re.compile(
    r"""
    (?:
        (?:materialize_figure_asset\.py|物化|复制|拷贝|写入|权限|permission|工具|copy)
        [^。；\n]{0,40}
        (?:失败|不足|拒绝|denied|blocked|error|报错)
        |
        (?:失败|不足|拒绝|denied|blocked|error|报错)
        [^。；\n]{0,40}
        (?:materialize|物化|复制|拷贝|写入|权限|permission|copy)
    )
    """,
    flags=re.IGNORECASE | re.VERBOSE,
)

MISSING_ASSET_MATERIALIZATION_RE = re.compile(
    r"""
    (?:
        (?:资产缺失|未找到|没有|缺少|asset_candidate_missing|candidate\s+missing)
        [^。；\n]{0,50}
        (?:materialize_figure_asset\.py|物化|复制|拷贝|写入|权限|permission|copy|blocked)
        |
        (?:materialize_figure_asset\.py|物化|复制|拷贝|写入|权限|permission|copy|blocked)
        [^。；\n]{0,50}
        (?:资产缺失|未找到|没有|缺少|asset_candidate_missing|candidate\s+missing)
    )
    """,
    flags=re.IGNORECASE | re.VERBOSE,
)

MARKDOWN_IMAGE_EMBED_RE = re.compile(r"^!\[[^\]]*\]\([^)]+\)\s*$")
FIGURE_CALLOUT_TITLE_RE = re.compile(r"^>\s*\[!figure\][+-]?\s*(.*)$")
HTTP_URL_RE = re.compile(r"https?://\S+")
ENGLISH_METADATA_SOURCE_SPAN_PATTERNS = (
    re.compile(r"`[^`\n]+`"),
    re.compile(r"\[[^\]\n]+\]\([^)\n]+\)"),
)
ENGLISH_CJK_ENTITY_LINK_RE = re.compile(
    r"\[[^\]\n]*[\u4e00-\u9fff][^\]\n]*\]\(https?://[^)\n]+\)"
    r"|\[\[[^\]\n]*[\u4e00-\u9fff][^\]\n]*\]\]"
)
ENGLISH_CJK_MATH_LABEL_RE = re.compile(
    r"\\operatorname\{(?:输入|输出|损失|状态|动作|奖励|标签|样本|预测|目标)\}"
)
ENGLISH_MATH_SPAN_RE = re.compile(r"\$\$[^$\n]+\$\$|\$[^$\n]+\$")

RUNTIME_ARTIFACT_REFERENCE_PATTERNS = [
    re.compile(
        r"\b[A-Za-z0-9][A-Za-z0-9._-]*_"
        r"(?:source_manifest|note_plan|raw_sections)\."
        r"(?:json|jsonl)\b",
        flags=re.IGNORECASE,
    ),
    re.compile(r"\b[A-Za-z0-9][A-Za-z0-9._-]*_(?:bundle|lint)\.json\b", flags=re.IGNORECASE),
    re.compile(r"(?:/private)?/tmp/[^\s)\]>\"']*"),
    re.compile(r"(?<![A-Za-z0-9:])/?artifacts/[^\s)\]>\"']*"),
    re.compile(r"\b(?:source_manifest|note_plan|raw_sections)\b", flags=re.IGNORECASE),
]

MECHANICAL_TRANSLATION_ARTIFACT_RE = re.compile(
    r"""
    (?:
        [\u4e00-\u9fff]+(?:ing|ed|s)\b
        |
        \b[A-Za-z]{2,}相关\b
        |
        [\u4e00-\u9fff](?:缓存|块)?\s+(?:of|with|for|on|in|from|and)\b
        |
        \b(?:of|with|for|on|in|from|and)\s+[\u4e00-\u9fff]
        |
        [\u4e00-\u9fff]\s+(?:table|translation|example|candidate|caption|slot|query|block|token|input|layout|serving|memory|management|overhead|latency|dependent|preemption)\b
        |
        \b(?:block|caption|slot|query|input|layout|serving|memory|management|overhead|latency|dependent|preemption)\s+[\u4e00-\u9fff]
        |
        \b(?:Single|Shared|Performance|Storing|Illustration)\b[^\n。；]{0,60}[\u4e00-\u9fff]
    )
    """,
    flags=re.IGNORECASE | re.VERBOSE,
)


def parser() -> argparse.ArgumentParser:
    p = argparse.ArgumentParser(description=__doc__ or "lint note")
    p.add_argument("--input", required=True, help="Markdown note path.")
    p.add_argument("--plan-file", default="", help="Optional note_plan JSON path. Defaults to sibling <note>.plan.json.")
    p.add_argument("--output", default="", help="Output JSON path.")
    p.add_argument("--paper-id", default="", help="Canonical paper id.")
    p.add_argument("--language", default="", help="Run Override for output language: en or zh-CN.")
    return p


def resolve_note_plan_path(note_path: Path, plan_file: str) -> Path:
    if plan_file:
        return Path(plan_file).expanduser().resolve()
    return note_path.with_suffix(".plan.json")


def inspect_note_plan(
    plan_path: Path,
    output_language: str | None = None,
) -> tuple[bool, list[str]]:
    if not plan_path.exists():
        return False, ["planning_artifact_missing"]

    try:
        plan = json.loads(plan_path.read_text(encoding="utf-8"))
    except (OSError, UnicodeDecodeError, json.JSONDecodeError):
        return True, ["planning_artifact_invalid_json"]

    if not isinstance(plan, dict):
        return True, ["planning_required_fields_invalid"]

    issues: list[str] = []
    try:
        require_artifact_output_language(
            plan,
            "Note Plan",
            output_language or ACTIVE_LANGUAGE,
        )
    except ValueError as exc:
        issues.append(f"planning_output_language_contract_failed: {exc}")
    missing_fields = [field for field in NOTE_PLAN_REQUIRED_FIELDS if field not in plan]
    if missing_fields:
        issues.append("planning_required_fields_missing")

    required_checks = WRITING_CONTRACT_RULES["note_plan_required_field_checks"]
    has_invalid_fields = False
    for field in NOTE_PLAN_REQUIRED_FIELDS:
        if field not in plan:
            continue
        validation_error = required_field_value_error(
            plan[field], NOTE_PLAN_FIELD_TYPES[field], required_checks
        )
        if validation_error == "invalid":
            has_invalid_fields = True
        elif validation_error == "empty":
            issues.append(f"planning_{field}_empty")
    if isinstance(plan.get("paper_type"), str) and plan["paper_type"].strip():
        if plan["paper_type"].strip() not in PAPER_TYPE_VALUES:
            issues.append("planning_paper_type_invalid")
    if has_invalid_fields:
        issues.append("planning_required_fields_invalid")
    issues.extend(inspect_central_claims_plan(plan.get("central_claims")))

    return True, issues


def inspect_central_claims_plan(value: object) -> list[str]:
    if value is None:
        return []
    if not isinstance(value, list):
        return ["planning_central_claims_invalid"]

    issues: list[str] = []
    contract = WRITING_CONTRACT_RULES["analysis_coverage_contract"]
    required_fields = contract["central_claim_fields"]
    field_types = contract["central_claim_field_types"]
    required_checks = contract["central_claim_required_field_checks"]
    for item in value:
        if not isinstance(item, dict):
            if "planning_central_claims_invalid" not in issues:
                issues.append("planning_central_claims_invalid")
            continue
        for field in required_fields:
            field_value = item.get(field)
            field_type = field_types[field]
            if required_field_value_error(field_value, field_type, required_checks):
                code = (
                    "planning_central_claims_supporting_evidence_missing"
                    if field == "supporting_evidence"
                    else f"planning_central_claims_{field}_missing"
                )
                if code not in issues:
                    issues.append(code)
    return issues


def extract_headers(text: str) -> list[str]:
    return [match.group(2).strip() for match in re.finditer(r"^(#{1,3})\s+(.+)$", text, flags=re.MULTILINE)]


def find_missing_sections(text: str) -> list[str]:
    missing = []
    for section in REQUIRED_SECTIONS:
        if f"## {section}" not in text:
            missing.append(section)
    return missing


def front_matter_order_warnings(text: str) -> list[str]:
    warnings: list[str] = []
    required_order = [f"## {section('abstract')}", f"## {section('contributions')}", f"## {section('one_sentence_summary')}"]
    positions = []
    for required_heading in required_order:
        idx = text.find(required_heading)
        if idx < 0:
            return warnings
        positions.append(idx)
    if positions != sorted(positions):
        warnings.append("front_matter_order_invalid")
    return warnings


def english_top_level_section_warnings(text: str) -> list[str]:
    if ACTIVE_LANGUAGE != "en":
        return []
    actual = re.findall(r"^##\s+(.+?)\s*$", text, flags=re.MULTILINE)
    return [] if actual == list(REQUIRED_SECTIONS) else ["top_level_section_profile_invalid"]


def _inside_any_span(start: int, end: int, spans: list[tuple[int, int]]) -> bool:
    return any(span_start <= start and end <= span_end for span_start, span_end in spans)


def inspect_reference_hygiene(text: str) -> list[dict[str, object]]:
    issues: list[dict[str, object]] = []
    seen: set[tuple[int, str]] = set()
    for idx, line in enumerate(text.splitlines(), start=1):
        url_spans = [(match.start(), match.end()) for match in HTTP_URL_RE.finditer(line)]
        for pattern in RUNTIME_ARTIFACT_REFERENCE_PATTERNS:
            for match in pattern.finditer(line):
                matched_text = match.group(0)
                if _inside_any_span(match.start(), match.end(), url_spans):
                    continue
                key = (idx, matched_text)
                if key in seen:
                    continue
                seen.add(key)
                issues.append(
                    {
                        "line_number": idx,
                        "line": line.strip(),
                        "reason": "runtime_artifact_reference",
                        "match": matched_text,
                    }
                )
    return issues


METHOD_PAPER_SIGNAL_KEYWORDS = [
    "模型",
    "框架",
    "系统",
    "模块",
    "编码器",
    "解码器",
    "预融合",
    "attention",
    "encoder",
    "decoder",
    "pipeline",
    "framework",
    "model",
    "system",
    "module",
]

MECHANISM_IO_TOKENS = [
    "输入",
    "输出",
    "送入",
    "送到",
    "生成",
    "得到",
    "input",
    "output",
    "produces",
    "returns",
]

MECHANISM_ACTION_TOKENS = [
    "融合",
    "投影",
    "压缩",
    "对齐",
    "池化",
    "提取",
    "编码",
    "解码",
    "拼接",
    "查询",
    "更新",
    "align",
    "compute",
    "estimate",
    "extract",
    "encode",
    "decode",
    "update",
    "aggregate",
]


ENGLISH_FUNCTION_WORDS = {
    "a",
    "an",
    "and",
    "are",
    "as",
    "at",
    "be",
    "both",
    "by",
    "for",
    "from",
    "in",
    "into",
    "is",
    "it",
    "its",
    "of",
    "on",
    "or",
    "our",
    "that",
    "the",
    "their",
    "these",
    "this",
    "those",
    "to",
    "we",
    "when",
    "which",
    "with",
}

PLACEHOLDER_ONLY_PATTERNS = [
    r"^待补充[。.!！]*$",
    r"^todo[。.!！]*$",
    r"^暂无[。.!！]*$",
    r"^略[。.!！]*$",
    r"^参见原论文[。.!！]*$",
    r"^这里记录.*[。.!！]*$",
    r"^本节记录.*[。.!！]*$",
]

GENERIC_INNOVATION_PATTERNS = [
    r"本文提出(?:了)?一种新方法",
    r"具有创新性",
    r"novel approach",
    r"首次实现",
]

GENERIC_KEY_RESULT_PATTERNS = [
    r"实验结果表明方法有效",
    r"结果表明.*有效",
    r"取得(?:了)?较好效果",
    r"性能.*优越",
]

GENERIC_LIMITATION_PATTERNS = [
    r"未来工作.*更多数据",
    r"需要更多数据",
    r"future work can",
    r"more data",
    r"后续.*扩展",
]

HONEST_MISSING_TOKENS = ("本文未给出", "论文未给出", "未报告", "没有报告", "未提供")
HONEST_MISSING_BASIS_TOKENS = ("依据", "正文", "附录", "表格", "coverage", "作者")
HONEST_MISSING_IMPACT_TOKENS = ("影响", "限制", "受限", "不能", "无法", "结论强度")

DOUBLE_ESCAPED_TEX_COMMANDS = {
    "alpha",
    "bar",
    "begin",
    "beta",
    "end",
    "exp",
    "frac",
    "gamma",
    "ge",
    "hat",
    "left",
    "le",
    "log",
    "mathcal",
    "mathrm",
    "prod",
    "right",
    "sum",
    "tau",
    "tilde",
}


def is_metadata_line(line: str) -> bool:
    stripped = line.strip()
    prefixes = [f"- {field}:" for field in (*CORE_INFO_FIELDS, *CORE_INFO_FIELD_ALIASES)]
    return any(stripped.startswith(prefix) for prefix in prefixes)


def is_exempt_line(line: str) -> bool:
    stripped = line.strip()
    if not stripped:
        return True
    if stripped.startswith("#"):
        return True
    if is_metadata_line(stripped):
        return True
    if (
        stripped.startswith("> [!figure]")
        or stripped.startswith(figure_prefix("location"))
        or stripped.startswith(figure_prefix("reason"))
        or stripped.startswith(figure_prefix("status"))
    ):
        return True
    if re.search(r"https?://", stripped):
        return True
    if re.search(r"`10\.\d{4,9}/", stripped):
        return True
    return False


def section_name_for_line(lines: list[str], line_index: int) -> str:
    current_section = ""
    for idx in range(0, line_index + 1):
        stripped = lines[idx].strip()
        match = re.match(r"^##\s+(.+)$", stripped)
        if match:
            current_section = match.group(1).strip()
    return current_section


def subsection_name_for_line(lines: list[str], line_index: int) -> str:
    current_subsection = ""
    for idx in range(0, line_index + 1):
        stripped = lines[idx].strip()
        if re.match(r"^##\s+.+$", stripped):
            current_subsection = ""
            continue
        match = re.match(r"^###\s+(.+)$", stripped)
        if match:
            current_subsection = match.group(1).strip()
    return current_subsection


def mixed_language_issues(text: str) -> list[dict[str, object]]:
    issues: list[dict[str, object]] = []
    lines = text.splitlines()
    fenced_code_lines: set[int] = set()
    fence_start: int | None = None
    for line_index, line in enumerate(lines, start=1):
        if not line.strip().startswith("```"):
            continue
        if fence_start is None:
            fence_start = line_index
        else:
            fenced_code_lines.update(range(fence_start, line_index + 1))
            fence_start = None
    for idx, line in enumerate(lines, start=1):
        stripped = line.strip()
        if ACTIVE_LANGUAGE == "en":
            if idx in fenced_code_lines:
                continue
            checked = stripped
            section_name = section_name_for_line(lines, idx - 1)
            if section_name in {section("core_information"), section("references")}:
                for pattern in ENGLISH_METADATA_SOURCE_SPAN_PATTERNS:
                    checked = pattern.sub("", checked)
            if stripped.startswith(f"*{FIGURE_LABELS['original_caption']}"):
                checked = ENGLISH_METADATA_SOURCE_SPAN_PATTERNS[0].sub("", checked)
            checked = ENGLISH_CJK_ENTITY_LINK_RE.sub("", checked)
            checked = ENGLISH_MATH_SPAN_RE.sub(
                lambda match: ENGLISH_CJK_MATH_LABEL_RE.sub("", match.group(0)),
                checked,
            )
            checked = HTTP_URL_RE.sub("", checked)
            if re.search(r"[\u4e00-\u9fff]", checked):
                issues.append({"line_number": idx, "line": stripped, "reason": "non_english_text_present"})
            continue
        if is_exempt_line(line):
            continue
        section_name = section_name_for_line(lines, idx - 1)
        subsection_name = subsection_name_for_line(lines, idx - 1)
        if section_name in {section("core_information"), section("references")}:
            continue
        if not re.search(r"[\u4e00-\u9fff]", stripped):
            continue
        english_words = re.findall(r"\b[A-Za-z][A-Za-z0-9.-]*\b", stripped)
        if len(english_words) < 4:
            continue
        function_hits = [word for word in english_words if word.lower() in ENGLISH_FUNCTION_WORDS]
        if not function_hits and len(english_words) < 7:
            continue
        issues.append(
            {
                "line_number": idx,
                "line": stripped,
                "english_word_count": len(english_words),
                "function_word_hits": function_hits[:6],
            }
        )
    return issues


def mechanical_translation_artifact_issues(text: str) -> list[dict[str, object]]:
    if ACTIVE_LANGUAGE == "en":
        return []
    issues: list[dict[str, object]] = []
    for idx, line in enumerate(text.splitlines(), start=1):
        stripped = line.strip()
        if not stripped:
            continue
        if stripped in {"---"} or re.search(r"https?://", stripped):
            continue
        match = MECHANICAL_TRANSLATION_ARTIFACT_RE.search(stripped)
        if not match:
            continue
        issues.append(
            {
                "line_number": idx,
                "line": stripped,
                "artifact": match.group(0),
            }
        )
    return issues


def inspect_figure_callouts(text: str) -> list[str]:
    warnings: list[str] = []
    lines = text.splitlines()
    i = 0
    saw_legacy_block = False
    while i < len(lines):
        stripped = lines[i].strip()
        if stripped.startswith("[FIGURE_PLACEHOLDER]"):
            saw_legacy_block = True
        if not stripped.startswith("> [!figure]"):
            i += 1
            continue
        if not figure_callout_title(stripped):
            warnings.append("figure_callout_missing_title")
        has_location = False
        has_reason = False
        has_status = False
        j = i + 1
        while j < len(lines):
            nxt = lines[j].strip()
            if not nxt.startswith(">"):
                break
            if nxt.startswith(figure_prefix("location")):
                has_location = True
            if nxt.startswith(figure_prefix("reason")):
                has_reason = True
            if nxt.startswith(figure_prefix("status")):
                has_status = True
            j += 1
        if not has_location:
            warnings.append("figure_callout_missing_location")
        if not has_reason:
            warnings.append("figure_callout_missing_reason")
        if not has_status:
            warnings.append("figure_callout_missing_status")
        i = j
    if saw_legacy_block:
        warnings.append("legacy_figure_placeholder_block_used")
    return warnings


def figure_callout_title(line: str) -> str:
    match = FIGURE_CALLOUT_TITLE_RE.match(line.strip())
    if not match:
        return ""
    return match.group(1).strip()


def figure_status_text(line: str) -> str:
    stripped = line.strip()
    prefix = figure_prefix("status")
    if not stripped.startswith(prefix):
        return ""
    return stripped.removeprefix(prefix).strip()


def has_accepted_usable_placeholder_reason(status_text: str) -> bool:
    return bool(
        USABLE_CANDIDATE_VISUAL_DEFECT_RE.search(status_text)
        or USABLE_CANDIDATE_MATERIALIZATION_BLOCKED_RE.search(status_text)
    )


def usable_candidate_decision_is_unresolved(status_text: str) -> bool:
    return bool(
        USABLE_CANDIDATE_STATUS_RE.search(status_text)
        and not has_accepted_usable_placeholder_reason(status_text)
    )


def is_figure_bucket_heading(title: str) -> bool:
    normalized = title.strip().lower()
    has_chinese_residue = any(token in normalized for token in FIGURE_BUCKET_RESIDUE_TOKENS)
    has_chinese_visual = any(token in normalized for token in FIGURE_BUCKET_VISUAL_TOKENS)
    if has_chinese_residue and has_chinese_visual:
        return True
    has_english_residue = any(token in normalized for token in ENGLISH_FIGURE_BUCKET_RESIDUE_TOKENS)
    has_english_visual = any(token in normalized for token in ENGLISH_FIGURE_BUCKET_VISUAL_TOKENS)
    return has_english_residue and has_english_visual


def figure_bucket_heading_issues(text: str) -> list[dict[str, object]]:
    issues: list[dict[str, object]] = []
    for idx, line in enumerate(text.splitlines(), start=1):
        match = re.match(r"^(#{2,3})\s+(.+)$", line.strip())
        if not match:
            continue
        heading = match.group(2).strip()
        if is_figure_bucket_heading(heading):
            issues.append(
                {
                    "line_number": idx,
                    "heading": heading,
                    "reason": "figure_placeholder_bucket_heading",
                }
            )
    return issues


def nonstandard_figure_placeholder_issues(text: str) -> list[dict[str, object]]:
    issues: list[dict[str, object]] = []
    for idx, line in enumerate(text.splitlines(), start=1):
        stripped = line.strip()
        if not stripped or stripped.startswith(">"):
            continue
        if NONSTANDARD_FIGURE_PLACEHOLDER_RE.match(stripped):
            issues.append(
                {
                    "line_number": idx,
                    "line": stripped,
                    "reason": "nonstandard_figure_placeholder_format",
                }
            )
    return issues


def figure_callout_placement_issues(text: str) -> list[dict[str, object]]:
    issues: list[dict[str, object]] = []
    lines = text.splitlines()
    for idx, line in enumerate(lines):
        stripped = line.strip()
        if stripped.startswith("[FIGURE_PLACEHOLDER]"):
            issues.append(
                {
                    "line_number": idx + 1,
                    "line": stripped,
                    "reason": "legacy_figure_placeholder_block_used",
                }
            )
            continue
        if not stripped.startswith("> [!figure]"):
            continue

        title = figure_callout_title(stripped)
        if not title:
            issues.append(
                {
                    "line_number": idx + 1,
                    "callout": stripped,
                    "reason": "figure_callout_missing_title",
                }
            )

        current_section = section_name_for_line(lines, idx)
        current_subsection = subsection_name_for_line(lines, idx)
        location = ""
        j = idx + 1
        while j < len(lines):
            nxt = lines[j].strip()
            if not nxt.startswith(">"):
                break
            prefix = figure_prefix("location")
            if nxt.startswith(prefix):
                location = nxt.removeprefix(prefix).strip()
                break
            j += 1

        if not location:
            issues.append(
                {
                    "line_number": idx + 1,
                    "callout": stripped,
                    "current_section": current_section,
                    "current_subsection": current_subsection,
                    "reason": "figure_callout_missing_location",
                }
            )
            continue

        if current_subsection and current_subsection in location:
            continue
        target_sections = [section for section in FIGURE_TARGET_SECTIONS if section in location]
        if target_sections and current_section not in target_sections:
            issues.append(
                {
                    "line_number": idx + 1,
                    "callout": stripped,
                    "current_section": current_section,
                    "current_subsection": current_subsection,
                    "declared_location": location,
                    "target_sections": target_sections,
                    "reason": "figure_callout_placement_mismatch",
                }
            )
    return issues


def figure_callout_real_image_status_issues(text: str) -> list[dict[str, object]]:
    issues: list[dict[str, object]] = []
    lines = text.splitlines()
    for idx, line in enumerate(lines):
        stripped = line.strip()
        if not stripped.startswith("> [!figure]"):
            continue
        j = idx + 1
        while j < len(lines):
            nxt = lines[j].strip()
            if not nxt.startswith(">"):
                break
            status_text = figure_status_text(nxt)
            if status_text and REAL_IMAGE_STATUS_RE.search(nxt):
                issues.append(
                    {
                        "line_number": j + 1,
                        "line": nxt,
                        "reason": "inserted_figure_redundant_callout",
                    }
                )
                break
            j += 1
    return issues


def figure_callout_usable_candidate_status_issues(text: str) -> list[dict[str, object]]:
    issues: list[dict[str, object]] = []
    lines = text.splitlines()
    for idx, line in enumerate(lines):
        stripped = line.strip()
        if not stripped.startswith("> [!figure]"):
            continue
        j = idx + 1
        while j < len(lines):
            nxt = lines[j].strip()
            if not nxt.startswith(">"):
                break
            status_text = figure_status_text(nxt)
            if status_text and usable_candidate_decision_is_unresolved(status_text):
                issues.append(
                    {
                        "line_number": j + 1,
                        "line": nxt,
                        "reason": "usable_candidate_unresolved_decision",
                    }
                )
                break
            j += 1
    return issues


def figure_callout_missing_asset_materialization_issues(text: str) -> list[dict[str, object]]:
    issues: list[dict[str, object]] = []
    lines = text.splitlines()
    for idx, line in enumerate(lines):
        stripped = line.strip()
        if not stripped.startswith("> [!figure]"):
            continue
        j = idx + 1
        while j < len(lines):
            nxt = lines[j].strip()
            if not nxt.startswith(">"):
                break
            status_text = figure_status_text(nxt)
            if status_text and MISSING_ASSET_MATERIALIZATION_RE.search(status_text):
                issues.append(
                    {
                        "line_number": j + 1,
                        "line": nxt,
                        "reason": "missing_asset_misreported_as_materialization_blocked",
                    }
                )
                break
            j += 1
    return issues


def is_image_embed_line(line: str) -> bool:
    stripped = line.strip()
    return stripped.startswith("![[") or bool(MARKDOWN_IMAGE_EMBED_RE.match(stripped))


def has_figure_marker(text: str) -> bool:
    return (
        "[!figure]" in text
        or "[FIGURE_PLACEHOLDER]" in text
        or any(is_image_embed_line(line) for line in text.splitlines())
    )


def is_italic_caption_line(line: str) -> bool:
    stripped = line.strip()
    if len(stripped) < 3:
        return False
    if stripped.startswith("*") and stripped.endswith("*") and not stripped.startswith("**"):
        return True
    return stripped.startswith("_") and stripped.endswith("_") and not stripped.startswith("__")


def image_embed_caption_issues(text: str) -> list[dict[str, object]]:
    issues: list[dict[str, object]] = []
    lines = text.splitlines()
    for idx, line in enumerate(lines):
        stripped = line.strip()
        if not is_image_embed_line(stripped):
            continue
        if idx + 1 < len(lines) and is_italic_caption_line(lines[idx + 1]):
            continue
        issues.append(
            {
                "line_number": idx + 1,
                "line": stripped,
                "reason": "inserted_figure_missing_caption",
            }
        )
    return issues


def figure_structure_issues(text: str) -> list[dict[str, object]]:
    return (
        figure_bucket_heading_issues(text)
        + nonstandard_figure_placeholder_issues(text)
        + figure_callout_placement_issues(text)
        + figure_callout_real_image_status_issues(text)
        + figure_callout_usable_candidate_status_issues(text)
        + figure_callout_missing_asset_materialization_issues(text)
        + image_embed_caption_issues(text)
    )


def figure_structure_passes(text: str) -> bool:
    return not figure_structure_issues(text)


def core_info_structure_issues(text: str) -> list[dict[str, object]]:
    core_heading = section("core_information")
    body = section_body(text, core_heading)
    if not body:
        return []

    issues: list[dict[str, object]] = []
    seen_fields: set[str] = set()
    last_known_index = -1
    base_line = _line_number_from_offset(text, text.find(f"## {core_heading}"))

    for offset, raw_line in enumerate(body.splitlines(), start=1):
        stripped = raw_line.strip()
        if not stripped:
            continue
        line_number = base_line + offset
        match = re.match(r"^-\s*([^:：]+)\s*[:：]\s*(.*)$", stripped)
        if not match:
            issues.append(
                {
                    "line_number": line_number,
                    "line": stripped,
                    "reason": "core_info_non_metadata_line",
                }
            )
            continue

        field = match.group(1).strip()
        field = CORE_INFO_FIELD_ALIASES.get(field, field)
        if field not in CORE_INFO_FIELD_INDEX:
            issues.append(
                {
                    "line_number": line_number,
                    "line": stripped,
                    "reason": "core_info_unknown_field",
                    "field": field,
                }
            )
            continue

        if field in seen_fields:
            issues.append(
                {
                    "line_number": line_number,
                    "line": stripped,
                    "reason": "core_info_duplicate_field",
                    "field": field,
                }
            )
            continue

        field_index = CORE_INFO_FIELD_INDEX[field]
        if field_index < last_known_index:
            issues.append(
                {
                    "line_number": line_number,
                    "line": stripped,
                    "reason": "core_info_field_order_invalid",
                    "field": field,
                }
            )
        seen_fields.add(field)
        last_known_index = max(last_known_index, field_index)

    return issues


def is_prose_line(line: str) -> bool:
    stripped = line.strip()
    if not stripped:
        return False
    if stripped.startswith(("#", "-", "*", "> ", "```", "![[", f"*{FIGURE_LABELS['original_caption']}")):
        return False
    if stripped.startswith("`") and stripped.endswith("`"):
        return False
    return True


def suspicious_mid_sentence_linebreaks(text: str) -> list[dict[str, object]]:
    issues: list[dict[str, object]] = []
    lines = text.splitlines()
    for idx in range(len(lines) - 1):
        current = lines[idx].rstrip()
        nxt = lines[idx + 1].lstrip()
        if not is_prose_line(current) or not is_prose_line(nxt):
            continue
        if is_metadata_line(current) or is_metadata_line(nxt):
            continue
        if re.search(r"[。！？.!?：:]$", current):
            continue
        if not re.search(r"[，,；;、）)\]」』]$", current):
            if not re.search(r"[A-Za-z0-9`\u4e00-\u9fff]$", current):
                continue
        if not re.match(r"^[A-Za-z0-9`\u4e00-\u9fff(（“‘\"]", nxt):
            continue
        issues.append(
            {
                "line_number": idx + 1,
                "line": current.strip(),
                "next_line": nxt.strip(),
            }
        )
    return issues


def suspicious_code_formatted_math(text: str) -> list[dict[str, object]]:
    issues: list[dict[str, object]] = []
    lines = text.splitlines()
    in_fence = False
    fence_start = 0
    fence_lines: list[str] = []

    for idx, line in enumerate(lines, start=1):
        stripped = line.strip()
        if stripped.startswith("```"):
            if not in_fence:
                in_fence = True
                fence_start = idx
                fence_lines = []
            else:
                fence_text = "\n".join(fence_lines)
                if re.search(r"(?:^|\\n)\s*(?:[A-Za-z][A-Za-z0-9_]*\s*=|O\(|\\sum|\\prod|\\mathcal|\\log|\\frac)", fence_text):
                    issues.append(
                        {
                            "line_number": fence_start,
                            "line": "```",
                            "next_line": fence_lines[0].strip() if fence_lines else "",
                            "kind": "fenced_math_like_block",
                        }
                    )
                in_fence = False
                fence_start = 0
                fence_lines = []
            continue
        if in_fence:
            fence_lines.append(line)
            continue
        for match in re.finditer(r"`([^`\n]{3,120})`", line):
            content = match.group(1).strip()
            if re.search(r"(=|O\(|\\sum|\\prod|\\mathcal|\\log|\\frac)", content):
                issues.append(
                    {
                        "line_number": idx,
                        "line": line.strip(),
                        "next_line": content,
                        "kind": "inline_code_math_like",
                    }
                )
                break
    return issues


def _line_number_from_offset(text: str, offset: int) -> int:
    return text.count("\n", 0, offset) + 1


def _formula_snippet(content: str, limit: int = 120) -> str:
    normalized = " ".join(content.split())
    if len(normalized) <= limit:
        return normalized
    return normalized[: limit - 3] + "..."


def _strip_fenced_code_preserve_newlines(text: str) -> str:
    return re.sub(r"```.*?```", lambda m: "\n" * m.group(0).count("\n"), text, flags=re.DOTALL)


def _extract_math_blocks(text: str) -> tuple[list[dict[str, object]], list[dict[str, object]]]:
    sanitized = _strip_fenced_code_preserve_newlines(text)
    if ACTIVE_LANGUAGE == "en":
        # Currency amounts such as "$25 billion" are prose, not LaTeX delimiters.
        sanitized = re.sub(
            r"\$(?=\d+(?:\.\d+)?\s+(?:thousand|million|billion|trillion|dollars?|usd)\b)",
            r"\$",
            sanitized,
            flags=re.IGNORECASE,
        )
    blocks: list[dict[str, object]] = []
    issues: list[dict[str, object]] = []
    consumed_lines: set[int] = set()

    block_pattern = re.compile(r"(?<!\\)\$\$(.+?)(?<!\\)\$\$", flags=re.DOTALL)
    for match in block_pattern.finditer(sanitized):
        start = match.start()
        line_number = _line_number_from_offset(sanitized, start)
        content = match.group(1).strip()
        blocks.append(
            {
                "kind": "block",
                "line_number": line_number,
                "content": content,
                "snippet": _formula_snippet(content),
            }
        )
        line_span = match.group(0).count("\n")
        for extra in range(line_span + 1):
            consumed_lines.add(line_number + extra)

    delimiter_positions = [m.start() for m in re.finditer(r"(?<!\\)\$\$", sanitized)]
    if len(delimiter_positions) % 2 == 1:
        offset = delimiter_positions[-1]
        issues.append(
            {
                "line_number": _line_number_from_offset(sanitized, offset),
                "snippet": "$$",
                "reason": "unclosed_math_delimiter",
            }
        )

    inline_pattern = re.compile(r"(?<!\\)(?<!\$)\$(?!\$)(.+?)(?<!\\)\$(?!\$)")
    for idx, line in enumerate(sanitized.splitlines(), start=1):
        if idx in consumed_lines:
            continue
        for match in inline_pattern.finditer(line):
            content = match.group(1).strip()
            if not content:
                continue
            blocks.append(
                {
                    "kind": "inline",
                    "line_number": idx,
                    "content": content,
                    "snippet": _formula_snippet(content),
                }
            )
        if len(re.findall(r"(?<!\\)(?<!\$)\$(?!\$)", line)) % 2 == 1:
            issues.append(
                {
                    "line_number": idx,
                    "snippet": line.strip(),
                    "reason": "unclosed_math_delimiter",
                }
            )
    return blocks, issues


def _find_unbalanced_braces(expr: str) -> bool:
    depth = 0
    for char in expr:
        if char == "{":
            depth += 1
        elif char == "}":
            depth -= 1
            if depth < 0:
                return True
    return depth != 0


def _parse_group_argument(expr: str, start: int) -> int | None:
    idx = start
    while idx < len(expr) and expr[idx].isspace():
        idx += 1
    if idx >= len(expr) or expr[idx] != "{":
        return None
    depth = 0
    while idx < len(expr):
        if expr[idx] == "{":
            depth += 1
        elif expr[idx] == "}":
            depth -= 1
            if depth == 0:
                return idx + 1
        idx += 1
    return None


def _has_invalid_frac_arguments(expr: str) -> bool:
    for match in re.finditer(r"(?<!\\)\\frac\b", expr):
        next_index = _parse_group_argument(expr, match.end())
        if next_index is None:
            return True
        final_index = _parse_group_argument(expr, next_index)
        if final_index is None:
            return True
    return False


def _has_environment_mismatch(expr: str) -> bool:
    stack: list[str] = []
    pattern = re.compile(r"(?<!\\)\\(begin|end)\{([A-Za-z*]+)\}")
    for kind, env in pattern.findall(expr):
        if kind == "begin":
            stack.append(env)
            continue
        if not stack or stack[-1] != env:
            return True
        stack.pop()
    return bool(stack)


def _has_left_right_mismatch(expr: str) -> bool:
    return len(re.findall(r"(?<!\\)\\left\b", expr)) != len(re.findall(r"(?<!\\)\\right\b", expr))


def _has_double_escaped_tex_command(expr: str) -> bool:
    pattern = r"(?<!\\)\\\\(" + "|".join(sorted(DOUBLE_ESCAPED_TEX_COMMANDS)) + r")\b"
    return bool(re.search(pattern, expr))


def math_render_issues(text: str) -> list[dict[str, object]]:
    issues: list[dict[str, object]] = []
    blocks, delimiter_issues = _extract_math_blocks(text)
    issues.extend(delimiter_issues)

    for block in blocks:
        content = str(block["content"])
        line_number = int(block["line_number"])
        snippet = str(block["snippet"])

        if _has_double_escaped_tex_command(content):
            issues.append(
                {
                    "line_number": line_number,
                    "snippet": snippet,
                    "reason": "double_escaped_tex_command",
                }
            )
        if _find_unbalanced_braces(content):
            issues.append(
                {
                    "line_number": line_number,
                    "snippet": snippet,
                    "reason": "unbalanced_braces",
                }
            )
        if _has_environment_mismatch(content):
            issues.append(
                {
                    "line_number": line_number,
                    "snippet": snippet,
                    "reason": "environment_mismatch",
                }
            )
        if _has_left_right_mismatch(content):
            issues.append(
                {
                    "line_number": line_number,
                    "snippet": snippet,
                    "reason": "left_right_mismatch",
                }
            )
        if _has_invalid_frac_arguments(content):
            issues.append(
                {
                    "line_number": line_number,
                    "snippet": snippet,
                    "reason": "invalid_frac_arguments",
                }
            )

    deduped: list[dict[str, object]] = []
    seen: set[tuple[int, str, str]] = set()
    for issue in issues:
        key = (int(issue["line_number"]), str(issue["snippet"]), str(issue["reason"]))
        if key in seen:
            continue
        seen.add(key)
        deduped.append(issue)
    return deduped


def section_body(text: str, heading: str) -> str:
    pattern = rf"^##\s+{re.escape(heading)}\s*$"
    match = re.search(pattern, text, flags=re.MULTILINE)
    if not match:
        return ""
    start = match.end()
    next_match = re.search(r"^##\s+.+$", text[start:], flags=re.MULTILINE)
    if not next_match:
        return text[start:]
    return text[start : start + next_match.start()]


def subsection_body(text: str, section_heading: str, subsection_heading: str) -> str:
    body = section_body(text, section_heading)
    if not body:
        return ""
    pattern = rf"^###\s+{re.escape(subsection_heading)}\s*$"
    match = re.search(pattern, body, flags=re.MULTILINE)
    if not match:
        return ""
    start = match.end()
    next_match = re.search(r"^(?:##|###)\s+.+$", body[start:], flags=re.MULTILINE)
    if not next_match:
        return body[start:]
    return body[start : start + next_match.start()]


def cleaned_section_lines(body: str) -> list[str]:
    lines: list[str] = []
    for raw_line in body.splitlines():
        stripped = raw_line.strip()
        if not stripped:
            continue
        if (
            stripped.startswith("> [!figure]")
            or stripped.startswith(figure_prefix("location"))
            or stripped.startswith(figure_prefix("reason"))
            or stripped.startswith(figure_prefix("status"))
        ):
            continue
        if stripped.startswith("!["):
            continue
        if stripped.startswith(f"*{FIGURE_LABELS['original_caption']}") and stripped.endswith("*"):
            continue
        if stripped.startswith("> "):
            stripped = stripped[2:].strip()
        lines.append(stripped)
    return lines


def normalized_section_content(body: str) -> str:
    return normalize_lint_whitespace(" ".join(cleaned_section_lines(body)))


def normalize_lint_whitespace(text: str) -> str:
    return re.sub(r"\s+", " ", text).strip()


def matches_any_pattern(text: str, patterns: list[str]) -> bool:
    normalized = normalize_lint_whitespace(text)
    return any(re.search(pattern, normalized, flags=re.IGNORECASE) for pattern in patterns)


def is_placeholder_like(text: str) -> bool:
    normalized = normalize_lint_whitespace(text)
    if not normalized:
        return True
    return matches_any_pattern(normalized, PLACEHOLDER_ONLY_PATTERNS)


def issue(section: str, reason: str, severity: str, snippet: str) -> dict[str, object]:
    return {
        "section": section,
        "reason": reason,
        "severity": severity,
        "snippet": normalize_lint_whitespace(snippet)[:160],
    }


def text_units(body: str) -> list[str]:
    units: list[str] = []
    for line in cleaned_section_lines(body):
        stripped = re.sub(r"^(?:[-*+]|\d+[.)、])\s*", "", line).strip()
        if stripped:
            units.append(stripped)
    if units:
        return units
    return [
        part.strip()
        for part in re.split(r"[。！？!?]\s*", normalized_section_content(body))
        if part.strip()
    ]


def meaningful_units(body: str, generic_patterns: list[str] | None = None) -> list[str]:
    generic_patterns = generic_patterns or []
    kept: list[str] = []
    for unit in text_units(body):
        if is_placeholder_like(unit):
            continue
        if generic_patterns and matches_any_pattern(unit, generic_patterns):
            continue
        compact = re.sub(r"[\s，。,.；;：:、\-*+()（）【】\[\]]+", "", unit)
        if len(compact) < 12:
            continue
        kept.append(unit)
    return kept


def has_number_token(text: str) -> bool:
    return bool(re.search(r"\d+(?:\.\d+)?\s*(?:%|％|[A-Za-z\u4e00-\u9fff]{0,8})", text))


def is_honest_missing_declaration(text: str) -> bool:
    normalized = normalize_lint_whitespace(text)
    if not any(token in normalized for token in HONEST_MISSING_TOKENS):
        return False
    if not any(token in normalized for token in HONEST_MISSING_BASIS_TOKENS):
        return False
    if not any(token in normalized for token in HONEST_MISSING_IMPACT_TOKENS):
        return False
    return len(normalized) >= 30


def has_reference_entry(text: str) -> bool:
    normalized = normalize_lint_whitespace(text)
    if re.search(r"10\.\d{4,9}/\S+", normalized):
        return True
    if re.search(r"\barXiv[:：]?\s*\d{4}\.\d{4,5}", normalized, flags=re.IGNORECASE):
        return True
    if re.search(r"\[\[[^\]]+\]\]", normalized):
        return True
    if re.search(r"\[[0-9]+\]", normalized):
        return True
    if re.search(r"\b[A-Z][A-Za-z-]+ et al\.?\s*,?\s*(?:19|20)\d{2}\b", normalized):
        return True
    if re.search(r"\b[A-Z][A-Za-z-]+(?:\s+(?:and|&|et al\.?|[A-Z][A-Za-z-]+))*\s*\((?:19|20)\d{2}\)", normalized):
        return True
    if re.search(r"(?:19|20)\d{2}.*(?:DOI|doi|会议|期刊|arXiv)", normalized):
        return True
    return False


def inspect_substantive_content(text: str) -> list[dict[str, object]]:
    issues: list[dict[str, object]] = []
    for section_heading in REQUIRED_SECTIONS:
        body = section_body(text, section_heading)
        content = normalized_section_content(body)
        if is_placeholder_like(content):
            issues.append(issue(section_heading, "section_empty_shell", "error", content or section_heading))
        if section_heading not in {section("key_results"), section("references")} and is_honest_missing_declaration(content):
            issues.append(issue(section_heading, "section_honest_missing_not_allowed", "error", content))

    contributions_heading = section("contributions")
    innovation = section_body(text, contributions_heading)
    innovation_content = normalized_section_content(innovation)
    innovation_units = meaningful_units(innovation, GENERIC_INNOVATION_PATTERNS)
    if not innovation_units:
        issues.append(issue(contributions_heading, "innovation_empty_shell", "error", innovation_content))
    elif len(innovation_units) < 2:
        issues.append(issue(contributions_heading, "innovation_too_few_specific_points", "warning", innovation_content))

    key_results_heading = section("key_results")
    key_results = section_body(text, key_results_heading)
    key_results_content = normalized_section_content(key_results)
    if is_honest_missing_declaration(key_results_content):
        issues.append(
            issue(
                key_results_heading,
                "key_results_honest_missing_not_allowed",
                "error",
                key_results_content,
            )
        )
    elif not meaningful_units(key_results, GENERIC_KEY_RESULT_PATTERNS):
        issues.append(issue(key_results_heading, "key_results_empty_shell", "error", key_results_content))
    elif not has_number_token(key_results_content):
        issues.append(
            issue(
                key_results_heading,
                "key_results_quantitative_result_missing",
                "warning",
                key_results_content,
            )
        )

    references_heading = section("references")
    references = section_body(text, references_heading)
    references_content = normalized_section_content(references)
    if is_honest_missing_declaration(references_content):
        issues.append(issue(references_heading, "references_unavailable_declared", "warning", references_content))
    elif is_placeholder_like(references_content) or not has_reference_entry(references_content):
        issues.append(issue(references_heading, "references_placeholder", "error", references_content))

    limitations_heading = section("limitations")
    limitations = section_body(text, limitations_heading)
    limitations_content = normalized_section_content(limitations)
    if not meaningful_units(limitations, GENERIC_LIMITATION_PATTERNS):
        issues.append(issue(limitations_heading, "limitations_empty_shell", "error", limitations_content))

    for section_heading in (section("method"), section("deep_analysis")):
        body = section_body(text, section_heading)
        content = normalized_section_content(body)
        if not meaningful_units(body):
            issues.append(issue(section_heading, "section_empty_shell", "error", content or section_heading))

    deduped: list[dict[str, object]] = []
    seen: set[tuple[str, str, str]] = set()
    for item in issues:
        key = (str(item["section"]), str(item["reason"]), str(item["severity"]))
        if key in seen:
            continue
        seen.add(key)
        deduped.append(item)
    return deduped


def method_section_requires_mechanism_flow(text: str) -> bool:
    body = section_body(text, section("method"))
    if not body:
        return False
    lower = body.lower()
    keyword_hits = sum(1 for token in METHOD_PAPER_SIGNAL_KEYWORDS if token.lower() in lower)
    has_formula = "$$" in body or bool(re.search(r"\$[^$\n]{4,}\$", body))
    return has_formula or keyword_hits >= 2


def mechanism_flow_warnings(text: str) -> list[str]:
    warnings: list[str] = []
    if ACTIVE_LANGUAGE == "en":
        for heading in re.findall(r"^###\s+(.+?)\s*$", text, flags=re.MULTILINE):
            if heading.casefold() == MECHANISM_FLOW_HEADING.casefold() and heading != MECHANISM_FLOW_HEADING:
                warnings.append("mechanism_flow_heading_invalid")
                break
    if not method_section_requires_mechanism_flow(text):
        return warnings
    heading_pattern = rf"^###\s+{re.escape(MECHANISM_FLOW_HEADING)}\s*$"
    if not re.search(heading_pattern, text, flags=re.MULTILINE):
        warnings.append("mechanism_flow_subsection_missing")
        return warnings

    body = subsection_body(text, section("method"), MECHANISM_FLOW_HEADING)
    if not body:
        warnings.append("mechanism_flow_subsection_empty")
        return warnings

    step_lines = [line.strip() for line in body.splitlines() if re.match(r"^\d+\.\s+", line.strip())]
    if not 3 <= len(step_lines) <= 4:
        warnings.append("mechanism_flow_step_count_unexpected")

    step_text = " ".join(step_lines)
    step_text_lower = step_text.lower()
    has_io_signal = any(token.lower() in step_text_lower for token in MECHANISM_IO_TOKENS)
    has_action_signal = any(token.lower() in step_text_lower for token in MECHANISM_ACTION_TOKENS)
    if not (has_io_signal and has_action_signal):
        warnings.append("mechanism_flow_too_abstract")

    return warnings


def strip_frontmatter(text: str) -> str:
    """Remove a leading YAML frontmatter block (---...---) if present.

    Tolerates CRLF (\\r\\n) line endings so notes written on Windows are
    stripped the same way as LF notes.
    """
    return re.sub(r"^---\r?\n.*?\r?\n---\r?\n?", "", text, count=1, flags=re.DOTALL)


def main() -> None:
    from common import emit, runtime_config

    args = parser().parse_args()
    config = runtime_config(cli_overrides={"output_language": args.language})
    output_language = configure_output_language(str(config["output_language"]))
    path = Path(args.input).expanduser().resolve()
    # utf-8-sig strips a leading BOM and the replace() normalizes CRLF so
    # Windows-authored notes are linted identically to LF/BOM-less notes;
    # otherwise the BOM/`\r` cause spurious title/frontmatter/style failures.
    text = path.read_text(encoding="utf-8-sig").replace("\r\n", "\n")
    body_text = strip_frontmatter(text)
    headers = extract_headers(text)
    missing_sections = find_missing_sections(text)
    warnings: list[str] = []
    mixed_issues = mixed_language_issues(text)
    mechanical_artifact_issues = mechanical_translation_artifact_issues(text)
    linebreak_issues = suspicious_mid_sentence_linebreaks(body_text)
    code_math_issues = suspicious_code_formatted_math(text)
    math_issues = math_render_issues(text)
    figure_issues = figure_structure_issues(text)
    core_info_issues = core_info_structure_issues(text)
    reference_hygiene_issues = inspect_reference_hygiene(text)
    substantive_issues = inspect_substantive_content(text)
    planning_artifact_found, planning_artifact_issues = inspect_note_plan(
        resolve_note_plan_path(path, args.plan_file),
        output_language,
    )
    warnings.extend(inspect_figure_callouts(text))
    for issue in figure_issues:
        reason = str(issue.get("reason", ""))
        if reason and reason not in warnings:
            warnings.append(reason)
    for issue in core_info_issues:
        reason = str(issue.get("reason", ""))
        if reason and reason not in warnings:
            warnings.append(reason)
    for issue in planning_artifact_issues:
        if issue not in warnings:
            warnings.append(issue)
    for issue in substantive_issues:
        reason = str(issue.get("reason", ""))
        if reason and reason not in warnings:
            warnings.append(reason)
    warnings.extend(front_matter_order_warnings(text))
    warnings.extend(english_top_level_section_warnings(text))
    warnings.extend(mechanism_flow_warnings(text))
    if not body_text.lstrip().startswith("# "):
        warnings.append("title_heading_missing")
    if "## " not in text:
        warnings.append("no_level2_sections")
    if "### " not in text:
        warnings.append("no_level3_headings")
    if len(headers) < 5:
        warnings.append("too_few_headings")
    if not has_figure_marker(text):
        warnings.append("no_figure_markers")
    if len(text.splitlines()) < 20:
        warnings.append("note_too_short")
    if mixed_issues:
        warnings.append("mixed_language_lines_present")
    if mechanical_artifact_issues:
        warnings.append("mechanical_translation_artifacts_present")
    if linebreak_issues:
        warnings.append("suspicious_mid_sentence_linebreaks")
    if code_math_issues:
        warnings.append("suspicious_code_formatted_math")
    if math_issues:
        warnings.append("math_render_issues_present")
    if reference_hygiene_issues:
        warnings.append("runtime_artifact_references_present")

    payload = {
        "status": "ok",
        "script": "lint_note.py",
        "output_language": output_language,
        "paper_id": args.paper_id,
        "input_path": str(path),
        "note_sha256": hashlib.sha256(text.encode("utf-8")).hexdigest(),
        "headers": headers,
        "missing_sections": missing_sections,
        "warnings": warnings,
        "mixed_language_issues": mixed_issues,
        "mechanical_translation_artifact_issues": mechanical_artifact_issues,
        "linebreak_issues": linebreak_issues,
        "code_math_issues": code_math_issues,
        "math_render_issues": math_issues,
        "figure_structure_issues": figure_issues,
        "core_info_structure_issues": core_info_issues,
        "reference_hygiene_issues": reference_hygiene_issues,
        "substantive_content_issues": substantive_issues,
        "planning_artifact_found": planning_artifact_found,
        "planning_artifact_issues": planning_artifact_issues,
        "passes_basic_structure": (
            not missing_sections
            and not core_info_issues
            and not {
                "title_heading_missing",
                "no_level2_sections",
                "front_matter_order_invalid",
                "top_level_section_profile_invalid",
                "mechanism_flow_heading_invalid",
            }
            & set(warnings)
        ),
        "passes_style_gate": (
            not mixed_issues
            and not mechanical_artifact_issues
            and not linebreak_issues
            and not code_math_issues
        ),
        "passes_math_gate": not math_issues,
        "passes_figure_gate": not figure_issues,
        "passes_reference_hygiene_gate": not reference_hygiene_issues,
        "passes_plan_gate": planning_artifact_found and not planning_artifact_issues,
        "passes_substantive_content": not any(
            str(issue.get("severity", "")) == "error" for issue in substantive_issues
        ),
    

#!/usr/bin/env python3
"""Language schemas shared by DeepPaperNote contracts and validators."""

from __future__ import annotations

import os
from copy import deepcopy
from typing import Any

DEFAULT_OUTPUT_LANGUAGE = "zh-CN"
SUPPORTED_OUTPUT_LANGUAGES = ("zh-CN", "en")
_ALIASES = {"zh": "zh-CN", "zh-cn": "zh-CN", "zh_cn": "zh-CN", "chinese": "zh-CN", "en": "en", "en-us": "en", "en_us": "en", "english": "en"}
_SCHEMAS: dict[str, dict[str, Any]] = {
    "zh-CN": {
        "sections": {"core_information": "核心信息", "abstract": "原文摘要翻译", "contributions": "创新点", "one_sentence_summary": "一句话总结", "research_questions": "研究问题", "data_and_task": "数据与任务定义", "method": "方法主线", "key_results": "关键结果", "deep_analysis": "深度分析", "limitations": "局限", "my_notes": "我的笔记", "references": "引用"},
        "core_info_fields": ("标题", "标题翻译", "作者", "机构", "发表时间", "发表渠道", "DOI", "arXiv", "论文链接", "代码 / 项目", "数据 / 资源", "论文类型"),
        "core_info_aliases": {},
        "figure_labels": {"location": "建议位置：", "reason": "放置原因：", "status": "当前状态：", "original_caption": "论文原图编号："},
        "mechanism_flow": "机制流程",
    },
    "en": {
        "sections": {"core_information": "Core Information", "abstract": "Abstract", "contributions": "Contributions", "one_sentence_summary": "One-Sentence Summary", "research_questions": "Research Question", "data_and_task": "Data and Task Definition", "method": "Method", "key_results": "Key Results", "deep_analysis": "Deep Analysis", "limitations": "Limitations", "my_notes": "Research Notes", "references": "References"},
        "core_info_fields": ("Title", "Translated title", "Authors", "Institutions", "Publication date", "Venue", "DOI", "arXiv", "Paper link", "Code / Project", "Data / Resources", "Paper type"),
        "core_info_aliases": {},
        "figure_labels": {"location": "Suggested location:", "reason": "Why it matters:", "status": "Current status:", "original_caption": "Original paper item:"},
        "mechanism_flow": "Mechanism Flow",
        "abstract_contract": {
            "source": "source_abstract",
            "requirement": "faithful_rendering_in_output_language",
            "forbidden_additions": [
                "later_contribution_claims",
                "later_result_interpretation",
                "hindsight_judgment",
            ],
        },
    },
}

def normalize_output_language(value: str | None = None) -> str:
    raw = (value if value is not None else os.environ.get("DEEPPAPERNOTE_OUTPUT_LANGUAGE", "")).strip()
    if not raw:
        return DEFAULT_OUTPUT_LANGUAGE
    normalized = _ALIASES.get(raw.lower(), raw)
    if normalized not in SUPPORTED_OUTPUT_LANGUAGES:
        raise ValueError(f"Unsupported DeepPaperNote output language: {raw}. Choose one of: {', '.join(SUPPORTED_OUTPUT_LANGUAGES)}.")
    return normalized


def require_artifact_output_language(
    artifact: dict[str, Any],
    artifact_name: str,
    expected: str,
) -> str:
    language = artifact.get("output_language")
    if language not in SUPPORTED_OUTPUT_LANGUAGES:
        raise ValueError(
            f"{artifact_name} requires output_language with one of: "
            f"{', '.join(SUPPORTED_OUTPUT_LANGUAGES)}."
        )
    resolved = normalize_output_language(expected)
    if language != resolved:
        raise ValueError(
            f"{artifact_name} output_language {language} does not match "
            f"resolved output_language {resolved}."
        )
    return str(language)

def note_schema(language: str | None = None) -> dict[str, Any]:
    return deepcopy(_SCHEMAS[normalize_output_language(language)])

def required_sections(language: str | None = None) -> tuple[str, ...]:
    return tuple(note_schema(language)["sections"].values())

#!/usr/bin/env python3
"""Locate a Zotero attachment file inside common local Zotero storage directories."""

from __future__ import annotations

import argparse
from pathlib import Path

from common import emit, normalize_whitespace


DEFAULT_STORAGE_ROOTS = [
    "~/Zotero/storage",
    "~/Library/Application Support/Zotero/storage",
]


def parser() -> argparse.ArgumentParser:
    p = argparse.ArgumentParser(description=__doc__ or "locate zotero attachment")
    p.add_argument("--attachment-key", default="", help="Zotero attachment key such as GBH8G5JP.")
    p.add_argument("--filename", default="", help="Attachment filename when known.")
    p.add_argument(
        "--storage-roots",
        nargs="*",
        default=[],
        help="Optional Zotero storage roots. Defaults to common local Zotero paths.",
    )
    p.add_argument("--output", default="", help="Output JSON path.")
    return p


def iter_storage_roots(custom_roots: list[str]) -> list[Path]:
    roots = custom_roots or DEFAULT_STORAGE_ROOTS
    resolved: list[Path] = []
    for root in roots:
        path = Path(root).expanduser().resolve()
        if path.exists() and path.is_dir():
            resolved.append(path)
    return resolved


def choose_pdf_file(directory: Path, filename_hint: str) -> Path | None:
    if filename_hint:
        candidate = directory / filename_hint
        if candidate.exists() and candidate.is_file():
            return candidate
    pdfs = sorted(
        [
            path
            for path in directory.iterdir()
            if path.is_file() and path.suffix.lower() == ".pdf"
        ]
    )
    if len(pdfs) == 1:
        return pdfs[0]
    if filename_hint:
        hint = normalize_whitespace(filename_hint).lower()
        for pdf in pdfs:
            if normalize_whitespace(pdf.name).lower() == hint:
                return pdf
    return pdfs[0] if pdfs else None


def locate_attachment(attachment_key: str, filename: str, storage_roots: list[Path]) -> tuple[Path | None, Path | None]:
    filename = normalize_whitespace(filename)
    if attachment_key:
        for root in storage_roots:
            directory = root / attachment_key
            if directory.exists() and directory.is_dir():
                return directory, choose_pdf_file(directory, filename)

    if filename:
        lower_filename = filename.lower()
        for root in storage_roots:
            for child in root.iterdir():
                if not child.is_dir():
                    continue
                candidate = child / filename
                if candidate.exists() and candidate.is_file():
                    return child, candidate
                for pdf in child.glob("*.pdf"):
                    if pdf.name.lower() == lower_filename:
                        return child, pdf
    return None, None


def main() -> None:
    args = parser().parse_args()
    attachment_key = normalize_whitespace(args.attachment_key)
    filename = normalize_whitespace(args.filename)
    roots = iter_storage_roots(list(args.storage_roots))

    if not attachment_key and not filename:
        raise SystemExit("locate_zotero_attachment.py requires --attachment-key or --filename.")

    directory, pdf_path = locate_attachment(attachment_key, filename, roots)
    payload = {
        "status": "ok" if pdf_path else "not_found",
        "script": "locate_zotero_attachment.py",
        "attachment_key": attachment_key,
        "filename": filename,
        "searched_roots": [str(root) for root in roots],
        "storage_dir": str(directory) if directory else "",
        "local_pdf_path": str(pdf_path) if pdf_path else "",
    }
    emit(payload, args.output)


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""Copy a chosen figure candidate into the Obsidian vault and return embed markup."""

from __future__ import annotations

import argparse
import shutil
from pathlib import Path

from common import (
    emit,
    maybe_load_json_record,
    require_path_within,
    resolve_domain_subdir,
    resolve_note_asset_dir,
    resolve_note_output_mode,
    resolve_obsidian_note_path,
    runtime_config,
)


def parser() -> argparse.ArgumentParser:
    p = argparse.ArgumentParser(description=__doc__ or "materialize figure asset")
    p.add_argument("--source-image", required=True, help="Source image path selected by the model.")
    p.add_argument("--input", default="", help="Optional metadata JSON path or JSON string.")
    p.add_argument("--title", default="", help="Explicit paper title.")
    p.add_argument("--vault", default="", help="Target Obsidian vault path.")
    p.add_argument("--save-mode", choices=("workspace", "obsidian"), default="")
    p.add_argument("--papers-dir", default="", help="Vault-relative paper directory.")
    p.add_argument("--subdir", default="", help="Vault-relative note subdirectory.")
    p.add_argument("--filename", default="", help="Optional note filename override.")
    p.add_argument("--asset-subdir", default="images", help="Asset folder name relative to the note directory.")
    p.add_argument("--label", default="", help="Optional human-readable label for the figure.")
    p.add_argument("--output", default="", help="JSON output path.")
    return p


def main() -> None:
    args = parser().parse_args()
    record = maybe_load_json_record(args.input) or {}
    title = args.title or str(record.get("title", "")).strip()
    if not title:
        raise SystemExit("materialize_figure_asset.py requires --title or metadata with a title.")

    config = runtime_config(
        cli_overrides={
            "save_mode": args.save_mode or ("obsidian" if args.vault else ""),
            "obsidian_vault": args.vault,
            "papers_dir": args.papers_dir,
        }
    )
    resolved_subdir = resolve_domain_subdir(
        config,
        title=title,
        abstract=str(record.get("abstract", "")),
        subdir=args.subdir,
    )

    note_path = resolve_obsidian_note_path(
        config,
        title=title,
        subdir=resolved_subdir,
        filename=args.filename,
    )
    source_image = Path(args.source_image).expanduser().resolve()
    if not source_image.exists():
        raise SystemExit(f"Source image does not exist: {source_image}")

    asset_dir = resolve_note_asset_dir(note_path, args.asset_subdir)
    asset_dir.mkdir(parents=True, exist_ok=True)
    dest_image = require_path_within(asset_dir, asset_dir / source_image.name, "asset path")
    shutil.copy2(source_image, dest_image)

    output_mode, root_root = resolve_note_output_mode(config)
    relative_from_note = dest_image.relative_to(note_path.parent)
    relative_markdown_embed = f"![{args.label or source_image.stem}]({relative_from_note.as_posix()})"
    # as_posix() keeps forward slashes; str(Path) would emit backslashes on
    # Windows, producing a broken Markdown image link.
    absolute_markdown_embed = f"![{args.label or source_image.stem}]({dest_image.as_posix()})"

    payload = {
        "status": "ok",
        "script": "materialize_figure_asset.py",
        "title": title,
        "note_path": str(note_path),
        "source_image": str(source_image),
        "dest_image_path": str(dest_image),
        "absolute_markdown_embed": absolute_markdown_embed,
        "relative_markdown_embed": relative_markdown_embed,
        "label": args.label,
        "output_mode": output_mode,
        "subdir": resolved_subdir,
    }

    if output_mode == "obsidian":
        vault_relative = dest_image.relative_to(root_root)
        payload["vault_relative_image_path"] = vault_relative.as_posix()
        payload["obsidian_embed"] = f"![[{vault_relative.as_posix()}]]"
    emit(payload, args.output)


if __name__ == "__main__":
    main()

"""Shared paper-directory contract. Vendored unchanged by DeepPaperNote Connector.

Copyright (c) 2026 DeepPaperNote contributors. SPDX-License-Identifier: MIT
"""

from __future__ import annotations

import copy
import hashlib
import json
import os
import re
import tempfile
import time
from contextlib import contextmanager
from pathlib import Path, PureWindowsPath

SIDECAR = ".deeppapernote.json"


class ArchiveError(ValueError):
    def __init__(self, code: str, paths=()):
        self.code = code
        self.paths = [str(p) for p in paths]
        super().__init__(f"{code}: {', '.join(self.paths)}")


def sha256(path: Path) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as stream:
        for chunk in iter(lambda: stream.read(1024 * 1024), b""):
            digest.update(chunk)
    return digest.hexdigest()


HASH = re.compile(r"[0-9a-f]{64}")
ARXIV = re.compile(r"(?:arxiv[:/\s]+|arxiv\.org/(?:abs|pdf)/)(\d{4}\.\d{4,5}(?:v\d+)?)", re.I)
DOI = re.compile(r'10\.\d{4,9}/[^\s<>"]+', re.I)


def safe_path(root: Path, value: str) -> Path:
    raw = str(value)
    if (
        not raw
        or "\\" in raw
        or PureWindowsPath(raw).drive
        or Path(raw).is_absolute()
        or ".." in Path(raw).parts
    ):
        raise ArchiveError("unsafe_archive_path", [raw])
    target = root / raw
    if target.is_symlink() or not target.resolve().is_relative_to(root.resolve()):
        raise ArchiveError("unsafe_archive_path", [target])
    return target


def normalized_title(value: str) -> str:
    return re.sub(r"[^\w]", "", value, flags=re.UNICODE).casefold()


def work_identity(record: dict) -> dict:
    contract = record.get("identity_contract", {}) or {}
    canonical = (
        contract.get("work_level_identity", {}) or record.get("work_level_identity", {}) or {}
    )
    data = {**record, **canonical}
    if isinstance(record.get("work"), dict):
        return copy.deepcopy(record["work"])
    identifiers = {}
    doi = DOI.search(
        " ".join(str(data.get(k) or "") for k in ("doi", "DOI", "paper_id", "source_url", "url"))
    )
    if doi:
        doi_value = doi.group().rstrip(".,;)").lower()
        if doi_value.startswith("10.48550/arxiv."):
            identifiers["arxiv"] = re.sub(r"v\d+$", "", doi_value.split("arxiv.", 1)[1])
        else:
            identifiers["doi"] = doi_value
    arxiv = str(data.get("arxiv_id") or data.get("arXiv") or "")
    if re.fullmatch(r"\d{4}\.\d{4,5}(?:v\d+)?", arxiv):
        identifiers["arxiv"] = re.sub(r"v\d+$", "", arxiv)
    else:
        match = ARXIV.search(
            " ".join(str(data.get(k, "")) for k in ("paper_id", "source_url", "url", "pdf_url"))
            + " "
            + arxiv
        )
        if match:
            identifiers["arxiv"] = re.sub(r"v\d+$", "", match[1])
    evidence = {"kind": "canonical_identity" if canonical else "supplied_metadata"}
    source_url = str(data.get("source_url") or data.get("url") or "")
    if source_url:
        evidence["source_url"] = source_url
    return {
        "title": str(data.get("title", "")),
        "identifiers": identifiers,
        "provenance": {key: [dict(evidence)] for key in identifiers},
    }


def record_work_evidence(work: dict, digest: str, observed: dict) -> None:
    """Keep identifier evidence separate from per-source PDF and note records."""
    ids = work.setdefault("identifiers", {})
    provenance = work.setdefault("provenance", {})
    for key, value in observed["identifiers"].items():
        if key in ids and ids[key] != value:
            raise ArchiveError("pdf_work_identity_mismatch")
        ids[key] = value
        evidence = {"kind": "pdf_first_page", "source_sha256": digest}
        entries = provenance.setdefault(key, [])
        if evidence not in entries:
            entries.append(evidence)


def same_work(left: dict, right: dict) -> bool:
    a, b = left.get("identifiers", {}), right.get("identifiers", {})
    shared = set(a) & set(b)
    return bool(shared) and all(a[k] == b[k] for k in shared)


def pdf_identity(path: Path) -> dict:
    try:
        import pymupdf as fitz

        with fitz.open(path) as doc:
            text = doc[0].get_text() if len(doc) else ""
    except (ImportError, RuntimeError, ValueError, IndexError) as exc:
        raise ArchiveError("pdf_identity_unavailable", [path]) from exc
    # Only the first page is identity evidence; references later in the PDF are not anchors.
    arxiv = ARXIV.search(text)
    doi = re.search(r"(?:doi\s*:\s*|doi\.org/)(10\.\d{4,9}/[^\s<>]+)", text, re.I)
    ids = {}
    if arxiv:
        ids["arxiv"] = re.sub(r"v\d+$", "", arxiv[1])
    if doi:
        ids.update(work_identity({"doi": doi[1]})["identifiers"])
    return {"identifiers": ids, "arxiv_id": arxiv[1] if arxiv else "", "text": text}


def verify_pdf(path: Path, record: dict, *, exact_version: bool = False) -> dict:
    observed = pdf_identity(path)
    expected = work_identity(record)
    a, b = observed["identifiers"], expected["identifiers"]
    if any(a[k] != b[k] for k in set(a) & set(b)):
        raise ArchiveError("pdf_work_identity_mismatch", [path])
    if not same_work(observed, expected):
        title = normalized_title(expected["title"])
        authors = (
            record.get("authors")
            or record.get("creators")
            or (record.get("identity_contract", {}).get("work_level_identity", {}) or {}).get(
                "authors"
            )
            or []
        )
        names = [
            str(x.get("lastName") or x.get("name") or "") if isinstance(x, dict) else str(x)
            for x in authors
        ]
        text = normalized_title(observed["text"])
        if (
            len(title) < 15
            or title not in text
            or not any(normalized_title(n) in text for n in names if n)
        ):
            raise ArchiveError("pdf_work_identity_unverified", [path])
    if exact_version:
        requested = source_details(record).get("arxiv_id", "")
        actual = observed.get("arxiv_id", "")
        if re.search(r"v\d+$", requested) and actual != requested:
            raise ArchiveError("pdf_version_mismatch", [path])
    return observed


def source_details(record: dict, pdf_path: Path | None = None) -> dict:
    manifestation = (
        record.get("source_manifestation", {})
        or (record.get("identity_contract", {}) or {}).get("source_manifestation", {})
        or {}
    )
    data = {**record, **manifestation}
    arxiv = str(data.get("arxiv_id") or data.get("arXiv") or "")
    match = ARXIV.search(
        " ".join(str(data.get(k, "")) for k in ("source_url", "pdf_url", "url", "paper_id"))
    )
    if match and not re.search(r"v\d+$", arxiv):
        arxiv = match[1]
    if pdf_path:
        try:
            observed = pdf_identity(pdf_path)
            arxiv = observed["arxiv_id"] or arxiv
        except ArchiveError:
            pass
    return {
        "arxiv_id": arxiv,
        "source_url": str(data.get("source_url") or data.get("url") or ""),
        "pdf_url": str(data.get("pdf_url") or ""),
    }


def read_record(directory: Path) -> dict | None:
    path = directory / SIDECAR
    if not path.exists() and not path.is_symlink():
        return None
    safe_path(directory, SIDECAR)
    try:
        record = json.loads(path.read_text(encoding="utf-8-sig"))
        if (
            not isinstance(record, dict)
            or record.get("artifact_type") != "deeppapernote_paper_directory"
        ):
            raise ValueError()
        record = copy.deepcopy(record)
        if record.get("schema_version") == 1:
            digest = record["source_sha256"]
            record["sources"] = {
                digest: {
                    "notes": copy.deepcopy(record.get("notes", {})),
                    "note_stem": record["note_stem"],
                    "asset_subdir": "images",
                }
            }
            record["work"] = work_identity(record)
            record["work"]["provenance"] = {
                key: [{"kind": "legacy_record"}] for key in record["work"]["identifiers"]
            }
            record["schema_version"] = 2
            record.pop("source_sha256", None)
            record.pop("notes", None)
        if record.get("schema_version") != 2 or not isinstance(record.get("sources"), dict):
            raise ValueError()
        if not isinstance(record.get("work"), dict) or not isinstance(
            record["work"].get("identifiers"), dict
        ):
            raise ValueError()
        provenance = record["work"].setdefault("provenance", {})
        if not isinstance(provenance, dict):
            raise ValueError()
        for key, value in record["work"]["identifiers"].items():
            if not isinstance(value, str) or not value:
                raise ValueError()
            entries = provenance.setdefault(key, [{"kind": "legacy_record"}])
            if not isinstance(entries, list) or not entries:
                raise ValueError()
            for evidence in entries:
                if not isinstance(evidence, dict) or not isinstance(evidence.get("kind"), str):
                    raise ValueError()
                if evidence.get("source_sha256") and not HASH.fullmatch(
                    str(evidence["source_sha256"])
                ):
                    raise ValueError()
        if Path(record["note_stem"]).name != record["note_stem"]:
            raise ValueError()
        safe_path(directory, record["note_stem"])
        for digest, source in record["sources"].items():
            if (
                not HASH.fullmatch(digest)
                or not isinstance(source, dict)
                or not isinstance(source.get("notes"), dict)
            ):
                raise ValueError()
            if source.get("pdf_path"):
                safe_path(directory, source["pdf_path"])
            if source.get("note_stem"):
                if Path(source["note_stem"]).name != source["note_stem"]:
                    raise ValueError()
                safe_path(directory, source["note_stem"])
            if source.get("asset_subdir"):
                safe_path(directory, source["asset_subdir"])
            for language, note in source["notes"].items():
                if language not in ("en", "zh-CN") or not isinstance(note, dict):
                    raise ValueError()
                filename = note.get("filename", "")
                if Path(filename).name != filename or not filename.endswith(".md"):
                    raise ValueError()
                safe_path(directory, filename)
                if not HASH.fullmatch(str(note.get("note_sha256", ""))):
                    raise ValueError()
        return record
    except (OSError, ValueError, KeyError, TypeError) as exc:
        raise ArchiveError("invalid_directory_record", [directory]) from exc


def new_record(title: str, note_stem: str, record: dict) -> dict:
    return {
        "artifact_type": "deeppapernote_paper_directory",
        "schema_version": 2,
        "title": title,
        "note_stem": note_stem,
        "work": work_identity(record),
        "sources": {},
    }


def source_stem(record: dict, digest: str, details: dict) -> str:
    previous = record["sources"].get(digest, {})
    if previous.get("note_stem"):
        return previous["note_stem"]
    if not record["sources"]:
        return record["note_stem"]
    version = re.search(r"v\d+$", str(details.get("arxiv_id", "")))
    label = "arxiv-" + version[0] if version else "source"
    return f"{record['note_stem']}.{label}-{digest[:12]}"


def register_source(record: dict, digest: str, details: dict) -> dict:
    if not HASH.fullmatch(digest):
        raise ArchiveError("invalid_source_hash")
    stem = source_stem(record, digest, details)
    source = record["sources"].setdefault(
        digest,
        {
            "notes": {},
            "note_stem": stem,
            "asset_subdir": "images" if not record["sources"] else f"images/{digest}",
        },
    )
    for key, value in details.items():
        if value and key not in ("notes", "note_stem", "asset_subdir"):
            source[key] = value
    return source


def write_record(directory: Path, record: dict) -> None:
    path = safe_path(directory, SIDECAR)
    fd, name = tempfile.mkstemp(prefix=SIDECAR + ".", suffix=".tmp", dir=directory)
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as stream:
            json.dump(record, stream, ensure_ascii=False, indent=2)
            stream.write("\n")
            stream.flush()
            os.fsync(stream.fileno())
        os.replace(name, path)
    finally:
        Path(name).unlink(missing_ok=True)


@contextmanager
def archive_lock(vault: Path):
    # ponytail: serialize short Vault writes; use directory locks if throughput requires it.
    path = safe_path(vault, ".deeppapernote.lock")
    with path.open("a+b") as stream:
        deadline = time.monotonic() + 15
        while True:
            try:
                if os.name == "nt":
                    import msvcrt

                    stream.seek(0)
                    if not stream.read(1):
                        stream.write(b"\0")
                        stream.flush()
                    stream.seek(0)
                    msvcrt.locking(stream.fileno(), msvcrt.LK_NBLCK, 1)
                else:
                    import fcntl

                    fcntl.flock(stream, fcntl.LOCK_EX | fcntl.LOCK_NB)
                break
            except OSError as exc:
                if time.monotonic() >= deadline:
                    raise ArchiveError("archive_busy", [vault]) from exc
                time.sleep(0.05)
        try:
            yield
        finally:
            if os.name == "nt":
                stream.seek(0)
                msvcrt.locking(stream.fileno(), msvcrt.LK_UNLCK, 1)
            else:
                fcntl.flock(stream, fcntl.LOCK_UN)


def directory_candidates(vault: Path, record: dict, digest: str = "", name: str = "") -> list[dict]:
    work = work_identity(record)
    matches = []
    for directory in sorted(p for p in vault.rglob("*") if p.is_dir() and not p.is_symlink()):
        if not directory.resolve().is_relative_to(vault.resolve()):
            continue
        title_match = normalized_title(directory.name) in {
            normalized_title(name),
            normalized_title(work["title"]),
        } - {""}
        try:
            registry = read_record(directory)
        except ArchiveError:
            if title_match:
                matches.append(
                    {
                        "path": str(directory),
                        "confidence": "invalid",
                        "pdf_count": len(list(directory.glob("*.pdf"))),
                        "note_count": len(list(directory.glob("*.md"))),
                    }
                )
            continue
        verified = bool(
            registry and (digest in registry["sources"] or same_work(registry["work"], work))
        )
        if registry and normalized_title(registry.get("title", "")) == normalized_title(
            work["title"]
        ):
            title_match = True
        pdfs = [
            p
            for p in directory.iterdir()
            if p.suffix.lower() == ".pdf" and p.is_file() and not p.is_symlink()
        ]
        if not registry and not verified:
            for path in pdfs:
                if digest and sha256(path) == digest:
                    verified = True
                    break
                try:
                    if same_work(pdf_identity(path), work):
                        verified = True
                        break
                except ArchiveError:
                    continue
        if verified or title_match:
            matches.append(
                {
                    "path": str(directory),
                    "confidence": "verified" if verified else "candidate",
                    "pdf_count": len(pdfs),
                    "note_count": len(list(directory.glob("*.md"))),
                }
            )
    verified = [candidate for candidate in matches if candidate["confidence"] == "verified"]
    return verified or matches


def select_directory(candidates: list[dict], selected: str = "") -> Path | None:
    if selected:
        target = Path(selected).resolve()
        if target not in [Path(c["path"]).resolve() for c in candidates]:
            raise ArchiveError("selected_directory_not_candidate", [target])
        if (
            next(c for c in candidates if Path(c["path"]).resolve() == target)["confidence"]
            == "invalid"
        ):
            raise ArchiveError("invalid_directory_record", [target])
        return target
    if len(candidates) > 1:
        raise ArchiveError("multiple_source_directories", [c["path"] for c in candidates])
    if candidates and candidates[0]["confidence"] == "invalid":
        raise ArchiveError("invalid_directory_record", [candidates[0]["path"]])
    return Path(candidates[0]["path"]) if candidates else None


def admit_directory(directory: Path, record: dict, digest: str) -> dict:
    registry = read_record(directory)
    work = work_identity(record)
    if registry:
        if digest not in registry["sources"] and not same_work(registry["work"], work):
            raise ArchiveError("same_name_different_source", [directory])
        if any(
            registry["work"]["identifiers"][k] != work["identifiers"][k]
            for k in set(registry["work"]["identifiers"]) & set(work["identifiers"])
        ):
            raise ArchiveError("pdf_work_identity_mismatch", [directory])
    else:
        registry = new_record(record.get("title", directory.name), directory.name, record)
        for path in directory.iterdir():
            if path.name == ".DS_Store" or (
                path.name.startswith(".deeppapernote-") and path.suffix == ".part"
            ):
                continue
            if path.is_symlink() or not path.is_file() or path.suffix.lower() != ".pdf":
                raise ArchiveError("unidentified_same_name_directory", [directory])
    for source_hash, source in registry["sources"].items():
        if source.get("pdf_path"):
            path = safe_path(directory, source["pdf_path"])
            if not path.is_file() or sha256(path) != source_hash:
                raise ArchiveError("archived_pdf_changed", [path])
    for path in sorted(directory.iterdir()):
        if path.suffix.lower() != ".pdf":
            continue
        safe_path(directory, path.name)
        actual = sha256(path)
        if actual != digest and actual not in registry["sources"]:
            observed = verify_pdf(path, record)
            record_work_evidence(registry["work"], actual, observed)
        details = source_details(record if actual == digest else {}, path)
        details["pdf_path"] = path.name
        register_source(registry, actual, details)
    return registry


def find_source(
    vault: Path,
    record: dict,
    reference: str = "",
    selected_directory: str = "",
    selected_hash: str = "",
) -> dict | None:
    candidates = directory_candidates(vault, record)
    directory = select_directory(candidates, selected_directory)
    if directory is None:
        return None
    registry = admit_directory(directory, record, "")
    sources = []
    for digest, source in registry["sources"].items():
        if not source.get("pdf_path"):
            continue
        path = safe_path(directory, source["pdf_path"])
        if not path.is_file() or sha256(path) != digest:
            raise ArchiveError("archived_pdf_changed", [path])
        sources.append(
            {
                **source,
                "pdf_path": str(path),
                "source_sha256": digest,
                "target_directory": str(directory),
            }
        )
    if selected_hash:
        sources = [s for s in sources if s["source_sha256"] == selected_hash]
        if not sources:
            raise ArchiveError("selected_source_not_found", [directory])
    explicit = ARXIV.search(
        "arxiv:" + reference if re.fullmatch(r"\d{4}\.\d{4,5}(?:v\d+)?", reference) else reference
    )
    if explicit and re.search(r"v\d+$", explicit[1]):
        sources = [s for s in sources if s.get("arxiv_id") == explicit[1]]
    if not sources:
        if selected_hash:
            raise ArchiveError("selected_source_version_mismatch", [directory])
        return None
    if len(sources) > 1:
        versions = [
            re.fullmatch(r"(\d{4}\.\d{4,5})v(\d+)", str(s.get("arxiv_id", ""))) for s in sources
        ]
        if all(versions) and len({v[1] for v in versions}) == 1:
            latest = max(int(v[2]) for v in versions)
            sources = [s for s, v in zip(sources, versions) if int(v[2]) == latest]
        if len(sources) != 1:
            raise ArchiveError("ambiguous_archived_sources", [s["pdf_path"] for s in sources])
    return sources[0]


def archive_location() -> tuple[Path, str]:
    path = Path(
        os.environ.get("DEEPPAPERNOTE_CONFIG_PATH") or Path.home() / ".deeppapernote/config.json"
    ).expanduser()
    try:
        config = json.loads(path.read_text(encoding="utf-8-sig"))
        vault = Path(config["obsidian_vault"]).expanduser()
        papers_dir = config["papers_dir"]
        if not vault.is_absolute() or not vault.is_dir():
            raise ValueError()
        safe_path(vault, papers_dir)
        return vault.resolve(), papers_dir
    except (OSError, ValueError, KeyError, TypeError) as exc:
        raise ArchiveError("shared_archive_configuration_required", [path]) from exc

#!/usr/bin/env python3
"""Plan figure/table placeholders and attach deterministic asset candidates."""

from __future__ import annotations

import argparse
import re

from common import caption_preference_score, maybe_load_json_record, normalize_whitespace, runtime_config
from localization import normalize_output_language


def parser() -> argparse.ArgumentParser:
    p = argparse.ArgumentParser(description=__doc__ or "plan figures")
    p.add_argument("--input", default="", help="Primary JSON path or string.")
    p.add_argument("--evidence", default="", help="Evidence JSON path or string.")
    p.add_argument("--assets", default="", help="PDF assets JSON path or string.")
    p.add_argument("--output", default="", help="Output JSON path.")
    p.add_argument("--paper-id", default="", help="Canonical paper id.")
    p.add_argument("--language", default="", help="Output language: en or zh-CN.")
    p.add_argument("--max-items", type=int, default=12, help="Maximum number of figure/table items to keep. 0 means keep all.")
    return p


def merge_inputs(primary: dict | None, evidence: dict | None, assets: dict | None) -> dict:
    merged: dict = {}
    for item in [primary, evidence, assets]:
        if isinstance(item, dict):
            merged.update(item)
    if evidence and evidence.get("evidence_pack"):
        merged["evidence_pack"] = evidence["evidence_pack"]
    if assets and assets.get("page_assets"):
        merged["page_assets"] = assets["page_assets"]
        merged["image_assets"] = assets.get("image_assets", [])
        merged["figure_assets"] = assets.get("figure_assets", [])
    return merged


def _classify_caption_kind_zh(item_id: str, caption: str) -> tuple[str, str, str]:
    text = f"{item_id} {caption}".lower()
    if any(
        token in text
        for token in [
            "accuracy",
            "score",
            "performance",
            "comparison",
            "win-rate",
            "results",
            "recall",
            "latency",
            "throughput",
            "request rate",
            "batched request",
            "batch size",
            "memory saving",
            "ablation",
            "ablated",
            "overhead",
            "microbenchmark",
            "block size",
            "single sequence generation",
            "parallel generation",
        ]
    ) or re.search(
        r"\b(?:model|system|architecture|method|approach)\s+"
        r"(?:produces|achieves|outperforms|improves|reduces|increases)\b",
        text,
    ):
        return "main_result", "关键结果", "这张图或表直接承载主结果，适合放在关键结果部分。"
    if any(
        token in text
        for token in [
            "prisma",
            "literature flow",
            "flow diagram",
            "study selection",
            "screening flow",
            "screened records",
            "records screened",
            "identification of studies",
        ]
    ):
        return "data_or_task_overview", "数据与任务定义", "这张图解释文献筛选或纳入流程；如果候选图质量足够，适合放在数据与任务定义部分帮助读者理解综述证据来源。"
    if any(
        token in text
        for token in [
            "pipeline",
            "framework",
            "overview",
            "architecture",
            "system",
            "workflow",
            "stage",
            "procedure",
            "process",
        ]
    ):
        return "method_overview", "机制流程", "这张图概括了整体方法或系统流程；如果匹配置信度足够高，最适合放在 `### 机制流程` 帮助快速建立执行链理解。"
    if any(
        token in text
        for token in [
            "created from",
            "sources",
            "connect",
            "collection",
            "curation",
            "filtering",
            "pull request",
            "issue",
        ]
    ):
        return "data_or_task_overview", "数据与任务定义", "这张图解释任务或数据集如何被构造；如果候选图质量足够，适合放在数据与任务定义部分帮助读者理解数据来源。"
    if any(
        token in text
        for token in [
            "dataset",
            "data",
            "corpus",
            "participants",
            "recordings",
            "setup",
            "distribution",
            "quality",
            "task",
            "instance",
            "attribute",
        ]
    ):
        return "data_or_task", "数据与任务定义", "这张图更像任务设定或数据说明，放在数据与任务定义最合适。"
    if any(
        token in text
        for token in [
            "algorithm",
            "block table",
            "kv cache",
            "key-value cache",
            "copy-on-write",
            "parallel sampling",
            "beam search",
            "shared prompt",
            "shared prefix",
            "memory management",
            "block translation",
        ]
    ):
        return "method_detail", "方法主线", "这张图解释方法内部机制或关键执行状态，适合放在方法主线部分作为机制细节占位。"
    if item_id.lower().startswith("table"):
        return "table_result", "关键结果", "这是关键结果表，适合放在关键结果部分辅助定位核心数值。"
    return "supporting_figure", "深度分析", "这张图更适合作为补充图，放在深度分析部分帮助解释作者论点。"


ENGLISH_FIGURE_PLACEMENT: dict[str, tuple[str, str]] = {
    "main_result": ("Key Results", "This figure or table carries a primary result and belongs in Key Results."),
    "data_or_task_overview": ("Data and Task Definition", "This visual explains the source, construction, screening, or scope of the data and task."),
    "method_overview": ("Mechanism Flow", "This visual summarizes the method or system flow and belongs in Mechanism Flow when the match is reliable."),
    "data_or_task": ("Data and Task Definition", "This visual clarifies the task setting, sample, or dataset."),
    "method_detail": ("Method", "This visual explains an internal mechanism or execution state and belongs in Method."),
    "table_result": ("Key Results", "This result table helps readers locate the central quantitative evidence."),
    "supporting_figure": ("Deep Analysis", "This supporting visual helps explain the authors' argument in Deep Analysis."),
}


def classify_caption_kind(item_id: str, caption: str, language: str | None = None) -> tuple[str, str, str]:
    result = _classify_caption_kind_zh(item_id, caption)
    if normalize_output_language(language) != "en":
        return result
    kind = result[0]
    section, reason = ENGLISH_FIGURE_PLACEMENT[kind]
    return kind, section, reason


def build_figure_items(evidence_pack: dict, *, limit: int = 12, language: str | None = None) -> list[dict]:
    raw_items = []
    for item in evidence_pack.get("figure_captions", []) or []:
        if isinstance(item, dict):
            raw_items.append({"id": item.get("id", ""), "caption": item.get("caption", ""), "source": "figure"})
    for item in evidence_pack.get("table_captions", []) or []:
        if isinstance(item, dict):
            raw_items.append({"id": item.get("id", ""), "caption": item.get("caption", ""), "source": "table"})

    grouped: dict[str, dict] = {}
    grouped_scores: dict[str, int] = {}
    order: list[str] = []
    for item in raw_items:
        item_id = normalize_whitespace(str(item.get("id", "")))
        caption = normalize_whitespace(str(item.get("caption", "")))
        if not item_id:
            continue
        key = _normalize_label_for_match(item_id) or item_id.lower()
        current = grouped.get(key)
        candidate = {"id": item_id, "caption": caption, "source": item.get("source", "")}
        if current is None:
            grouped[key] = candidate
            grouped_scores[key] = caption_preference_score(item_id, caption)
            order.append(key)
            continue
        # Prefer true caption-looking lines over later body references such as
        # "Fig. 14 shows ..."; fall back to richness when both look caption-like.
        score = caption_preference_score(item_id, caption)
        if score > grouped_scores[key]:
            grouped[key] = candidate
            grouped_scores[key] = score

    picked: list[dict] = []
    for key in order:
        item = grouped[key]
        item_id = normalize_whitespace(str(item.get("id", "")))
        caption = normalize_whitespace(str(item.get("caption", "")))
        kind, section, reason = classify_caption_kind(item_id, caption, language)
        priority = 3
        if kind == "method_overview":
            priority = 1
        elif kind in {"data_or_task_overview", "main_result", "table_result", "method_detail"}:
            priority = 2
        picked.append(
            {
                "id": item_id,
                "caption": caption,
                "kind": kind,
                "section": section,
                "reason": reason,
                "priority": priority,
                "anchor_text": section,
                "insert_mode": "placeholder",
            }
        )
    picked.sort(key=lambda item: (item["priority"], item["id"]))
    if limit and limit > 0:
        high_priority = [item for item in picked if int(item.get("priority", 99)) <= 2]
        supporting = [item for item in picked if int(item.get("priority", 99)) > 2]
        if len(high_priority) >= limit:
            return high_priority
        return high_priority + supporting[: limit - len(high_priority)]
    return picked


STOPWORDS = {
    "the",
    "and",
    "for",
    "with",
    "that",
    "this",
    "from",
    "our",
    "study",
    "figure",
    "table",
    "results",
    "result",
    "shows",
    "showing",
    "overview",
}


def label_variants(label: str) -> list[str]:
    normalized = normalize_whitespace(label).lower()
    if not normalized:
        return []
    variants = {normalized}
    short = normalized.replace("figure", "fig").replace("table.", "table").replace("fig.", "fig")
    variants.add(short)
    extended_match = re.match(r"^extended data (fig(?:ure)?|table)\.?\s*(\d+[a-z]?)$", normalized)
    if extended_match:
        prefix = "extended data table" if extended_match.group(1) == "table" else "extended data fig"
        variants.update(
            {
                f"{prefix} {extended_match.group(2)}",
                f"{prefix}. {extended_match.group(2)}",
            }
        )
    scheme_match = re.match(r"^(scheme|algorithm)\.?\s*(\d+[a-z]?)$", normalized)
    if scheme_match:
        variants.update(
            {
                f"{scheme_match.group(1)} {scheme_match.group(2)}",
                f"{scheme_match.group(1)}. {scheme_match.group(2)}",
            }
        )
    digits = re.findall(r"\d+[a-z]?", normalized)
    if digits:
        number = digits[0]
        if normalized.startswith("fig"):
            variants.update({f"fig {number}", f"fig. {number}", f"figure {number}"})
        if normalized.startswith("table"):
            variants.update({f"table {number}", f"table. {number}"})
    return sorted(variants)


def caption_keywords(caption: str, *, limit: int = 5) -> list[str]:
    words = re.findall(r"[A-Za-z][A-Za-z-]{3,}", caption.lower())
    picked: list[str] = []
    for word in words:
        if word in STOPWORDS or word in picked:
            continue
        picked.append(word)
        if len(picked) >= limit:
            break
    return picked


def match_snippet(page_text: str, needle: str, *, radius: int = 90) -> str:
    lower = page_text.lower()
    idx = lower.find(needle.lower())
    if idx < 0:
        return ""
    start = max(0, idx - radius)
    end = min(len(page_text), idx + len(needle) + radius)
    snippet = normalize_whitespace(page_text[start:end])
    return snippet[:220]


def _normalize_label_for_match(label: str) -> str:
    """Normalize a figure/table label to a canonical form for matching.

    'Figure 3' / 'Fig. 3' / 'fig 3' / 'Figure. 3' all become 'fig 3'.
    'Table 2' / 'table. 2' / 'Table. 2' all become 'table 2'.
    """
    text = normalize_whitespace(label).lower()
    extended_figure_match = re.match(
        r"^extended\s+data\s+fig(?:ure)?\.?\s*(\d+[a-z]?)$",
        text,
    )
    if extended_figure_match:
        return f"extended data fig {extended_figure_match.group(1)}"
    extended_table_match = re.match(
        r"^extended\s+data\s+table\.?\s*(\d+[a-z]?)$",
        text,
    )
    if extended_table_match:
        return f"extended data table {extended_table_match.group(1)}"
    scheme_match = re.match(r"^(scheme|algorithm)\.?\s*(\d+[a-z]?)$", text)
    if scheme_match:
        return f"{scheme_match.group(1)} {scheme_match.group(2)}"
    supplementary_match = re.match(
        r"^supplementary\s+(fig(?:ure)?|table)\.?\s*(\d+[a-z]?)$",
        text,
    )
    if supplementary_match:
        prefix = "table" if supplementary_match.group(1) == "table" else "fig"
        return f"{prefix} s{supplementary_match.group(2)}"
    label_match = re.match(r"^(fig(?:ure)?|table)\.?\s*([as]?\d+[a-z]?)$", text)
    if label_match:
        prefix = "table" if label_match.group(1) == "table" else "fig"
        return f"{prefix} {label_match.group(2)}"
    return normalize_whitespace(text)


def _match_figure_asset(item_id: str, figure_assets: list[dict]) -> dict | None:
    """Find a figure-level asset whose label matches the plan item id."""
    target = _normalize_label_for_match(item_id)
    if not target:
        return None
    for asset in figure_assets:
        asset_label = _normalize_label_for_match(str(asset.get("label", "")))
        if asset_label == target:
            return asset
    return None


def _candidate_status_for_quality(asset: dict) -> str:
    signals = asset.get("quality_signals")
    status = signals.get("visual_quality_status", "") if isinstance(signals, dict) else ""
    if status == "usable":
        return "usable_candidate"
    if status == "reject":
        return "reject_visual_quality"
    return "needs_visual_quality_check"


def _asset_candidate(asset: dict, *, include_label: bool = False) -> dict:
    candidate = {
        "filename": asset.get("filename", ""),
        "path": asset.get("path", ""),
        "width": asset.get("width", 0),
        "height": asset.get("height", 0),
        "size_bytes": asset.get("size_bytes", 0),
    }
    if include_label:
        candidate["label"] = asset.get("label", "")
        candidate["extraction_level"] = asset.get("extraction_level", "figure")
        if isinstance(asset.get("quality_signals"), dict):
            candidate["quality_signals"] = asset.get("quality_signals")
        candidate["candidate_status"] = _candidate_status_for_quality(asset)
    return candidate


def attach_candidate_images(
    items: list[dict],
    page_assets: list[dict],
    image_assets: list[dict],
    figure_assets: list[dict] | None = None,
) -> list[dict]:
    figure_assets = figure_assets or []

    image_map: dict[int, list[dict]] = {}
    for image in image_assets:
        if not isinstance(image, dict):
            continue
        page_number = int(image.get("page_number", 0) or 0)
        if page_number <= 0:
            continue
        image_map.setdefault(page_number, []).append(image)

    figure_map: dict[int, list[dict]] = {}
    for asset in figure_assets:
        if not isinstance(asset, dict):
            continue
        page_number = int(asset.get("page_number", 0) or 0)
        if page_number <= 0:
            continue
        figure_map.setdefault(page_number, []).append(asset)

    has_visual = set()
    for page in page_assets:
        if not isinstance(page, dict):
            continue
        pn = int(page.get("page_number", 0) or 0)
        img_count = int(page.get("image_count", 0) or 0)
        fig_count = int(page.get("figure_count", 0) or 0)
        if img_count > 0 or fig_count > 0:
            has_visual.add(pn)
    pages_with_images = [
        page for page in page_assets
        if isinstance(page, dict) and int(page.get("page_number", 0) or 0) in has_visual
    ]

    for index, item in enumerate(items):
        item_id = str(item.get("id", ""))

        fig_match = _match_figure_asset(item_id, figure_assets)
        if fig_match:
            item["figure_asset_candidate"] = _asset_candidate(fig_match, include_label=True)

        variants = label_variants(item_id)
        keywords = caption_keywords(str(item.get("caption", "")))
        candidates: list[dict] = []
        for page in pages_with_images:
            page_number = int(page.get("page_number", 0) or 0)
            page_text = normalize_whitespace(str(page.get("page_text", "")))
            lower = page_text.lower()
            score = 0
            matched_terms: list[str] = []
            snippets: list[str] = []

            for variant in variants:
                if variant and variant in lower:
                    score += 5
                    matched_terms.append(variant)
                    snippet = match_snippet(page_text, variant)
                    if snippet:
                        snippets.append(snippet)
                    break

            keyword_hits = 0
            for keyword in keywords:
                if keyword in lower:
                    keyword_hits += 1
                    matched_terms.append(keyword)
                    snippet = match_snippet(page_text, keyword)
                    if snippet:
                        snippets.append(snippet)
            score += min(keyword_hits, 3)

            if score <= 0:
                continue

            candidates.append(
                {
                    "page_number": page_number,
                    "score": score,
                    "matched_terms": matched_terms[:6],
                    "snippet": snippets[0] if snippets else normalize_whitespace(str(page.get("text_preview", "")))[:220],
                    "images": [
                        _asset_candidate(img)
                        for img in image_map.get(page_number, [])[:3]
                    ],
                    "figure_assets": [
                        _asset_candidate(asset, include_label=True)
                        for asset in figure_map.get(page_number, [])[:3]
                    ],
                }
            )

        candidates.sort(key=lambda candidate: (-candidate["score"], candidate["page_number"]))
        item["candidate_pages"] = candidates[:3]
        if fig_match:
            item["matching_strategy"] = "figure-asset-candidate"
        elif candidates:
            item["matching_strategy"] = "page-proximity-and-caption-cues"
        else:
            item["candidate_status"] = "no_match_found"
            item["matching_strategy"] = "no-match-found"
    return items


def main() -> None:
    from common import emit

    args = parser().parse_args()
    primary = maybe_load_json_record(args.input) if args.input else None
    evidence = maybe_load_json_record(args.evidence) if args.evidence else None
    assets = maybe_load_json_record(args.assets) if args.assets else None
    data = merge_inputs(primary, evidence, assets)
    if not data:
        raise SystemExit("plan_figures.py requires at least one JSON input.")

    evidence_pack = data.get("evidence_pack", {}) if isinstance(data.get("evidence_pack"), dict) else {}
    page_assets = data.get("page_assets", []) if isinstance(data.get("page_assets"), list) else []
    image_assets = data.get("image_assets", []) if isinstance(data.get("image_assets"), list) else []
    figure_assets = data.get("figure_assets", []) if isinstance(data.get("figure_assets"), list) else []
    language = normalize_output_language(
        runtime_config(cli_overrides={"output_language": args.language})["output_language"]
    )
    items = build_figure_items(evidence_pack, limit=args.max_items, language=language)
    items = attach_candidate_images(items, page_assets, image_assets, figure_assets)
    payload = {
        "status": "ok",
        "script": "plan_figures.py",
        "paper_id": args.paper_id or data.get("paper_id", ""),
        "output_language": language,
        "figure_plan": {
            "paper_id": args.paper_id or data.get("paper_id", ""),
            "figures": items,
        },
    }
    emit(payload, args.output)


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""Create a full figure/table decision table from source captions and figure planning."""

from __future__ import annotations

import argparse
import re
from pathlib import Path
from typing import Any

from common import (
    caption_label_key,
    caption_preference_score,
    emit,
    file_sha256,
    maybe_load_json_record,
    normalize_whitespace,
    runtime_config,
)
from contracts import WRITING_CONTRACT_RULES
from extract_pdf_assets import _render_crop, save_image_bytes
from localization import normalize_output_language, require_artifact_output_language
from source_corpus import SourceCorpusLoadError, load_source_corpus

DECISION_VALUES = set(WRITING_CONTRACT_RULES["figure_decision_values"])
INSERTABLE_KINDS = set(WRITING_CONTRACT_RULES["usable_insert_candidate"]["kinds"])
REVIEW_RENDER_DPI = int(
    WRITING_CONTRACT_RULES["visual_review_contract"]["selected_render_dpi"]
)


def parser() -> argparse.ArgumentParser:
    p = argparse.ArgumentParser(description=__doc__ or "plan figure/table decisions")
    p.add_argument("--source-manifest", default="", help="Source manifest JSON path or string.")
    p.add_argument("--figures", default="", help="Figure plan JSON path or string.")
    p.add_argument("--assets", default="", help="PDF assets JSON path or string.")
    p.add_argument(
        "--review-decisions",
        default="",
        help="Existing decision JSON whose requested bounded repairs should be applied.",
    )
    p.add_argument(
        "--asset-subdir",
        default="images",
        help="Paper-relative images directory from save preflight.",
    )
    p.add_argument("--output", default="", help="Output JSON path.")
    p.add_argument("--paper-id", default="", help="Canonical paper id.")
    p.add_argument(
        "--language",
        default="",
        help="Run Override for output language: en or zh-CN.",
    )
    return p


def load_record(value: str) -> dict[str, Any]:
    record = maybe_load_json_record(value)
    if record is None:
        raise SystemExit(f"Expected JSON object for {value!r}.")
    return record


def normalize_label(label: str) -> str:
    text = normalize_whitespace(label).lower()
    text = text.replace("figure", "fig")
    text = re.sub(r"\bfig\.\s*", "fig ", text)
    text = re.sub(r"\btable\.\s*", "table ", text)
    return normalize_whitespace(text)


def input_is_file(value: str) -> bool:
    if not value or value.strip().startswith("{"):
        return False
    try:
        return Path(value).expanduser().is_file()
    except OSError:
        return False


def normalized_caption_items(raw_items: list[dict[str, Any]]) -> list[dict[str, Any]]:
    grouped: dict[str, dict[str, Any]] = {}
    scores: dict[str, int] = {}
    order: list[str] = []
    for item in raw_items:
        if not isinstance(item, dict):
            continue
        label = normalize_whitespace(str(item.get("id") or item.get("label") or ""))
        caption = normalize_whitespace(str(item.get("caption", "")))
        if not label and not caption:
            continue
        page = item.get("page") or item.get("page_number") or 0
        group_key = caption_label_key(label)
        if not group_key:
            continue
        candidate = {
            "kind": item.get("kind", ""),
            "label": label,
            "caption": caption,
            "page": page,
            "pages": item.get("pages") or ([page] if page else []),
            "section_id": item.get("section_id", ""),
        }
        score = caption_preference_score(label, caption)
        if group_key not in grouped:
            grouped[group_key] = candidate
            scores[group_key] = score
            order.append(group_key)
            continue
        if score > scores[group_key]:
            grouped[group_key] = candidate
            scores[group_key] = score
    return [grouped[key] for key in order]


def caption_items(source_manifest: dict[str, Any]) -> list[dict[str, Any]]:
    captions = (
        source_manifest.get("captions", {})
        if isinstance(source_manifest.get("captions"), dict)
        else {}
    )
    raw_items: list[dict[str, Any]] = []
    for kind, key in (("figure", "figures"), ("table", "tables")):
        for item in captions.get(key, []) or []:
            if isinstance(item, dict):
                raw_items.append({**dict(item), "kind": kind})
    return normalized_caption_items(raw_items)


def source_caption_items(
    source_manifest: dict[str, Any],
    source_manifest_input: str = "",
) -> list[dict[str, Any]]:
    if input_is_file(source_manifest_input):
        try:
            corpus = load_source_corpus(source_manifest_input)
        except SourceCorpusLoadError as exc:
            raise SystemExit(str(exc)) from exc
        return normalized_caption_items(corpus.caption_items())
    return caption_items(source_manifest)


def planned_items(figures_wrapper: dict[str, Any]) -> dict[str, dict[str, Any]]:
    figure_plan = (
        figures_wrapper.get("figure_plan", {})
        if isinstance(figures_wrapper.get("figure_plan"), dict)
        else {}
    )
    planned: dict[str, dict[str, Any]] = {}
    for item in figure_plan.get("figures", []) or []:
        if not isinstance(item, dict):
            continue
        label = normalize_whitespace(str(item.get("id", "")))
        if label:
            planned[normalize_label(label)] = item
    return planned


def quality_status(plan_item: dict[str, Any]) -> str:
    candidate = plan_item.get("figure_asset_candidate")
    if not isinstance(candidate, dict):
        return ""
    status = normalize_whitespace(str(candidate.get("candidate_status", "")))
    if status:
        return status
    signals = candidate.get("quality_signals", {})
    if isinstance(signals, dict):
        return normalize_whitespace(str(signals.get("visual_quality_status", "")))
    return ""


def figure_asset_candidate(plan_item: dict[str, Any]) -> dict[str, Any]:
    candidate = plan_item.get("figure_asset_candidate")
    return candidate if isinstance(candidate, dict) else {}


def source_image_path(plan_item: dict[str, Any]) -> str:
    return normalize_whitespace(str(figure_asset_candidate(plan_item).get("path", "")))


def source_image_filename(plan_item: dict[str, Any]) -> str:
    candidate = figure_asset_candidate(plan_item)
    filename = normalize_whitespace(str(candidate.get("filename", "")))
    if filename:
        return filename
    path = source_image_path(plan_item)
    # Split on both separators so a Windows-style backslash path does not get
    # returned whole (rsplit("/") alone leaves "C:\\...\\fig.png" intact).
    return re.split(r"[\\/]", path)[-1] if path else ""


def should_review(caption: dict[str, Any], plan_item: dict[str, Any], status: str) -> bool:
    if status != "usable_candidate":
        return False
    if caption.get("kind") not in INSERTABLE_KINDS:
        return False
    path = source_image_path(plan_item)
    return bool(path and Path(path).expanduser().is_file())


def matched_figure_asset(assets_wrapper: dict[str, Any], label: str) -> dict[str, Any]:
    target = normalize_label(label)
    for asset in assets_wrapper.get("figure_assets", []) or []:
        if isinstance(asset, dict) and normalize_label(str(asset.get("label", ""))) == target:
            return asset
    return {}


def normalized_bbox(page_rect, bbox: list[float]) -> list[float]:
    width = max(float(page_rect.width), 1.0)
    height = max(float(page_rect.height), 1.0)
    return [
        round((bbox[0] - page_rect.x0) / width, 6),
        round((bbox[1] - page_rect.y0) / height, 6),
        round((bbox[2] - page_rect.x0) / width, 6),
        round((bbox[3] - page_rect.y0) / height, 6),
    ]


def prepare_review_candidate(
    caption: dict[str, Any],
    plan_item: dict[str, Any],
    assets_wrapper: dict[str, Any],
) -> dict[str, Any]:
    image_path = source_image_path(plan_item)
    filename = source_image_filename(plan_item)
    asset = matched_figure_asset(assets_wrapper, str(caption.get("label", "")))
    pdf_path = Path(str(assets_wrapper.get("pdf_path", ""))).expanduser()
    bbox = asset.get("bbox_pt", []) if isinstance(asset.get("bbox_pt"), list) else []
    page_number = int(asset.get("page_number", 0) or 0)
    evidence = {
        "candidate_path": image_path,
        "page_preview_path": str(asset.get("page_preview_path", "")),
        "source_pdf_path": str(pdf_path) if str(pdf_path) != "." else "",
        "source_page": page_number,
        "caption": caption.get("caption", ""),
        "bbox_pt": bbox,
        "normalized_bbox": [],
        "render_dpi": 0,
    }
    if pdf_path.is_file() and page_number > 0 and len(bbox) == 4:
        doc = None
        try:
            from common import fitz

            if fitz is not None:
                doc = fitz.open(pdf_path.resolve())
                page = doc[page_number - 1]
                source = Path(image_path).expanduser()
                output_path = source.with_name(f"{source.stem}_review.png")
                save_image_bytes(
                    output_path,
                    _render_crop(page, tuple(float(value) for value in bbox), REVIEW_RENDER_DPI),
                )
                image_path = str(output_path)
                filename = output_path.name
                evidence["candidate_path"] = image_path
                evidence["normalized_bbox"] = normalized_bbox(page.rect, bbox)
                evidence["render_dpi"] = REVIEW_RENDER_DPI
        except (OSError, RuntimeError, ValueError, IndexError):
            pass
        finally:
            if doc is not None:
                doc.close()
    return {
        "path": image_path,
        "filename": filename,
        "sha256": file_sha256(image_path),
        "review_evidence": evidence,
    }


def validate_normalized_bbox(value: Any) -> list[float]:
    if not isinstance(value, list) or len(value) != 4:
        raise SystemExit("revised_bbox must be [x0, y0, x1, y1].")
    if not all(
        isinstance(item, (int, float)) and not isinstance(item, bool)
        for item in value
    ):
        raise SystemExit("revised_bbox coordinates must be numeric.")
    bbox = [float(item) for item in value]
    x0, y0, x1, y1 = bbox
    if not (0.0 <= x0 < x1 <= 1.0 and 0.0 <= y0 < y1 <= 1.0):
        raise SystemExit(
            "revised_bbox coordinates must be ordered and within [0, 1]."
        )
    return bbox


def apply_requested_repairs(wrapper: dict[str, Any]) -> dict[str, Any]:
    asset_subdir = checked_asset_subdir(wrapper.get("asset_subdir", "images"))
    decisions = wrapper.get("decisions", [])
    if not isinstance(decisions, list):
        raise SystemExit("--review-decisions requires a decisions list.")
    contract = WRITING_CONTRACT_RULES["visual_review_contract"]
    repairable = set(contract["repairable_failure_reasons"])
    repair_limit = int(contract["repair_limit"])

    for item in decisions:
        if not isinstance(item, dict):
            continue
        review = item.get("visual_review", {})
        if not isinstance(review, dict) or review.get("status") != "repair_requested":
            continue
        source_path = Path(str(item.get("source_image_path", ""))).expanduser()
        current_sha256 = file_sha256(str(source_path))
        if not current_sha256 or review.get("reviewed_asset_sha256") != current_sha256:
            raise SystemExit("Bounded crop repair requires a current reviewed asset SHA-256.")
        attempts = int(review.get("repair_attempts", 0) or 0)
        if attempts >= repair_limit:
            review.update(
                {
                    "status": "fail",
                    "failure_reason": "repair_limit_exhausted",
                    "reviewed_asset_sha256": current_sha256,
                }
            )
            item["decision"] = "visual_defect"
            item["skip_reason"] = "repair_limit_exhausted"
            continue
        failure_reason = normalize_whitespace(str(review.get("failure_reason", "")))
        if failure_reason not in repairable:
            raise SystemExit(
                f"Failure reason is not repairable by recropping: {failure_reason}"
            )
        revised_bbox = validate_normalized_bbox(review.get("revised_bbox"))
        evidence = item.get("review_evidence", {})
        if not isinstance(evidence, dict):
            raise SystemExit("Bounded crop repair requires review_evidence.")
        pdf_path = Path(str(evidence.get("source_pdf_path", ""))).expanduser()
        page_number = int(evidence.get("source_page", 0) or 0)
        if not pdf_path.is_file() or page_number <= 0:
            raise SystemExit("Bounded crop repair requires a source PDF and page.")

        from common import fitz

        if fitz is None:
            raise SystemExit("Bounded crop repair requires PyMuPDF (`fitz`).")
        doc = fitz.open(pdf_path.resolve())
        try:
            page = doc[page_number - 1]
            page_rect = page.rect
            x0, y0, x1, y1 = revised_bbox
            bbox_pt = [
                page_rect.x0 + x0 * page_rect.width,
                page_rect.y0 + y0 * page_rect.height,
                page_rect.x0 + x1 * page_rect.width,
                page_rect.y0 + y1 * page_rect.height,
            ]
            repair_path = source_path.with_name(
                f"{source_path.stem}_repair{attempts + 1}.png"
            )
            save_image_bytes(
                repair_path,
                _render_crop(page, tuple(bbox_pt), REVIEW_RENDER_DPI),
            )
        finally:
            doc.close()

        repaired_sha256 = file_sha256(str(repair_path))
        item["decision"] = "review_pending"
        item["source_image_path"] = str(repair_path)
        item["source_image_filename"] = repair_path.name
        item["source_image_sha256"] = repaired_sha256
        if item.get("relative_markdown_embed"):
            item["relative_markdown_embed"] = (
                f"![{item.get('source_id') or repair_path.name}]"
                f"({asset_subdir}/{repair_path.name})"
            )
        evidence.update(
            {
                "candidate_path": str(repair_path),
                "bbox_pt": [round(value, 6) for value in bbox_pt],
                "normalized_bbox": revised_bbox,
                "render_dpi": REVIEW_RENDER_DPI,
            }
        )
        item["review_evidence"] = evidence
        item["visual_review"] = {
            "status": "pending",
            "reviewed_asset_sha256": "",
            "preserved_scientific_elements": [],
            "omitted_scientific_elements": [],
            "notes": "",
            "failure_reason": "",
            "repair_attempts": attempts + 1,
            "revised_bbox": revised_bbox,
        }
    wrapper["decisions"] = decisions
    return wrapper


def decide(
    caption: dict[str, Any],
    plan_item: dict[str, Any] | None,
    assets_wrapper: dict[str, Any] | None = None,
    asset_subdir: str = "images",
) -> dict[str, Any]:
    label = normalize_whitespace(str(caption.get("label", "")))
    fallback_caption = normalize_whitespace(str(caption.get("caption", "")))[:40]
    base = {
        "item_id": f"{caption.get('kind', 'item')}:{label or fallback_caption}",
        "source_id": label,
        "kind": caption.get("kind", ""),
        "label": label,
        "caption": caption.get("caption", ""),
        "pages": caption.get("pages", []),
        "section_id": caption.get("section_id", ""),
        "decision": "low_priority",
        "reason": "caption_detected_but_not_selected_by_figure_plan",
        "skip_reason": "",
        "visual_quality_status": "",
        "priority": 99,
        "target_section": "",
    }
    if not plan_item:
        return base

    status = quality_status(plan_item)
    base["priority"] = int(plan_item.get("priority", 99) or 99)
    base["target_section"] = plan_item.get("section", "")
    base["plan_kind"] = plan_item.get("kind", "")
    base["visual_quality_status"] = status
    base["reason"] = plan_item.get("reason", "") or "selected_by_figure_plan"
    assets_wrapper = assets_wrapper or {}
    prepared = (
        prepare_review_candidate(caption, plan_item, assets_wrapper)
        if status == "usable_candidate"
        else {}
    )
    filename = str(prepared.get("filename") or source_image_filename(plan_item))
    image_path = str(prepared.get("path") or source_image_path(plan_item))
    if filename:
        base["source_image_filename"] = filename
        base["relative_markdown_embed"] = f"![{label or filename}]({asset_subdir}/{filename})"
    if image_path:
        base["source_image_path"] = image_path
    if status in {"reject", "reject_visual_quality"}:
        base["decision"] = "visual_defect"
        base["skip_reason"] = "visual_quality_gate_rejected_candidate"
    elif should_review(caption, plan_item, status):
        base["decision"] = "review_pending"
        base["materialization_status"] = "pending"
        base["skip_reason"] = ""
        base["source_image_sha256"] = str(prepared.get("sha256") or file_sha256(image_path))
        base["review_evidence"] = prepared.get("review_evidence", {})
        base["visual_review"] = {
            "status": "pending",
            "reviewed_asset_sha256": "",
            "preserved_scientific_elements": [],
            "omitted_scientific_elements": [],
            "notes": "",
            "failure_reason": "",
            "repair_attempts": 0,
            "revised_bbox": [],
        }
    else:
        base["decision"] = "placeholder"
        if status == "usable_candidate":
            base["skip_reason"] = "asset_candidate_missing"
        elif status:
            base["skip_reason"] = "visual_quality_requires_review"
        else:
            base["skip_reason"] = "asset_candidate_missing"
    return base


def build_decisions(
    source_manifest: dict[str, Any],
    figures_wrapper: dict[str, Any],
    source_manifest_input: str = "",
    assets_wrapper: dict[str, Any] | None = None,
    asset_subdir: str = "images",
) -> list[dict[str, Any]]:
    planned = planned_items(figures_wrapper)
    decisions = [
        decide(
            caption,
            planned.get(normalize_label(str(caption.get("label", "")))),
            assets_wrapper,
            checked_asset_subdir(asset_subdir),
        )
        for caption in source_caption_items(source_manifest, source_manifest_input)
    ]
    for decision in decisions:
        if decision["decision"] not in DECISION_VALUES:
            decision["decision"] = "skip"
            decision["skip_reason"] = "invalid_decision_normalized_to_skip"
    return decisions


def checked_asset_subdir(value: str) -> str:
    if not isinstance(value, str) or not re.fullmatch(r"images(?:/[0-9a-f]{64})?", value):
        raise SystemExit("Asset directory must be images or images/<source-sha256> from preflight.")
    return value


def main() -> None:
    args = parser().parse_args()
    language = normalize_output_language(
        runtime_config(cli_overrides={"output_language": args.language})["output_language"]
    )
    if args.review_decisions:
        payload = load_record(args.review_decisions)
        require_artifact_output_language(payload, "Figure/Table Decisions", language)
        payload = apply_requested_repairs(payload)
        payload["status"] = "ok"
        payload["script"] = "plan_figure_table_decisions.py"
        emit(payload, args.output)
        return
    if not args.source_manifest:
        raise SystemExit(
            "plan_figure_table_decisions.py requires --source-manifest "
            "or --review-decisions."
        )
    source_manifest = load_record(args.source_manifest)
    figures = load_record(args.figures) if args.figures else {}
    require_artifact_output_language(figures, "Figure Plan", language)
    assets = load_record(args.assets) if args.assets else {}
    decisions = build_decisions(
        source_manifest,
        figures,
        args.source_manifest,
        assets,
        checked_asset_subdir(args.asset_subdir),
    )
    payload = {
        "status": "ok",
        "script": "plan_figure_table_decisions.py",
        "output_language": language,
        "paper_id": args.paper_id or source_manifest.get("paper_id", ""),
        "asset_subdir": args.asset_subdir,
        "decisions": decisions,
        "summary": {
            "total_items": len(decisions),
            "by_decision": {
                value: sum(1 for item in decisions if item.get("decision") == value)
                for value in sorted(DECISION_VALUES)
            },
        },
    }
    emit(payload, args.output)


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""Resolve a title, DOI, URL, arXiv ID, local PDF, or Zotero item into one paper identity."""

from __future__ import annotations

from typing import Any

from _zotero_local import lookup_zotero_local
from common import (
    base_parser,
    emit,
    infer_source_type,
    maybe_load_json_record,
    paper_id_for_record,
    require_ok_input_artifact,
    resolve_reference,
)

ZOTERO_MODES = ("auto", "off", "required")
ZOTERO_QUERY_SOURCE_TYPES = {
    "arxiv_id",
    "arxiv_url",
    "doi",
    "doi_url",
    "title",
    "title_query",
    "zotero_key",
}
ZOTERO_BYPASS_SOURCE_TYPES = {"local_pdf"}


class ZoteroResolutionError(RuntimeError):
    """Raised when Zotero mode requires the resolver to stop fail-closed."""

    def __init__(self, message: str, *, lookup: dict[str, Any], failure_class: str) -> None:
        super().__init__(message)
        self.lookup = lookup
        self.failure_class = failure_class


def parser():
    p = base_parser(__doc__ or "resolve paper")
    p.add_argument(
        "--zotero-mode",
        choices=ZOTERO_MODES,
        default="auto",
        help=(
            "Local Zotero lookup policy: auto prefers unique local matches, off preserves "
            "the web-only path, and required fails unless Zotero resolves the reference."
        ),
    )
    return p


def zotero_lookup_summary(result: dict[str, Any], *, mode: str) -> dict[str, Any]:
    summary: dict[str, Any] = {
        "mode": mode,
        "status": str(result.get("status", "error")),
        "match_kind": str(result.get("match_kind", "")),
    }
    if result.get("candidate_count") is not None:
        summary["candidate_count"] = result["candidate_count"]
    probe = result.get("probe", {})
    if isinstance(probe, dict) and probe:
        summary["api_status"] = str(probe.get("status", ""))
        summary["api_version"] = str(probe.get("api_version", ""))
        summary["schema_version"] = str(probe.get("schema_version", ""))
    error = result.get("error", {})
    if isinstance(error, dict) and error:
        summary["error"] = dict(error)
    return summary


def _zotero_failure(result: dict[str, Any], *, mode: str) -> ZoteroResolutionError:
    status = str(result.get("status", "error"))
    summary = zotero_lookup_summary(result, mode=mode)
    if status == "ambiguous":
        return ZoteroResolutionError(
            "Zotero returned multiple equally plausible local items; provide a DOI, arXiv ID, "
            "or exact Zotero item key.",
            lookup=summary,
            failure_class="zotero_ambiguous_match",
        )
    error = result.get("error", {})
    error_code = str(error.get("code", "")) if isinstance(error, dict) else ""
    if str(result.get("match_kind", "")) == "unsupported_reference":
        failure_class = "zotero_unsupported_reference"
    else:
        failure_class = error_code or (
            "zotero_item_not_found" if status == "not_found" else "zotero_lookup_failed"
        )
    return ZoteroResolutionError(
        "Zotero Local API could not uniquely resolve this reference while "
        f"--zotero-mode={mode} is active.",
        lookup=summary,
        failure_class=failure_class,
    )


def resolve_scalar_reference(reference: str, *, zotero_mode: str = "auto") -> dict[str, Any]:
    if zotero_mode not in ZOTERO_MODES:
        raise ValueError(f"Unsupported Zotero mode: {zotero_mode}")
    source_type = infer_source_type(reference)
    if zotero_mode == "off" or source_type in ZOTERO_BYPASS_SOURCE_TYPES:
        return resolve_reference(reference)

    if source_type not in ZOTERO_QUERY_SOURCE_TYPES:
        if zotero_mode == "required":
            result = {
                "status": "not_found",
                "match_kind": "unsupported_reference",
            }
            raise _zotero_failure(result, mode=zotero_mode)
        return resolve_reference(reference)

    lookup = lookup_zotero_local(reference, reference_type=source_type)
    lookup_status = str(lookup.get("status", "error"))
    explicit_zotero_reference = source_type == "zotero_key" or reference.strip().lower().startswith(
        "zotero://select/"
    )
    if lookup_status == "match":
        record = lookup.get("record", {})
        if not isinstance(record, dict):
            lookup = {
                "status": "error",
                "match_kind": source_type,
                "error": {
                    "code": "zotero_invalid_response",
                    "message": "Zotero lookup returned an invalid record.",
                    "retryable": False,
                },
            }
        else:
            if explicit_zotero_reference:
                resolved = dict(record)
                resolved["zotero_lookup"] = zotero_lookup_summary(
                    lookup,
                    mode=zotero_mode,
                )
                return resolved
            resolved = resolve_reference(reference)
            resolved["identity_observations"] = [
                {
                    "provider": "zotero",
                    "retrieved_by": {"kind": source_type, "value": reference.strip()},
                    "relation": {
                        "kind": "zotero_lookup",
                        "match_kind": str(lookup.get("match_kind", source_type)),
                        "match_resolution": "unique_exact",
                    },
                    "record": dict(record),
                }
            ]
            resolved["zotero_lookup"] = zotero_lookup_summary(
                lookup,
                mode=zotero_mode,
            )
            return resolved

    if lookup_status == "ambiguous" or zotero_mode == "required" or explicit_zotero_reference:
        raise _zotero_failure(lookup, mode=zotero_mode)

    resolved = resolve_reference(reference)
    resolved["zotero_lookup"] = zotero_lookup_summary(lookup, mode=zotero_mode)
    return resolved


def main(argv: list[str] | None = None) -> None:
    args = parser().parse_args(argv)

    if not args.input:
        raise SystemExit("resolve_paper.py requires --input.")

    input_record = maybe_load_json_record(args.input)
    if input_record is not None:
        resolved = dict(require_ok_input_artifact(input_record, "resolve_paper.py"))
    else:
        try:
            resolved = resolve_scalar_reference(args.input, zotero_mode=args.zotero_mode)
        except ZoteroResolutionError as exc:
            emit(
                {
                    "status": "error",
                    "run_status": "failed",
                    "script": "resolve_paper.py",
                    "failure_class": exc.failure_class,
                    "failure_summary": str(exc),
                    "zotero_lookup": exc.lookup,
                },
                args.output,
            )
            raise SystemExit(str(exc)) from exc

    resolved["paper_id"] = (
        args.paper_id or resolved.get("paper_id") or paper_id_for_record(resolved)
    )
    resolved["status"] = resolved.get("status") or "ok"
    resolved["script"] = "resolve_paper.py"
    emit(resolved, args.output)


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""Run the deterministic DeepPaperNote stages sequentially for one paper."""

from __future__ import annotations

import argparse
import os
import subprocess
import sys
from pathlib import Path

from common import runtime_config


def parser() -> argparse.ArgumentParser:
    p = argparse.ArgumentParser(description=__doc__ or "run pipeline")
    p.add_argument(
        "--input",
        required=True,
        help="Paper title, DOI, URL, arXiv id, local PDF path, or JSON artifact.",
    )
    p.add_argument(
        "--workdir",
        default="tmp/DeepPaperNote_runs",
        help="Directory for intermediate artifacts.",
    )
    p.add_argument("--prefix", default="run", help="Filename prefix for artifacts.")
    p.add_argument(
        "--zotero-mode",
        choices=("auto", "off", "required"),
        default="auto",
        help="Local Zotero lookup policy used by the resolve stage.",
    )
    p.add_argument(
        "--language",
        default="",
        choices=("", "en", "zh-CN"),
        help="Run Override for the output language contract.",
    )
    p.add_argument("--save-mode", choices=("workspace", "obsidian"), default="")
    p.add_argument("--vault", default="", help="Run Override for the Obsidian Vault.")
    p.add_argument("--target-directory", default="", help="Chosen existing paper directory.")
    p.add_argument("--source-sha256", default="", help="Chosen archived source.")
    p.add_argument("--papers-dir", default="", help="Run Override for the Vault paper directory.")
    return p


def run_step(cmd: list[str], *, env: dict[str, str]) -> None:
    subprocess.run(cmd, check=True, env=env)


def main() -> None:
    args = parser().parse_args()
    try:
        config = runtime_config(
            cli_overrides={
                "output_language": args.language,
                "save_mode": args.save_mode,
                "obsidian_vault": args.vault,
                "papers_dir": args.papers_dir,
            }
        )
    except RuntimeError as exc:
        raise SystemExit(str(exc)) from exc
    args.language = config["output_language"]
    run_environment = os.environ.copy()
    for field, name in {
        "output_language": "DEEPPAPERNOTE_OUTPUT_LANGUAGE",
        "save_mode": "DEEPPAPERNOTE_SAVE_MODE",
        "obsidian_vault": "DEEPPAPERNOTE_OBSIDIAN_VAULT",
        "papers_dir": "DEEPPAPERNOTE_PAPERS_DIR",
    }.items():
        value = str(config.get(field, "")).strip()
        if value:
            run_environment[name] = value
        else:
            run_environment.pop(name, None)
    scripts_dir = Path(__file__).resolve().parent
    workdir = Path(args.workdir).expanduser().resolve()
    workdir.mkdir(parents=True, exist_ok=True)

    resolve_json = workdir / f"{args.prefix}_resolve.json"
    metadata_json = workdir / f"{args.prefix}_metadata.json"
    identity_json = workdir / f"{args.prefix}_identity.json"
    identity_trace_json = workdir / f"{args.prefix}_identity_repair_trace.json"
    fetch_json = workdir / f"{args.prefix}_fetch.json"
    pdf_dir = workdir / f"{args.prefix}_pdfs"
    source_manifest_json = workdir / f"{args.prefix}_source_manifest.json"
    raw_sections_jsonl = workdir / f"{args.prefix}_raw_sections.jsonl"
    full_text_md = workdir / f"{args.prefix}_full_text.md"
    evidence_json = workdir / f"{args.prefix}_evidence.json"
    assets_json = workdir / f"{args.prefix}_assets.json"
    assets_dir = workdir / f"{args.prefix}_assets"
    figures_json = workdir / f"{args.prefix}_figures.json"
    figure_decisions_json = workdir / f"{args.prefix}_figure_table_decisions.json"
    bundle_json = workdir / f"{args.prefix}_bundle.json"
    py = sys.executable
    run_step(
        [
            py,
            str(scripts_dir / "resolve_paper.py"),
            "--input",
            args.input,
            "--zotero-mode",
            args.zotero_mode,
            "--output",
            str(resolve_json),
        ],
        env=run_environment,
    )
    run_step(
        [
            py,
            str(scripts_dir / "collect_metadata.py"),
            "--input",
            str(resolve_json),
            "--output",
            str(metadata_json),
        ],
        env=run_environment,
    )
    run_step(
        [
            py,
            str(scripts_dir / "build_identity_contract.py"),
            "--input",
            str(metadata_json),
            "--resolve",
            str(resolve_json),
            "--trace-output",
            str(identity_trace_json),
            "--output",
            str(identity_json),
        ],
        env=run_environment,
    )
    run_step(
        [
            py,
            str(scripts_dir / "fetch_pdf.py"),
            "--input",
            str(metadata_json),
            "--identity",
            str(identity_json),
            "--reference",
            args.input,
            "--target-directory",
            args.target_directory,
            "--source-sha256",
            args.source_sha256,
            "--dest-dir",
            str(pdf_dir),
            "--output",
            str(fetch_json),
        ],
        env=run_environment,
    )
    run_step(
        [
            py,
            str(scripts_dir / "extract_source_text.py"),
            "--input",
            str(fetch_json),
            "--output",
            str(source_manifest_json),
            "--raw-sections-output",
            str(raw_sections_jsonl),
            "--full-text-output",
            str(full_text_md),
        ],
        env=run_environment,
    )
    run_step(
        [
            py,
            str(scripts_dir / "extract_evidence.py"),
            "--input",
            str(fetch_json),
            "--source-manifest",
            str(source_manifest_json),
            "--output",
            str(evidence_json),
        ],
        env=run_environment,
    )
    run_step(
        [
            py,
            str(scripts_dir / "extract_pdf_assets.py"),
            "--input",
            str(fetch_json),
            "--assets-dir",
            str(assets_dir),
            "--output",
            str(assets_json),
        ],
        env=run_environment,
    )
    run_step(
        [
            py,
            str(scripts_dir / "plan_figures.py"),
            "--evidence",
            str(evidence_json),
            "--assets",
            str(assets_json),
            "--language",
            args.language,
            "--output",
            str(figures_json),
        ],
        env=run_environment,
    )
    run_step(
        [
            py,
            str(scripts_dir / "plan_figure_table_decisions.py"),
            "--source-manifest",
            str(source_manifest_json),
            "--figures",
            str(figures_json),
            "--assets",
            str(assets_json),
            "--language",
            args.language,
            "--output",
            str(figure_decisions_json),
        ],
        env=run_environment,
    )
    bundle_command = [
            py,
            str(scripts_dir / "build_synthesis_bundle.py"),
            "--metadata",
            str(metadata_json),
            "--evidence",
            str(evidence_json),
            "--figures",
            str(figures_json),
            "--assets",
            str(assets_json),
            "--source-manifest",
            str(source_manifest_json),
            "--figure-decisions",
            str(figure_decisions_json),
            "--output",
            str(bundle_json),
        ]
    if args.language:
        bundle_command.extend(["--language", args.language])
    run_step(bundle_command, env=run_environment)

    print(
        "\n".join(
            [
                f"resolve={resolve_json}",
                f"metadata={metadata_json}",
                f"identity={identity_json}",
                f"identity_repair_trace={identity_trace_json}",
                f"fetch={fetch_json}",
                f"source_manifest={source_manifest_json}",
                f"raw_sections={raw_sections_jsonl}",
                f"full_text_md={full_text_md}",
                f"evidence={evidence_json}",
                f"assets={assets_json}",
                f"figures={figures_json}",
                f"figure_table_decisions={figure_decisions_json}",
                f"bundle={bundle_json}",
            ]
        )
    )


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""Source Corpus loader, validator, and deterministic derived views."""

from __future__ import annotations

import hashlib
import json
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any


SOURCE_CORPUS_ISSUE_SEVERITY = "error"


class SourceCorpusLoadError(RuntimeError):
    """Raised when the source manifest itself cannot be constructed."""


def _sha256_text(text: str) -> str:
    return hashlib.sha256((text or "").encode("utf-8")).hexdigest()


def _issue(code: str, *, message: str, path: str = "", **details: Any) -> dict[str, Any]:
    issue: dict[str, Any] = {
        "code": code,
        "severity": SOURCE_CORPUS_ISSUE_SEVERITY,
        "message": message,
    }
    if path:
        issue["path"] = path
    issue.update(details)
    return issue


def _copy_record(value: Any) -> dict[str, Any]:
    return dict(value) if isinstance(value, dict) else {}


def _copy_list(value: Any) -> list[Any]:
    return list(value) if isinstance(value, list) else []


def _positive_int(value: Any) -> int | None:
    if isinstance(value, bool):
        return None
    try:
        number = int(value)
    except (TypeError, ValueError):
        return None
    return number if number >= 1 else None


def _record_has_appendix_hint(record: dict[str, Any]) -> bool:
    searchable = " ".join(
        str(record.get(key, ""))
        for key in ("kind", "title", "section_id")
    ).lower()
    return any(
        token in searchable
        for token in ("appendix", "appendices", "supplementary", "附录", "补充材料")
    )


def _resolve_manifest_path(source_manifest_path: str | Path) -> Path:
    return Path(source_manifest_path).expanduser().resolve()


def _resolve_raw_sections_path(manifest_path: Path, manifest: dict[str, Any]) -> Path | None:
    raw_path_value = str(manifest.get("raw_sections_path", "")).strip()
    if not raw_path_value:
        return None
    raw_path = Path(raw_path_value).expanduser()
    if not raw_path.is_absolute():
        raw_path = manifest_path.parent / raw_path
    return raw_path.resolve()


def _load_manifest(path: Path) -> dict[str, Any]:
    if not path.is_file():
        raise SourceCorpusLoadError(f"source manifest not found: {path}")
    try:
        data = json.loads(path.read_text(encoding="utf-8"))
    except json.JSONDecodeError as exc:
        raise SourceCorpusLoadError(f"source manifest is invalid JSON: {path}") from exc
    if not isinstance(data, dict):
        raise SourceCorpusLoadError(f"source manifest must be a JSON object: {path}")
    return data


def _load_raw_sections(
    raw_sections_path: Path | None,
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
    if raw_sections_path is None:
        return [], [
            _issue(
                "source_corpus_raw_sections_missing",
                message="source_manifest.raw_sections_path is missing",
                path="raw_sections_path",
            )
        ]
    if not raw_sections_path.is_file():
        return [], [
            _issue(
                "source_corpus_raw_sections_missing",
                message="raw sections JSONL file is missing",
                path=str(raw_sections_path),
            )
        ]

    records: list[dict[str, Any]] = []
    issues: list[dict[str, Any]] = []
    for line_number, line in enumerate(raw_sections_path.read_text(encoding="utf-8").splitlines(), 1):
        if not line.strip():
            continue
        try:
            parsed = json.loads(line)
        except json.JSONDecodeError:
            issues.append(
                _issue(
                    "source_corpus_raw_sections_invalid_jsonl",
                    message="raw sections JSONL line is invalid JSON",
                    path=str(raw_sections_path),
                    line=line_number,
                )
            )
            continue
        if not isinstance(parsed, dict):
            issues.append(
                _issue(
                    "source_corpus_raw_sections_invalid_jsonl",
                    message="raw sections JSONL line must be a JSON object",
                    path=str(raw_sections_path),
                    line=line_number,
                )
            )
            continue
        records.append(parsed)
    return records, issues


@dataclass(frozen=True)
class SourceCorpus:
    manifest_path: Path
    manifest: dict[str, Any]
    raw_sections_path: Path | None
    raw_sections: list[dict[str, Any]]
    load_issues: list[dict[str, Any]] = field(default_factory=list)

    def full_text(self) -> str:
        return "\n\n".join(
            str(record.get("text", "")).strip()
            for record in self.raw_sections
            if str(record.get("text", "")).strip()
        )

    def section_map(self) -> dict[str, dict[str, Any]]:
        raw_by_section = {
            str(record.get("section_id", "")): record
            for record in self.raw_sections
            if str(record.get("section_id", "")).strip()
        }
        sections: dict[str, dict[str, Any]] = {}
        for section in _copy_list(self.manifest.get("sections")):
            if not isinstance(section, dict):
                continue
            section_id = str(section.get("section_id", "")).strip()
            if not section_id:
                continue
            record = dict(section)
            raw_record = raw_by_section.get(section_id)
            if raw_record is not None:
                record.update({"text": raw_record.get("text", ""), "raw_record": dict(raw_record)})
            sections[section_id] = record
        for section_id, raw_record in raw_by_section.items():
            if section_id not in sections:
                sections[section_id] = dict(raw_record)
        return sections

    def caption_items(self) -> list[dict[str, Any]]:
        captions = self.manifest.get("captions", {})
        if not isinstance(captions, dict):
            return []
        items: list[dict[str, Any]] = []
        for manifest_key, kind in (("figures", "figure"), ("tables", "table")):
            for item in _copy_list(captions.get(manifest_key)):
                if isinstance(item, dict):
                    items.append({"kind": kind, **dict(item)})
        return items

    def appendix_pages(self) -> list[dict[str, Any]]:
        appendix_index = self.manifest.get("appendix_index", {})
        coverage = self.manifest.get("coverage", {})
        start_page = None
        if isinstance(appendix_index, dict):
            start_page = _positive_int(appendix_index.get("start_page"))
        if start_page is None and isinstance(coverage, dict):
            start_page = _positive_int(coverage.get("appendix_start_page"))
        if start_page is None:
            return []
        return [
            dict(page)
            for page in _copy_list(self.manifest.get("pages"))
            if isinstance(page, dict) and (_positive_int(page.get("page")) or 0) >= start_page
        ]

    def appendix_text_pages(self) -> list[dict[str, Any]]:
        appendix_index = self.manifest.get("appendix_index", {})
        coverage = self.manifest.get("coverage", {})
        start_page = None
        if isinstance(appendix_index, dict):
            start_page = _positive_int(appendix_index.get("start_page"))
        if start_page is None and isinstance(coverage, dict):
            start_page = _positive_int(coverage.get("appendix_start_page"))

        pages: list[dict[str, Any]] = []
        seen: set[tuple[int, str, str]] = set()
        for record in self.raw_sections:
            text = str(record.get("text", "")).strip()
            page_start = _positive_int(record.get("page_start"))
            page_end = _positive_int(record.get("page_end")) or page_start
            if not text or page_start is None or page_end is None:
                continue

            is_appendix = _record_has_appendix_hint(record)
            if start_page is not None:
                if page_end < start_page:
                    continue
                page_number = max(page_start, start_page)
            elif is_appendix:
                page_number = page_start
            else:
                continue

            section_id = str(record.get("section_id", "")).strip()
            title = str(record.get("title", "")).strip()
            marker = (page_number, section_id, text)
            if marker in seen:
                continue
            seen.add(marker)
            page = {"page": page_number, "text": text}
            if section_id:
                page["section_id"] = section_id
            if title:
                page["title"] = title
            pages.append(page)
        return pages

    def coverage(self) -> dict[str, Any]:
        return _copy_record(self.manifest.get("coverage"))

    def truncation(self) -> dict[str, Any]:
        coverage = self.coverage()
        text_truncated = bool(
            coverage.get("text_truncated") or coverage.get("truncated_due_to_page_limit")
        )
        return {
            "total_pages": coverage.get("total_pages"),
            "text_max_pages": coverage.get("text_max_pages"),
            "text_pages_extracted": coverage.get("text_pages_extracted"),
            "text_pages_scanned": coverage.get("text_pages_scanned"),
            "text_truncated": text_truncated,
            "truncated_due_to_page_limit": bool(
                coverage.get("truncated_due_to_page_limit") or text_truncated
            ),
            "partial_reading_accepted": False,
        }

    def language_hint(self) -> str:
        return str(self.manifest.get("language_hint", "") or "unknown")

    def math_index(self) -> list[Any]:
        return _copy_list(self.manifest.get("math_index"))

    def text_hash_metadata(self) -> dict[str, Any]:
        return {
            "manifest_text_hash_sha256": str(self.manifest.get("text_hash_sha256", "")),
            "raw_full_text_hash_sha256": _sha256_text(self.full_text()),
            "section_hashes_sha256": {
                str(record.get("section_id", "")): str(record.get("text_hash_sha256", ""))
                for record in self.raw_sections
                if str(record.get("section_id", "")).strip()
            },
        }


def load_source_corpus(source_manifest_path: str | Path) -> SourceCorpus:
    manifest_path = _resolve_manifest_path(source_manifest_path)
    manifest = _load_manifest(manifest_path)
    raw_sections_path = _resolve_raw_sections_path(manifest_path, manifest)
    raw_sections, load_issues = _load_raw_sections(raw_sections_path)
    return SourceCorpus(
        manifest_path=manifest_path,
        manifest=manifest,
        raw_sections_path=raw_sections_path,
        raw_sections=raw_sections,
        load_issues=load_issues,
    )


def _section_location_issue(record: dict[str, Any], *, source: str, index: int) -> dict[str, Any] | None:
    section_id = str(record.get("section_id", "")).strip()
    page_start = _positive_int(record.get("page_start"))
    page_end = _positive_int(record.get("page_end"))
    if not section_id or page_start is None or page_end is None or page_start > page_end:
        return _issue(
            "source_corpus_section_location_invalid",
            message="section record has invalid section id or page range",
            path=f"{source}[{index}]",
            section_id=section_id,
        )
    return None


def _validate_section_locations(corpus: SourceCorpus) -> list[dict[str, Any]]:
    issues: list[dict[str, Any]] = []
    manifest_sections = _copy_list(corpus.manifest.get("sections"))
    for index, section in enumerate(manifest_sections):
        if not isinstance(section, dict):
            issues.append(
                _issue(
                    "source_corpus_section_location_invalid",
                    message="manifest section must be a JSON object",
                    path=f"sections[{index}]",
                )
            )
            continue
        issue = _section_location_issue(section, source="sections", index=index)
        if issue is not None:
            issues.append(issue)
    for index, record in enumerate(corpus.raw_sections):
        issue = _section_location_issue(record, source="raw_sections", index=index)
        if issue is not None:
            issues.append(issue)
    return issues


def _validate_page_locations(corpus: SourceCorpus) -> list[dict[str, Any]]:
    sections = set(corpus.section_map())
    issues: list[dict[str, Any]] = []
    for index, page in enumerate(_copy_list(corpus.manifest.get("pages"))):
        if not isinstance(page, dict):
            issues.append(
                _issue(
                    "source_corpus_page_location_invalid",
                    message="page record must be a JSON object",
                    path=f"pages[{index}]",
                )
            )
            continue
        page_number = _positive_int(page.get("page"))
        section_ids = page.get("section_ids", [])
        if (
            page_number is None
            or not isinstance(section_ids, list)
            or any(str(section_id) not in sections for section_id in section_ids)
        ):
            issues.append(
                _issue(
                    "source_corpus_page_location_invalid",
                    message="page record has invalid page number or section ids",
                    path=f"pages[{index}]",
                    page=page.get("page"),
                )
            )
    return issues


def _caption_page_values(item: dict[str, Any]) -> list[Any]:
    pages = item.get("pages")
    if isinstance(pages, list) and pages:
        return pages
    if "page" in item:
        return [item.get("page")]
    return []


def _validate_caption_locations(corpus: SourceCorpus) -> list[dict[str, Any]]:
    sections = set(corpus.section_map())
    coverage = corpus.coverage()
    total_pages = _positive_int(coverage.get("total_pages")) if coverage else None
    issues: list[dict[str, Any]] = []
    for index, item in enumerate(corpus.caption_items()):
        pages = _caption_page_values(item)
        valid_pages = [_positive_int(page) for page in pages]
        section_id = str(item.get("section_id", "")).strip()
        page_out_of_range = total_pages is not None and any(
            page is not None and page > total_pages for page in valid_pages
        )
        if (
            not pages
            or any(page is None for page in valid_pages)
            or page_out_of_range
            or (section_id and section_id not in sections)
        ):
            issues.append(
                _issue(
                    "source_corpus_caption_location_invalid",
                    message="caption record has invalid page or section location",
                    path=f"captions[{index}]",
                    caption_id=item.get("id", ""),
                )
            )
    return issues


def _validate_hashes(corpus: SourceCorpus) -> list[dict[str, Any]]:
    issues: list[dict[str, Any]] = []
    raw_by_section = {
        str(record.get("section_id", "")): record
        for record in corpus.raw_sections
        if str(record.get("section_id", "")).strip()
    }
    for index, record in enumerate(corpus.raw_sections):
        expected_hash = str(record.get("text_hash_sha256", "")).strip()
        if expected_hash and expected_hash != _sha256_text(str(record.get("text", ""))):
            issues.append(
                _issue(
                    "source_corpus_section_hash_mismatch",
                    message="raw section text hash does not match raw section text",
                    path=f"raw_sections[{index}].text_hash_sha256",
                    section_id=record.get("section_id", ""),
                )
            )
    for index, section in enumerate(_copy_list(corpus.manifest.get("sections"))):
        if not isinstance(section, dict):
            continue
        section_id = str(section.get("section_id", "")).strip()
        manifest_hash = str(section.get("text_hash_sha256", "")).strip()
        raw_record = raw_by_section.get(section_id)
        if raw_record is None or not manifest_hash:
            continue
        raw_hash = str(raw_record.get("text_hash_sha256", "")).strip()
        if raw_hash and manifest_hash != raw_hash:
            issues.append(
                _issue(
                    "source_corpus_section_hash_mismatch",
                    message="manifest section hash does not match raw section hash",
                    path=f"sections[{index}].text_hash_sha256",
                    section_id=section_id,
                )
            )
    return issues


def validate_source_corpus(corpus: SourceCorpus) -> list[dict[str, Any]]:
    issues: list[dict[str, Any]] = [dict(issue) for issue in corpus.load_issues]
    if not corpus.full_text():
        issues.append(
            _issue(
                "source_corpus_empty_text",
                message="raw sections do not contain extracted source text",
                path="raw_sections",
            )
        )
    issues.extend(_validate_section_locations(corpus))
    issues.extend(_validate_page_locations(corpus))
    issues.extend(_validate_caption_locations(corpus))
    issues.extend(_validate_hashes(corpus))
    return issues

#!/usr/bin/env python3
"""Inspect and persist DeepPaperNote's device-local User Configuration."""

from __future__ import annotations

import argparse
import json
import os
import re
import shutil
import tempfile
from datetime import datetime, timezone
from pathlib import Path, PureWindowsPath
from typing import Any, Mapping, Sequence

OUTPUT_LANGUAGES = {"zh-CN", "en"}
SAVE_MODES = {"workspace", "obsidian"}
KNOWN_FIELDS = ("output_language", "save_mode", "obsidian_vault", "papers_dir")
ALWAYS_REQUIRED = ("output_language", "save_mode")
ENV_FIELDS = {
    "output_language": "DEEPPAPERNOTE_OUTPUT_LANGUAGE",
    "save_mode": "DEEPPAPERNOTE_SAVE_MODE",
    "obsidian_vault": "DEEPPAPERNOTE_OBSIDIAN_VAULT",
    "papers_dir": "DEEPPAPERNOTE_PAPERS_DIR",
}


class ConfigurationWriteError(RuntimeError):
    """Raised when a Preference Change cannot be durably verified."""


class ConfigurationValidationError(ConfigurationWriteError):
    """Raised when a Preference Change still needs user input or repair."""

    def __init__(
        self,
        result: dict[str, Any],
        message: str = "Preference Change is incomplete or invalid.",
    ) -> None:
        super().__init__(message)
        self.result = result


def user_config_path() -> Path:
    override = os.environ.get("DEEPPAPERNOTE_CONFIG_PATH", "").strip()
    if override:
        return Path(override).expanduser()
    return Path.home() / ".deeppapernote" / "config.json"


def default_shell_paths() -> tuple[Path, ...]:
    home = Path.home()
    return tuple(
        home / name
        for name in (".zshenv", ".zprofile", ".zshrc", ".bash_profile", ".bashrc")
    )


def _clean_values(values: Mapping[str, Any] | None) -> dict[str, str]:
    if not values:
        return {}
    return {
        field: str(values[field]).strip()
        for field in KNOWN_FIELDS
        if field in values and str(values[field]).strip()
    }


def _migration_candidates(
    environ: Mapping[str, str], shell_paths: Sequence[Path]
) -> dict[str, dict[str, str]]:
    candidates: dict[str, dict[str, str]] = {}
    for field, name in ENV_FIELDS.items():
        value = environ.get(name, "").strip()
        if value and _migration_value_supported(field, value):
            candidates[field] = {"value": value, "source": "process_environment"}
    for path in shell_paths:
        if not path.is_file():
            continue
        try:
            lines = path.read_text(encoding="utf-8-sig").splitlines()
        except OSError:
            continue
        for field, name in ENV_FIELDS.items():
            if field in candidates:
                continue
            pattern = re.compile(rf"^\s*(?:export\s+)?{re.escape(name)}=(.*)$")
            for line in reversed(lines):
                match = pattern.match(line)
                if not match:
                    continue
                value = match.group(1).strip()
                if len(value) >= 2 and value[0] == value[-1] and value[0] in {'"', "'"}:
                    value = value[1:-1]
                if value and _migration_value_supported(field, value):
                    candidates[field] = {"value": value, "source": str(path)}
                break
    return candidates


def _issue(field: str, code: str, message: str) -> dict[str, str]:
    return {"field": field, "code": code, "message": message}


def _safe_relative_path(value: str) -> bool:
    native = Path(value)
    windows = PureWindowsPath(value)
    if native.is_absolute() or windows.is_absolute() or windows.drive or windows.root:
        return False
    parts = tuple(part for part in re.split(r"[\\/]+", value) if part and part != ".")
    return bool(parts) and ".." not in parts


def _migration_value_supported(field: str, value: str) -> bool:
    if field == "output_language":
        return value in OUTPUT_LANGUAGES
    if field == "save_mode":
        return value in SAVE_MODES
    if field == "obsidian_vault":
        try:
            vault = Path(value).expanduser()
        except (OSError, RuntimeError):
            return False
        return vault.is_absolute() and vault.is_dir()
    if field == "papers_dir":
        return _safe_relative_path(value)
    return False


def _validate(configuration: Mapping[str, Any]) -> tuple[list[dict[str, str]], list[str]]:
    issues: list[dict[str, str]] = []
    missing: list[str] = []
    for field in ALWAYS_REQUIRED:
        if not str(configuration.get(field, "")).strip():
            missing.append(field)
    language = str(configuration.get("output_language", "")).strip()
    save_mode = str(configuration.get("save_mode", "")).strip()
    if language and language not in OUTPUT_LANGUAGES:
        issues.append(_issue("output_language", "invalid_enum", "Expected zh-CN or en."))
    if save_mode and save_mode not in SAVE_MODES:
        issues.append(_issue("save_mode", "invalid_enum", "Expected workspace or obsidian."))
    if save_mode == "obsidian":
        vault = str(configuration.get("obsidian_vault", "")).strip()
        papers_dir = str(configuration.get("papers_dir", "")).strip()
        if not vault:
            missing.append("obsidian_vault")
        else:
            vault_path = Path(vault).expanduser()
            if not vault_path.is_absolute() or not vault_path.is_dir():
                issues.append(
                    _issue(
                        "obsidian_vault",
                        "missing_vault",
                        "Expected an existing absolute directory.",
                    )
                )
        if not papers_dir:
            missing.append("papers_dir")
        elif not _safe_relative_path(papers_dir):
            issues.append(
                _issue(
                    "papers_dir",
                    "unsafe_path",
                    "Expected a safe relative path inside the Vault.",
                )
            )
    return issues, missing


def _blocked_result(path: Path, code: str, message: str) -> dict[str, Any]:
    return {
        "state": "blocked",
        "config_path": str(path),
        "affected_fields": ["configuration"],
        "prompt_fields": [],
        "migration_candidates": {},
        "warnings": [],
        "issues": [_issue("configuration", code, message)],
    }


def _repair_result(
    path: Path,
    configuration: Mapping[str, Any],
    issues: list[dict[str, str]],
    missing: list[str],
) -> dict[str, Any]:
    affected = [issue["field"] for issue in issues]
    for field in missing:
        if field not in affected:
            affected.append(field)
    result: dict[str, Any] = {
        "state": "invalid" if issues else "needs_input" if missing else "ready",
        "config_path": str(path),
        "affected_fields": affected,
        "prompt_fields": [field for field in KNOWN_FIELDS if field in affected],
        "migration_candidates": {},
        "warnings": [],
        "configuration": dict(configuration),
    }
    if issues:
        result["issues"] = issues
    return result


def _read_configuration(path: Path) -> tuple[dict[str, Any] | None, str]:
    try:
        raw = path.read_text(encoding="utf-8-sig")
    except OSError as exc:
        return None, f"unreadable:{exc}"
    try:
        value = json.loads(raw)
    except (json.JSONDecodeError, UnicodeError) as exc:
        return None, f"malformed:{exc}"
    if not isinstance(value, dict):
        return None, "non_object:Expected a JSON object."
    return value, ""


def _path_is_writable(path: Path) -> bool:
    candidate = path.parent
    while not candidate.exists() and candidate != candidate.parent:
        candidate = candidate.parent
    return os.access(candidate, os.W_OK)


def inspect_configuration(
    *,
    config_path: str | Path | None = None,
    environ: Mapping[str, str] | None = None,
    shell_paths: Sequence[Path] | None = None,
) -> dict[str, Any]:
    path = Path(config_path or user_config_path()).expanduser()
    environment = os.environ if environ is None else environ
    if not _path_is_writable(path):
        return _blocked_result(path, "unwritable", "User Configuration is not writable.")
    if not path.exists():
        return {
            "state": "needs_input",
            "config_path": str(path),
            "affected_fields": list(ALWAYS_REQUIRED),
            "prompt_fields": list(ALWAYS_REQUIRED),
            "migration_candidates": _migration_candidates(
                environment, default_shell_paths() if shell_paths is None else shell_paths
            ),
            "warnings": [],
        }

    configuration, read_error = _read_configuration(path)
    if read_error.startswith("unreadable:"):
        return _blocked_result(path, "unreadable", read_error.partition(":")[2])
    if read_error:
        code, _, message = read_error.partition(":")
        return {
            "state": "invalid",
            "config_path": str(path),
            "affected_fields": ["configuration"],
            "prompt_fields": list(ALWAYS_REQUIRED),
            "migration_candidates": {},
            "warnings": [],
            "issues": [_issue("configuration", code, message)],
        }

    assert configuration is not None
    issues, missing = _validate(configuration)
    return _repair_result(path, configuration, issues, missing)


def resolve_preferences(
    *,
    config_path: str | Path | None = None,
    explicit_overrides: Mapping[str, Any] | None = None,
    cli_overrides: Mapping[str, Any] | None = None,
    environ: Mapping[str, str] | None = None,
) -> dict[str, Any]:
    path = Path(config_path or user_config_path()).expanduser()
    configuration, read_error = _read_configuration(path)
    if read_error or configuration is None:
        raise RuntimeError(f"User Configuration is not readable and valid: {read_error}")
    resolved = resolve_run_overrides(
        explicit_overrides=explicit_overrides,
        cli_overrides=cli_overrides,
        environ=environ,
    )
    values = _clean_values(configuration)
    sources = {field: "user_configuration" for field in values}
    values.update(resolved["values"])
    sources.update(resolved["sources"])
    issues, missing = _validate(values)
    return {"values": values, "sources": sources, "issues": issues, "missing": missing}


def resolve_run_overrides(
    *,
    explicit_overrides: Mapping[str, Any] | None = None,
    cli_overrides: Mapping[str, Any] | None = None,
    environ: Mapping[str, str] | None = None,
) -> dict[str, Any]:
    environment = os.environ if environ is None else environ
    layers = (
        (
            "process_environment",
            {
                field: environment.get(name, "").strip()
                for field, name in ENV_FIELDS.items()
                if environment.get(name, "").strip()
            },
        ),
        ("cli", _clean_values(cli_overrides)),
        ("explicit_request", _clean_values(explicit_overrides)),
    )
    values: dict[str, str] = {}
    sources: dict[str, str] = {}
    for source, layer in layers:
        for field, value in layer.items():
            values[field] = value
            sources[field] = source
    issues, missing = _validate(values)
    return {"values": values, "sources": sources, "issues": issues, "missing": missing}


def _invalid_backup_path(path: Path) -> Path:
    stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S%fZ")
    candidate = path.with_name(f"{path.stem}.invalid-{stamp}{path.suffix}")
    counter = 1
    while candidate.exists():
        candidate = path.with_name(f"{path.stem}.invalid-{stamp}-{counter}{path.suffix}")
        counter += 1
    return candidate


def _atomic_write(path: Path, configuration: Mapping[str, Any]) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    temporary_name = ""
    try:
        with tempfile.NamedTemporaryFile(
            mode="w",
            encoding="utf-8",
            dir=path.parent,
            prefix=f".{path.name}.",
            suffix=".tmp",
            delete=False,
        ) as handle:
            temporary_name = handle.name
            json.dump(configuration, handle, ensure_ascii=False, indent=2)
            handle.write("\n")
            handle.flush()
            os.fsync(handle.fileno())
        os.chmod(temporary_name, 0o600)
        os.replace(temporary_name, path)
    except OSError as exc:
        if temporary_name:
            try:
                Path(temporary_name).unlink()
            except OSError:
                pass
        raise ConfigurationWriteError(f"User Configuration write is blocked: {exc}") from exc


def persist_preferences(
    preferences: Mapping[str, Any],
    *,
    config_path: str | Path | None = None,
    environ: Mapping[str, str] | None = None,
    replace_invalid: bool = False,
) -> dict[str, Any]:
    path = Path(config_path or user_config_path()).expanduser()
    if not _path_is_writable(path):
        raise ConfigurationWriteError("User Configuration is not writable.")
    existing: dict[str, Any] = {}
    if path.exists():
        loaded, read_error = _read_configuration(path)
        if read_error.startswith("unreadable:"):
            raise ConfigurationWriteError(f"User Configuration is unreadable: {read_error}")
        if read_error:
            if not replace_invalid:
                raise ConfigurationValidationError(
                    inspect_configuration(config_path=path, environ=environ),
                    "Invalid User Configuration requires an explicitly confirmed replacement.",
                )
            backup = _invalid_backup_path(path)
            try:
                shutil.copy2(path, backup)
            except OSError as exc:
                raise ConfigurationWriteError(
                    f"Could not preserve invalid User Configuration: {exc}"
                ) from exc
        else:
            assert loaded is not None
            existing = loaded
    updates = _clean_values(preferences)
    candidate = {**existing, **updates}
    issues, missing = _validate(candidate)
    if issues or missing:
        raise ConfigurationValidationError(_repair_result(path, candidate, issues, missing))
    unknown = sorted(set(existing) - set(KNOWN_FIELDS))
    _atomic_write(path, candidate)
    reread, read_error = _read_configuration(path)
    if read_error or reread != candidate:
        raise ConfigurationWriteError(
            "User Configuration readback did not match the persisted values."
        )
    result = inspect_configuration(config_path=path, environ=environ)
    if result["state"] != "ready":
        raise ConfigurationWriteError("Persisted User Configuration did not pass validation.")
    if unknown:
        result["warnings"] = [
            f"Preserved unknown configuration fields: {', '.join(unknown)}"
        ]
    return result


def parser() -> argparse.ArgumentParser:
    argument_parser = argparse.ArgumentParser(description=__doc__)
    argument_parser.add_argument("--set-output-language", choices=sorted(OUTPUT_LANGUAGES))
    argument_parser.add_argument("--set-save-mode", choices=sorted(SAVE_MODES))
    argument_parser.add_argument("--set-vault")
    argument_parser.add_argument("--set-papers-dir")
    argument_parser.add_argument("--replace-invalid", action="store_true")
    return argument_parser


def main() -> None:
    args = parser().parse_args()
    preferences = {
        field: value
        for field, value in {
            "output_language": args.set_output_language,
            "save_mode": args.set_save_mode,
            "obsidian_vault": args.set_vault,
            "papers_dir": args.set_papers_dir,
        }.items()
        if value is not None
    }
    try:
        result = (
            persist_preferences(preferences, replace_invalid=args.replace_invalid)
            if preferences
            else inspect_configuration()
        )
    except ConfigurationValidationError as exc:
        result = exc.result
    except ConfigurationWriteError as exc:
        result = _blocked_result(user_config_path(), "write_failed", str(exc))
    print(json.dumps(result, ensure_ascii=False, indent=2))
    if result["state"] != "ready":
        raise SystemExit(2)


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""Write the final Markdown note into an Obsidian-style vault."""

from __future__ import annotations

import argparse
import hashlib
import json
import os
import re
import shutil
import sys
import tempfile
from contextlib import nullcontext
from pathlib import Path

from common import (
    emit,
    ensure_parent,
    file_sha256,
    maybe_load_json_record,
    resolve_domain_subdir,
    resolve_note_asset_dir,
    resolve_note_output_mode,
    resolve_obsidian_note_path,
    runtime_config,
)
from lint_note import inspect_reference_hygiene
from localization import normalize_output_language, require_artifact_output_language
from paper_archive import (
    ArchiveError,
    admit_directory,
    archive_lock,
    directory_candidates,
    new_record,
    pdf_identity,
    read_record,
    record_work_evidence,
    register_source,
    safe_path,
    select_directory,
    source_details,
    verify_pdf,
    work_identity,
    write_record,
)

PAPER_DIRECTORY_SIDECAR = ".deeppapernote.json"
SOURCE_SHA256_RE = re.compile(r"^[0-9a-f]{64}$")
WINDOWS_FILE_ATTRIBUTE_HIDDEN = 0x2
WINDOWS_INVALID_FILE_ATTRIBUTES = 0xFFFFFFFF


def ensure_sidecar_hidden(
    path: Path,
    *,
    platform: str | None = None,
    get_attributes=None,
    set_attributes=None,
) -> None:
    if (platform or os.name) != "nt":
        return
    if (get_attributes is None) != (set_attributes is None):
        raise ValueError("get_attributes and set_attributes must be provided together")

    if get_attributes is None:
        import ctypes

        kernel32 = ctypes.WinDLL("kernel32", use_last_error=True)
        get_attributes = kernel32.GetFileAttributesW
        get_attributes.argtypes = [ctypes.c_wchar_p]
        get_attributes.restype = ctypes.c_uint32
        set_attributes = kernel32.SetFileAttributesW
        set_attributes.argtypes = [ctypes.c_wchar_p, ctypes.c_uint32]
        set_attributes.restype = ctypes.c_int

        def fail() -> None:
            raise ctypes.WinError(ctypes.get_last_error())

    else:

        def fail() -> None:
            raise OSError(f"Could not set Windows Hidden attribute: {path}")

    path_value = str(path)
    current = int(get_attributes(path_value))
    if current == WINDOWS_INVALID_FILE_ATTRIBUTES:
        fail()
    if not set_attributes(path_value, current | WINDOWS_FILE_ATTRIBUTE_HIDDEN):
        fail()
    verified = int(get_attributes(path_value))
    if (
        verified == WINDOWS_INVALID_FILE_ATTRIBUTES
        or not verified & WINDOWS_FILE_ATTRIBUTE_HIDDEN
    ):
        fail()


def parser() -> argparse.ArgumentParser:
    p = argparse.ArgumentParser(description=__doc__ or "write obsidian note")
    p.add_argument("--input", default="", help="Metadata JSON path or JSON string.")
    p.add_argument("--content-file", default="", help="Path to the final Markdown content.")
    p.add_argument("--content", default="", help="Inline Markdown content.")
    p.add_argument("--stdin", action="store_true", help="Read Markdown content from stdin.")
    p.add_argument(
        "--lint-json",
        default="",
        help="Required Final Note Lint JSON for Formal Save.",
    )
    p.add_argument(
        "--figure-decisions",
        default="",
        help="Required Figure/Table Decisions JSON for Formal Save.",
    )
    p.add_argument(
        "--source-manifest",
        default="",
        help="Required Source Manifest JSON for Obsidian Save Target Admission.",
    )
    p.add_argument("--title", default="", help="Explicit title override.")
    p.add_argument("--output", default="", help="JSON status output path.")
    p.add_argument("--vault", default="", help="Target Obsidian vault path.")
    p.add_argument("--save-mode", choices=("workspace", "obsidian"), default="")
    p.add_argument("--papers-dir", default="", help="Vault-relative paper directory.")
    p.add_argument("--subdir", default="", help="Vault-relative subdirectory.")
    p.add_argument("--filename", default="", help="Explicit note filename.")
    p.add_argument("--asset-subdir", default="images", help="Asset folder name relative to the note directory.")
    p.add_argument("--target-directory", default="", help="Previously selected existing paper directory.")
    p.add_argument("--paper-id", default="", help="Canonical paper id.")
    p.add_argument("--language", default="", help="Run Override for output language: en or zh-CN.")
    p.add_argument(
        "--preflight",
        action="store_true",
        help="Resolve the Obsidian Save Target without writing files.",
    )
    p.add_argument(
        "--overwrite-existing-note",
        action="store_true",
        help="Overwrite a same-language note after explicit user confirmation.",
    )
    p.add_argument(
        "--expected-existing-note-sha256",
        default="",
        help="SHA-256 returned by the conflict that the user approved overwriting.",
    )
    return p


def insert_decisions(decisions: dict) -> list[dict]:
    items = decisions.get("decisions", []) if isinstance(decisions, dict) else []
    if not isinstance(items, list):
        return []
    return [
        item
        for item in items
        if isinstance(item, dict) and str(item.get("decision", "")).strip() == "insert"
    ]


def safe_image_filename(filename: str, source_image: Path) -> str:
    candidate = filename.strip() or source_image.name
    if (
        not candidate
        or candidate in {".", ".."}
        or "/" in candidate
        or "\\" in candidate
        or Path(candidate).is_absolute()
    ):
        raise SystemExit(f"Unsafe figure image filename in insert decision: {candidate}")
    return candidate


def embed_target_matches(target: str, expected_relative: str) -> bool:
    normalized = target.strip().strip("<>").split("|", 1)[0]
    if normalized == expected_relative:
        return True
    return normalized.endswith(f"/{expected_relative}")


def note_references_image_embed(note_text: str, expected_relative: str) -> bool:
    markdown_targets = re.findall(r"!\[[^\]]*\]\(([^)\s]+)(?:\s+\"[^\"]*\")?\)", note_text)
    obsidian_targets = re.findall(r"!\[\[([^\]]+)\]\]", note_text)
    return any(
        embed_target_matches(target, expected_relative)
        for target in markdown_targets + obsidian_targets
    )


def require_reference_hygiene(note_text: str, stage: str) -> None:
    issues = inspect_reference_hygiene(note_text)
    if not issues:
        return
    first = issues[0]
    match = str(first.get("match", "")).strip()
    line_number = first.get("line_number", "")
    detail = f": {match}" if match else ""
    line_detail = f" on line {line_number}" if line_number else ""
    raise SystemExit(
        f"write_obsidian_note.py refused to write note because reference hygiene gate failed"
        f" {stage}{line_detail}{detail}."
    )


def materialize_insert_decisions(
    note_text: str,
    target_path: Path,
    decisions: dict,
    asset_subdir: str,
    created_paths: list[Path] | None = None,
) -> list[dict]:
    asset_dir = target_path.parent / asset_subdir
    pending: list[tuple[Path, Path, str, dict]] = []
    planned_hashes: dict[Path, str] = {}
    for item in insert_decisions(decisions):
        source_value = str(item.get("source_image_path", "")).strip()
        source_image = Path(source_value).expanduser()
        if not source_value or not source_image.is_file():
            label = item.get("source_id") or item.get("label") or item.get("item_id") or "unknown"
            raise SystemExit(f"Insert decision source image does not exist for {label}: {source_value}")
        current_sha256 = file_sha256(source_image)
        review = item.get("visual_review", {})
        if (
            not isinstance(review, dict)
            or str(review.get("status", "")).strip() != "pass"
            or str(review.get("reviewed_asset_sha256", "")).strip() != current_sha256
            or str(item.get("source_image_sha256", "")).strip() != current_sha256
        ):
            label = item.get("source_id") or item.get("label") or item.get("item_id") or "unknown"
            raise SystemExit(
                f"Insert decision for {label} does not match its reviewed asset SHA-256."
            )
        filename = safe_image_filename(
            str(item.get("source_image_filename", "")),
            source_image,
        )
        expected_relative = f"{asset_subdir}/{filename}"
        if not note_references_image_embed(note_text, expected_relative):
            label = item.get("source_id") or item.get("label") or item.get("item_id") or filename
            raise SystemExit(
                f"Insert decision for {label} is not referenced as an image embed: {expected_relative}."
            )
        dest_image = asset_dir / filename
        if dest_image.resolve().parent != asset_dir.resolve():
            raise SystemExit(f"Unsafe figure image destination: {dest_image}")
        existing_planned_sha256 = planned_hashes.get(dest_image)
        if existing_planned_sha256 and existing_planned_sha256 != current_sha256:
            raise SystemExit(
                f"Insert decisions assign different bytes to the same image: {filename}"
            )
        planned_hashes[dest_image] = current_sha256
        if dest_image.is_file() and file_sha256(dest_image) != current_sha256:
            raise SystemExit(
                f"Refusing to save because an existing paper-local image has different bytes: "
                f"{filename}"
            )
        pending.append(
            (
                source_image,
                dest_image,
                current_sha256,
                {
                    "source_id": item.get("source_id")
                    or item.get("label")
                    or item.get("item_id")
                    or "",
                    "source_image": str(source_image.resolve()),
                    "dest_image_path": str(dest_image),
                    "relative_markdown_path": expected_relative,
                    "reviewed_asset_sha256": current_sha256,
                },
            )
        )

    if pending:
        asset_dir.mkdir(parents=True, exist_ok=True)
    materialized: list[dict] = []
    for source_image, dest_image, current_sha256, record in pending:
        if dest_image.is_file():
            if file_sha256(dest_image) != current_sha256:
                raise SystemExit(
                    f"Refusing to save because an existing paper-local image has different bytes: "
                    f"{dest_image.name}"
                )
        elif source_image.resolve() != dest_image.resolve():
            if created_paths is not None:
                created_paths.append(dest_image)
            shutil.copy2(source_image, dest_image)
        if file_sha256(dest_image) != current_sha256:
            raise SystemExit(
                f"Materialized image bytes do not match the reviewed asset SHA-256: "
                f"{dest_image.name}"
            )
        materialized.append(record)
    return materialized


def lint_failure_message(lint: dict, gate: str, lint_path: str) -> str:
    detail_parts: list[str] = []
    for warning in lint.get("warnings", []) or []:
        if warning:
            detail_parts.append(str(warning))
    for issue_key in (
        "core_info_structure_issues",
        "figure_structure_issues",
        "planning_artifact_issues",
        "substantive_content_issues",
        "mixed_language_issues",
        "mechanical_translation_artifact_issues",
        "linebreak_issues",
        "code_math_issues",
        "math_render_issues",
        "reference_hygiene_issues",
    ):
        issues = lint.get(issue_key, []) or []
        if not issues:
            continue
        first = issues[0]
        if isinstance(first, dict):
            reason = first.get("reason") or first.get("line") or first.get("snippet") or first
            detail_parts.append(f"{issue_key}: {reason}")
        else:
            detail_parts.append(f"{issue_key}: {first}")
    details = "; ".join(detail_parts[:4])
    suffix = f" Details: {details}." if details else ""
    return f"write_obsidian_note.py refused to write note because {gate} gate failed.{suffix} See lint JSON: {lint_path}"


def require_lint_gate(lint: dict, key: str, gate: str, lint_path: str) -> None:
    if not lint.get(key, False):
        raise SystemExit(lint_failure_message(lint, gate, lint_path))


def require_source_manifest(path_value: str) -> dict:
    if not path_value:
        raise SystemExit(
            "Obsidian Save Target Admission requires a Source Manifest with source_sha256."
        )
    manifest = maybe_load_json_record(path_value)
    if manifest is None or str(manifest.get("status", "")).strip() != "ok":
        raise SystemExit(f"Expected successful Source Manifest JSON: {path_value}")
    source_sha256 = str(manifest.get("source_sha256", "")).strip().lower()
    if not SOURCE_SHA256_RE.fullmatch(source_sha256):
        raise SystemExit("Source Manifest requires a 64-character source_sha256.")
    manifest["source_sha256"] = source_sha256
    return manifest


def language_note_path(target_path: Path, output_language: str) -> Path:
    suffix = target_path.suffix or ".md"
    stem = target_path.stem
    if stem.endswith(f".{output_language}"):
        return target_path
    return target_path.with_name(f"{stem}.{output_language}{suffix}")


def block_directory_conflict(
    args: argparse.Namespace,
    *,
    conflict_code: str,
    target_directories: list[Path],
    source_sha256: str,
    output_language: str,
) -> None:
    payload = {
        "status": "blocked",
        "script": "write_obsidian_note.py",
        "conflict_code": conflict_code,
        "requires_user_confirmation": False,
        "target_directory": str(target_directories[0]),
        "matching_directories": [str(path) for path in target_directories],
        "source_sha256": source_sha256,
        "output_language": output_language,
    }
    emit(payload, args.output)
    raise SystemExit(2)


def block_note_conflict(
    args: argparse.Namespace,
    target_path: Path,
    *,
    conflict_code: str,
    source_sha256: str,
    output_language: str,
) -> None:
    emit(
        {
            "status": "blocked",
            "script": "write_obsidian_note.py",
            "conflict_code": conflict_code,
            "requires_user_confirmation": True,
            "target_directory": str(target_path.parent),
            "existing_note_path": str(target_path),
            "existing_note_sha256": file_sha256(target_path),
            "source_sha256": source_sha256,
            "output_language": output_language,
        },
        args.output,
    )
    raise SystemExit(2)


def resolve_obsidian_save_target(
    args: argparse.Namespace,
    config: dict,
    vault: Path,
    *,
    title: str,
    abstract: str,
    source_sha256: str,
    output_language: str,
) -> tuple[Path, str, str, bool, str]:
    archive_input = getattr(args, "archive_input", {"title": title})
    details = getattr(args, "source_details", {})
    identity_path = resolve_obsidian_note_path(config, title=title, filename=args.filename)
    resolved_subdir = ""
    try:
        candidates = directory_candidates(
            vault, archive_input, source_sha256, identity_path.parent.name
        )
        existing_dir = select_directory(candidates, getattr(args, "target_directory", ""))
        if existing_dir:
            registry = admit_directory(existing_dir, archive_input, source_sha256)
            if getattr(args, 'pdf_identity', None):
                record_work_evidence(registry['work'], source_sha256, args.pdf_identity)
            source = register_source(registry, source_sha256, details)
            notes = source["notes"]
            if output_language in notes:
                target_path = safe_path(existing_dir, notes[output_language]["filename"])
                if not target_path.is_file():
                    raise ArchiveError("recorded_language_note_missing", [existing_dir])
            else:
                target_path = safe_path(existing_dir, f"{source['note_stem']}.{output_language}.md")
            admission = (
                "reuse_source_directory"
                if read_record(existing_dir)
                else (
                    "reuse_pdf_directory"
                    if any(existing_dir.iterdir())
                    else "reuse_empty_same_name_directory"
                )
            )
            domain_routing_skipped = True
        else:
            resolved_subdir = resolve_domain_subdir(
                config, title=title, abstract=abstract, subdir=args.subdir
            )
            target_path = resolve_obsidian_note_path(
                config, title=title, subdir=resolved_subdir, filename=args.filename
            )
            registry = new_record(title, target_path.stem, archive_input)
            source = register_source(registry, source_sha256, details)
            target_path = language_note_path(target_path, output_language)
            admission = "new_directory"
            domain_routing_skipped = False
        args.archive_registry = registry
        args.admitted_asset_subdir = source["asset_subdir"]
        if target_path.is_symlink() or not target_path.resolve().is_relative_to(vault.resolve()):
            raise ArchiveError("unsafe_archive_path", [target_path])
    except ArchiveError as exc:
        block_directory_conflict(
            args,
            conflict_code=exc.code,
            target_directories=[Path(p) for p in exc.paths] or [identity_path.parent],
            source_sha256=source_sha256,
            output_language=output_language,
        )

    approved_existing_note_sha256 = ""
    if target_path.is_file():
        existing_note_sha256 = file_sha256(target_path)
        if not args.overwrite_existing_note:
            block_note_conflict(
                args,
                target_path,
                conflict_code="same_language_note_exists",
                source_sha256=source_sha256,
                output_language=output_language,
            )
        expected_sha256 = args.expected_existing_note_sha256.strip().lower()
        if not SOURCE_SHA256_RE.fullmatch(expected_sha256):
            raise SystemExit(
                "--overwrite-existing-note requires a 64-character "
                "--expected-existing-note-sha256 from the conflict response."
            )
        if existing_note_sha256 != expected_sha256:
            block_note_conflict(
                args,
                target_path,
                conflict_code="stale_overwrite_confirmation",
                source_sha256=source_sha256,
                output_language=output_language,
            )
        approved_existing_note_sha256 = existing_note_sha256
        admission = "overwrite_same_language_note"
    elif args.overwrite_existing_note:
        raise SystemExit(
            "--overwrite-existing-note is only valid when the target language note exists."
        )

    return (
        target_path,
        resolved_subdir,
        admission,
        domain_routing_skipped,
        approved_existing_note_sha256,
    )


def write_paper_directory_sidecar(
    target_path: Path,
    *,
    paper_id: str,
    title: str,
    source_sha256: str,
    output_language: str,
    note_sha256: str,
    registry: dict | None = None,
) -> Path:
    sidecar_path = target_path.parent / PAPER_DIRECTORY_SIDECAR
    payload = (
        registry
        or read_record(target_path.parent)
        or new_record(
            title,
            target_path.name.removesuffix(f".{output_language}{target_path.suffix}"),
            {"title": title, "paper_id": paper_id},
        )
    )
    source = register_source(payload, source_sha256, {})
    source["notes"][output_language] = {"filename": target_path.name, "note_sha256": note_sha256}
    write_record(target_path.parent, payload)
    ensure_sidecar_hidden(sidecar_path)
    return sidecar_path


def atomic_write_note(path: Path, note_text: str) -> None:
    fd, temp_name = tempfile.mkstemp(
        dir=path.parent,
        prefix=f".{path.name}.",
        suffix=".tmp",
    )
    temp_path = Path(temp_name)
    try:
        with os.fdopen(fd, "w", encoding="utf-8", newline="\n") as handle:
            handle.write(note_text)
        if path.is_file():
            shutil.copymode(path, temp_path)
        os.replace(temp_path, path)
    finally:
        temp_path.unlink(missing_ok=True)


def backup_file(path: Path) -> Path | None:
    if not path.is_file():
        return None
    fd, backup_name = tempfile.mkstemp(
        dir=path.parent,
        prefix=f".{path.name}.",
        suffix=".backup",
    )
    os.close(fd)
    backup_path = Path(backup_name)
    try:
        shutil.copy2(path, backup_path)
    except BaseException:
        backup_path.unlink(missing_ok=True)
        raise
    return backup_path


def restore_file(path: Path, backup_path: Path | None) -> None:
    if backup_path is None:
        path.unlink(missing_ok=True)
    else:
        os.replace(backup_path, path)


def main() -> None:
    args = parser().parse_args()
    config = runtime_config(
        cli_overrides={
            "output_language": args.language,
            "save_mode": args.save_mode or ("obsidian" if args.vault else ""),
            "obsidian_vault": args.vault,
            "papers_dir": args.papers_dir,
        }
    )
    output_language = normalize_output_language(
        args.language or str(config.get("output_language", "")) or None
    )
    output_mode, root_path = resolve_note_output_mode(config)

    record = maybe_load_json_record(args.input) or {}
    title = args.title or str(record.get("title", "")).strip()
    if not title:
        raise SystemExit("write_obsidian_note.py requires --title or metadata with a title.")

    source_manifest = (
        require_source_manifest(args.source_manifest) if output_mode == "obsidian" else {}
    )
    args.archive_input = {**record, **source_manifest, "title": title}
    if not args.target_directory:
        args.target_directory = (source_manifest.get("archive_source") or {}).get(
            "target_directory", ""
        )
    pdf_value = (source_manifest.get("pdf") or {}).get("path") or source_manifest.get(
        "pdf_path", ""
    )
    pdf_path = Path(pdf_value).expanduser() if pdf_value else None
    if pdf_path and (
        not pdf_path.is_file() or file_sha256(pdf_path) != source_manifest["source_sha256"]
    ):
        raise SystemExit("Source Manifest PDF does not match source_sha256.")
    if pdf_path:
        expected_work = work_identity(args.archive_input)
        try:
            observed = (
                verify_pdf(pdf_path, args.archive_input)
                if expected_work["identifiers"]
                else pdf_identity(pdf_path)
            )
            args.pdf_identity = observed
            record_work_evidence(expected_work, source_manifest['source_sha256'], observed)
            args.archive_input["work"] = expected_work
        except ArchiveError as exc:
            raise SystemExit(str(exc)) from exc
    args.source_details = source_details(args.archive_input, pdf_path)
    admission = "workspace"
    domain_routing_skipped = False
    approved_existing_note_sha256 = ""
    if output_mode == "obsidian":
        (
            target_path,
            resolved_subdir,
            admission,
            domain_routing_skipped,
            approved_existing_note_sha256,
        ) = resolve_obsidian_save_target(
            args,
            config,
            root_path,
            title=title,
            abstract=str(record.get("abstract", "")),
            source_sha256=str(source_manifest["source_sha256"]),
            output_language=output_language,
        )
    else:
        if args.preflight:
            raise SystemExit("--preflight requires Obsidian save mode.")
        resolved_subdir = resolve_domain_subdir(
            config,
            title=title,
            abstract=str(record.get("abstract", "")),
            subdir=args.subdir,
        )
        target_path = resolve_obsidian_note_path(
            config,
            title=title,
            subdir=resolved_subdir,
            filename=args.filename,
        )

    if args.preflight:
        emit(
            {
                "status": "ok",
                "script": "write_obsidian_note.py",
                "phase": "save_target_admission",
                "admission": admission,
                "domain_routing_skipped": domain_routing_skipped,
                "target_directory": str(target_path.parent),
                "note_path": str(target_path),
                "asset_subdir": args.admitted_asset_subdir,
                "source_sha256": str(source_manifest["source_sha256"]),
                "output_language": output_language,
            },
            args.output,
        )
        return

    if not args.lint_json:
        raise SystemExit("Formal Save requires Final Note Lint with output_language.")
    lint_path = str(Path(args.lint_json).expanduser().resolve())
    # utf-8-sig tolerates a BOM in the lint JSON (e.g. produced/edited on
    # Windows) that would otherwise crash json.loads before any gate check.
    lint = json.loads(Path(lint_path).read_text(encoding="utf-8-sig"))
    try:
        require_artifact_output_language(lint, "lint artifact", output_language)
    except ValueError as exc:
        raise SystemExit(str(exc)) from exc
    require_lint_gate(lint, "passes_basic_structure", "basic structure", lint_path)
    require_lint_gate(lint, "passes_style_gate", "style", lint_path)
    require_lint_gate(lint, "passes_math_gate", "math", lint_path)
    if "passes_figure_gate" in lint and not lint.get("passes_figure_gate", False):
        raise SystemExit(lint_failure_message(lint, "figure", lint_path))
    if "passes_plan_gate" in lint and not lint.get("passes_plan_gate", False):
        raise SystemExit(lint_failure_message(lint, "plan", lint_path))
    if "passes_substantive_content" in lint and not lint.get("passes_substantive_content", False):
        raise SystemExit(lint_failure_message(lint, "substantive content", lint_path))
    if "passes_reference_hygiene_gate" in lint and not lint.get("passes_reference_hygiene_gate", False):
        raise SystemExit(lint_failure_message(lint, "reference hygiene", lint_path))

    if not args.figure_decisions:
        raise SystemExit("Formal Save requires Figure/Table Decisions with output_language.")
    figure_decisions = maybe_load_json_record(args.figure_decisions)
    if figure_decisions is None:
        raise SystemExit(f"Expected JSON object for --figure-decisions: {args.figure_decisions}")
    try:
        require_artifact_output_language(
            figure_decisions,
            "Figure/Table Decisions",
            output_language,
        )
    except ValueError as exc:
        raise SystemExit(str(exc)) from exc

    if args.content_file:
        # utf-8-sig strips a leading BOM so it is never written into the saved
        # note (a leading BOM breaks Obsidian frontmatter / the H1 title).
        note_text = Path(args.content_file).expanduser().resolve().read_text(encoding="utf-8-sig")
    elif args.content:
        note_text = args.content
    elif args.stdin:
        note_text = sys.stdin.read()
    else:
        raise SystemExit("write_obsidian_note.py requires --content-file, --content, or --stdin.")
    note_text = note_text.replace("\r\n", "\n")
    lint_note_sha256 = str(lint.get("note_sha256", "")).strip()
    if not lint_note_sha256:
        raise SystemExit("lint artifact requires note_sha256.")
    note_sha256 = hashlib.sha256(note_text.encode("utf-8")).hexdigest()
    if lint_note_sha256 != note_sha256:
        raise SystemExit(
            "write_obsidian_note.py refused to write note because the final note "
            "changed after Final Note Lint; rerun lint under the same output_language."
        )
    require_reference_hygiene(note_text, "before save")

    with archive_lock(root_path) if output_mode == "obsidian" else nullcontext():
        if pdf_path and file_sha256(pdf_path) != source_manifest["source_sha256"]:
            raise SystemExit("Source Manifest PDF changed before save.")
        overwrote_existing_note = bool(approved_existing_note_sha256)
        if output_mode == "obsidian":
            rechecked_target, _, _, _, rechecked_existing_sha256 = resolve_obsidian_save_target(
                args,
                config,
                root_path,
                title=title,
                abstract=str(record.get("abstract", "")),
                source_sha256=str(source_manifest["source_sha256"]),
                output_language=output_language,
            )
            if (
                rechecked_target.resolve() != target_path.resolve()
                or rechecked_existing_sha256 != approved_existing_note_sha256
            ):
                block_directory_conflict(
                    args,
                    conflict_code="save_target_changed",
                    target_directories=[target_path.parent, rechecked_target.parent],
                    source_sha256=str(source_manifest["source_sha256"]),
                    output_language=output_language,
                )

        asset_subdir = args.admitted_asset_subdir if output_mode == 'obsidian' else args.asset_subdir
        asset_dir = resolve_note_asset_dir(target_path, asset_subdir)
        asset_subdir = asset_dir.relative_to(target_path.parent).as_posix()
        asset_directory_existed = asset_dir.exists()
        ensure_parent(target_path)
        paper_id = (
            args.paper_id
            or str(source_manifest.get("paper_id", ""))
            or str(record.get("paper_id", ""))
        )
        sidecar_path = (
            target_path.parent / PAPER_DIRECTORY_SIDECAR
            if output_mode == "obsidian"
            else None
        )
        note_backup = None
        sidecar_backup = None
        created_assets: list[Path] = []
        note_attempted = False
        sidecar_attempted = False
        try:
            note_backup = backup_file(target_path)
            sidecar_backup = backup_file(sidecar_path) if sidecar_path is not None else None
            materialized_figures = (
                materialize_insert_decisions(
                    note_text,
                    target_path,
                    figure_decisions,
                    asset_subdir,
                    created_assets,
                )
                if figure_decisions
                else []
            )
            if (
                approved_existing_note_sha256
                and file_sha256(target_path) != approved_existing_note_sha256
            ):
                block_note_conflict(
                    args,
                    target_path,
                    conflict_code="stale_overwrite_confirmation",
                    source_sha256=str(source_manifest["source_sha256"]),
                    output_language=output_language,
                )
            note_attempted = True
            atomic_write_note(target_path, note_text)
            require_reference_hygiene(
                target_path.read_text(encoding="utf-8"),
                "after save",
            )
            asset_dir.mkdir(parents=True, exist_ok=True)
            if sidecar_path is not None:
                sidecar_attempted = True
                write_paper_directory_sidecar(
                    target_path,
                    paper_id=paper_id,
                    title=title,
                    source_sha256=str(source_manifest["source_sha256"]),
                    output_language=output_language,
                    note_sha256=note_sha256,
                    registry=args.archive_registry,
                )
        except BaseException:
            if sidecar_attempted and sidecar_path is not None:
                restore_file(sidecar_path, sidecar_backup)
                if sidecar_backup is not None:
                    try:
                        ensure_sidecar_hidden(sidecar_path)
                    except OSError:
                        pass
            if note_attempted:
                restore_file(target_path, note_backup)
            for created_asset in reversed(created_assets):
                created_asset.unlink(missing_ok=True)
            if not asset_directory_existed:
                try:
                    asset_dir.rmdir()
                except OSError:
                    pass
            raise
        finally:
            if note_backup is not None:
                note_backup.unlink(missing_ok=True)
            if sidecar_backup is not None:
                sidecar_backup.unlink(missing_ok=True)

        payload = {
            "status": "ok",
            "script": "write_obsidian_note.py",
            "output_language": output_language,
            "paper_id": paper_id,
            "title": title,
            "note_path": str(target_path),
            "subdir": resolved_subdir,
            "images_dir": str(asset_dir),
            "materialized_figures": materialized_figures,
            "overwrote_existing_note": overwrote_existing_note,
            "admission": admission,
            "domain_routing_skipped": domain_routing_skipped,
        }
        if sidecar_path is not None:
            payload["sidecar_path"] = str(sidecar_path)
            payload["source_sha256"] = str(source_manifest["source_sha256"])
        payload["output_mode"] = output_mode
        payload["base_output_root"] = str(root_path)
        if config.get("obsidian_vault"):
            payload["vault"] = str(Path(config["obsidian_vault"]).expanduser().resolve())
        emit(payload, args.output)


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""Read-only client and matching helpers for Zotero's desktop Local API."""

from __future__ import annotations

import json
import os
import re
import socket
import urllib.error
import urllib.parse
import urllib.request
from dataclasses import dataclass
from pathlib import Path, PurePosixPath, PureWindowsPath
from typing import Any, Callable, Mapping

from common import (
    DEFAULT_USER_AGENT,
    apply_identity_confidence,
    extract_arxiv_id,
    extract_doi,
    infer_source_type,
    normalize_identity_title,
    normalize_whitespace,
    paper_id_for_record,
)

ZOTERO_LOCAL_BASE_URL = "http://127.0.0.1:23119/api"
ZOTERO_API_VERSION = "3"
DEFAULT_TIMEOUT_SECONDS = 1.5
MAX_JSON_RESPONSE_BYTES = 2 * 1024 * 1024
MAX_FILE_URL_BYTES = 16 * 1024
ZOTERO_ITEM_KEY_PATTERN = re.compile(r"^[A-Z0-9]{8}$")
ZOTERO_SELECT_KEY_PATTERN = re.compile(
    r"^zotero://select/library/items/([A-Z0-9]{8})$",
    flags=re.IGNORECASE,
)
ZOTERO_GROUP_SELECT_PATTERN = re.compile(
    r"^zotero://select/groups/\d+/items/[A-Z0-9]{8}$",
    flags=re.IGNORECASE,
)


@dataclass(frozen=True)
class HttpResult:
    """Small transport-neutral HTTP response used by tests and the stdlib client."""

    status: int
    headers: Mapping[str, str]
    body: bytes


class ZoteroLocalError(RuntimeError):
    """Stable error contract for Local API failures."""

    def __init__(
        self,
        code: str,
        message: str,
        *,
        http_status: int | None = None,
        retryable: bool = False,
    ) -> None:
        super().__init__(message)
        self.code = code
        self.http_status = http_status
        self.retryable = retryable

    def as_dict(self) -> dict[str, Any]:
        payload: dict[str, Any] = {
            "code": self.code,
            "message": str(self),
            "retryable": self.retryable,
        }
        if self.http_status is not None:
            payload["http_status"] = self.http_status
        return payload


Transport = Callable[[str, dict[str, str], float], HttpResult]


class _NoRedirectHandler(urllib.request.HTTPRedirectHandler):
    """Keep every Local API transport request on the validated loopback endpoint."""

    def redirect_request(self, req, fp, code, msg, headers, newurl):  # type: ignore[no-untyped-def]
        return None


def _normalized_headers(headers: Mapping[str, str]) -> dict[str, str]:
    return {str(key).lower(): str(value) for key, value in headers.items()}


def _read_limited(response: Any, limit: int) -> bytes:
    body = response.read(limit + 1)
    if len(body) > limit:
        raise ZoteroLocalError(
            "zotero_invalid_response",
            "Zotero Local API returned an unexpectedly large response.",
        )
    return body


def _default_transport(url: str, headers: dict[str, str], timeout: float) -> HttpResult:
    # Local library queries should never inherit a corporate/system HTTP proxy.
    opener = urllib.request.build_opener(
        urllib.request.ProxyHandler({}),
        _NoRedirectHandler(),
    )
    request = urllib.request.Request(url, headers=headers, method="GET")
    try:
        with opener.open(request, timeout=timeout) as response:
            return HttpResult(
                status=int(getattr(response, "status", 200)),
                headers=_normalized_headers(response.headers),
                body=_read_limited(response, MAX_JSON_RESPONSE_BYTES),
            )
    except urllib.error.HTTPError as exc:
        body = b""
        try:
            body = _read_limited(exc, MAX_JSON_RESPONSE_BYTES)
        except Exception:
            pass
        return HttpResult(
            status=int(exc.code),
            headers=_normalized_headers(exc.headers or {}),
            body=body,
        )
    except urllib.error.URLError as exc:
        if isinstance(exc.reason, (socket.timeout, TimeoutError)):
            raise ZoteroLocalError(
                "zotero_timeout",
                "Timed out while connecting to the Zotero Local API.",
                retryable=True,
            ) from exc
        raise ZoteroLocalError(
            "zotero_not_running",
            "Could not connect to the Zotero Local API on this computer.",
            retryable=True,
        ) from exc
    except (socket.timeout, TimeoutError) as exc:
        raise ZoteroLocalError(
            "zotero_timeout",
            "Timed out while connecting to the Zotero Local API.",
            retryable=True,
        ) from exc
    except OSError as exc:
        raise ZoteroLocalError(
            "zotero_not_running",
            "Could not connect to the Zotero Local API on this computer.",
            retryable=True,
        ) from exc


def _validate_base_url(base_url: str) -> str:
    parsed = urllib.parse.urlsplit(base_url.rstrip("/"))
    if (
        parsed.scheme != "http"
        or parsed.hostname not in {"127.0.0.1", "localhost", "::1"}
        or parsed.username is not None
        or parsed.password is not None
        or parsed.query
        or parsed.fragment
        or parsed.path.rstrip("/") != "/api"
    ):
        raise ValueError("Zotero Local API base URL must be a loopback HTTP /api endpoint.")
    return urllib.parse.urlunsplit((parsed.scheme, parsed.netloc, parsed.path.rstrip("/"), "", ""))


def _error_for_http_status(status: int) -> ZoteroLocalError:
    if status == 400:
        return ZoteroLocalError(
            "zotero_bad_request",
            "Zotero Local API rejected the request.",
            http_status=status,
        )
    if status == 403:
        return ZoteroLocalError(
            "zotero_api_disabled",
            "Zotero Local API access is disabled in Zotero settings.",
            http_status=status,
        )
    if status == 404:
        return ZoteroLocalError(
            "zotero_item_not_found",
            "The requested Zotero item was not found.",
            http_status=status,
        )
    if status == 501:
        return ZoteroLocalError(
            "zotero_unsupported_version",
            "This Zotero installation does not support the requested Local API version.",
            http_status=status,
        )
    if status >= 500:
        return ZoteroLocalError(
            "zotero_server_error",
            "Zotero Local API returned a server error.",
            http_status=status,
            retryable=True,
        )
    return ZoteroLocalError(
        "zotero_invalid_response",
        f"Zotero Local API returned unexpected HTTP status {status}.",
        http_status=status,
    )


class ZoteroLocalClient:
    """Minimal read-only Zotero Local API v3 client."""

    def __init__(
        self,
        *,
        timeout: float = DEFAULT_TIMEOUT_SECONDS,
        transport: Transport | None = None,
        base_url: str = ZOTERO_LOCAL_BASE_URL,
    ) -> None:
        self.timeout = timeout
        self.transport = transport or _default_transport
        self.base_url = _validate_base_url(base_url)
        self._probe_result: dict[str, Any] | None = None

    def _url(self, path: str = "", params: Mapping[str, Any] | None = None) -> str:
        url = f"{self.base_url}/{path.lstrip('/')}" if path else f"{self.base_url}/"
        if params:
            url = f"{url}?{urllib.parse.urlencode(params)}"
        return url

    def _request(
        self,
        path: str = "",
        *,
        params: Mapping[str, Any] | None = None,
        accept: str = "application/json",
        max_bytes: int = MAX_JSON_RESPONSE_BYTES,
    ) -> HttpResult:
        headers = {
            "Accept": accept,
            "User-Agent": DEFAULT_USER_AGENT,
            "Zotero-API-Version": ZOTERO_API_VERSION,
        }
        result = self.transport(self._url(path, params), headers, self.timeout)
        if not 200 <= result.status < 300:
            raise _error_for_http_status(result.status)
        if len(result.body) > max_bytes:
            raise ZoteroLocalError(
                "zotero_invalid_response",
                "Zotero Local API returned an unexpectedly large response.",
            )
        return result

    def probe(self) -> dict[str, Any]:
        if self._probe_result is not None:
            return dict(self._probe_result)
        result = self._request()
        headers = _normalized_headers(result.headers)
        api_version = normalize_whitespace(headers.get("zotero-api-version", ""))
        if api_version != ZOTERO_API_VERSION:
            raise ZoteroLocalError(
                "zotero_unsupported_version",
                "Zotero Local API did not report compatible API version 3.",
            )
        self._probe_result = {
            "status": "available",
            "reachable": True,
            "ready": True,
            "base_url": self.base_url,
            "api_version": api_version,
            "schema_version": normalize_whitespace(headers.get("zotero-schema-version", "")),
        }
        return dict(self._probe_result)

    def _ensure_ready(self) -> None:
        if self._probe_result is None:
            self.probe()

    def _request_json(
        self,
        path: str,
        *,
        params: Mapping[str, Any] | None = None,
    ) -> Any:
        self._ensure_ready()
        result = self._request(path, params=params)
        try:
            return json.loads(result.body.decode("utf-8"))
        except (UnicodeDecodeError, json.JSONDecodeError) as exc:
            raise ZoteroLocalError(
                "zotero_invalid_response",
                "Zotero Local API returned invalid JSON.",
            ) from exc

    def get_item(self, item_key: str) -> dict[str, Any]:
        key = normalize_whitespace(item_key).upper()
        if not ZOTERO_ITEM_KEY_PATTERN.fullmatch(key):
            raise ValueError("Zotero item keys must contain exactly eight letters or digits.")
        payload = self._request_json(
            f"users/0/items/{key}",
            params={"format": "json", "include": "data"},
        )
        if not isinstance(payload, dict):
            raise ZoteroLocalError(
                "zotero_invalid_response",
                "Zotero item response was not a JSON object.",
            )
        return payload

    def search_top_items(
        self,
        query: str,
        *,
        qmode: str,
    ) -> list[dict[str, Any]]:
        if qmode not in {"everything", "titleCreatorYear"}:
            raise ValueError("Unsupported Zotero quick-search mode.")
        payload = self._request_json(
            "users/0/items/top",
            params={
                "format": "json",
                "include": "data",
                "itemType": "-attachment",
                "q": query,
                "qmode": qmode,
            },
        )
        if not isinstance(payload, list) or not all(isinstance(item, dict) for item in payload):
            raise ZoteroLocalError(
                "zotero_invalid_response",
                "Zotero search response was not a JSON item list.",
            )
        return payload

    def get_children(self, parent_key: str) -> list[dict[str, Any]]:
        key = normalize_whitespace(parent_key).upper()
        if not ZOTERO_ITEM_KEY_PATTERN.fullmatch(key):
            raise ValueError("Zotero item keys must contain exactly eight letters or digits.")
        payload = self._request_json(
            f"users/0/items/{key}/children",
            params={"format": "json", "include": "data"},
        )
        if not isinstance(payload, list) or not all(isinstance(item, dict) for item in payload):
            raise ZoteroLocalError(
                "zotero_invalid_response",
                "Zotero children response was not a JSON item list.",
            )
        return payload

    def get_attachment_file_url(self, attachment_key: str) -> str:
        key = normalize_whitespace(attachment_key).upper()
        if not ZOTERO_ITEM_KEY_PATTERN.fullmatch(key):
            raise ValueError("Zotero item keys must contain exactly eight letters or digits.")
        self._ensure_ready()
        result = self._request(
            f"users/0/items/{key}/file/view/url",
            accept="text/plain",
            max_bytes=MAX_FILE_URL_BYTES,
        )
        try:
            return result.body.decode("utf-8").strip()
        except UnicodeDecodeError as exc:
            raise ZoteroLocalError(
                "zotero_invalid_response",
                "Zotero attachment URL was not UTF-8 text.",
            ) from exc


def _item_data(raw: Mapping[str, Any]) -> dict[str, Any]:
    data = raw.get("data", {})
    return dict(data) if isinstance(data, dict) else {}


def zotero_item_key(raw: Mapping[str, Any]) -> str:
    data = _item_data(raw)
    return normalize_whitespace(str(raw.get("key") or data.get("key") or "")).upper()


def _creator_name(creator: Mapping[str, Any]) -> str:
    corporate = normalize_whitespace(str(creator.get("name", "")))
    if corporate:
        return corporate
    return normalize_whitespace(f"{creator.get('firstName', '')} {creator.get('lastName', '')}")


def _publication_year(value: str) -> str:
    match = re.search(r"(?<!\d)((?:18|19|20|21)\d{2})(?!\d)", value or "")
    return match.group(1) if match else ""


def normalize_zotero_item(raw: Mapping[str, Any]) -> dict[str, Any]:
    """Map a Zotero item wrapper into DeepPaperNote's metadata vocabulary."""

    data = _item_data(raw)
    item_type = normalize_whitespace(str(data.get("itemType", "")))
    key = zotero_item_key(raw)
    authors: list[str] = []
    for creator in data.get("creators", []) or []:
        if not isinstance(creator, dict):
            continue
        if creator.get("creatorType") not in {"author", "bookAuthor"}:
            continue
        name = _creator_name(creator)
        if name and name not in authors:
            authors.append(name)

    doi = (extract_doi(str(data.get("DOI", ""))) or "").lower()
    arxiv_text = " ".join(
        str(data.get(field, "")) for field in ("archiveLocation", "extra", "url", "DOI")
    )
    arxiv_id = extract_arxiv_id(arxiv_text) or ""
    venue = next(
        (
            normalize_whitespace(str(data.get(field, "")))
            for field in (
                "publicationTitle",
                "conferenceName",
                "proceedingsTitle",
                "bookTitle",
                "websiteTitle",
                "repository",
                "publisher",
            )
            if normalize_whitespace(str(data.get(field, "")))
        ),
        "",
    )
    source_url = normalize_whitespace(str(data.get("url", "")))
    parent_url_error = ""
    if source_url:
        try:
            parsed_source_url = _split_attachment_url(source_url)
        except ZoteroLocalError as exc:
            source_url = ""
            parent_url_error = exc.code
        else:
            if parsed_source_url.scheme.lower() not in {"http", "https"}:
                source_url = ""
    if not source_url and doi:
        source_url = f"https://doi.org/{doi}"

    record: dict[str, Any] = {
        "status": "ok",
        "source_type": "zotero",
        "metadata_sources": ["zotero"],
        "zotero_key": key,
        "zotero_item_type": item_type,
        "title": normalize_whitespace(str(data.get("title", ""))),
        "authors": authors,
        "affiliations": [],
        "abstract": normalize_whitespace(str(data.get("abstractNote", ""))),
        "venue": venue,
        "year": _publication_year(str(data.get("date", ""))),
        "doi": doi,
        "arxiv_id": arxiv_id,
        "source_url": source_url,
    }
    version = raw.get("version")
    if version not in (None, ""):
        record["zotero_version"] = version
    if parent_url_error:
        record["zotero_parent_url_error"] = parent_url_error
    return record


def normalize_zotero_attachment(raw: Mapping[str, Any]) -> dict[str, Any]:
    data = _item_data(raw)
    return {
        "key": zotero_item_key(raw),
        "parent_key": normalize_whitespace(str(data.get("parentItem", ""))).upper(),
        "item_type": normalize_whitespace(str(data.get("itemType", ""))),
        "link_mode": normalize_whitespace(str(data.get("linkMode", ""))),
        "title": normalize_whitespace(str(data.get("title", ""))),
        "content_type": normalize_whitespace(str(data.get("contentType", ""))).lower(),
        "filename": normalize_whitespace(str(data.get("filename", ""))),
        "url": normalize_whitespace(str(data.get("url", ""))),
    }


def is_pdf_attachment(attachment: Mapping[str, Any]) -> bool:
    if attachment.get("item_type") != "attachment":
        return False
    if attachment.get("link_mode") == "linked_url":
        return False
    return attachment.get("content_type") == "application/pdf" or any(
        str(attachment.get(field, "")).lower().endswith(".pdf") for field in ("filename", "title")
    )


def _split_attachment_url(raw_url: str) -> urllib.parse.SplitResult:
    try:
        parsed = urllib.parse.urlsplit(raw_url)
        _ = parsed.port, parsed.hostname
    except ValueError as exc:
        raise ZoteroLocalError(
            "zotero_unsafe_file_url",
            "Zotero attachment URL contained an invalid network location.",
        ) from exc
    return parsed


def file_url_to_local_path(raw_url: str, *, platform: str | None = None) -> str:
    """Convert a safe local ``file://`` URL without touching the filesystem."""

    value = (raw_url or "").strip()
    if not value or "\x00" in value:
        raise ZoteroLocalError(
            "zotero_unsafe_file_url",
            "Zotero returned an empty or invalid attachment file URL.",
        )
    parsed = _split_attachment_url(value)
    port = parsed.port
    if (
        parsed.scheme.lower() != "file"
        or parsed.username is not None
        or parsed.password is not None
        or port is not None
        or parsed.query
        or parsed.fragment
        or parsed.hostname not in {None, "", "localhost"}
    ):
        raise ZoteroLocalError(
            "zotero_unsafe_file_url",
            "Zotero attachment URL did not identify a local file.",
        )
    try:
        decoded = urllib.parse.unquote(parsed.path, encoding="utf-8", errors="strict")
    except UnicodeDecodeError as exc:
        raise ZoteroLocalError(
            "zotero_unsafe_file_url",
            "Zotero attachment URL contained invalid UTF-8 path data.",
        ) from exc
    if "\x00" in decoded:
        raise ZoteroLocalError(
            "zotero_unsafe_file_url",
            "Zotero attachment URL contained an invalid path.",
        )
    target_platform = platform or os.name
    if target_platform == "nt":
        windows_path = decoded.replace("/", "\\")
        if windows_path.startswith("\\\\"):
            raise ZoteroLocalError(
                "zotero_unsafe_file_url",
                "Zotero attachment URL identified a remote UNC path.",
            )
        if re.match(r"^/[A-Za-z]:/", decoded):
            decoded = decoded[1:]
        path = PureWindowsPath(decoded.replace("/", "\\"))
    else:
        if decoded.startswith("//"):
            raise ZoteroLocalError(
                "zotero_unsafe_file_url",
                "Zotero attachment URL identified an ambiguous network path.",
            )
        path = PurePosixPath(decoded)
    if not path.is_absolute():
        raise ZoteroLocalError(
            "zotero_unsafe_file_url",
            "Zotero attachment URL was not an absolute local path.",
        )
    return str(path)


def existing_pdf_from_file_url(raw_url: str) -> str:
    path_value = file_url_to_local_path(raw_url)
    try:
        path = Path(path_value).expanduser().resolve(strict=True)
    except (OSError, RuntimeError):
        return ""
    if not path.is_file() or path.suffix.lower() != ".pdf":
        return ""
    return str(path)


def normalize_lookup_title(title: str) -> str:
    return normalize_identity_title(title)


def _match_search_results(
    raw_items: list[dict[str, Any]],
    *,
    match_kind: str,
    query: str,
) -> dict[str, Any]:
    candidates = [
        normalize_zotero_item(item)
        for item in raw_items
        if _item_data(item).get("itemType") not in {"", "attachment", "note", "annotation"}
        and ZOTERO_ITEM_KEY_PATTERN.fullmatch(zotero_item_key(item))
    ]
    if match_kind == "doi":
        expected = (extract_doi(query) or "").lower()
        matches = [item for item in candidates if item.get("doi", "").lower() == expected]
    elif match_kind == "arxiv_id":
        expected = extract_arxiv_id(query) or ""
        matches = [item for item in candidates if item.get("arxiv_id") == expected]
    else:
        expected_title = normalize_lookup_title(query)
        matches = [
            item
            for item in candidates
            if normalize_lookup_title(str(item.get("title", ""))) == expected_title
        ]
    if not matches:
        return {"status": "not_found", "match_kind": match_kind, "candidate_count": 0}
    if len(matches) > 1:
        return {
            "status": "ambiguous",
            "match_kind": match_kind,
            "candidate_count": len(matches),
        }
    return {"status": "match", "match_kind": match_kind, "record": matches[0]}


def _attachment_priority(attachment: Mapping[str, Any]) -> tuple[int, int, int, int]:
    label = f"{attachment.get('title', '')} {attachment.get('filename', '')}".casefold()
    supplementary = any(
        term in label
        for term in ("supplementary", "supplemental", "supporting information", "appendix")
    )
    link_mode_score = {
        "imported_file": 3,
        "linked_file": 2,
        "imported_url": 1,
    }.get(str(attachment.get("link_mode", "")), 0)
    explicit_pdf = int(attachment.get("content_type") == "application/pdf")
    full_text = int("full text" in label or "pdf" in label)
    return (int(not supplementary), link_mode_score, explicit_pdf, full_text)


def _attach_local_pdf(
    client: ZoteroLocalClient,
    record: dict[str, Any],
    raw_children: list[dict[str, Any]],
    *,
    preferred_attachment: Mapping[str, Any] | None = None,
) -> dict[str, Any]:
    parent_key = str(record.get("zotero_key", ""))
    raw_candidates = list(raw_children)
    if preferred_attachment and zotero_item_key(preferred_attachment) not in {
        zotero_item_key(item) for item in raw_candidates
    }:
        raw_candidates.append(dict(preferred_attachment))
    candidates = [normalize_zotero_attachment(item) for item in raw_candidates]
    candidates = [
        item
        for item in candidates
        if is_pdf_attachment(item)
        and item.get("parent_key") == parent_key
        and ZOTERO_ITEM_KEY_PATTERN.fullmatch(str(item.get("key", "")))
    ]
    preferred_key = zotero_item_key(preferred_attachment or {})
    if preferred_key:
        candidates = [item for item in candidates if item.get("key") == preferred_key]
    if not candidates:
        record["zotero_attachment_status"] = "not_found"
        return record

    local_candidates: list[tuple[dict[str, Any], str]] = []
    attachment_errors: list[str] = []
    for attachment in sorted(candidates, key=lambda item: str(item.get("key", ""))):
        try:
            file_url = client.get_attachment_file_url(str(attachment.get("key", "")))
            local_path = existing_pdf_from_file_url(file_url)
        except ZoteroLocalError as exc:
            attachment_errors.append(exc.code)
            continue
        if local_path:
            local_candidates.append((attachment, local_path))

    if preferred_key:
        preferred = [pair for pair in local_candidates if pair[0].get("key") == preferred_key]
        if preferred:
            local_candidates = preferred
    if local_candidates:
        best_priority = max(_attachment_priority(item) for item, _ in local_candidates)
        best = [pair for pair in local_candidates if _attachment_priority(pair[0]) == best_priority]
        distinct_paths = {path for _, path in best}
        if len(distinct_paths) == 1:
            attachment, local_path = best[0]
            record["zotero_attachment_status"] = "available"
            record["zotero_attachment_key"] = attachment["key"]
            record["zotero_attachment_filename"] = attachment["filename"]
            record["local_pdf_path"] = local_path
            return record
        record["zotero_attachment_status"] = "ambiguous"
        return record

    remote_pdf_urls: set[str] = set()
    for item in candidates:
        attachment_url = str(item.get("url", ""))
        if not attachment_url:
            continue
        try:
            parsed_url = _split_attachment_url(attachment_url)
        except ZoteroLocalError as exc:
            attachment_errors.append(exc.code)
            continue
        if parsed_url.scheme == "https" and parsed_url.path.lower().endswith(".pdf"):
            remote_pdf_urls.add(attachment_url)
    pdf_urls = sorted(remote_pdf_urls)
    if len(pdf_urls) == 1:
        record["pdf_url"] = pdf_urls[0]
        record["zotero_attachment_status"] = "remote_only"
    else:
        record["zotero_attachment_status"] = "unavailable"
    if attachment_errors:
        record["zotero_attachment_error"] = attachment_errors[0]
    return record


def _zotero_key_from_reference(reference: str, *, reference_type: str) -> str:
    value = normalize_whitespace(reference)
    if ZOTERO_GROUP_SELECT_PATTERN.fullmatch(value):
        raise ZoteroLocalError(
            "zotero_unsupported_library",
            "Zotero group-library select links are not supported by the local user-library client.",
        )
    match = ZOTERO_SELECT_KEY_PATTERN.fullmatch(value)
    if match:
        return match.group(1).upper()
    if reference_type == "zotero_key" or ZOTERO_ITEM_KEY_PATTERN.fullmatch(value):
        return value.upper()
    return ""


def _resolve_parent_item(
    client: ZoteroLocalClient,
    raw_item: dict[str, Any],
) -> tuple[dict[str, Any], dict[str, Any] | None]:
    data = _item_data(raw_item)
    if data.get("itemType") != "attachment":
        return raw_item, None
    parent_key = normalize_whitespace(str(data.get("parentItem", ""))).upper()
    if not ZOTERO_ITEM_KEY_PATTERN.fullmatch(parent_key):
        raise ZoteroLocalError(
            "zotero_item_not_bibliographic",
            "The Zotero attachment is not linked to a bibliographic parent item.",
        )
    return client.get_item(parent_key), raw_item


def lookup_zotero_local(
    reference: str,
    *,
    reference_type: str = "",
    client: ZoteroLocalClient | None = None,
) -> dict[str, Any]:
    """Resolve one scalar reference against the local Zotero library."""

    local_client = client or ZoteroLocalClient()
    source_type = reference_type or infer_source_type(reference)
    try:
        probe = local_client.probe()
        explicit_key = _zotero_key_from_reference(reference, reference_type=source_type)
        preferred_attachment: dict[str, Any] | None = None
        if explicit_key:
            raw_item = local_client.get_item(explicit_key)
            if zotero_item_key(raw_item) != explicit_key:
                raise ZoteroLocalError(
                    "zotero_invalid_response",
                    "Zotero returned a different item than the requested key.",
                )
            raw_parent, preferred_attachment = _resolve_parent_item(local_client, raw_item)
            record = normalize_zotero_item(raw_parent)
            parent_key = str(record.get("zotero_key", ""))
            if record.get("zotero_item_type") in {"attachment", "note", "annotation", ""}:
                raise ZoteroLocalError(
                    "zotero_item_not_bibliographic",
                    "The Zotero item is not a bibliographic parent item.",
                )
            if not ZOTERO_ITEM_KEY_PATTERN.fullmatch(parent_key):
                raise ZoteroLocalError(
                    "zotero_invalid_response",
                    "Zotero bibliographic item did not contain a valid item key.",
                )
            match_kind = "zotero_key"
        else:
            doi = extract_doi(reference) or ""
            arxiv_id = extract_arxiv_id(reference) or ""
            if source_type in {"doi", "doi_url"} or doi:
                match_kind = "doi"
                query = doi
                qmode = "everything"
            elif source_type in {"arxiv_id", "arxiv_url"} or arxiv_id:
                match_kind = "arxiv_id"
                query = arxiv_id
                qmode = "everything"
            elif source_type in {"title", "title_query"}:
                match_kind = "title"
                query = reference
                qmode = "titleCreatorYear"
            else:
                return {
                    "status": "not_found",
                    "match_kind": "unsupported_reference",
                    "probe": probe,
                }
            raw_items = local_client.search_top_items(query, qmode=qmode)
            match = _match_search_results(raw_items, match_kind=match_kind, query=query)
            if match["status"] != "match":
                match["probe"] = probe
                return match
            record = dict(match["record"])

        try:
            raw_children = local_client.get_children(str(record.get("zotero_key", "")))
            record = _attach_local_pdf(
                local_client,
                record,
                raw_children,
                preferred_attachment=preferred_attachment,
            )
        except ZoteroLocalError as exc:
            record["zotero_attachment_status"] = "unavailable"
            record["zotero_attachment_error"] = exc.code
        record["paper_id"] = paper_id_for_record(record)
        record = apply_identity_confidence(record)
        return {
            "status": "match",
            "match_kind": match_kind,
            "record": record,
            "probe": probe,
        }
    except ZoteroLocalError as exc:
        if exc.code == "zotero_item_not_found":
            return {
                "status": "not_found",
                "match_kind": source_type,
                "error": exc.as_dict(),
            }
        return {
            "status": "error",
            "match_kind": source_type,
            "error": exc.as_dict(),
        }


def probe_zotero_local_api(
    *,
    client: ZoteroLocalClient | None = None,
) -> dict[str, Any]:
    """Return a non-raising environment diagnostic for the Local API."""

    local_client = client or ZoteroLocalClient()
    try:
        return local_client.probe()
    except ZoteroLocalError as exc:
        if exc.code == "zotero_api_disabled":
            status = "disabled"
        elif exc.code == "zotero_unsupported_version":
            status = "incompatible"
        elif exc.code in {"zotero_not_running", "zotero_timeout", "zotero_server_error"}:
            status = "unavailable"
        else:
            status = "error"
        return {
            "status": status,
            "reachable": exc.code not in {"zotero_not_running", "zotero_timeout"},
            "ready": False,
            "base_url": local_client.base_url,
            "api_version": "",
            "schema_version": "",
            "error": exc.as_dict(),
        }