interface:
  display_name: "Creator Content Ingest"
  short_description: "Build evidence and four source-extracted asset types"
  default_prompt: "Use $creator-content-ingest to preserve this source, build navigable chapters, and extract evidence-backed quote, viewpoint, hook, and structure assets."

{
  "schema": "creator-asr-lexicon/v1",
  "version": 1,
  "terms": [
    {
      "term": "WebCode",
      "aliases": ["webcode", "web code"],
      "category": "brand_or_product",
      "status": "verified",
      "source": "seeded_creator_vocabulary"
    },
    {
      "term": "Vibe Coding",
      "aliases": ["vibe coding", "vibecoding"],
      "category": "english_term",
      "status": "verified",
      "source": "seeded_creator_vocabulary"
    },
    {
      "term": "Codex",
      "aliases": [],
      "category": "brand_or_product",
      "status": "verified",
      "source": "seeded_creator_vocabulary"
    },
    {
      "term": "Claude Code",
      "aliases": [],
      "category": "brand_or_product",
      "status": "verified",
      "source": "seeded_creator_vocabulary"
    },
    {
      "term": "Gemini",
      "aliases": [],
      "category": "brand_or_product",
      "status": "verified",
      "source": "seeded_creator_vocabulary"
    },
    {
      "term": "MiniMax",
      "aliases": [],
      "category": "brand_or_product",
      "status": "verified",
      "source": "seeded_creator_vocabulary"
    },
    {
      "term": "PRD",
      "aliases": [],
      "category": "english_term",
      "status": "verified",
      "source": "seeded_creator_vocabulary"
    }
  ]
}

{
  "schema_version": "1.0",
  "job_id": "CCI-20260803-120000-a1b2c3d4",
  "created_at": "2026-08-03T12:00:00+08:00",
  "source_agent": "hermes",
  "input_channel": "user-direct",
  "raw_input": "这篇小红书的封面和观点都值得学习 https://xhslink.cn/o/example",
  "source": {
    "url": "https://xhslink.cn/o/example",
    "local_files": [],
    "platform_hint": "xhs",
    "content_form_hint": "unknown"
  },
  "instruction": {
    "raw": "完整保存，逐图 OCR，再拆观点和版式",
    "mode": "full",
    "requested_outputs": ["images", "ocr", "viewpoints", "visual_layout"]
  },
  "routing": {
    "platform": "xhs",
    "content_form": "unknown",
    "priority": "high"
  },
  "state": {
    "status": "pending",
    "stage": "resolve",
    "attempts": 0,
    "claimed_by": null,
    "claimed_at": null,
    "updated_at": "2026-08-03T12:00:00+08:00"
  },
  "resolution": {},
  "outputs": {},
  "errors": []
}

# Cross-agent conformance and handoff

Hermes, Claudian, WorkBuddy, and Codex are four runtime adapters for the same skill. They are not quality tiers and are not restricted to separate pipeline stages. Any one of them may execute a full job when its environment exposes the required tools.

## Equal-result contract

For the same source, instruction, and mode, every adapter must produce:

- the same canonical job and package schemas;
- the same four evidence/standard/processed/card layers;
- the same carrier-specific artifact types;
- the same locator precision;
- the same nine quality gates and completion rules, including independent `comments` and `cover` gates;
- the same uncertainty disclosures and card admission threshold.

An artifact is conformant when another adapter can validate and continue it without knowing which platform created it.

## Runtime differences are adapters, not lowered standards

- Hermes may receive links fastest in chat and also execute the complete pipeline.
- Claudian may process persistent local batches and also execute the complete pipeline.
- WorkBuddy may run in an authorized workspace or through connectors and also execute the complete pipeline.
- Codex may handle individual or difficult jobs and also execute the complete pipeline.

These are convenience and environment differences only. No adapter may replace high-accuracy transcription, per-image OCR, full visual inspection, source locators, or evidence-backed analysis with a generic summary while still returning `completed`.

## Capability declaration and recovery

Before execution, an adapter declares its current filesystem, browser/auth, acquisition, media probe, transcription, OCR, and visual/audio verification capabilities. If a required capability is absent:

1. complete and preserve every valid earlier artifact;
2. set the blocked gate to `review` or `partial`;
3. leave an exact handoff receipt;
4. allow any other conformant adapter to resume the same package.

This is continuation of one skill, not transfer to a lower/higher-quality role.

## Optional queue shortcut

Any adapter may enqueue/defer when the user asks or the runtime lacks a required capability. Queueing is an operating mode, not Hermes's default responsibility. Each independent source still gets one portable job preserving the user's exact message.

## Handoff receipt

```json
{
  "worker": "hermes",
  "stage": "extract",
  "result": "review",
  "artifacts_added": ["evidence/ocr/image-003.raw.txt"],
  "gates_changed": {"extraction": "review"},
  "next_action": "Visually verify two stylized title lines on image 3",
  "updated_at": "2026-08-03T13:00:00+08:00"
}
```

“处理失败” is not adequate. State exact stage, artifact, error, attempted recovery, and next action. Never alter raw evidence during a handoff.

## Deterministic adapter commands

Every adapter should expose equivalent commands for:

- `capture_source.py`: platform acquisition, original media/images, body, OCR, manifest skeleton;
- `transcribe_package.py`: `large-v3-turbo` speech evidence, caption OCR, full-duration keyframes;
- `finalize_transcript.py`: corrected-transcript coverage, lexical checks, unresolved-item and quotation-readiness gate;
- `merge_comments.py`: rank and retain up to ten qualifying comments with explicit sample scope;
- `validate_social_analysis.py`: controlled comment/cover semantics and gate commit;
- `validate_package.py`: final artifact and gate conformance.

If an adapter cannot execute one command directly, it may implement the same input/output contract with its native tools. It may not omit the corresponding artifacts or quietly weaken the gate.

# Article and webpage pipeline

Use for WeChat public-account articles, newsletters, essays, web pages, long notes, and text-led documents. The primary value is usually argument, structure, evidence, title, and reusable viewpoints—not blindly mirroring every byte.

## Stage A0 — Resolve and identify

1. Preserve the user's share message and original URL.
2. Resolve canonical URL and record redirects without retaining secret-bearing query data.
3. Record title, author/account, publication/update time, platform/site, stable ID if present.
4. Record the source channel: `公众号文章` for `mp.weixin.qq.com`; otherwise `网页文章` plus site/domain. This is a carrier/source field, not a content track.
5. Read `content-taxonomy.md` and record one `primary_track`, up to three `secondary_tracks`, and one `content_mechanism`. The same track can occur in either source channel.
6. Identify content boundaries and whether the page is complete, truncated, paywalled, dynamically loaded, or reconstructed.
7. Check duplicate/revision identity.

## Stage A1 — Acquire source evidence

Preserve enough evidence to verify later analysis:

- clean main body with heading/paragraph order;
- meaningful inline images, captions, tables, charts, footnotes, and outbound citations;
- source HTML or reader-mode snapshot when lawful and useful;
- body SHA-256 and image inventory.

Exclude or label navigation, recommendations, advertisements, comments, unrelated author cards, and footer boilerplate. Do not let them enter the article body unnoticed.

Keep acquisition diagnostics out of the reader layer. HTML comments, paragraph locator tokens, download errors, and missing-image placeholders belong in `evidence/` JSON or `manifest.json`, never among the paragraphs shown to the user. If an inline image cannot be archived, omit it from the readable flow and retain the failure only in `source.image_errors`/manifest errors.

Also omit low-information page decoration: publisher logos, icons, loading animations, QR codes, thin separators, and small numbered section-marker GIFs. Use DOM/alt/source semantics first, then conservative dimension/animation checks. Record each deliberate omission in `source.image_omissions`; do not count it as an acquisition failure. A numbered marker immediately followed by a compact section title is structural evidence: preserve its `chapter_number`/`chapter_title`, promote the title to `### 01｜标题`, and omit only the bitmap. Preserve substantial photos, diagrams, screenshots, charts, and image cards.

Full offline mirroring is optional unless the user asks for it. A standard package must still preserve decisive passages and stable locators sufficient to audit claims and quotes.

## Stage A2 — Standardize with locators

Create preserved ordered paragraph evidence, then create `evidence/article/正文-章节版.json` and `.md` according to `chapter-contract.md`. `标准内容.md` renders those chapters with:

- completeness/source note;
- metadata;
- heading hierarchy;
- source-authored chapter number and title in canonical `### 01｜标题` form; keep unnumbered lead-in prose as a preface;
- clean reader-visible paragraphs, with durable IDs and line/range locators stored in the chapter/body JSON rather than printed into the prose;
- inline image/caption markers;
- archived inline images and captions in their original reading position; inaccessible media is reported in evidence metadata, not rendered as a fake paragraph.

An image caption is source content, not an inventory label. Put verified source-visible caption text in Markdown image alt text and render it once below the corresponding image. For a source image without a caption, emit empty alt text (`![](path)`). Never expose generated labels such as `Image 001`, `Image 002`, or filenames as reader captions. When one source caption follows a visual group, associate it with the group in evidence and display it once after the final image.

Each semantic chapter groups adjacent paragraph IDs and displays the complete continuous original/correction-only prose. It is not a rewritten synopsis. Correct only extraction artifacts such as broken line wraps, duplicated menus, or encoding errors. Do not silently copyedit the author's prose.

## Stage A3 — Argument and structure analysis

### Core reasoning

- problem/question, thesis, subclaims, premises, evidence, examples, causal links, conclusion;
- alternative explanations, counterarguments, missing evidence, domain boundaries;
- what is original/non-obvious versus familiar framing.

### Article design

- title promise and audience tension;
- opening device;
- section order and transitions;
- evidence placement and abstraction/example alternation;
- payoff and ending;
- what could be compressed, expanded, or rearranged.

### Reusable material

- strong viewpoints expressed as paraphrase plus source locator;
- exact quote candidates, verified against source;
- evidence/cases and their provenance limitations;
- title/hook patterns and structural moves;
- connections to the user's existing ideas, scripts, topics, or disagreements.

### Why saved

Separate:

- the user's explicit reason, if given;
- an evidence-based inference about likely attraction;
- the actual strongest reusable dimensions;
- reasons not to reuse or claims needing fact-checking.

## Stage A4 — Four automatic content assets

- Viewpoint card: claim, reasoning/evidence, context, counterpoint, source locator, possible use.
- Quote card: exact wording, speaker/author, paragraph/heading locator, surrounding context.
- Structure card: reusable article move with section mapping and conditions.
- Title/hook card: syntax/tension mechanism, promised payoff, transfer boundary.
- Archived inline visuals may support understanding, hook, or structure review,
  but become image assets only after explicit user interaction.

Do not promote a summary sentence written by the executor as a source quote.

## Article completion checklist

- Main-body boundaries and completeness are declared.
- Every quote/viewpoint can be traced to a paragraph or heading.
- Images/charts that carry arguments are archived or explicitly missing.
- Source claims are not automatically endorsed as facts.
- Analysis focuses on the article's real strengths, not a fixed template.

# Asset decomposition system

Read this reference before the carrier profile for every `analyze`, `cards`, or
`full` job. It defines what is shared, what varies by carrier, and where mature
open-source components may be reused.

## One system, five adapters

Implement the decomposition loop once:

`canonical capture → normalized evidence → carrier adapter → shared semantic core → independent type gates → standard cards`

The shared semantic core always performs:

1. a brief whole-source understanding: what happened or was explained, the
   central judgment, intended audience takeaway, and commercial assessment;
2. hook reconstruction as an opening promise plus a later payoff;
3. an ordered movement map based on meaningful changes rather than equal slices;
4. quote and viewpoint recall across distinct semantic roles;
5. evidence reconstruction, type-specific falsification, independent review,
   and materialization of admitted cards only.

Do not create five Agents, five schemas, or five renderers. Carrier adapters may
change only:

- which evidence channels are read;
- which boundary candidates are proposed;
- where an opening contract may occur;
- which speaker, time, page, or paragraph locators are required;
- which asset types are applicable and what falsifies them.

Ownership, storage types, review states, compact reader projection, and the
isolation of `agent_generated` from `user_injected` remain shared.

## Mature-component reuse matrix

External component output is a signal, not an asset or semantic conclusion.
Record its provenance and confidence, then let the carrier adapter and shared
semantic core test it against the archived source.

| Carrier | Mature basis | Reuse policy | Forbidden inference |
|---|---|---|---|
| Video | [PySceneDetect](https://github.com/Breakthrough/PySceneDetect) | Optional cut/transition boundary proposals. Merge with reviewed speech, on-screen text, visible action, and result changes. | A camera cut is not automatically a chapter, argument turn, hook, or payoff. |
| Podcast / spoken audio | [pyannote.audio](https://github.com/pyannote/pyannote-audio) concepts; current Sherpa-ONNX runtime with a pyannote segmentation model | Keep the existing runtime. Reuse diarization only for “who spoke when,” attribution, overlap, and conversational-turn evidence. Do not add a parallel Python diarization stack without a measured quality need. | A speaker change is not automatically a topic or structure boundary; diarization does not identify a person's real identity. |
| Image post / document image | Direct image inspection plus source-authored text first; Apple Vision for the baseline page pass; [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR) as a portable/layout-aware fallback | Run one bounded baseline OCR pass per page so the archive has a portable text layer. Trigger retries, crops, alternate engines, or full-resolution OCR only when the baseline is insufficient or exact decision-relevant text must be verified. Preserve page order, OCR boxes, raw/corrected text, and confidence. | OCR prominence is not authorship, importance, viewpoint support, or page function. |
| Article / WeChat / web document | Platform-specific canonical capture first; [Trafilatura](https://github.com/adbar/trafilatura) as generic-web extraction fallback; [Argdown](https://github.com/argdown/argdown) relation concepts | Recover body, metadata, headings, and paragraph order without replacing a higher-fidelity captured body. Model internal relations as claim, support, objection, reply, and conclusion; no Argdown dependency is required. | A heading is not necessarily a claim; extracted text alone does not establish the author's stance or causal support. |
| Music | Verified, rights-compatible lyrics evidence for quotes; [MSAF](https://github.com/urinieto/msaf) or [All-In-One](https://github.com/mir-aidj/all-in-one) as optional structure enhancement | Scan short exact lyric lines for quote value when evidence is available. Use section/boundary estimates only when permitted audio exists. | Audio section estimates do not license or reconstruct lyrics, infer unsupported lyrical meaning, or create viewpoint/content-hook assets. |

## Adoption levels

- **Reuse now:** existing canonical capture, ASR, diarization, on-demand Apple Vision OCR,
  evidence locators, schemas, gates, renderer, and user-card isolation.
- **Optional adapter:** PySceneDetect boundaries, PaddleOCR fallback,
  Trafilatura fallback, and music-structure estimators. Add one only after a
  representative evaluation shows that it improves recall or locator quality
  without weakening fidelity.
- **Conceptual reuse:** Argdown-style argument relations and pyannote's
  separation of segmentation from identity. Express these in the internal
  evidence schema; do not add a dependency merely to copy terminology.

Never install an optional dependency during an ordinary capture job. Runtime
capabilities are detected once; absence must produce a bounded fallback or an
explicit review state, not fabricated structure.

## Boundary-to-meaning rule

A detector proposes *where to inspect*. The Agent decides *what changed* only
after reopening the bounded evidence. Admit a movement boundary only when at
least one audience-relevant variable changes: question, claim, evidence,
speaker position, action, scene purpose, stakes, result, or resolution.

This rule keeps carrier-specific detection useful without allowing mechanical
cuts, speaker turns, page breaks, headings, or musical estimates to masquerade
as content understanding.

## Evaluation before adoption

Evaluate an optional adapter on a small carrier-specific benchmark and compare
it with the current pipeline. Track at least:

- evidence recall and false-boundary rate;
- locator accuracy and source fidelity;
- whole-source understanding agreement;
- hook promise/payoff recovery;
- admitted-card precision by type;
- runtime cost and new review burden.

Adopt the adapter only when it improves a named failure mode. Keep the old path
available until migrated packages validate, and never rewrite `user_injected`
cards during regeneration.

# Asset review loop

资产处理不是“提取一次 -> 审阅一次 -> 改几句文案”。它是一个由审阅结果驱动的闭环。审阅 Agent 的职责不是替提取 Agent 把卡片润色到能看，而是判断问题发生在哪一层，并把候选退回正确的处理阶段。

## 处理阶段

### A. Evidence readiness

先确认来源证据是否足以支持对应类型：

- 文章：正文、章节边界、图片及图片说明；
- 图文：封面、页面顺序、逐图文字和最终行动入口；
- 视频：逐字稿、章节、开头、后续兑现段和必要的镜头证据。

证据不够时，停止在 `review`，不要让提取 Agent 用标题、简介或摘要补全。

### B. Whole-source understanding

在判断钩子和结构之前，先生成一段可供读者和审阅 Agent 共用的简短整体认知。它不是资产，也不是一句标题改写：

```yaml
source_understanding:
  status: reviewed
  summary: "用 1–3 句话说清文章具体发生了什么、视频展示/论述了什么，或图文按什么顺序想讲什么"
  core_topic: "来源真正处理的主题或问题"
  core_message: "来源最后建立的核心观点、结论或主要认知"
  content_intent: "作者想解释、证明、展示、说服或触发什么行动"
  audience_takeaway: "观众看完后最应该明白、感到或记住什么"
  evidence: ["覆盖来源主要部分的证据定位"]
```

`summary` 要短，但必须覆盖完整内容，不能只根据标题、开头或结尾猜测。教程和项目展示不一定有价值判断，但仍须写清它展示了什么过程、证明了什么结果。纯叙事不强行概括成普遍观点，应保留人物、事件、变化和结局。

### C. Source map

整体认知通过复核后，再写来源地图；不能直接开始列卡：

```yaml
source_main_line: "从 source_understanding.core_message 收敛出的审阅主线"
promise:
  surface: "开头给观众的具体承诺或问题"
  locator: "承诺位置"
payoff:
  result: "后文怎样兑现、复杂化或反转"
  locator: "兑现位置"
audience_path:
  attract: "观众为什么停下来"
  carry: "什么让观众继续看"
  leave: "最后留下什么"
candidate_space:
  quote: "可能承载主线的原话"
  viewpoint: "来源真正建立的判断"
  hook: "承诺与兑现的组合"
  structure: "从注意到回收的实际路线"
```

`source_understanding` 的五个简短字段可显示在读者界面，来源地图其余字段只作内部控制。钩子和结构候选必须写清它们怎样服务整体认知；只描述开头技巧或章节顺序而不解释其内容作用，不能准入。

图像和镜头在这里只作为理解内容、验证钩子或解释结构的证据，不成为自动候选。只有用户在后续交互中主动选择、补充或形成判断，才进入标准化注入流程。

### D. Candidate discovery

按金句、观点、钩子、结构四类完整扫描，允许 0-N。每个候选先记录：

- 类型；
- 钩子或结构的候选机制类型（仅作候选，待复原完整承诺/兑现或推进路线后确认）；
- 最小证据定位；
- 它服务来源主线的方式；
- 初步的观众作用；
- 可能与哪张候选重复。

此阶段禁止生成最终的“推荐原因”和“适用场景”，也禁止把摘要直接包装成观点。

### E. Independent reconstruction

审阅 Agent 必须重新打开候选定位，独立复原：

- 金句：原文是否真的这样说；
- 观点：来源是否真的用事实建立了这个判断；
- 钩子：承诺后面是否有真实兑现；
- 结构：顺序、转折和收束是否来自来源；
- 图像/镜头：视觉机制是否由实际证据支持。

上面最后一项只适用于用户交互形成的图像/镜头卡片，不属于本轮自动候选复原。

复原失败时，反馈必须退回 `D`，而不是在审阅阶段替候选改写。

### F. Story-led review

对每个候选依次回答：

1. 抓住人：观众为什么停；
2. 带着走：下一段为什么值得看；
3. 留下来：最后记住什么；
4. 拿得走：以后复制哪种机制，边界是什么。

审阅结果必须标明问题属于哪一类：

- `evidence_gap`：证据或定位不够；
- `source_understanding_missing`：尚未形成完整内容总括；
- `source_understanding_mismatch`：整体认知只覆盖局部、误解载体内容，或与证据冲突；
- `main_line_mismatch`：没有抓住来源主线；
- `type_mismatch`：内容应该归另一类；
- `audience_path_missing`：只有结果，没有注意推进；
- `payoff_missing`：有开头，没有兑现；
- `scope_missing`：观点没有边界；
- `duplicate_mechanism`：和更强候选重复；
- `reader_copy_issue`：内部判断成立，但前端文案不自然。

除 `reader_copy_issue` 外都退回提取或审阅前置阶段；只有文案问题才进入读者-facing 校正。

### G. Repair and rerun

每个退回候选必须生成机器可执行的反馈：

```yaml
asset_id: "source:viewpoint:01"
verdict: "review"
failure_stage: "candidate_discovery"
failure_code: "main_line_mismatch"
repair_request: "重新围绕来源最后的判断扫描观点，不要只概括主题"
evidence_to_reuse:
  - "chapter-03"
  next_stage: "candidate_discovery"
```

重跑规则：

- `evidence_gap`：回到 A；
- `source_understanding_missing`、`source_understanding_mismatch`：回到 B；
- `main_line_mismatch`、`type_mismatch`、`duplicate_mechanism`：回到 C/D；
- `audience_path_missing`、`payoff_missing`、`scope_missing`：回到 C/E；
- `reader_copy_issue`：只重写 reader-facing 字段，不重做候选；
- 已通过证据与语义审核、仅缺少或误用了钩子/结构类型名时，也按
  `reader_copy_issue` 处理：复用原审核，只更新 `hook_type` / `structure_type`
  与 `类型｜具体做法` 标题，不重复扫描来源；
- 已通过的来源证据和其他独立候选继续复用，不整包重跑。
- 每次重跑声明 `settled_types`，只替换本轮真正结算的类型。每类把
  `reviewed_evidence_revision` 绑定到当前 `type_revisions`；证据指纹未变
  时复用原审核，不再次通读。

### H. Promotion

某一类型满足以下条件后，就独立同步该类型通过的结果到 `manifest.document.path → 素材卡片`，不等待其他类型：

- 证据已验证；
- 整体内容理解已复核，并先于钩子和结构显示；
- 候选与来源主线直接相关，或是不可替代的支撑；
- 类型标准通过；
- 观众路径说得清；
- 复用条件、边界和防照抄说明完整；
- 没有被更强候选重复覆盖。

`admitted` 是处理链路的终点，不是提取 Agent 的默认输出。

## 监督原则

- 审阅 Agent 不替提取 Agent 修正主线判断。
- 提取 Agent 不替审阅 Agent 自己宣布通过。
- 文案自然度问题不能掩盖证据、类型或结构问题。
- 反馈必须指向一个阶段和一个动作，不能只写“需要优化”。
- 同一来源可以某一类为 0；0 是扫描后的结果，不是失败。
- 金句、观点、钩子、结构分别拥有独立审核状态、审核人和审核时间。某一类处于 `review` 只阻塞该类；顶层 `review.status` 只是四类状态的汇总，不能作为所有卡片的统一显示门。
- Agent 只能重写 `origin: agent_generated` 卡片；`origin: user_injected` 卡片不进入 Agent 候选池，重跑时必须原样保留。
- 候选发现和准入审核分别写 `generation_receipt` 与
  `review_receipts.<type>`，不得使用同一次执行身份或同一份自我辩护上下文。
- 四类内容资产既可由 Agent 提取，也可由用户交互后标准化注入；图像和镜头只允许用户交互注入。

## 隔离执行协议

每个待结算类型必须经过两个一次性执行上下文：

1. `generation` 只读取复核包与定点证据，写
   `evidence/assets/staged-agent-candidates.json`。顶层 schema 为
   `agent-asset-candidates/v1`，并原样携带 `source_id`、`asset_type`、
   `type_revision`、调度器分配的 `generated_by`、`generated_at` 与
   `candidates`。候选不得包含 verdict、admission 或 reviewer identity。
2. 调度器校验并记录候选文件 SHA-256 后，启动新的 `review` 执行。
   它把候选当作待证伪的召回列表，按 locator 重开来源证据，只写
   `evidence/assets/staged-agent-submit.json`。`generation_receipt` 必须绑定
   原 `generated_by` 与 `candidate_sha256`，该类型的 `reviewed_by` 必须是
   本轮新的 execution identity。即使本轮只审核一个类型，
   `review.type_scan` 仍必须是只含一个对象的数组，且该对象的 `type`
   等于当前类型；不得写成以类型名为键的对象。
3. 两个 worker 都不能修改 `assets.json`、`manifest.json` 或读者 Markdown。
   generation 不能调用 check/apply/validate；review 只能调用执行清单给出的
   只读 `check-agent-assets`，并在退出前把 staged 修正到 preflight 通过，
   不能调用 apply/validate。review 退出后，调度器再次核对候选指纹与两个
   身份并重复权威 `check-agent-assets`，才执行一次
   `apply-agent-assets` 和一次增量校验。调度器必须在正式 apply 前持久化
   提交检查点；若进程在 apply 附近中断，只能先核对 canonical 回执：已
   落盘时只补跑校验，无法证明已落盘时停止并等待恢复，不能猜测性地
   再次 apply。

调度器下发给 review worker 的紧凑执行契约必须直接展开所有有限枚举，
尤其是 `quality.strength`、`quality.admission`、
`quality.evidence_confidence`、`quality.originality` 与 locator
`verification`；不得只写“使用 creator-assets/v1 枚举”并假设一次性 worker
会额外读取完整参考文档。确定性 `check-agent-assets` 的任何失败都属于
`preflight` staged-output 问题，按 checkpoint stage 进入一次有界提交修复，
不得依赖自然语言错误文案来判断是否可修。只有候选文件 SHA、类型 revision、
source identity 或 generation identity 真正变化时，才把临时提交判为 stale 并
退回生成阶段。

纯结构预检错误必须保留原候选指纹、generation identity 和 review
identity，优先对现有 staged 文件做白名单确定性修复；不得为修复数组、
locator 包装等无语义变化的问题重新启动生成或复核 Agent。

来源或类型 revision 变化、候选文件被修改、身份相同、回执未绑定候选
指纹时，丢弃该临时检查点并回到 generation；不得沿用旧审核上下文。

# Carrier-specific asset frameworks

Read `asset-decomposition-system.md` first, then use this reference after classifying the actual carrier. The storage model remains shared—quote, viewpoint, hook, structure—so assets stay searchable across sources. Candidate discovery, evidence, falsification, and applicability change by carrier. Detector output is only a boundary or evidence signal; it never replaces whole-source understanding or semantic review.

Select exactly one profile:

| Carrier | Profile | Default Agent types |
|---|---|---|
| video | `video_timeline` | quote, viewpoint, hook, structure |
| audio / podcast | `spoken_longform` | quote, viewpoint, hook, structure |
| image_post | `carousel_sequence` | quote, viewpoint, hook, structure |
| article / WeChat / document | `argument_article` | quote, viewpoint, hook, structure |
| music | `music_form` | quote, structure |

A platform does not select the profile. A Xiaohongshu note may be video or image-post; a WeChat page may be an article or an embedded video. Classify the evidence carrier first.

## Video — `video_timeline`

- **Understand from:** title/cover, source-authored description, reviewed speech, and visible action across the timeline. Do not summarize from transcript alone when demonstrations or on-screen comparisons change the meaning.
- **Quote:** verify the exact bounded speech span or source-authored text channel. Attribute the speaker when more than one person speaks.
- **Viewpoint:** require the spoken claim plus its demonstration, comparison, example, or later qualification when those carry the reasoning.
- **Hook:** treat title + cover + first visible/spoken beat as a multimodal opening contract. The payoff must be located later in speech, action, result, or visual proof; a strong cover without in-video delivery is not admitted.
- **Structure:** map changes in question, evidence, action, scene, stakes, or result. Do not manufacture chapters from equal time intervals.
- **Commercial:** distinguish a whole-video promotion from an embedded sponsored segment. Product presence or a review subject alone is not sponsorship evidence.

## Podcast / spoken audio — `spoken_longform`

- **Understand from:** episode title/show notes, reviewed transcript, speaker roster, and the full conversational arc. Long-form audio may establish its promise gradually; do not impose a video-first-seconds rule.
- **Quote:** require speaker attribution, exact words, and enough adjacent context to avoid turning a conversational response into a universal claim.
- **Viewpoint:** separate host framing, guest judgment, disagreement, and jointly developed conclusions. Do not merge different speakers into one synthetic viewpoint.
- **Hook:** the contract may live in title/show notes, a cold open, the host's framing question, or an early unresolved story. Locate the later answer or changed understanding.
- **Structure:** map question arcs, topic turns, disagreement, story/example, synthesis, and callbacks. Speaker changes alone are not structure; fixed-duration slicing is invalid.
- **Commercial:** mark host-read ads, inserted ads, affiliate plugs, and editorial discussion separately. An embedded ad segment does not automatically make the whole episode a hard ad.

## Image post / carousel — `carousel_sequence`

- **Understand from:** source-authored title/body/caption and every page in order. Inspect the images directly and reuse the single baseline OCR pass required by the image-post pipeline. Additional OCR retries, crops, alternate engines, and full-resolution passes are on demand only when the baseline is too sparse or exact decision-relevant page text needs verification.
- **Quote:** admit exact source-authored title/body/caption text directly when authorship is verified. Use page OCR only for a visually identified quote candidate or when source-authored text is insufficient.
- **Viewpoint:** reconstruct the judgment from page sequence and caption support; do not treat a visually prominent slogan as a supported viewpoint.
- **Hook:** cover and first page create the opening contract; later pages must deliver the promised list, transformation, reveal, route, result, or emotional turn.
- **Structure:** map page-turn jobs—orient, open a gap, reveal, compare, demonstrate, escalate, recap, or close. Every unit cites real page indices; repeated decorative pages may be grouped.
- **Commercial:** inspect brand recurrence, campaign language, product benefits, participation instructions, tags, links, and disclosure across both pages and caption.
- **Boundary:** images remain evidence. They become image assets only after explicit user interaction.

## Article / WeChat / document — `argument_article`

- **Understand from:** headline, lead, complete body, subheads, cited evidence, counterarguments, and conclusion. A headline is not the article's final claim.
- **Quote:** use exact authored body text with paragraph locators. Preserve qualification and attribution.
- **Viewpoint:** require a visible argument path: claim plus data, case, comparison, causal explanation, or counterargument. Separate the author's position from quoted sources.
- **Hook:** headline/lead may open a question, contradiction, scene, or news event. Admit only when the body materially resolves or reframes it.
- **Structure:** map the argument rather than forcing a story arc: question/problem → thesis/hypothesis → evidence/cases → complication/counterpoint → conclusion/implication. Preserve the article's actual order.
- **Commercial:** inspect disclosure, brand concentration, product links/QR codes, benefit language, campaign CTA, and whether criticism or alternatives remain. Media reporting about products is not automatically advertising.

## Music — `music_form`

- **Understand from:** available metadata, permitted audio, and rights-compatible exact lyrics or artist-authored text. Describe audible form, mood movement, instrumentation, performance facts, and the bounded lyrical ideas actually evidenced; do not invent lyrical meaning when lyrics are unavailable.
- **Applicability:** Agent extraction defaults to quote and structure. Viewpoint and content-hook gates are `not_applicable`; “hook” in music theory is not the same asset as an attention-contract hook. Missing lyric evidence settles quote as `no_asset` or `review`, not `not_applicable`.
- **Quote:** actively scan verified lyrics or artist-authored text for short, self-contained lines that compress an emotional recognition, image, tension, or turn of meaning. Preserve the exact bounded wording and context; do not reconstruct lyrics from listening or admit a line from an unverified lyrics page.
- **Structure:** locate audible section changes such as intro, verse, chorus, break, bridge, solo, drop, and outro only when supported by audio evidence. Use functional descriptions when formal labels are uncertain.
- **Rights:** accessibility alone does not establish authorship or permission. Keep automatic quote cards short and retrieval-oriented, cite the archived exact-text evidence, and never reproduce a substantial portion of the lyrics. A user-added interpretation remains a protected `user_injected` card.
- **Commercial:** ordinary release links, label metadata, or artist promotion are not enough to label the song itself a commercial cooperation.

## Commercial scope

When evidence supports it, add optional `commercial_assessment.scope`:

- `whole_source` — promotion is the main content job;
- `embedded_segment` — a bounded sponsor/ad/affiliate segment sits inside otherwise editorial content;
- `none` — no commercial evidence was found;
- `unclear` — scope cannot be established.

This field refines status; it does not replace `confirmed_commercial | suspected_commercial | no_commercial_evidence | uncertain`.

## Shared reader projection

Keep the compact reader surface across profiles so users can compare assets:

- 讲了什么;
- 核心判断;
- 商业属性;
- hook contract when applicable;
- ordered movement map;
- one reusable mechanism.

Carrier-specific evidence and reasoning stay in `assets.json` and the review packet.

# Chaptered source-content contract

Chapters are a navigable fidelity layer, not a summary or rewrite. Build them during extraction/standardization. The frontend only renders and seeks from them; it must not invent boundaries or titles.

## Video and audio

Write `evidence/transcript/逐字稿-章节版.json` and `.md` using `schema_version: chaptered-transcript/v3`.

Each chapter requires:

- stable `id`;
- concise semantic `title` describing the section topic, not `第 N 章` and not a blindly copied opening sentence;
- numeric `start` and `end`, with `start < end`;
- one or more `paragraphs` containing the complete continuous corrected transcript for that range;
- ordered `source_segment_ids` and `source_segment_count` linking granular ASR evidence;
- `title_status: reviewed` or `review`;
- `text_fidelity: verbatim_corrected`, `verbatim_raw`, or `review`.

Use `chapter_number` only when the source supplies a number or stable section label. Otherwise the frontend may display a sequential navigation number for video/audio, but it must not write that display number back as source evidence.

Each chapter must contain a short `summary` between its title and transcript paragraphs. The deterministic builder selects a source-grounded provisional sentence from the complete chapter and records `summary_status: provisional` plus `summary_basis: extractive_full_chapter_context`. A full-chapter Agent pass may replace it with a more useful overview and set `summary_status: reviewed` plus `summary_basis: full_chapter_review`. The summary is navigation/analysis and never substitutes for `paragraphs`.

Every chapter timestamp is a media-seek control. Render it with a visible play icon plus the time label so the action is understandable without hover or prior knowledge.

Chapters must cover every reader-eligible non-empty source transcript segment exactly once and in order. A QA-declared `reader_exclusions` entry may omit a local catastrophic decoder artifact or strictly recognized music/watermark hallucination from prose only when the raw segment remains untouched and the chapter document preserves its source index, timestamps, text, and reason in the exclusion audit list. Boundaries may follow topic transitions, argument moves, scene changes, or musical sections—not an arbitrary fixed duration. Keep sentence/word timestamps in raw evidence for audit and fine seeking. Clicking a chapter seeks to `start`; the reader displays the whole chapter paragraph block.

Before composing chapter prose, the builder may apply confirmed exact-match rules from `evidence/transcript/transcript-review.json` to an in-memory copy of the ASR segments. It must report `lexical_correction_count`, preserve every source segment ID and timestamp, and never modify `transcript-raw.json`. A correction changes wording fidelity only; it must not create, delete, merge, or reorder source segments. Pending or in-progress lexical review keeps the transcript non-quotation-ready even though confirmed corrections are visible in the reader.

Inside a chapter, group speech into readable `paragraph_blocks` using sentence completion, semantic/discourse turns, meaningful pauses, and verified speaker changes. Each block retains start/end and source segment IDs. Speaker IDs are optional: assign `A/B/C...` only from real diarization or upstream speaker metadata, never from textual guesswork. Keep the speaker ID stable while names remain user-editable mappings.

Carrier policy narrows this further: ordinary `video` never exposes speaker IDs, even when the ASR payload contains them. Its `paragraph_blocks` use a compact video profile with natural sentence punctuation, semantic turns, and pauses so the whole transcript cannot collapse into one wall of text. Podcast/audio may use verified speaker changes in addition to those same paragraph signals.

Use this boundary routing order:

1. source-authored structured chapters or publisher timeline;
2. parseable shownote timestamps and section labels;
3. only when neither exists, ASR inference from pauses, explicit topic changes, local lexical change, and discourse transitions.

Preserve a clear source label verbatim as the canonical navigation title and initially mark it `title_status: review`. The full-chapter Agent pass verifies that the author title matches the real chapter and writes the chapter summary; after verification the title may become `reviewed`, but `title_source` remains the publisher/shownotes source and the wording must not be replaced. If the first source timestamp begins materially after the recording starts, retain the preceding transcript as a separate chapter and generate a semantic title and summary for that uncovered range. Duration may prevent pathological tiny or enormous fallback chapters, but it must not be the primary boundary signal. Record document-level `boundary_source: source_timeline` or `asr_semantic`, and keep the source timestamp on each timeline-backed chapter.

Shownote navigation is adaptive rather than uniformly nested. A plain timestamp list remains a single-level chapter list. When the source clearly provides a parent topic followed by at least two timed child items, preserve `group_title`, optional `group_number`, `group_start`, `group_evidence`, and `outline_level: 2` on those child chapters so the frontend can render a clickable two-level outline. Mixed single-level and grouped runs are valid. Never invent a parent topic for visual consistency. Follow `shownote-timeline-adaptation.md` for section classification, supported evidence forms, conservative fallback, and the required regression matrix.

Summary depth must exactly follow the source-authored outline depth. A single-level timeline has one reviewed summary per timed chapter. A two-level run additionally has one reviewed parent-group summary covering the complete ordered child range, while every child retains its own reviewed summary. Mixed outlines add parent summaries only to the runs that actually have parent groups. Store parent records in document-level `groups` with ordered `chapter_ids`, a coverage fingerprint, `summary`, `summary_status`, and `summary_basis: full_group_review`; never synthesize a parent merely to make the layout uniform.

Classify the shownote section before interpreting its timestamps. Entries under `时间轴`/`本期你会听到` may define transcript boundaries. Timestamped entries under `猜你想看`, references, links, definitions, or highlights remain separately typed `timestamp_notes` with playback locators; they may appear in the full shownotes view but must not split the transcript.

A deterministic builder may create `status: provisional` and `title_status: review`, but `full` completion requires an Agent to review all titles and boundaries in complete context and set document `status: reviewed`. Do not mark a copied first sentence as reviewed merely because it is grammatical.

## Article, blog, and text document

Write `evidence/article/正文-章节版.json` and `.md` using `schema_version: chaptered-article/v1`.

First preserve the clean body as ordered, immutable paragraph records (`P0001`, `P0002`…) with heading paths. Then group only adjacent paragraphs. Each chapter requires:

- stable `id` and semantic `title`;
- integer `paragraph_start` and `paragraph_end` referring to the preserved body;
- ordered `source_paragraph_ids`;
- `paragraphs` containing the complete continuous original/correction-only text in that range;
- `title_status` and `text_fidelity` with the same semantics as transcript chapters.

Also record:

- `kind: chapter`, `preface`, or `unsectioned`;
- `chapter_number` when the source explicitly provides one; otherwise `null`.

Use canonical Markdown headings such as `### 01｜重组飞书，拼出业务链`. Keep leading prose as `kind: preface` without inventing a number. If the source has no reliable section boundary, retain one `kind: unsectioned` body instead of manufacturing chapter titles.

Every body paragraph must appear exactly once, in source order. Preserve inline image markers and captions in their original paragraph positions. A chapter can use a clean source heading when it accurately names the section. For articles, do not invent boundaries merely to create navigation. Do not replace article paragraphs with an executive summary, rewritten prose, or bullet abstraction.

Some publishers render a section number as a tiny image followed by a text title. Treat that image as structure, not a content illustration: record the omission, preserve the recovered number/title pair, promote the title to the canonical Markdown heading, and exclude the tiny image from the reader gallery. If OCR cannot recover the printed number but several verified marker images occur in source order, a sequential inferred number is allowed only with `structure_preserved: true` in omission evidence.

## Titles and boundaries

Good titles state the section's role or topic: `为什么企业 AI 项目停在试点`, `从脑暴到 Skill 的工作流`, or `副歌如何释放前段张力`. Bad titles are `第一部分`, `继续讨论`, a truncated first sentence, or an analyst conclusion absent from the source.

Source timelines provide both boundary evidence and author-written navigation titles. Preserve a usable source title verbatim; full-chapter review checks its factual fit and supplies the chapter introduction instead of creating a duplicate Agent title. Only chapters without a source title require an independently written semantic title. Copied opening sentences remain invalid for those Agent-generated titles.

Prefer 3–12 Chinese words. Keep titles unique within a source. Merge tiny fragments that do not form an independent semantic unit; split long chapters when the claim, scene, or argumentative job changes. Record material uncertainty rather than forcing a confident label.

## Validation and completion

The source-content validator rejects missing/duplicate/out-of-order source IDs, empty chapter prose, invalid ranges, generic titles, and chapter records that do not declare fidelity. In strict mode it also rejects `review` title/text status. Asset locators may cite only chapters that pass at least handoff validation; admitted quote assets require reviewed fidelity.

# Top-comment evidence and topic signals

When an eligible public payload does not contain comments, stop with `confirmed_no_public_source`; do not escalate to a login/browser retry. Account-bearing and dedicated rendered-browser routes are retired. Douyin may reuse only the comment nodes already present in the disposable anonymous detail snapshot required for source capture.

Comment enrichment is adaptive. For Douyin and Bilibili, apply it only to `analyze`, `cards`, and `full` sources with at least 20 visible total comments; the threshold is configurable through `CREATOR_INGEST_COMMENT_MIN_TOTAL`. A Douyin page whose total is unavailable may qualify when the already-open anonymous detail snapshot contains a visible sample. Set `skipped_by_policy` for `capture`, `standard`, and below-threshold sources, and settle the gate without a comment request. Comments are a separate user-generated evidence layer; they are not part of the creator's source text and do not automatically establish facts.

## Acquisition target

Capture up to ten highest-liked **top-level** comments available through a no-login public context. Never import an existing logged-in browser session as a comment fallback. Retain only comments with a verified like count of at least 10 and enough complete information to stand on their own; reject emoji-only reactions, generic cheering, and fragmentary remarks.

Preferred order:

1. Bilibili hot page 1 only, capped at 20 returned items;
2. Douyin top-level comments already rendered in the disposable detail snapshot, without scroll or pagination;
3. another platform response explicitly bounded and ranked as hot/top.

Never imply global completeness when only a visible or paginated sample was captured.

## Required fields

Save `evidence/comments/top-comments.json`:

```json
{
  "status": "passed",
  "ranking_scope": "platform_hot_page_1",
  "sample_size": 20,
  "has_more": true,
  "acquired_at": "2026-08-03T12:00:00+08:00",
  "method": "anonymous_public_page_dom",
  "comments": [
    {
      "rank": 1,
      "comment_id": "...",
      "author_display": "...",
      "text": "...",
      "like_count": 1524,
      "like_count_raw": "1524",
      "reply_count": 12,
      "created_at": null,
      "locator": "comment:..."
    }
  ]
}
```

Allowed `status`: `passed`, `partial`, `unavailable`, `review`, `not_applicable`.

Do not archive private profile data, cookies, tokens, signed requests, or hidden user identifiers. Public display name and platform comment ID are sufficient.

## Ranking and parsing

- Normalize `万`/`w` counts conservatively and retain the visible raw value.
- Exclude replies when ranking the requested top-level three; record reply count separately.
- Deduplicate by comment ID, then normalized text when IDs are unavailable.
- Exclude UI labels, creator caption, recommended-search text, and empty/emoji-only items unless the emoji response is itself analytically important.
- If fewer than three comments are accessible, preserve what exists and state why.

## Topic-signal judgment

After preserving the comments, decide whether they add a meaningful creator topic signal. Use only evidence from captured comments and cite comment IDs/ranks.

Possible signals:

- `explicit_question`: audiences directly ask for an explanation, tutorial, comparison, source, or follow-up;
- `unmet_need`: a pain point or missing capability appears;
- `disagreement`: a recurring counterargument, trust concern, or polarizing premise;
- `missing_context`: audiences need prerequisites, boundary conditions, price, process, or proof;
- `use_case`: commenters supply a concrete scenario the original content omitted;
- `experience_evidence`: commenters add personal cases worth researching, not automatically treating as fact;
- `language_resonance`: a phrase or framing clearly triggers recognition and can inform title/angle design;
- `extension`: a related subtopic or sequel direction appears.

For each accepted signal record:

- signal type and concise judgment;
- supporting comment locators;
- why it matters for selection or creative framing;
- possible content angle;
- confidence and limitation;
- whether fact-checking is required.

## No-signal rule

It is valid and often correct to output:

```yaml
topic_signal_status: no_meaningful_signal
topic_signals: []
reason: "保留评论主要是情绪表达和社交互动，没有稳定问题、需求或争议。"
```

Do not convert generic praise, jokes, tagging friends, or isolated anecdotes into a forced topic conclusion.

## Failure handling

- Xiaohongshu public initial state has no comments: `unavailable`, reason `not_in_public_payload`, manifest marker `confirmed_no_public_source`; this is a terminal, non-blocking result.
- A configured non-Xiaohongshu browser route requires authorization but lacks it: `review`, next action specifies browser context.
- Platform shows a ranking but like counts are hidden: preserve order, set `ranking_scope` and `review`.
- Comments disabled or truly zero: `passed` with an empty list and `topic_signal_status: no_meaningful_signal`.

# Evidence-backed content deconstruction

Use this reference when reviewing whole-source understanding, hooks, and structures. It distills useful operational patterns from public content-analysis projects without treating their author-defined labels or timing heuristics as universal laws.

## What the external projects contribute

- [BrowserAct YouTube transcript analysis](https://github.com/browser-act/skills/blob/main/solutions/video-platforms/youtube-transcript-analysis-api-skill/SKILL.md) separates structure/hook, core message, evidence/credibility, and business/CTA. Adopt the separation: understand the claim before judging its presentation.
- [Storytelling Skills](https://github.com/yaxeen/storytelling-skills) treats a hook as a curiosity/tension contract that must be paid off, and long-form structure as a chain of opened and closed loops. Adopt promise-to-payoff tracing; treat exact second counts and named psychology levers as heuristics, not admission rules.
- [Viral Reverse-Engineering](https://github.com/social-media-skills/skills/tree/main/skills/viral-reverse-engineering) distinguishes surface, retention structure, emotional/share driver, and distribution confounds. Its counterfactual test—remove a feature and ask whether the piece still works—helps isolate one or two load-bearing mechanisms. A single successful example remains a hypothesis, not proof of a universal formula.

## The compact framework

Analyze in this order:

1. **Understand** — What concretely happens or is argued? What single judgment or recognition remains at the end? Is there commercial intent or integration?
2. **Hook contract** — What does the opening make the intended audience want resolved? State the opening move, the gap/tension, the promise, and the later payoff.
3. **Movement map** — Reduce the source to 3–6 ordered beats. Each beat must change at least one thing: the audience question, evidence, stakes, direction, or resolution. If removing a beat does not change the movement, merge or drop it.
4. **Load-bearing lesson** — Name at most two mechanisms the piece would materially weaken without. State the reusable move, not the topic or wording. Keep transfer limits and counterevidence in the audit record.

Do not force a named template such as PAS, Hero's Journey, or three-act structure when the source's actual order explains it better. After reconstructing the evidence-backed hook contract or movement, assign one controlled retrieval type and pair it with a source-specific name. The type answers “这是什么机制”; the second half answers “这篇内容具体怎样使用它”.

## Reader projection

Keep the detailed reasoning, evidence locators, boundary, anti-copy note, and review verdict in JSON. The reader-facing structure decomposition is intentionally compact:

```markdown
> [!abstract] 整体内容理解
> **标签：** `视频` `产品评测` `实测` `论证` `获得判断` `AI 助手`
>
> **讲了什么：** …
>
> **核心判断：** …
>
> **商业属性：** …

> [!tip] 钩子｜反常识纠偏｜亲测推翻流行说法
> **开场：** …
>
> **为什么想继续看：** …
>
> **承诺 → 兑现：** … → …
>
> **可复用：** …

> [!example] 结构｜递进论证型｜从能力实测推到退出成本
> **内容路线：** 〔1 …〕 → 〔2 …〕 → 〔3 …〕
>
> **可复用：** …
```

The flow line is the required lightweight diagram. Prefer concrete verbs such as “质疑、实测、转向、升级、兑现” over abstract chapter labels such as “背景、分析、总结”.

# Evolving content taxonomy and decomposition

Use this reference for `analyze`, `cards`, and `full` jobs. It defines how to classify the content mechanism and how to choose an analysis that fits the source. A platform or carrier is not a content track.

## Three-axis classification

Every package records:

- `carrier`: `video`, `image_post`, `article`, `audio`, or `document`; this controls evidence acquisition.
- `primary_track`: one short primary keyword for the main creative or reading problem.
- `secondary_tracks`: zero to three controlled child keywords belonging to the selected primary track.
- `content_mechanism`: the way the source creates value; this selects the decomposition template.
- `tags`: richer cross-cutting keywords for tools, objects, topics, techniques, places, people, works, visual language, or narrative moves.
- `navigation_tags`: one to four reviewed, reusable labels shown in the favorites browser. Use the primary track plus at most three stable child concepts; never copy the platform hashtag list into this field.

Keep source metadata and personal organization separate:

- `source.tags` preserves platform-provided hashtags unchanged as evidence and search metadata.
- `classification.tags` may retain richer analysis keywords.
- `classification.navigation_tags` is the small user-facing filter vocabulary. Remove campaign slogans, platform growth tags, generic calls such as “好视频扶持计划”, entire titles, author handles, one-off promotional phrases, and near-duplicates. Prefer an existing controlled label over inventing a synonym.
- A source may have no navigation tag when classification remains `待归类`; do not fill a quota.

Collection placement is a separate decision. On an uninitialized library, the favorites service may inspect the tracks actually present and create only a small set of broad, editable collection suggestions; it must not materialize the whole taxonomy as a folder tree. After initialization, read the shared library's `library.json` and choose only among its configured `collections` (both auto-suggested and user-created). Treat each collection's `keywords` as matching clues, not as permission to proliferate folders. If no existing collection fits with clear evidence, assign `待整理`.

`primary_track` is one of `AI工具与教程`, `AI产品与趋势`, `创作者经营与方法论`, `商业品牌与职业`, `产品与消费决策`, `项目展示与创作过程`, `文化与叙事`, `审美策略与灵感`, `生活成长与思考`, `社会人文观察`, or `待归类`. Each name must remain within nine Chinese characters and be stable enough for navigation.

This unified taxonomy was calibrated first against a legacy set of Xiaohongshu and Douyin reference videos, then extended with image-post samples. Every track applies to every carrier; video, image post, and article only change the acquisition and evidence workflow. It is a working model, not a fixed platform hierarchy:

- `AI工具与教程`: `工具推荐`, `入门教程`, `工作流`, `Skill`, `Agent`, `Vibe Coding`.
- `AI产品与趋势`: `AI产品`, `组织应用`, `行业趋势`, `市场变化`, `社会影响`, `技术判断`.
- `创作者经营与方法论`: `账号定位`, `内容策略`, `表达技巧`, `制作方法`, `创作效率`, `职业成长`.
- `商业品牌与职业`: `品牌营销`, `商业机会`, `定价`, `商业模式`, `职业发展`, `求职`.
- `产品与消费决策`: `产品评测`, `消费选择`, `需求洞察`, `产品体验`, `消费趋势`, `服务模式`.
- `项目展示与创作过程`: `AI项目`, `设计项目`, `作品展示`, `制作过程`, `复盘`, `开源`.
- `文化与叙事`: `城市叙事`, `人物故事`, `作品解读`, `文化观察`, `个人表达`, `旅行`.
- `审美策略与灵感`: `摄影`, `排版`, `视觉设计`, `策展`, `影像`, `灵感素材`.
- `生活成长与思考`: `日常生活`, `情绪管理`, `习惯方法`, `关系体验`, `个人成长`, `哲思`.
- `社会人文观察`: `社会观察`, `跨文化`, `性别`, `历史`, `公共议题`, `社会学`.

Secondary tracks may contain up to three values, must come from the selected primary track's list, and must never repeat the primary track. Use tags for all cross-track concepts. Do not invent a new primary track for a single unusual item. Add a proposed track only when at least three items need a different evidence or analysis contract; document its definition, boundary, counterexample, template, and three verified examples. Preserve older labels as aliases when tracks are renamed or merged.

For example, an AI-made music map is `primary_track: 项目展示与创作过程`, `secondary_tracks: [AI项目]`, `content_mechanism: 项目展示`, with tags such as `Vibe Coding`, `音乐`, `地图`, `策展`, `交互`, and `网页应用`. It is not assigned a cross-parent secondary track.

## Standard content versus processed content

Standard content is faithful evidence: source metadata, source body, original
images/media, corrected transcript, on-demand image OCR when needed, ordering,
locators, and declared uncertainty. It never includes unlabelled inferences.

Processed content is analysis: probable reason the user saved it, content map, source-specific mechanisms, strong reusable candidates, transfer conditions, and limitations. It never presents a paraphrase as source wording.

## Shared processed-content blocks

Every processed record contains:

1. `Why this may have been saved`: separate explicit user reason from evidence-backed inference.
2. `Content map`: follows the real source sequence and cites pages, paragraphs, or time ranges.
3. `Reusable mechanisms`: only genuine candidates with locators, reuse conditions, and boundaries.
4. `Uncertainty and limits`: missing sources, unverified claims, weak OCR/ASR, and non-transferable context.

## How to derive a decomposition module

Do not select a module from title keywords or platform. After standard content exists, classify four mechanisms:

1. `value`: what the viewer gains: an operational skill, a decision, an argument, a project process, a narrative experience, or an aesthetic rule.
2. `evidence`: what must be retained to support conclusions: steps/screens, verified speech, use experience, cases/data, image order, project process, or article locators.
3. `expression`: how the source moves: tutorial, argument, comparison, process, narrative, or curation.
4. `asset`: what can be reused: workflow, decision criteria, reasoning structure, process rule, narrative rhythm, visual rule, or evidence case.

The first two determine the track and quality gates; the second two determine `content_mechanism` and card candidates. The common processed-record blocks are deliberately fixed, but the mechanism analysis is not.

Each mechanism must specify `purpose`, `required_evidence`, `analysis_questions`, `standard_rendering`, `candidate_assets`, `transfer_boundary`, and `acceptance`.

When a source needs evidence, questions, or assets that no current module covers, set `content_mechanism: 待试运行`, create that seven-field contract for the first source, and verify it against a second independent source. Add a new primary track only after three sources still cannot be accurately represented by the existing tracks.

## Mechanism templates

### Tutorial, tool, skill, or workflow

Analyze the user problem, promised outcome, inputs, steps, tools, outputs, proof/demo, and operating boundaries. Candidate cards: hook/promise, workflow, step structure, interface/diagram layout.

For video, also inspect the relationship of voiceover, screen capture, demo, caption, and outcome. Preserve original video and granular machine timestamps as evidence, but present the reviewed transcript as semantic paragraphs with bounded time ranges.

### Business, product, or opportunity argument

Separate observed facts, author interpretation, causal reasoning, prediction, evidence, counterexample, and hidden condition. Candidate cards: argument, evidence case, reasoning structure. Never promote an unsupported forecast as a fact.

Apply this to `AI产品与趋势`, `商业品牌与职业`, and `产品与消费决策`. Do not add monetization advice unless the source itself provides evidence for a commercial mechanism.

### Creation method or knowledge management

Analyze the real creative block, minimum action sequence, decision points, success conditions, failure modes, and how the source makes the method learnable. Candidate cards: method, structure, visual layout, title promise.

For `创作者经营与方法论`, also analyze title promise, opening hook, information release, edit rhythm, shot/caption division, and CTA only when each is actually present.

### Aesthetic, cultural, or curation narrative

Analyze observation, point of view, title-image-body division, image sequence, rhythm, composition, color, typography, whitespace, density, and ending. Candidate cards: visual layout, narrative rhythm, curation rule. "Beautiful" without a describable mechanism is not a card.

Apply this to `审美策略与灵感`, `项目展示与创作过程`, and `文化与叙事` while selecting the relevant dimensions: materials, interaction, and making process for objects; sensory detail and place observation for narrative content; selection and ordering rules for curation.

### Personal essay or lived experience

Analyze concrete situation, tension, change, generalization, turning point, ending, and identity-specific constraints. Candidate cards: narrative structure and viewpoint. Quote cards require verbatim verification.

Apply this to `文化与叙事` and `生活成长与思考` when the source is a personal essay. Do not turn a personal conclusion into universal advice without preserving its original scope.

### Social or humanities observation

Analyze question framing, definition, evidence, explanatory frame, factual claim versus value judgment, missing context, and counterargument. Candidate cards: question frame, viewpoint, evidence case.

Apply this to `社会人文观察`. Do not collapse a source-backed social claim into a personal-life viewpoint.

### Practical life guidance

Analyze situation, target, constraints, actions, decision points, risks, and the list/map/routing structure that makes it saveable. Candidate cards: checklist structure, service experience, information design.

## Image-post requirements

For an image-led post, the processed record also evaluates the cover promise,
original image sequence, page roles, text-image division, readability, visual
system, and payoff. Every visual claim cites image numbers. Source body/caption
and any triggered image OCR remain separate in standard content.

## Card promotion

Emit candidates through `source-assets-contract.md`. Each candidate requires source evidence, a carrier-appropriate locator, transfer condition, boundary, anti-copy note, uncertainty and counterevidence assessment. Quote assets require reviewed verbatim text; viewpoint assets require scope and counterpoint; hooks require located payoff. Do not create or admit assets merely to satisfy a count.

# Xiaohongshu/Douyin cover analysis

Apply to every Xiaohongshu or Douyin video/image post in `analyze`, `cards`, and `full` modes. This controlled vocabulary was initialized from the user's existing library of 86 Xiaohongshu video covers and 7 Douyin video covers, then generalized for future samples.

The cover is a distinct creative asset. Capture the original cover, analyze its promise and visual mechanism, and compare it with the actual content payoff. Video covers use bounded OCR when text is present. For image posts, use verified source-authored title/body plus direct cover inspection first; OCR only when that evidence cannot recover the cover promise or exact decision-relevant cover text. Do not infer cover effectiveness from aesthetics alone.

## Evidence

Save:

- original cover as `evidence/cover/cover.<ext>` or identify `evidence/images/image-001.*` as the cover;
- dimensions, MIME, SHA-256, acquisition method;
- an OCR decision; when OCR is triggered, raw text, corrected text, and visual-review status;
- `封面分析.md` with controlled tags and evidence-based judgment.

## Controlled tag dimensions

Choose one primary value per dimension and at most two secondary values. Use `other` with a note rather than inventing near-duplicate tags.

### `promise_mechanism`

- `how_to`: promises a method, workflow, or teachable procedure;
- `result_reveal`: foregrounds a finished artifact, transformation, or outcome;
- `problem_solution`: names a pain and implies relief;
- `counterintuitive_claim`: overturns a familiar assumption;
- `question_gap`: uses a question whose answer is withheld;
- `numbered_value`: uses a number/list/time/cost to bound value;
- `identity_relevance`: calls out a role, group, life stage, or “people like me”;
- `experience_confession`: personal lesson, review, mistake, or lived conclusion;
- `comparison_choice`: A/B, before/after, worth-it, versus, or purchase decision;
- `novel_artifact`: an unusual product/project/map/site/object is itself the hook;
- `emotional_scene`: mood, relationship, memory, aspiration, or narrative moment;
- `spectacle_curiosity`: visually strange, dramatic, cute, or impressive event drives the click.

### `visual_archetype`

- `talking_head_big_text`
- `talking_head_subtitle_frame`
- `person_plus_product`
- `person_plus_result_overlay`
- `product_hero`
- `artifact_or_demo_hero`
- `interface_or_screenshot`
- `before_after_split`
- `collage_evidence`
- `poster_typography`
- `diagram_map_infographic`
- `documentary_scene`
- `cinematic_still`
- `lifestyle_scene`
- `tutorial_action`

### `text_structure`

- `no_added_text`
- `single_keyword`
- `short_proposition`
- `question`
- `number_plus_benefit`
- `problem_plus_solution`
- `main_title_plus_subtitle`
- `multi_label_annotation`
- `quote_or_confession`
- `series_episode_label`

### `subject_role`

- `creator_identity`: face/person builds recognition or trust;
- `demonstrator`: person shows an action or points to proof;
- `scale_reference`: person makes product/space/result legible;
- `emotion_carrier`: expression/body conveys emotion;
- `object_is_hero`;
- `space_is_hero`;
- `screen_or_document_is_hero`;
- `no_dominant_subject`.

### `attention_device`

Allowed values: `oversized_type`, `high_contrast_block`, `bright_accent`, `outlined_cutout`, `arrow_or_circle`, `hand_gesture`, `facial_expression`, `scale_surprise`, `split_screen`, `dense_collage`, `negative_space`, `cinematic_light`, `unusual_object`, `UI_proof`, `before_after_boundary`.

### `information_density`

- `low`: one promise/subject, readable in under one second;
- `medium`: main promise plus one proof/subtitle/secondary subject;
- `high`: multiple screenshots, labels, steps, or competing focal points.

### `trust_device`

Allowed values: `face_identity`, `real_usage_scene`, `finished_result`, `before_after`, `screen_or_data_proof`, `credential_or_experience`, `specific_number`, `product_detail`, `social_proof`, `none_visible`.

### `tone`

Allowed values: `expert`, `practical`, `intimate`, `aspirational`, `playful`, `urgent`, `provocative`, `calm`, `cinematic`, `documentary`.

## Required judgments

### One-second comprehension

- What can a viewer understand in one second?
- Is the title legible at feed thumbnail size?
- Is there one dominant focal point?
- Does text compete with the face, product, or proof?

### Promise construction

- What exact click promise is made?
- What audience or situation is implied?
- Which word/image creates curiosity, specificity, or trust?
- Is the promise topic-led, result-led, identity-led, or emotion-led?

### Image-text division of labor

- Does the image prove the text, contextualize it, create mood, or merely decorate it?
- Is important information duplicated or complementary?
- Would the cover still work if text or image were removed?

### Promise/payoff alignment

Compare the cover with transcript/OCR/body:

- `fulfilled`: content directly delivers the promised answer/result;
- `partially_fulfilled`: useful content but narrower/weaker than cover;
- `mismatched`: cover promise is not materially delivered;
- `not_yet_verified`: downstream evidence incomplete.

### Transfer judgment

Record the reusable mechanism, minimum assets needed, production cost, suitable content types, and what would become imitation if copied literally.

## Cover strength

Rate separately:

- `click_clarity`
- `visual_hierarchy`
- `mobile_legibility`
- `specificity`
- `trust`
- `distinctiveness`
- `promise_payoff_alignment`
- `fit_for_user_creation`

Use `strong/useful/ordinary/weak/review`. A visually polished cover may still be weak if generic or misleading.

## Cover-card admission

Create a `封面卡` only when at least one mechanism is strong and transferable.
Include source, cover path, verified cover text when available, controlled tags,
mechanism, recreation recipe, suitable topics, misuse boundary, and payoff
result. Ordinary covers remain tagged in `封面分析.md` without a card.

# Versioned gate ledger and incremental resume

`manifest.gates` remains the compact compatibility summary. `manifest.gate_ledger` is the incremental-execution record for those same nine gates; it does not introduce an audit gate or duplicate evidence.

Each entry records `status`, `summary_status`, `rule_version`, `input_fingerprint`, `output_fingerprint`, dependencies, executor class, context policy, and validation/invalidation times. Evidence and judgments remain in their canonical artifacts. In particular, the two-pass asset audit remains inside `evidence/assets/assets.json` and G5/cards.

On repeated input:

- reuse a passed/not-applicable gate when its rule version, dependency fingerprints, and output fingerprint still match;
- run a `stale`, `missing`, or `failed` gate;
- do not repeat an unchanged `review` gate automatically—it is waiting for its declared reviewer/capability;
- when a rule version or dependency output changes, invalidate only that gate and its transitive dependants;
- never invalidate capture, OCR, or ASR merely because an unrelated analysis/card rule changed.

Comments are audience enrichment and do not invalidate source-derived asset
types by default. Cover changes invalidate hook/commercial evidence through the
asset type revisions, not quote or viewpoint. The aggregate analysis/cards
ledger remains UI/resume state; `assets.json.type_revisions` is the finer-grained
semantic invalidation authority.

Change one `GATE_SPECS[gate].rule_version` in `scripts/gate_ledger.py` only when that gate's acceptance rule or deterministic implementation materially changes. Do not bump all versions for a general Skill wording edit.

Token policy is stored per gate. Deterministic/local-compute gates use no LLM. Semantic gates read standard content or compact review packets first and open raw evidence only by locator. A changed semantic rule reuses unchanged evidence.

Use:

```bash
python3 scripts/gate_ledger.py plan --package <package>
python3 scripts/gate_ledger.py sync --package <package>
```

The save wrapper invokes this automatically. These commands are for diagnosis, migration, and cross-Agent handoff.

# Image-led post pipeline

Use for Xiaohongshu carousel notes, poster sets, comics, screenshot essays, infographics, and image-plus-caption posts. Images are first-class source content. Never reduce the post to its caption.

## Stage I0 — Resolve and inventory

1. Preserve the original share text and shared URL.
2. Record canonical URL, platform, stable note ID, title, author, publication time and body/caption if visible.
3. Determine expected image count and order from the source payload/page.
4. Classify the dominant image function:
   - photography/aesthetic reference;
   - text poster or screenshot essay;
   - tutorial/steps;
   - product or comparison;
   - infographic/diagram;
   - mixed carousel.
5. Read `content-taxonomy.md` and record one `primary_track`, up to three `secondary_tracks`, and one `content_mechanism`. Image function controls evidence review; the unified track and mechanism control analysis.
6. Check duplicate/revision identity.

## Stage I1 — Acquire every original image

Download the highest available original or near-original image, not thumbnails or browser screenshots unless no alternative exists. For each image record:

- ordered index;
- source URL only if it is stable and non-secret-bearing;
- local filename, MIME/extension, pixel dimensions, byte size, SHA-256;
- whether it is original, resized, screenshot fallback, or unavailable.

Verify acquired count against expected count. Do not call acquisition complete if images are missing or reordered. Preserve animated/live formats when available and also create an inspectable frame if needed.

For a Xiaohongshu item with `livePhoto: true`, or a Douyin image whose public
embedded detail carries its own `video`, preserve one compound object at the source
position: the normal static image plus the highest verified stream from that same
image item. Never use the carousel's background-music player as image motion. Assign
a stable `pair_id`; store the MP4 motion file under `evidence/media/`; and record the
pair in `live_photo_inventory`. A still image without its declared motion component
fails the acquisition gate. Download motion automatically in every mode that archives
the image post.

Keep completeness and fidelity separate. A complete pair can still contain platform
derivatives. When declared dimensions exceed the downloaded JPEG or only an `HD`
H.264 stream is exposed, mark fidelity `review` and describe it as the highest verified
platform version rather than an upload-original or lossless asset.

## Stage I2 — Scan every image once, then organize

Run one baseline OCR pass on every acquired source image. This is mandatory even
when the platform title/body is informative, direct inspection appears sufficient,
or a page contains only a few visible words. Title/body and visual inspection are
parallel evidence channels; neither may suppress the per-image OCR pass.

For every page, preserve a controlled text classification:

- `substantial_text`（大段文字）— coherent readable text is present;
- `sparse_text`（零星文字）— only short labels, isolated phrases, watermarks, or fragments are present;
- `no_text`（无文字）— no reliable visible text was recognized.

The baseline pass is exactly once per source image. A second method, crop, or
targeted retry is allowed only for low-confidence decision-relevant text. It must
reuse the baseline artifact and must not rescan the entire carousel.

For an existing package, run the shared `refresh-ocr` wrapper without limiting
coverage. Named pages may be passed as priority hints, but all source pages keep
their original `source_index` and are scanned; never renumber page 4 as page 1.

For each image:

1. Inspect orientation and resolution; deskew/upscale/contrast only on a derivative.
2. Detect text regions and reading order.
3. Save raw OCR with region/bounding-box or line order when supported.
4. Visually compare the result with the original.
5. Save corrected OCR separately. Correct only visible text.
6. Record illegible, covered, stylized, tiny, or uncertain spans in machine evidence and review notes; never guess them.
7. Identify the page role: cover, setup, claim, evidence, step, example, transition, payoff, CTA.

Run a second OCR method or crop-level retry for core-claim text when the first pass is weak. Never infer a missing phrase from the caption and present it as image text.

## Stage I3 — Standard reconstruction

`标准内容.md` keeps these channels separate and presents them for reading rather than debugging:

- post metadata and body/caption;
- image inventory;
- one ordered `逐图内容` section containing `封面`, `图 2`, `图 3`… groups;
- in every group, the archived original image first, then corrected visible text
  from the mandatory OCR pass when present, plus any useful human-facing image note.

Do not render OCR engine status, review enums, mean confidence, model names, thresholds, bounding-box dumps, `_no_text_`, or pipeline instructions in `标准内容.md`, the canonical `manifest.document.path` Markdown, or the frontend. Keep them in `evidence/ocr/ocr-raw.json`, manifest gates, and dedicated review artifacts. If an image has no text, show the image without a synthetic no-text paragraph. Use `封面文字` rather than `封面 OCR（raw…）` for a video cover's readable text.

Then provide a clearly labelled reading-order reconstruction only if useful. It is derived content, not raw source.

## Stage I4 — Analyze the carousel

### Meaning and viewpoint

- What claim/information exists only in images?
- What does the caption add, repeat, or contradict?
- Which viewpoint, quote, example, or evidence is strong enough to reuse?
- What is opinion, what is observation, and what would require fact checking?

### Cover and sequence

- Cover promise, title-image relationship, curiosity gap, audience cue.
- Page-by-page information release and reasons to swipe.
- Density curve, repetition, transitions, climax/payoff, final CTA.

### Visual system

- Image type, crop, focal point, color, light, texture, negative space.
- Typography hierarchy, font role, size/contrast, annotation style.
- Text-image division of labor and mobile readability.
- Reusable layout rule versus topic-specific decoration.
- Aesthetic references worth saving independently.

### Transferability

- What could transfer to the user's topic without copying the surface?
- What asset would be needed: photo style, template, layout, diagram grammar, page sequence?
- Minimum viable recreation and likely production cost.

## Stage I5 — Four automatic content assets

Promote only strong candidates:

- viewpoint/quote card from verified source-authored text, or with an exact page
  OCR locator when the candidate was first identified visually;
- title/hook card with cover evidence and promised payoff;
- structure card with carousel page roles;

Visual/layout and Live Photo observations remain evidence. Create image or
shot cards only after explicit user interaction and standardization.

“好看” is not enough. State the visual mechanism and why it works.

For Xiaohongshu/Douyin, execute `comment-signals.md`. Comments may reveal missing context, requested tutorials, alternative use cases, or disagreement with image text. Keep them as audience evidence with comment locators; do not merge them into image OCR or source body.

Use `scripts/merge_comments.py` after an anonymous public-page adapter exports a bounded sample. Never import a logged-in browser profile. The script retains up to ten complete-information comments with at least 10 verified likes, ranked by normalized likes, and preserves the exact ranking scope.

Treat the first platform-designated image as the cover and execute `cover-analysis.md`. Its cover role, feed readability, promise, and payoff comparison are separate from the full carousel sequence analysis.

Write the controlled result to `evidence/cover/analysis.json`; `scripts/validate_social_analysis.py --scope cover --commit` is the cover gate, not the presence of a prose paragraph alone. Comment signals use `--scope comments` as a separate checkpoint and cannot block cover completion.

## Image-post completion checklist

- Image count/order/dimensions/hashes pass G1.
- The OCR decision records `all_pages_ocr` with policy `mandatory_per_image_v1`.
- Every source image has raw and corrected OCR artifacts, including empty corrected
  text for a verified `no_text` page; core text is visually verified.
- Body and image text remain distinguishable.
- Analysis cites image numbers.
- Missing original images force `partial` or `review`, never silent completion.
- Every declared Live Photo has a passing static/motion pair; missing motion fails G1.

# Portable content job contract

Use UTF-8 JSON. One file represents one source. The packet is transportable across Hermes, Claudian, WorkBuddy, and Codex.

## Required fields

```json
{
  "schema_version": "1.0",
  "job_id": "CCI-20260803-120000-a1b2c3d4",
  "created_at": "2026-08-03T12:00:00+08:00",
  "source_agent": "hermes",
  "input_channel": "user-direct",
  "raw_input": "the user's exact share message",
  "source": {
    "url": "https://...",
    "local_files": [],
    "platform_hint": "xhs",
    "content_form_hint": "unknown"
  },
  "instruction": {
    "raw": "the user's exact instruction",
    "mode": "full",
    "requested_outputs": []
  },
  "routing": {
    "platform": "unknown",
    "content_form": "unknown",
    "priority": "high"
  },
  "state": {
    "status": "pending",
    "stage": "resolve",
    "attempts": 0,
    "claimed_by": null,
    "claimed_at": null,
    "updated_at": "2026-08-03T12:00:00+08:00"
  },
  "resolution": {},
  "outputs": {
    "chaptered_content": null,
    "asset_bundle": null
  },
  "errors": []
}
```

`raw_input` and `instruction.raw` are append-only evidence. Later agents may add normalized fields but must not replace them.

For chaptered carriers, `outputs.chaptered_content` points to the chapter JSON defined in `chapter-contract.md`. For `analyze/cards/full`, `outputs.asset_bundle` points to `evidence/assets/assets.json` defined in `source-assets-contract.md`. Handoffs preserve these paths and their validation state.

## Enumerations

- `instruction.mode`: `capture`, `standard`, `analyze`, `cards`, `full`
- `routing.platform`: `xhs`, `douyin`, `wechat`, `web`, `youtube`, `bilibili`, `podcast`, `local`, `unknown`
- `routing.content_form`: `video`, `image_post`, `article`, `audio`, `document`, `unknown`
- `state.status`: `pending`, `running`, `partial`, `review`, `completed`, `failed`
- `state.stage`: `resolve`, `acquire`, `extract`, `verify`, `standardize`, `analyze`, `cards`, `archive`, `done`

## State transitions

```text
pending -> running -> completed
                   -> partial
                   -> review
                   -> failed
```

Only a successful atomic claim may change `pending` to `running`. A stale claim may be recovered only after checking the worker is not active and recording the recovery in `errors` as a non-fatal event.

## Idempotency and identity

Prefer stable platform note/video/article IDs. Otherwise compute `sha256(canonical_url)`; for local files use the file SHA-256. Before acquisition, search manifests for the stable ID/hash:

- exact same source + completed compatible mode: reuse or extend existing package;
- same source + currently running: do not duplicate;
- same URL but changed media/body: create a revision and link the prior package.

## Error record

Each error is structured:

```json
{
  "at": "2026-08-03T12:05:00+08:00",
  "stage": "extract",
  "code": "OCR_LOW_CONFIDENCE",
  "message": "Images 3 and 5 require visual review",
  "recoverable": true,
  "artifact": "evidence/ocr/image-003.raw.txt"
}
```

Do not include credentials, cookie values, signed query strings, or full sensitive headers.

## Minimal portable handoff

When an intake agent cannot write files, return one fenced `content-job` JSON block using this schema. The desktop executor imports it unchanged and adds state fields; it does not re-summarize the user's message.

# Obsidian storage contract

## Queue

Default vault-relative queue root:

```text
【00.自媒体工作台】/处理队列/
  pending/
  running/
  completed/
  partial/
  review/
  failed/
```

Each directory contains portable job JSON files. Agents may use another root only when the user explicitly configures one.

## Canonical source package

Default package root:

```text
<library_root>/
  S-{platform}-{stable-id-or-urlhash}/
    manifest.json
    {content-form}-{platform}-{title}-{author}-{date}.md
    evidence/
      source/
      author/
      metrics/
      media/
      images/
      transcript/
      article/
      assets/
      ocr/
      frames/
      comments/
      cover/
```

The system-named file recorded in `manifest.document.path` is the only canonical human-editable content document. Its basename follows `内容形态-来源渠道-最多10字标题-最多10字作者-YYYYMMDD.md` and stays stable after assignment. It contains `原始资料`, `标准内容`, `内容拆解`, `素材卡片`, and `证据索引` as stable second-level sections. The frontend renders those sections and writes edits back to the same file with an optimistic revision check. Binary/raw machine evidence remains under `evidence/`. Legacy `收藏.md` is accepted only long enough to migrate it to the assigned name.

## File responsibilities

### `manifest.json`

Machine-readable source identity, acquisition/extraction metadata, hashes, artifact inventory, gate results, errors, revisions, and originating job ID.

Change time domains are independent. `created_at` is the immutable package intake time. `change_tracking.user_edited_at` is written only by an explicit user edit; `analysis_updated_at` is written by Agent transcript/understanding/asset review; `system_updated_at` is for capture, comment refresh, migration, and maintenance. `last_change_kind` and `last_changed_at` describe the latest write. Legacy `updated_at` may still mirror the latest write for compatibility, but UI code must never infer a user edit from it.

Every route uses the same source-identity contract:

- `source.author_profile.display_name`: public account name;
- `platform_user_id`: stable platform account ID when lawfully visible;
- `public_handle`: public-facing handle/number when visible;
- `avatar`: local artifact record under `evidence/author/`, including path, dimensions, MIME, bytes, and SHA-256; use `null` plus a reason when unavailable.

Visible engagement is stored as an immutable timestamped snapshot under `evidence/metrics/`. Use the common keys `likes`, `collects`, `comments`, and `shares`; unsupported or unavailable values are `null`, never inferred as zero. Platform-specific extra metrics may be added without changing these common keys.

For `analyze`, `cards`, or `full`, also record `classification.primary_track`, optional parent-bound `classification.secondary_tracks`, `classification.content_mechanism`, and `classification.tags`. Primary/secondary tracks are controlled, evolvable labels; tags are richer keywords. Never infer any of them from platform alone.

Also record zero to four `classification.navigation_tags` for the favorites UI. They are reviewed knowledge-navigation labels, not copied platform hashtags. Shared editable collections live at the package root's sibling `library.json`; an uninitialized library may derive a few broad suggestions from tracks actually present. Write a collection candidate only when it names an existing configured collection, otherwise use `待整理`.

### `canonical document → 原始资料`

Preserve the user's exact message and instruction, received time, intake agent, and any stated reason for saving. Clearly separate verbatim user text from later inference.

Index raw artifacts without pretending to be a clean transcript/article. Include the user's input, source URL, metadata, body/caption as acquired, media/image inventory, and links to raw transcript/OCR.

Also index the author/account record, local avatar path or explicit unavailable reason, engagement snapshot time, and comment-evidence path when the carrier exposes them.

### `canonical document → 标准内容`

Faithful, navigable corrected content. Video/audio exposes chapter title + start/end + continuous full transcript paragraphs; articles/blogs expose chapter title + adjacent paragraph range + continuous source body. Use image-number sections for image posts. Granular timestamps and paragraph IDs remain evidence, not the default reader surface. State completeness and uncertainty at the top.

For image-led content, always group the original images in source order under
`逐图内容`. Add corrected visible text only when OCR was needed; source-authored
title/body remains a separate channel. Human-facing Markdown never includes OCR
status enums, confidence scores, model/threshold data, `_no_text_`, or
pending-review instructions. Preserve those only in machine evidence and manifest gates.

### `canonical document → 内容拆解`

Use carrier-specific analysis. Minimum fields:

- Why this may have been saved: explicit reason / inferred reason / evidence.
- One-sentence core and content map.
- Strong/useful/ordinary/weak findings by applicable dimension.
- Viewpoints and quotes with locators.
- Xiaohongshu/Douyin top-comment evidence and any meaningful topic signals, or an explicit no-signal result.
- Xiaohongshu/Douyin controlled cover tags, promise mechanism, OCR decision,
  payoff comparison, and transfer judgment.
- Reusable patterns and conditions.
- Possible links to the user's collected-content graph and creation graph.
- What not to imitate or what remains uncertain.
- Classification (`primary_track`, parent-bound optional secondary tracks, content mechanism, and tags) and the reason for the choice.

### `canonical document → 素材卡片`

Use one third-level heading per promoted candidate. Storage supports `金句卡`, `观点卡`, `钩子卡`, `结构卡`, `图像卡`, and `镜头卡`, with machine values `quote`, `viewpoint`, `hook`, `structure`, `image`, and `shot` in `evidence/assets/assets.json`. Agent source digestion may promote only the first four. Image and shot require standardized user interaction, and any of the first four may also be `user_injected`. Evidence cases remain evidence within analysis or a locator attached to one of these six storage types; they are not a seventh asset class.

Each rendered card preserves its machine `asset_id` and presents the reader-facing content needed to understand and reuse it: type, origin, source or interaction reference, reusable payload, reuse condition, boundary, and created time. Detailed strength, admission, uncertainty, and review evidence remain in `assets.json`; Markdown prose cannot silently override those machine decisions.

## Naming and paths

- Use stable IDs when possible; otherwise first 16 characters of canonical URL SHA-256.
- Keep original file extensions and image order: `image-001.ext`.
- Never use an untrusted title directly as a directory path without sanitizing it.
- Keep package identity stable when the title changes.

## Minimal manifest gates

```json
{
  "package_schema": "1.1",
  "package_id": "S-xhs-abc123",
  "job_id": "CCI-...",
  "mode": "full",
  "content_form": "image_post",
  "source": {"canonical_url": "https://...", "stable_id": "abc123"},
  "artifacts": [],
  "gates": {
    "identity": "passed",
    "acquisition": "passed",
    "extraction": "passed",
    "fidelity": "passed",
    "analysis": "passed",
    "comments": "not_applicable",
    "cover": "not_applicable",
    "cards": "not_applicable",
    "archive": "passed"
  },
  "gate_ledger_schema": "creator-gates/v1",
  "gate_ledger": {
    "identity": {
      "status": "passed",
      "rule_version": "identity-v1",
      "input_fingerprint": "sha256...",
      "output_fingerprint": "sha256...",
      "depends_on": []
    }
  },
  "errors": []
}
```

The abbreviated ledger entry illustrates the shape; actual packages contain one entry for each existing gate. The ledger stores cache and invalidation metadata only. It never duplicates transcript, OCR, analysis, comment, cover, or asset-audit content.

# Quality gates

The nine gates below are summarized in `manifest.gates` and versioned in `manifest.gate_ledger`. Read `gate-ledger.md` for resume and invalidation behavior. Asset candidate discovery and independent admission audit stay inside G5; they are not an additional gate.

Every applicable gate must be `passed`, `failed`, `review`, or `not_applicable` in `manifest.json`. `completed` requires all required gates to pass.

## G0 Identity

- Canonical/resolved URL or local file identity is recorded.
- Platform stable ID or deterministic SHA-256 exists.
- Carrier classification is based on actual content, not just platform.
- Duplicate/revision check is recorded.

## G1 Acquisition

- Every claimed media/image/body artifact exists and is non-empty.
- Images have count, order, dimensions, MIME type, and SHA-256.
- Audio/video has duration where readable, MIME/container, and SHA-256.
- HTML/body capture identifies whether it is complete, partial, or reader-mode reconstruction.
- Temporary/signed URLs are not preserved as canonical sources.
- Public account display name is recorded; public avatar is archived locally with hash when accessible, otherwise unavailability is explicit.
- Visible engagement counts are stored in a timestamped snapshot. Missing/unsupported counts are `null`, not zero.

## G2 Extraction

Video/audio:

- Raw transcript exists with timestamps.
- Language/model/runtime are recorded.
- Corrected transcript preserves meaning and has an uncertainty list.
- Critical names, numbers, claims, and quotes are reviewed against audio or marked uncertain.
- Chapter JSON/Markdown uses semantic titles, bounded start/end, and complete continuous transcript paragraphs. It covers every reader-eligible non-empty raw segment once and in order; QA-declared `reader_exclusions` remain in raw evidence and in the chapter audit list but never enter reader prose. Granular ASR rows are not the reader surface.

Image post:

- Every source image receives one baseline OCR pass. The decision records
  `all_pages_ocr` and policy `mandatory_per_image_v1`; title/body sufficiency,
  `--skip-ocr`, or visual triage cannot suppress image-post OCR.
- Every inventory index has raw and corrected OCR evidence and one controlled
  classification: `substantial_text`, `sparse_text`, or `no_text`. A targeted
  retry is limited to low-confidence decision-relevant spans and never replaces
  or repeats the full baseline pass. Source page locators remain stable.
- OCR-derived core-claim text is visually checked; low-confidence regions are marked, not guessed.
- Body text is separate from image OCR.

Article:

- Title/body boundaries are identified.
- Navigation, recommendation, comment, and footer noise are excluded or labelled.
- Paragraph/heading locators survive into standard content.
- Article/blog chapter JSON/Markdown groups only adjacent preserved body paragraphs under semantic titles. It neither summarizes nor rewrites the chapter body.

## G3 Fidelity

- Standard content can be traced back to evidence.
- Every speech-bearing source has a lexical-review log. A pass requires all declared checks, reviewer, review time, coverage, and evidence method even when no correction was needed; file presence or automated ASR QA alone never passes this gate.
- Omissions and inaccessible areas are declared.
- Quotations are verbatim within correction limits and have locators.
- Paraphrases are labelled as paraphrases.
- No generated connective text is presented as source text.

## G4 Analysis

- Analysis answers why the user may have saved the item, with evidence rather than generic praise.
- Carrier-relevant dimensions are evaluated; irrelevant template sections are `not_applicable`.
- Strength labels use `strong`, `useful`, `ordinary`, `weak`, or `not_applicable`.
- Limitations and non-transferable context are explicit.
- Visual/shot/rhythm claims cite image numbers or timestamps.

## G1C Comment acquisition (sources with a comment layer)

- For a source eligible under `comment-signals.md`, the executor attempted to retain up to ten top-level comments with at least 10 verified likes and complete information.
- Ranking scope, sample size, has-more state, method, and capture time are explicit.
- Comment text, visible like counts, and locators are preserved separately from creator content.
- Apply comment acquisition only to `analyze`, `cards`, and `full` sources that meet the platform threshold in `comment-signals.md`. Set `skipped_by_policy` and settle the gate for `capture`, `standard`, and below-threshold sources; do not start a comment request for them.
- For Xiaohongshu, a completed public initial-state scan with no comment list satisfies the attempt requirement as `confirmed_no_public_source`; it must not request login or keep the package in review.

## G4C Comment topic signals (Xiaohongshu/Douyin analysis)

- Topic signals cite comments and distinguish audience evidence from verified fact.
- `no_meaningful_signal` is accepted; generic praise is not forced into a topic idea.
- If comment access is unavailable, the package records `unavailable/review` rather than claiming top-three completeness.

## G4V Cover (Xiaohongshu/Douyin)

- Original cover, dimensions, hash, and acquisition method exist.
- Cover evidence records whether OCR was needed. When OCR runs, raw text,
  corrected text, and visual-review status are preserved; an image post with
  sufficient source-authored text may use that text plus direct cover inspection.
- Controlled tags follow `cover-analysis.md`, not free-form near-duplicates.
- Analysis identifies one-second comprehension, promise mechanism, visual hierarchy, image-text division, trust device, and mobile legibility.
- Cover promise is compared with actual body/transcript/video payoff.
- A cover card is created only for a strong, transferable mechanism.

## G5 Asset admission

- `evidence/assets/assets.json` passes `creator-assets/v1` validation. Agent extraction uses only quote, viewpoint, hook, and structure. Image and shot require standardized `user_injected` provenance, and Agent reruns preserve every user-owned card.
- Every Agent-generated asset has carrier-appropriate source evidence, strength, admission decision, evidence confidence, originality, uncertainty, counterevidence, reuse condition, boundary, and anti-copy note. A user-injected card may instead cite the archived interaction that created it, but must still be standardized and carry the same reuse and boundary fields.
- A viewpoint states claim, reasoning/evidence, scope, and a credible counterpoint. A quote preserves exact reviewed wording and context.
- A hook locates both promise and payoff. A structure follows actual source order. User-injected image/shot assets state a reusable visual/production mechanism rather than aesthetic praise; Agent extraction does not scan or create these two types.
- Only strong/useful, verified, non-generic material may be admitted. Ordinary/weak material is rejected or left in analysis; never promote to fill a quota.
- The auditor reconstructs every admitted Agent-generated payload from the cited source artifact without relying on analyst prose. A user-injected payload must instead be reconstructable from its archived interaction record. Text excerpts absent from the cited evidence, evenly manufactured structure boundaries, unlocated hook payoff, generic visual/shot templates, or review placeholders fail G5.
- A `passed` bundle contains no unresolved `review` assets. If one applicable type remains unresolved while another has settled, keep `analysis_status: partial`; the settled type's admitted cards remain available.

## G6 Archive

- Canonical folder and manifest exist.
- Internal links are relative or vault-compatible.
- Job output path and manifest package ID agree.
- No secret-bearing URLs, cookies, keys, or raw headers are stored.
- Job state reflects real gate results.

## Completion rule

- `completed`: every required gate passed.
- Comment acquisition uses no-login public adapters only. Xiaohongshu and Douyin store a missing stable public list as `confirmed_no_public_source`; Bilibili may use an anonymous guest session; Xiaoyuzhou uses its public episode payload. No platform may retry comments through a logged-in or rendered browser. `not_applicable` remains reserved for comments disabled/zero or a carrier with no meaningful comment layer. For `analyze/cards/full`, cover and available comment-topic-signal review still apply. A Xiaohongshu collection child with only directory metadata/cover and no detail body or complete media has failed acquisition/extraction/fidelity; it is stopped, not awaiting review and not still running.
- `partial`: useful evidence exists but a non-critical required artifact is missing.
- `review`: evidence exists but human/strong-agent judgment is necessary before fidelity or card promotion.
- `failed`: no useful recoverable package was produced or source acquisition failed completely.

# Podcast shownote timeline adaptation

Use this contract when capturing, repairing, rebuilding, or rendering podcast shownotes with timestamps. The goal is not to recognize one publisher template. It is to preserve source evidence while adapting among flat, grouped, mixed, collapsed, and reference-only timestamp shapes.

## Decision order

1. Classify the shownote section before classifying its timestamps.
2. Normalize layout without changing wording or hierarchy.
3. Prefer a valid structured `source.timeline` over any later body re-parse.
4. Detect parent evidence and divide it into distinct source runs.
5. Promote a run to two levels only when one explicit parent has at least two ordered timed children.
6. Project the same structure into chapter building and the reader; neither layer may independently invent or rename it.

## Section classification

- Timeline labels such as `时间轴`, `时间线`, `本期你会听到`, and `本期节目清单` may introduce navigational boundaries.
- `猜你想看`, links, references, definitions, and highlight sections produce seekable `timestamp_notes`, never transcript boundaries.
- A headerless run of at least two increasing line-leading timestamps may be a flat timeline. One timestamp under link/reference metadata remains a reference unless another timestamp forms a clear increasing run.
- Preserve unknown sections and timestamps in the source body. Uncertainty is not permission to manufacture chapters.

## Layout normalization

Normalize full-width/ASCII colon variants and store one separator between a timestamp and its title. When a publisher container collapses several timestamp rows into one line, split it only when the line is inside a classified timeline section or begins with a timestamp and contains multiple line-leading-style timestamp/title pairs.

Normalization must not:

- split a timestamp into pieces;
- turn an inline narrative time reference into a new outline level;
- discard headings, links, images, or explanation paragraphs;
- change source wording.

## Parent evidence

The following may start a parent run when they are source-authored and occur in a timeline section:

- an upstream structured parent/group field;
- a timed bold-only row such as `00:02:13 **上野公园**`;
- an untimed Markdown heading;
- an untimed bold-only row;
- an explicit Arabic/keycap/circled or Chinese section number;
- a standalone bracketed heading.

Plain prose, a topic guessed from child wording, visual spacing alone, or a desire for a uniform layout are not parent evidence.

Each parent occurrence gets a distinct source-run identity. Repeated parent text later in the shownotes must not merge with an earlier run merely because the title is identical.

## Promotion and fallback

- Two or more increasing timed children in the same parent run: keep `group_title`, optional `group_number`, `group_start`, `group_evidence`, and `outline_level: 2` on every child.
- One child: flatten that child to `outline_level: 1`. If the parent itself had a timestamp, retain it as an ordinary single-level timestamp entry; an untimed parent does not become a synthetic chapter.
- No parent evidence: keep every valid entry at `outline_level: 1`.
- Mixed grouped and ungrouped runs are valid.
- Invalid, duplicate, or decreasing timestamps stay in raw shownotes for audit but do not silently become navigation boundaries.

Use the first child start as `group_start` when an untimed parent has no source-authored start. This is a playback locator, not evidence that the parent was spoken at an invented earlier time.

## Canonical projection

Capture writes a structured timeline once. Chapter building treats that structure as canonical and uses body parsing only for legacy packages that lack it. The API and reader consume the structured projection; Markdown splitting remains a compatibility fallback for old collapsed bodies.

Required entry fields:

- `start`, `title`, `source`, and `outline_level`;
- grouped entries additionally carry `group_title`, optional `group_number`, `group_start`, and `group_evidence`.

Do not require every historical package to contain the newer provenance fields. Derive display defaults conservatively, but write the complete current shape on new capture or a scoped shownote refresh.

## Regression matrix

Any parser change must test observable behavior across these independent axes:

- explicit timeline heading vs no heading;
- one timestamp per line vs multiple collapsed rows;
- ASCII vs full-width colons and missing whitespace;
- flat list vs grouped list vs mixed runs;
- numbered, timed-bold, Markdown, bold-only, and bracketed parent evidence;
- one-child false parent vs two-child valid parent;
- repeated identical parent titles in separate runs;
- timestamped reference/highlight sections;
- structured timeline conflicting with legacy body text;
- images, links, or prose between valid rows.

Add a new evidence form by extending this matrix and recording its provenance. Never add a rule keyed to one episode title, author, city, or screenshot.

# Source-to-assets contract

Read this reference for every `analyze`, `cards`, or `full` job. Emit one machine file at `evidence/assets/assets.json`; render only independently reviewed `admitted` candidates through two projections: whole-source understanding, hook, and structure under `manifest.document.path → 内容拆解`, and quote, viewpoint, plus interaction-only image/shot cards under `manifest.document.path → 素材卡片`. The JSON retains review/rejected audit records and is authoritative for filtering and validation, while Markdown remains the human-editable reading surface.

## Asset bundle

Use `schema_version: creator-assets/v1`:

```json
{
  "schema_version": "creator-assets/v1",
  "source_id": "S-xhs-example",
  "source_revision": "sha256-or-manifest-revision",
  "type_revisions": {
    "quote": "sha256:...", "viewpoint": "sha256:...",
    "hook": "sha256:...", "structure": "sha256:..."
  },
  "understanding_revision": "sha256:...",
  "carrier": "video",
  "generated_at": "2026-08-17T12:00:00+08:00",
  "analysis_status": "review",
  "source_understanding": {
    "status": "review",
    "summary": "",
    "core_topic": "",
    "core_message": "",
    "content_intent": "",
    "audience_takeaway": "",
    "classification": {
      "primary_type": "",
      "expression_modes": [],
      "audience_values": [],
      "topic_tags": []
    },
    "commercial_assessment": {
      "status": "uncertain",
      "promotion_style": "unclear",
      "disclosure": "unavailable",
      "scope": "unclear",
      "subject": "",
      "reason": "",
      "evidence": []
    },
    "evidence": []
  },
  "review": {
    "schema_version": "asset-review/v1",
    "status": "review",
    "reviewed_source_revision": "sha256-or-manifest-revision",
    "source_main_line": "",
    "audience_path": {"attract": "", "carry": "", "leave": "", "transfer": ""},
    "type_scan": [
      {"type": "quote", "status": "review", "reason": "等待独立审核", "evidence_revision": "sha256:...", "reviewed_evidence_revision": "", "reviewed_by": "", "reviewed_at": ""},
      {"type": "viewpoint", "status": "review", "reason": "等待独立审核", "reviewed_by": "", "reviewed_at": ""},
      {"type": "hook", "status": "review", "reason": "等待独立审核", "reviewed_by": "", "reviewed_at": ""},
      {"type": "structure", "status": "review", "reason": "等待独立审核", "reviewed_by": "", "reviewed_at": ""}
    ],
    "candidates": [],
    "reviewed_by": "",
    "reviewed_at": ""
  },
  "assets": []
}
```

`carrier` is one of `video`, `audio`, `image_post`, `article`, `document`, or `music`. `analysis_status` is `review`, `partial`, or `passed`: `partial` means at least one applicable type has produced an `admitted` or `no_asset` result while another remains in review. It is a usable incremental result, not a batch failure. A passed bundle may contain zero assets: scarcity is preferable to quota filling.

`evidence/assets/evidence-manifest.json` is the bounded carrier-aware input index.
It points to complete evidence rather than copying raw media. `source_revision` is
its aggregate digest; the four `type_revisions` hash only evidence relevant to
that type. A changed type becomes stale independently. Preserve stale Agent
cards for audit but never render or finalize them. User-injected cards do not
inherit source-evidence staleness.

## Whole-source understanding

Before discovering hook or structure candidates, read the complete source and
fill `source_understanding`. It gives the creator a short orientation before
mechanism analysis:

Read `content-deconstruction-framework.md` for the evidence-backed compact
framework. The normal reader projection is one overview callout card showing
classification tags, `summary` as “讲了什么”, `core_message` as “核心判断”,
and the commercial assessment. Keep intent, audience takeaway, locators, and
detailed reasoning in JSON instead of repeating them on the page.

- `summary`: one to three sentences describing what concretely happens, is
  argued, demonstrated, or sequenced;
- `core_topic`: the real problem or subject;
- `core_message`: the central judgment, conclusion, demonstrated result, or
  narrative recognition;
- `content_intent`: what the author is trying to explain, prove, show, persuade,
  or trigger;
- `audience_takeaway`: what the audience should finally understand, feel, or
  remember;
- `classification`: one controlled primary content type, 1–2 expression modes,
  1–2 audience values, and 1–3 concise topic tags;
- `commercial_assessment`: whether commercial cooperation is confirmed,
  suspected, unsupported by available evidence, or impossible to judge; when
  applicable, whether the presentation behaves like hard or soft advertising;
- `evidence`: locators covering the main body, not title metadata alone.

Use these commercial values:

- `status`: `confirmed_commercial`, `suspected_commercial`,
  `no_commercial_evidence`, or `uncertain`;
- `promotion_style`: `hard_ad`, `soft_ad`, `not_applicable`, or `unclear`;
- `disclosure`: `explicit`, `implicit`, `none_seen`, or `unavailable`.
- optional `scope`: `whole_source`, `embedded_segment`, `none`, or
  `unclear`. Use it to avoid calling an entire podcast or video an ad when only
  a bounded sponsor segment is commercial.

Hard advertising puts the product, offer, price, benefit list, purchase path,
coupon, campaign, or direct call to action at the center. Soft advertising
embeds promotion inside a tutorial, experience, story, review, comparison, or
ordinary-looking recommendation. `soft_ad` describes the presentation, not
proof of payment: without a disclosure, commercial link, repeated campaign
language, or other relationship evidence, use `suspected_commercial` rather
than claiming a confirmed deal. Product mention, critical review, ordinary
recommendation, event reference, or official-document link alone is not proof
of commercial cooperation. For `confirmed_commercial` and
`suspected_commercial`, record the promoted subject, one short reason, and
source locators. For `no_commercial_evidence`, `subject` may be empty because
there is no promoted object to name; the evidence-grounded reason and locators
remain required. Use `uncertain` when required evidence is unavailable.

Keep the summary concise and descriptive. Do not turn a tutorial,
demonstration, or personal narrative into a universal opinion merely to fill
`core_message`. Hook and structure candidates must state their
`main_line_relation` in the review receipt.

Use these classification values:

- `primary_type`: `新闻分析`, `观点评论`, `知识解释`, `教程方法`, `案例复盘`,
  `产品评测`, `体验记录`, `人物访谈`, `故事叙事`, `清单推荐`, `作品解读`, or
  `音乐作品`;
- `expression_modes` (1–2): `分析`, `论证`, `叙事`, `对谈`, `采访`, `演示`,
  `实测`, `清单`, `教程`, `观察`, `抒情`, or `复盘`;
- `audience_values` (1–2): `获得判断`, `理解机制`, `解决问题`, `行动参考`,
  `信息获取`, `情绪共鸣`, `审美体验`, or `选择参考`;
- `topic_tags`: 1–3 source-specific retrieval terms, each no longer than 16
  characters. Reuse an established term when it has the same meaning; do not
  add synonyms merely to make the card look rich.

Classify from the complete source after understanding review. Platform does
not determine content type, and a topic keyword does not determine expression
mode or audience value.

## Supervised loop

The two-pass extraction below runs inside the supervised loop in
`references/asset-review-loop.md`. The source map and review receipt are
required control artifacts even when the final bundle contains zero assets.
An audit finding must identify the stage to rerun; “rewrite the card” is not a
valid recovery instruction unless the candidate itself has already passed.

## Two-pass extraction and audit

Do not generate a finished asset in one pass.

1. **Discover candidates:** scan only quote, viewpoint, hook, and structure. Record the smallest source span that appears potentially reusable. For quote and viewpoint, complete a recall pass across the full source and cluster candidates by semantic role: central conclusion, supporting judgment, method/decision rule, cognitive turn, and bounded caveat. These roles are search prompts rather than a quota. Do not stop after one representative candidate, but also do not retain paraphrase duplicates. Quote discovery is semantic and must run even when a speech carrier's whole transcript is not `quotation_ready`; retain valuable spoken lines as review candidates with bounded spans. Scan source-authored title/caption/description/body as an independent exact-text channel. Do not assign `admitted` yet and do not manufacture missing payoff, counterpoint, or structure boundary. Images and shots remain evidence at this stage, never automatic asset candidates.
2. **Reconstruct from evidence:** reopen only the cited artifact and locator. Confirm that an independent reader can recover the proposed payload without seeing the analyst's summary. Text excerpts must occur in the cited artifact; visual claims must cite archived images, frames, media, or visual-analysis evidence.
3. **Falsify by type:** actively look for the failure condition listed below. If decisive evidence is absent, reject the candidate or keep the whole bundle in explicit review; record the failure stage/code and repair request; never render it as a production asset.
4. **Judge transfer value:** require a source-specific mechanism, concrete reuse condition, real boundary, and anti-copy note. Reject praise, topic labels, summaries, and templates that could describe many unrelated sources.
5. **Promote distinctly:** only `admitted` assets appear in the normal asset browser. Scarcity is still preferable to quota filling, but “one card per type” is not a quality rule. Keep multiple cards when their claims, source spans, audience effects, and future retrieval uses are materially different. Review/rejected records may remain in JSON as an audit trail, but are not user-facing assets. Reader-facing copy editing happens only after semantic promotion.

Before promotion, perform cross-type collision detection between admitted quote
and viewpoint candidates. The normalized `viewpoint.claim` must not equal an
admitted `quote.verbatim`. Exact wording with the same semantic job belongs to
the quote card; a viewpoint must be a bounded paraphrase with a distinct,
source-backed reasoning path. Reject unchanged duplicates as `type_mismatch`
and rerun viewpoint discovery only when the source contains a broader supported
judgment.

The top-level `review` object is the canonical Agent receipt, but its `status` is
only an aggregate and never an admission gate. Each quote, viewpoint, hook, and
structure entry in `type_scan` is its own gate with its own `status`, `reason`,
`reviewed_by`, and `reviewed_at`. Use `no_asset` with a concrete reason when a
completed scan finds nothing worth retaining, and `not_applicable` only when
the carrier lacks the required evidence mode. An `admitted` card is renderable
as soon as its own type gate is `admitted`; a different type remaining in
`review` cannot hide it. Every stored candidate has a matching review record.
Strict whole-bundle completion still requires all four gates to settle, while
partial rendering does not. User-injected cards never enter the Agent candidate
receipt and remain independently visible.

`type_scan.status: partial` is valid when one type contains both admitted cards
and unresolved valuable candidates. It renders the admitted cards and retains
the other candidate receipts for targeted repair. For quotes, separate three
states: `candidate_value` (is the line worth keeping), `verbatim_status` (is
the exact wording verified), and final admission. A global transcript flag may
limit unrestricted quoting from the transcript, but it must not erase quote
candidates or block a separately verified source-body or audio-span quote.

Type-specific falsification tests:

- `quote`: exact reviewed wording is absent from the cited source span, speaker/context is ambiguous, or the line only works after analyst rewriting. Source-authored body/caption and bounded reviewed speech spans are separate valid channels; do not require the entire transcript to pass when the cited channel itself is verified.
- `viewpoint`: reasoning does not support the claim, scope is broader than evidence, or no credible counterpoint/boundary can be stated.
- `hook`: require two evidence locators with `role: promise` and `role: payoff`; payoff must occur later, contain a distinct source excerpt, and materially answer the promise. Missing payoff means `rejected`, not a permanently pending hook.
- `structure`: every unit needs a distinct, ordered, source-backed locator and excerpt. Never divide total duration or paragraph count evenly to invent boundaries.

## Common asset record

Every asset uses this base shape:

```json
{
  "id": "S-xhs-example:quote:01",
  "type": "quote",
  "origin": "agent_generated",
  "title": "A short retrieval label",
  "payload": {},
  "evidence": [],
  "quality": {
    "strength": "strong",
    "admission": "admitted",
    "evidence_confidence": "verified",
    "originality": "source_distinctive"
  },
  "reuse": {
    "conditions": "Where the mechanism transfers",
    "boundary": "Where it stops transferring",
    "anti_copy": "What surface form must not be copied"
  },
  "uncertainties": [],
  "counterevidence": [],
  "created_at": "2026-08-17T12:00:00+08:00"
}
```

Allowed `type` values and Chinese UI names are:

- `quote` — 金句
- `viewpoint` — 观点
- `hook` — 钩子
- `structure` — 结构
- `image` — 图像
- `shot` — 镜头

Do not add a seventh storage type for evidence or generic notes. Agent extraction uses only the first four; image and shot remain interaction-only.

## Ownership and injection isolation

Every card has one write owner:

- `origin: agent_generated`: created by source digestion. Agent reruns may replace
  these cards through the protected merge command.
- `origin: user_injected`: formed through user interaction, standardized, then
  injected. Agent reruns must preserve it exactly.

Quote, viewpoint, hook, and structure may use either origin. Image and shot may
only use `user_injected`; captured visuals remain evidence until the user selects,
interprets, or develops them through interaction.

A user card requires:

```json
{
  "origin": "user_injected",
  "injection": {
    "status": "standardized",
    "created_by": "user",
    "protected_from_agent": true,
    "interaction_id": "stable-interaction-id",
    "injected_at": "2026-08-20T16:00:00+08:00"
  }
}
```

Do not write Agent results directly over canonical `assets.json`. A staged
incremental result declares `settled_types`, `generation_receipt.generated_by`,
and one independent `review_receipts.<type>.reviewed_by` per settled type. Apply
it with `save_link.py apply-agent-assets`; the merger replaces Agent-owned cards
only for those types, preserves other Agent types, and carries user-owned cards
forward unchanged. Candidate generation and review must be separate execution
identities/contexts even when they use the same model family.
In every incremental staged result, `review.type_scan` remains an array. A
single-type work unit writes exactly one entry whose `type` equals that unit;
it must never serialize the gate as an object keyed by `quote`, `viewpoint`,
`hook`, or `structure`.
The discovery execution first writes a fingerprinted
`evidence/assets/staged-agent-candidates.json`; the independent review execution
may only write `evidence/assets/staged-agent-submit.json`. Its
`generation_receipt.candidate_sha256` must match the dispatcher's recorded
candidate fingerprint. Neither execution may mutate the canonical bundle or
invoke apply; the dispatcher performs preflight, one atomic apply, and
incremental validation after both identities and the candidate fingerprint are
verified. The dispatcher durably records the apply boundary before invoking it.
After an interrupted or ambiguous apply, it may rerun validation only when the
canonical generation and review receipts prove that exact submission landed;
otherwise it must block for recovery instead of repeating apply.
Before apply, a malformed staged shape must return a named schema failure rather
than a runtime exception. A lossless structural repair reuses the original
candidate fingerprint and both execution identities; it does not rerun either
semantic execution.

Whole-source understanding has a narrower transaction boundary than asset
admission. Stage only `source_id`, the exact canonical
`understanding_revision`, and `source_understanding`, then commit it with
`save_link.py apply-source-understanding --expected-understanding-revision ...`.
That entry validates only the understanding schema, controlled enums,
commercial assessment, and locators. It must not validate, replace, or repair
unrelated cards, candidates, type scans, or user-injected assets. WeChat and
ordinary web articles both use this same article-carrier rule.

Before an older article enters this unit, `normalize-legacy-asset-review` may
reconcile an already admitted Agent card with its lost type-scan/candidate
projection only when an independent historical reviewer receipt exists and
every current locator still verifies. Record the result as an explicit
`legacy-admission-migration/v1` receipt; never infer admission from card
presence alone and never use this migration for a new judgment.
User editing and Agent commit share one short package mutation lock. The Agent
rereads canonical state only after it owns that lock. If the user edited an
Agent card during the run, its ownership changes to `user_injected` and that
exact card ID supersedes the staged Agent version; preserve the user version,
leave only the affected type checkpoint unresolved, and continue independent
types. This is not a whole-package validation failure and never authorizes
rerunning capture or settled evidence stages.
Use `inject-user-asset` only after a user idea has become a complete standard
card. A conversational draft is not yet an injected asset.

`strength` is `strong`, `useful`, `ordinary`, or `weak`. `admission` is `admitted`, `review`, or `rejected`. `evidence_confidence` is `verified`, `review`, or `uncertain`. `originality` is `source_distinctive`, `well_executed_familiar`, or `generic`.

These values are part of the runtime output schema, not optional prose guidance.
Every bounded review execution manifest must inline the complete sets above;
workers must never invent synonyms such as `high`, `original`, `distinctive`,
or `confident`. Evidence locator `verification` likewise uses only `verified`,
`review`, or `uncertain`.

Admission rules:

- Admit only `strong` or `useful` material with verified evidence, a non-empty transfer condition, boundary, and anti-copy note.
- Send an otherwise valuable candidate to `review` when wording, authorship, OCR/ASR, the visual mechanism, or claim support remains uncertain.
- Reject ordinary/weak/generic material or material whose value depends only on topic popularity. Record `rejection_reason` on rejected items.
- Never upgrade strength to fill a page, category, or requested count.

## Evidence locator

Each asset has at least one evidence object:

```json
{
  "kind": "time_range",
  "artifact": "evidence/transcript/逐字稿-章节版.json",
  "chapter_id": "chapter-03",
  "start": 125.2,
  "end": 151.8,
  "paragraph_start": null,
  "paragraph_end": null,
  "image_index": null,
  "region": null,
  "excerpt": "Short exact excerpt or null",
  "visual_description": null,
  "verification": "verified"
}
```

Allowed `kind` values:

- `time_range`: video, podcast, audio, or music; require numeric `start < end` and a chapter ID when chapter evidence exists.
- `paragraph_range`: article/blog/document; require integer `paragraph_start <= paragraph_end` and a chapter ID when chapter evidence exists.
- `image_region`: image post or article image; require positive `image_index`; use a normalized `[x, y, width, height]` region when the claim depends on a crop.
- `image_sequence`: carousel/visual sequence; require positive `image_index` and optionally `image_end_index`.
- `shot_range`: video/Live Photo; require numeric `start < end` plus a visual description.
- `source_metadata`: only for title, author, publication or platform facts; it cannot be the sole evidence for an analytical asset.
- `user_interaction`: the user's own added thought or decision, archived by `inject-user-asset`; require the stable interaction ID, the package-relative interaction record, and an exact user-authored excerpt. It is valid only for `user_injected` cards and does not replace source evidence required by hook, structure, image, or shot payloads.

Use a package-relative `artifact`; paths may not escape the package. `excerpt` is verbatim source wording, not an analyst paraphrase. Use `visual_description` for non-text evidence. `verification` is `verified`, `review`, or `uncertain` and cannot exceed the underlying transcript/OCR/body status.

## Type payloads

Required payload fields:

- `quote`: `quote_type`, `verbatim`, `context`, `why_it_lands`. New Agent cards choose exactly one `quote_type`: `情绪金句`, `概念金句`, `判断金句`, or `行动金句`. Choose it from the verified line's audience effect in context, never from isolated keywords. The evidence excerpt must contain the verbatim string after whitespace normalization. Never create from a summary, translation, or unreviewed ASR/OCR.
- `viewpoint`: `claim`, `reasoning`, `scope`, `counterpoint`. The claim is a concise, source-faithful proposition synthesized from the cited reasoning path. Remove non-load-bearing speaker, self-reporting, and publication-history wrappers (`我发现`、`作者表示`、`上个月我在……发了条表示`) while retaining real modality and scope. A compact load-bearing clause may retain source wording when it is materially shorter than the cited evidence and isolates the actual judgment. It must not copy a complete or near-complete cited source sentence or span; source wording belongs in a quote card. Adding explanation fields around unchanged source text does not satisfy the viewpoint type.
- `hook`: `hook_type`, `surface`, `mechanism`, `promise`, `payoff`, `transfer_pattern`. Its evidence array uses exactly one `role: promise` and one `role: payoff`. Admit only when the body actually pays off the promise; if payoff evidence is unavailable, reject it.
- `structure`: `structure_type`, `pattern`, `units`, `rhythm`, `transfer_pattern`. `units` is an ordered array with `label`, `function`, and a source locator. It must follow the source sequence rather than a generic template.
- `image`: `subject`, `composition`, `visual_system`, `text_image_role`, `transfer_pattern`. Cite the actual image(s); “好看/高级” is not a mechanism.
- `shot`: `subject_action`, `framing`, `camera`, `duration`, `transition`, `purpose`, `transfer_pattern`. Cite a shot range; a still frame without reusable production logic is not a shot asset.

Hook and structure retain these complete payloads internally, but their reader
projection is deliberately smaller and uses callout cards:

- hook: a title in `钩子｜<hook_type>｜<source-specific move>` form, followed by 开场、为什么想继续看、承诺 → 兑现、可复用;
- structure: a title in `结构｜<structure_type>｜<source-specific movement>` form, followed by a 3–6 beat arrow diagram built from ordered unit labels and one
  可复用 line. Keep pattern and rhythm in JSON.

Use one primary retrieval type per card. The controlled hook types are
`反常识纠偏`, `问题悬念`, `结果前置`, `风险预警`, `冲突反差`, `情绪共鸣`,
`身份代入`, `场景切入`, `事件切入`, and `新用途发现`. The controlled structure
types are `问题解决型`, `递进论证型`, `对比论证型`, `清单展开型`, `过程复盘型`,
`转变叙事型`, `循环变奏型`, `对话探索型`, `教程演示型`, and `并线汇合型`.
Choose the closest load-bearing mechanism after reconstructing the source; do
not infer a type from title keywords. Store the type separately in the payload,
then make the asset title exactly `类型｜具体做法`. The part after `｜` must name
this source's concrete promise or movement, not repeat the generic type.

Quote categories are stored only in `payload.quote_type`; do not force them into
the quote title. The reader renders the exact line as the primary text, then
shows source context and why the line lands. Category may control a restrained
background family, but must not reduce contrast or obscure exact wording.

Do not print evidence locators, review verdicts, boundary, anti-copy prose, or
every internal payload field in the normal structure-decomposition view. They
remain available in `assets.json` for audit. This projection rule does not
weaken admission requirements.

Carrier applicability for Agent extraction:

- Video, audio/podcast, image posts, articles/blogs, and documents may yield quote, viewpoint, hook, and structure when evidence supports them.
- Music defaults to quote and structure. Permitted audio may yield structure; rights-compatible verified lyrics or artist-authored text may yield short quote assets. If exact lyric evidence is missing, settle quote as `no_asset` or `review` rather than `not_applicable`; never reconstruct exact lyrics from audio alone.
- Image and shot are never part of carrier-driven automatic extraction. Their eligibility begins only with an explicit user interaction.

Carrier applicability is only the first routing layer. Read
`asset-decomposition-system.md`, then read
`carrier-specific-asset-frameworks.md` and select exactly one discovery/review
profile. Reuse the shared semantic core, schema, type gates, renderer, and
ownership rules once; the profile is an evidence adapter, not a separate Agent:

- `video_timeline` for video;
- `spoken_longform` for podcast and spoken audio;
- `carousel_sequence` for image posts;
- `argument_article` for articles, WeChat pages, and documents;
- `music_form` for music.

The profile changes candidate sources, hook location, movement boundaries,
speaker/page/paragraph requirements, commercial interpretation, and
falsification. It does not change card ownership, storage type names, or the
compact reader projection. Never make all four type gates applicable merely
for schema symmetry; music defaults to quote and structure, with viewpoint and
content-hook remaining inapplicable.

## Uncertainty and counterexample rules

Put every unresolved issue in `uncertainties` as an object with `code`, `detail`, `locator`, and `impact`. Examples include `ASR_NAME_UNCERTAIN`, `OCR_CORE_TEXT_UNCERTAIN`, `CLAIM_UNVERIFIED`, and `VISUAL_INTENT_INFERRED`.

Use `counterevidence` for source or captured-audience evidence that weakens the asset claim. Each entry needs a locator and explanation. An empty array means no captured counterevidence, not proof that none exists. A viewpoint without a credible counterpoint remains `review`; a hook whose payoff cannot be located is `rejected`.

## Deterministic validation

Run:

```bash
python scripts/validate_source_assets.py --package <package>
```

Use `--allow-review` for a partial handoff. It validates every settled type gate and allows its admitted cards to render while other types continue. Strict whole-bundle validation requires all four gates to settle, but it is not a shared card-admission gate.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "creator-assets/v1",
  "type": "object",
  "required": ["schema_version", "source_id", "source_revision", "carrier", "generated_at", "analysis_status", "source_understanding", "review", "assets"],
  "properties": {
    "schema_version": {"const": "creator-assets/v1"},
    "source_id": {"type": "string", "minLength": 1},
    "source_revision": {"type": "string", "minLength": 1},
    "type_revisions": {
      "type": "object",
      "required": ["quote", "viewpoint", "hook", "structure"],
      "properties": {
        "quote": {"type": "string", "minLength": 1},
        "viewpoint": {"type": "string", "minLength": 1},
        "hook": {"type": "string", "minLength": 1},
        "structure": {"type": "string", "minLength": 1}
      },
      "additionalProperties": false
    },
    "understanding_revision": {"type": "string", "minLength": 1},
    "carrier": {"enum": ["video", "audio", "image_post", "article", "document", "music"]},
    "generated_at": {"type": "string", "minLength": 1},
    "analysis_status": {"enum": ["review", "passed", "partial"]},
    "source_understanding": {
      "type": "object",
      "required": ["status", "summary", "core_topic", "core_message", "content_intent", "audience_takeaway", "evidence"],
      "properties": {
        "status": {"enum": ["review", "reviewed"]},
        "summary": {"type": "string", "maxLength": 500},
        "core_topic": {"type": "string"},
        "core_message": {"type": "string"},
        "content_intent": {"type": "string"},
        "audience_takeaway": {"type": "string"},
        "evidence": {"type": "array"}
      },
      "additionalProperties": true
    },
    "review": {
      "type": "object",
      "required": ["schema_version", "status", "reviewed_source_revision", "source_main_line", "audience_path", "type_scan", "candidates", "reviewed_by", "reviewed_at"],
      "properties": {
        "schema_version": {"const": "asset-review/v1"},
        "status": {"enum": ["review", "partial", "passed"]},
        "reviewed_source_revision": {"type": "string"},
        "source_main_line": {"type": "string"},
        "audience_path": {"type": "object"},
        "type_scan": {
          "type": "array",
          "items": {
            "type": "object",
            "required": ["type", "status", "reason", "reviewed_by", "reviewed_at"],
            "properties": {
              "type": {"enum": ["quote", "viewpoint", "hook", "structure"]},
              "status": {"enum": ["admitted", "partial", "no_asset", "not_applicable", "review"]},
              "reason": {"type": "string"},
              "evidence_revision": {"type": "string"},
              "reviewed_evidence_revision": {"type": "string"},
              "reviewed_by": {"type": "string"},
              "reviewed_at": {"type": "string"}
            }
          }
        },
        "candidates": {"type": "array"},
        "reviewed_by": {"type": "string"},
        "reviewed_at": {"type": "string"}
      },
      "additionalProperties": true
    },
    "assets": {
      "type": "array",
      "items": {
        "type": "object",
        "required": ["id", "type", "origin", "title", "payload", "evidence", "quality", "reuse", "uncertainties", "counterevidence", "created_at"],
        "properties": {
          "id": {"type": "string", "minLength": 1},
          "type": {"enum": ["quote", "viewpoint", "hook", "structure", "image", "shot"]},
          "origin": {"enum": ["agent_generated", "user_injected"]},
          "evidence_revision": {"type": "string"},
          "stale": {"type": "boolean"},
          "injection": {
            "type": "object",
            "properties": {
              "status": {"const": "standardized"},
              "created_by": {"const": "user"},
              "protected_from_agent": {"const": true},
              "interaction_id": {"type": "string", "minLength": 1},
              "injected_at": {"type": "string", "minLength": 1}
            }
          },
          "title": {"type": "string", "minLength": 1},
          "payload": {"type": "object"},
          "evidence": {"type": "array", "minItems": 1},
          "quality": {"type": "object"},
          "reuse": {"type": "object"},
          "uncertainties": {"type": "array"},
          "counterevidence": {"type": "array"},
          "created_at": {"type": "string", "minLength": 1}
        },
        "allOf": [
          {
            "if": {"properties": {"origin": {"const": "user_injected"}}},
            "then": {"required": ["injection"]}
          },
          {
            "if": {"properties": {"type": {"enum": ["image", "shot"]}}},
            "then": {"properties": {"origin": {"const": "user_injected"}}, "required": ["injection"]}
          }
        ],
        "additionalProperties": true
      }
    }
  },
  "additionalProperties": true
}

# Token-efficient evidence review

The archive may be large; the model context should not be. Preserve complete raw evidence on disk, then generate a compact review surface before semantic work.

## Default read policy

Agents must not load these whole artifacts into model context by default:

- `transcript-raw.json` with word timestamps;
- full-frame or per-second OCR JSON;
- every extracted video frame;
- binary media or image bytes represented as text/base64;
- duplicate raw and corrected versions of the same content.

Read in this order:

1. `manifest.json` and source metadata;
2. the compact standard content or machine transcript;
3. `逐字稿-复核包.md` for only high-risk ASR/OCR conflicts;
4. selected original audio/image/frame evidence needed to settle those conflicts;
5. corrected transcript/OCR for analysis.

This compact route is mandatory for every speech-bearing job. “No suspicious term found” may finish a check, but it must be recorded; token discipline reduces the evidence read, not the existence of the review stage.

Open raw JSON only for a specific timestamp, image, or region that cannot be resolved from the compact packet.

## Bounded semantic work units

Every semantic continuation processes exactly one work unit and checkpoints it before another unit is scheduled. A unit is one chapter-semantics batch, whole-source understanding, one asset type, or one named evidence repair; it is never all unresolved gates together.

The execution manifest must enforce both a total `open_first` byte budget and a per-artifact cap. An artifact above either cap moves to `on_demand` even when its role would normally be preferred. Opening `on_demand` evidence requires a named locator range; reaching the budget produces a review handoff, not a larger context window.

Long-audio lexical fidelity is not eligible for a whole-episode automatic semantic run. Review it through bounded listening checkpoints or leave the fidelity gate in review for the user. A process timeout preserves the latest canonical checkpoint and must not cause the same unchanged unit to loop.

## Video policy

Local MLX Whisper, ffmpeg sampling, and Vision OCR do not consume language-model tokens. They may produce large evidence files, which remain on disk.

The review packet should surface:

- names, product/tool names, English terms, numbers, units, and quotations;
- ASR segments with automated QA warnings;
- concurrent OCR candidates that disagree with or clarify ASR;
- only the representative frames needed for shot/visual analysis.

Do not pass per-second full-screen OCR to an LLM. For long videos, review by chapter and carry forward a compact decision log rather than reloading prior raw evidence.

## Image-post policy

Keep per-image OCR files. The analysis agent reads the corrected OCR and one image at a time or a bounded contact sheet. It should not load all raw OCR alternatives unless a core claim is uncertain.

## Article policy

Use clean heading/paragraph-located content. Do not simultaneously load raw HTML, cleaned Markdown, and the processed summary. Retrieve source paragraphs only when checking a quote or claim.

## Completion invariant

Token efficiency never lowers fidelity. If a compact packet is insufficient, open the necessary evidence and leave a locator-backed review note. Do not guess merely to save tokens.

# Video and audio-led pipeline

The goal is not merely a transcript. Produce trustworthy speech evidence and separately inspect viewpoint, content structure, delivery rhythm, shots, sound, and reusable creative assets.

## Stage V0 — Resolve and classify

1. Resolve the shared link without discarding the original share text.
2. Record canonical URL, platform, stable ID, title, author, publication time if visible, and description/caption.
3. Determine whether the item is truly video/audio-led.
4. Read `content-taxonomy.md`, then record one `primary_track`, up to three `secondary_tracks`, and one `content_mechanism`. The track states why the user studies this video; the mechanism selects the decomposition questions. Do not infer either from platform, narration style, or legacy folder name alone.
5. Record acquisition route and duplicate/revision result.

Pass G0 before download.

## Stage V1 — Acquire and verify

Acquire the best lawful, available media representation. Preserve source audio when possible.

Required checks:

- file exists, size > 0, playable/readable by probe;
- archive the original playable representation as `evidence/media/来源（小红书/抖音/B站）-作者-“标题”.mp4`; it is the retained source video, not a preview stream URL;
- duration, container/MIME, audio/video streams, resolution and frame rate if available;
- SHA-256;
- no login/session data is copied into the vault;
- source caption and visible metadata are stored separately from transcription.

If only audio is available, mark visual analysis `not_applicable`, not failed. If media acquisition fails but page text is available, create a partial package rather than inventing a transcript.

## Stage V2 — Speech evidence

For user-selected high-quality input, prefer a high-accuracy local Whisper-family model such as MLX Whisper `large-v3-turbo`; do not default to `base` when fidelity matters. Record implementation, model, language setting, and options.

The bundled reference command is `scripts/transcribe_package.py --package <package>`. It resolves a compatible reusable ASR Python independently of the calling agent's environment; use `scripts/transcribe_package.py --runtime-check` to inspect that choice. Do not install an ASR stack inside an active content job. It writes untouched ASR JSON, automated QA, a timestamped machine transcript, and a separate pending corrected transcript. Caption OCR or representative frames are generated only when a bounded visual-evidence task requires them. Automated QA passing never makes excerpts quotation-ready. Conversely, a whole-transcript review flag never suppresses semantic quote discovery. Retain strong spoken candidates with bounded timestamps, then review only those audio spans for exact admission. Source-authored title/caption/body text is a separate exact-text quote channel and does not inherit the transcript's pending state.

Speech and visual evidence are separate gates. If every compatible speech backend is unavailable, preserve the media and mark transcription blocked/review with the diagnostic. Do not claim that keyframe extraction, frame OCR, source captions, or a prose summary replaces a spoken-word transcript.

If QA rejects the audio as music or no reliable speech, treat that as a valid `no_reliable_speech` outcome: keep the transcript tab, show only `本视频无可识别人声`, emit no transcript chapters, and skip caption/per-frame OCR fallback.

For review, load `逐字稿-复核包.md` first. Keep full per-second OCR and word-level ASR JSON on disk; inspect them only for a named timestamp or unresolved term. This is a fidelity-preserving context rule, not a reason to skip verification.

Use `transcript-review.json` as the single lexical-review log for video and podcast speech. Run this review on every speech-bearing import, including when automated QA looks clean; a zero-correction pass still records completed checks, coverage, reviewer, time, and evidence methods. Confirm likely names, show/work titles, brands, products, English terms, and numbers against source metadata and nearby transcript context before replaying only the necessary audio range. A user-confirmed correction such as an ASR homophone is stored as an exact global rule and re-rendered into `逐字稿-章节版.*`, `逐字稿-校正版.md`, and the transcript block in the canonical document; the untouched machine transcript remains the audit source. Saving one correction moves lexical review to `in_progress`, not `passed`. Only `finalize_transcript.py` may mark quotation readiness after every required check is complete.

The reader does not need a separate correction form inside the transcript. Word-like in-place transcript edits are the preferred human review surface: convert the smallest safe exact before/after span into the same evidence-bearing correction command, rebuild the corrected/chapter artifacts, and preserve the raw ASR. Generic Markdown replacement must not bypass this log for transcript edits.

For ordinary video, include a bounded copy of the source-authored title/caption/body in both the ASR spelling prompt and review packet. It may confirm spelling and highlight likely ASR errors, but it must never replace the audio. Keep this route simple: restore natural punctuation, split by sentence meaning, discourse turns, and pauses into several readable paragraphs, and omit all speaker diarization and identity UI even if an upstream transcript happens to contain speaker fields. Podcast/audio may use the separate optional speaker route.

1. Extract/normalize audio without overwriting the original.
2. Transcribe with segment timestamps; word timestamps if supported.
3. Save untouched machine output as raw transcript.
4. Correct only clear recognition, punctuation, segmentation, named-entity, homophone, and numeric errors.
5. Maintain an uncertainty list with timestamps and alternatives.
6. Re-listen to:
   - title/opening thesis;
   - names, numbers, products, quoted phrases;
   - every excerpt considered for a card;
   - low-confidence or semantically broken spans.
7. Save corrected transcript separately. Never use analysis prose as a transcript repair source.

If the user asks for strict verbatim output, retain fillers, repetitions, self-corrections, and incomplete sentences. A readable edition may be produced as an additional file, clearly labelled.

## Stage V3 — Visual and sound timeline

Sample the full duration and use scene/shot changes, not a fixed handful of screenshots. Build a timeline whose granularity follows the content.

For each meaningful segment record as applicable:

- timestamp range and spoken function;
- shot size, camera position/movement, subject/action;
- on-screen text and caption role;
- B-roll, demonstration, screen recording, product detail, transition;
- cut frequency, hold time, pattern interrupt;
- music/sound effect/silence and relationship to speech;
- why the visual exists: evidence, explanation, emotion, pacing, proof, or decoration.

Run targeted frame OCR only for named, decision-relevant on-screen text when the task explicitly needs visual analysis. Keep the frame locator and raw/corrected OCR. Never run full-duration or per-frame OCR merely to fill an empty transcript.

## Stage V4 — Carrier-specific analysis

Always analyze only applicable dimensions.

### Viewpoint and cognition

- Central claim, subclaims, assumptions, evidence, causal chain, counterposition.
- What is genuinely non-obvious versus merely confidently phrased.
- Which exact lines are quote-worthy and why.

### Content and structure

- Hook promise, information gaps, section sequence, examples/proof, payoff, ending action.
- What knowledge the creator assumes.
- Compression: what could be removed without loss, and what needed more support.

### Delivery and rhythm

- Sentence length, pauses, emphasis, speed changes, repetitions, emotional curve.
- Information density versus processing time.
- How voice, subtitles, B-roll, and cuts share the explanatory load.

### Shots and production

- Reusable shot units, transition logic, coverage, continuity, prop/location/light use.
- Whether the production pattern transfers to a different category.
- Cost/complexity and minimum viable recreation.

### Track-specific emphasis

- `AI工具与教程`: inspect the problem, promised result, inputs, tools, steps, proof, reproducibility, and screen demonstration.
- `AI产品与趋势`: inspect facts, author judgment, causal chain, evidence, counterexample, prediction, and conditions.
- `创作者经营与方法论`: inspect target audience, creation problem, opening promise, method, information release, edit rhythm, shots/captions, and CTA where present.
- `商业品牌与职业`: inspect market observations, mechanism judgment, evidence, counterexample, action conditions, and risks.
- `产品与消费决策`: inspect use or purchase setting, decision criteria, experience or case evidence, comparison basis, suitability, risk, and conditions.
- `项目展示与创作过程`: inspect motivation, artifact/result, key choices, making process, proof, constraints, and retrospective insight.
- `文化与叙事`: inspect scene or person, observation/material, narrative movement, viewpoint or emotional turn, ending, and source boundaries.
- `审美策略与灵感`: inspect visual proposition, selection/order rule, visual system, sound-image relation, emotional/informational rhythm, and transfer conditions.
- `生活成长与思考`: inspect concrete situation, tension, experience or method, cognitive turn, scoped conclusion, and identity-specific constraint.
- `社会人文观察`: inspect question framing, concepts, evidence, explanatory frame, factual claim versus value judgment, missing context, and counterargument.

## Stage V5 — Package and cards

Before lexical/audio review, `标准内容.md` may expose only a clearly labelled machine timeline. The finished, user-facing transcript is a single reviewed document organized into semantic paragraphs: each paragraph has a bounded time range and a concise topic label, followed by its full corrected wording. Retain granular ASR timestamps in evidence; do not make readers navigate line-by-line timestamps.

For the chapter reading layer, follow `chapter-contract.md`: review the chapter title independently from its complete continuous transcript paragraphs. A chapter is not an executive summary, and the frontend does not split ASR rows into reader items. `梳理后内容.md` contains separate analysis using the chosen track and mechanism rather than a generic ten-section template.

Before inferring any boundary from ASR, inspect structured source chapters and the complete shownote/description for a valid time axis. Source chapters win over machine inference. Preserve exact shownote labels and their timestamps, including an explicit `开场` for preceding speech; use ASR semantic boundary scoring only when the source supplies no usable chapter structure.

Automatic candidate assets use `source-assets-contract.md` and may include only:

- viewpoint or quote cards tied to timestamps;
- title/hook card tied to opening promise and payoff;
- structure/rhythm card with a reproducible beat sheet;
Images, frames, and shots remain source evidence during automatic extraction.
They become image/shot cards only after explicit user interaction produces a
standardized `user_injected` asset.

For an `analyze`, `cards`, or `full` Xiaohongshu/Douyin source eligible under `comment-signals.md`, execute that adaptive comment step before completion. `capture`, `standard`, and below-threshold sources settle as `skipped_by_policy` without a request. Keep commenter viewpoints distinct from the creator's transcript. Add useful audience questions, disagreements, missing proof, or follow-up demands to the processed content; write `no_meaningful_signal` when the retained comments add no useful direction.

Also execute `cover-analysis.md`: preserve and OCR the platform cover, classify its promise/visual mechanism, and verify whether the video actually pays it off. Cover and opening frame may differ; analyze the chosen platform cover, not an arbitrary first frame.

## Video completion checklist

- G0–G4 pass for `analyze`; G5 additionally passes for `cards/full`.
- The independent comment and cover gates pass for Xiaohongshu/Douyin `analyze/cards/full` jobs.
- Transcript excerpts used in analysis are audio-verified. Unreviewed ASR may support only provisional structure and visual review, never quote cards or source-wording claims.
- Visual claims cover the entire relevant video, not only the opening.
- Audio-only and visually inaccessible areas are declared.
- No generic ten-section template overrides the chosen track, mechanism, and source structure.

#!/usr/bin/env python3
"""Commit a page-complete image-post OCR fidelity review transactionally."""

from __future__ import annotations

import argparse
import hashlib
import json
import os
import re
import tempfile
from datetime import datetime
from pathlib import Path

from gate_ledger import sync_gate_ledger
from validate_source_assets import validate_asset_bundle


SCHEMA = "image-text-fidelity-review/v1"


def sha256(path: Path) -> str:
    value = hashlib.sha256()
    with path.open("rb") as handle:
        for chunk in iter(lambda: handle.read(1024 * 1024), b""):
            value.update(chunk)
    return value.hexdigest()


def atomic_json(path: Path, value: dict) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    descriptor, temporary = tempfile.mkstemp(prefix=f".{path.name}.", suffix=".tmp", dir=path.parent)
    try:
        with os.fdopen(descriptor, "w", encoding="utf-8") as handle:
            json.dump(value, handle, ensure_ascii=False, indent=2)
            handle.write("\n")
        os.replace(temporary, path)
    finally:
        if os.path.exists(temporary):
            os.unlink(temporary)


def corrected_pages(package: Path) -> dict[int, Path]:
    pages: dict[int, Path] = {}
    for path in (package / "evidence/ocr").glob("image-*.corrected.md"):
        match = re.fullmatch(r"image-(\d+)\.corrected\.md", path.name)
        if match:
            pages[int(match.group(1))] = path
    return pages


def validate_receipt(package: Path, manifest: dict, receipt: dict) -> tuple[list[str], dict]:
    errors: list[str] = []
    if receipt.get("schema_version") != SCHEMA:
        errors.append(f"schema_version must be {SCHEMA}")
    if not str(receipt.get("reviewed_by") or "").strip():
        errors.append("reviewed_by is required")
    if not str(receipt.get("reviewed_at") or "").strip():
        errors.append("reviewed_at is required")
    if manifest.get("content_form") != "image_post":
        errors.append("package content_form must be image_post")

    inventory = {
        int(item.get("index")): item
        for item in manifest.get("image_inventory") or []
        if isinstance(item, dict) and str(item.get("index") or "").isdigit()
    }
    corrected = corrected_pages(package)
    raw_pages = {
        int(match.group(1))
        for path in (package / "evidence/ocr").glob("image-*.raw.txt")
        if (match := re.fullmatch(r"image-(\d+)\.raw\.txt", path.name))
    }
    expected = sorted(inventory)
    missing_raw = sorted(set(expected) - raw_pages)
    missing_corrected = sorted(set(expected) - set(corrected))
    unexpected_raw = sorted(raw_pages - set(expected))
    unexpected_corrected = sorted(set(corrected) - set(expected))
    if not expected:
        errors.append("image_inventory must contain at least one source image")
    if missing_raw:
        errors.append(f"source images lack raw OCR: {missing_raw}")
    if missing_corrected:
        errors.append(f"source images lack corrected OCR: {missing_corrected}")
    if unexpected_raw:
        errors.append(f"raw OCR contains unknown source indices: {unexpected_raw}")
    if unexpected_corrected:
        errors.append(f"corrected OCR contains unknown source indices: {unexpected_corrected}")
    decision_path = package / "evidence/ocr/decision.json"
    try:
        decision = json.loads(decision_path.read_text("utf-8"))
    except (OSError, json.JSONDecodeError):
        decision = {}
    if decision.get("status") != "all_pages_ocr" or decision.get("policy") != "mandatory_per_image_v1":
        errors.append("OCR decision must use mandatory_per_image_v1 for every image post")
    rows = receipt.get("pages")
    if not isinstance(rows, list):
        errors.append("pages must be an array")
        rows = []
    by_index = {int(row.get("image_index")): row for row in rows if isinstance(row, dict) and str(row.get("image_index") or "").isdigit()}
    if sorted(by_index) != expected:
        errors.append(f"pages must cover corrected OCR indices exactly once: {expected}")

    normalized_rows = []
    for index in expected:
        row = by_index.get(index) or {}
        item = inventory.get(index) or {}
        if index not in corrected:
            continue
        source_relative = str(item.get("relative_path") or "")
        source_path = package / source_relative
        corrected_path = corrected[index]
        if not source_relative or not source_path.is_file():
            errors.append(f"page {index} source image is missing")
            continue
        source_digest = sha256(source_path)
        corrected_digest = sha256(corrected_path)
        if item.get("sha256") and item.get("sha256") != source_digest:
            errors.append(f"page {index} source image no longer matches manifest")
        if row.get("source_sha256") != source_digest:
            errors.append(f"page {index} source_sha256 does not match current image")
        if row.get("corrected_sha256") != corrected_digest:
            errors.append(f"page {index} corrected_sha256 does not match current text")
        if row.get("status") != "verified":
            errors.append(f"page {index} status must be verified")
        normalized_rows.append({
            "image_index": index,
            "status": "verified",
            "source_artifact": source_relative,
            "source_sha256": source_digest,
            "corrected_artifact": str(corrected_path.relative_to(package)),
            "corrected_sha256": corrected_digest,
            "notes": str(row.get("notes") or "").strip(),
        })
    normalized = {
        "schema_version": SCHEMA,
        "reviewed_by": str(receipt.get("reviewed_by") or "").strip(),
        "reviewed_at": str(receipt.get("reviewed_at") or "").strip(),
        "coverage": "all_source_images",
        "pages": normalized_rows,
    }
    return errors, normalized


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--package", required=True, type=Path)
    parser.add_argument("--input", required=True, type=Path)
    args = parser.parse_args()
    package = args.package.expanduser().resolve()
    receipt = json.loads(args.input.expanduser().resolve().read_text("utf-8"))
    manifest_path = package / "manifest.json"
    manifest = json.loads(manifest_path.read_text("utf-8"))
    errors, normalized = validate_receipt(package, manifest, receipt)
    if errors:
        print("\n".join(errors))
        return 2

    atomic_json(package / "evidence/ocr/fidelity-review.json", normalized)
    manifest = json.loads(manifest_path.read_text("utf-8"))
    manifest.setdefault("gates", {})["extraction"] = "passed"
    manifest["gates"]["fidelity"] = "passed"
    completed = {"extraction", "fidelity"}
    asset_path = package / "evidence/assets/assets.json"
    asset_errors = ["asset bundle is missing"]
    if asset_path.is_file():
        assets = json.loads(asset_path.read_text("utf-8"))
        protected_user_ids = {
            str(item.get("id"))
            for item in assets.get("assets") or []
            if isinstance(item, dict) and item.get("origin") == "user_injected" and item.get("id")
        }
        asset_errors = validate_asset_bundle(
            package,
            assets,
            allow_review=False,
            preserved_user_asset_ids=protected_user_ids,
        )
    if not asset_errors:
        for gate in ("analysis", "cards"):
            if manifest["gates"].get(gate) == "passed":
                completed.add(gate)
    manifest["updated_at"] = datetime.now().astimezone().isoformat(timespec="seconds")
    sync_gate_ledger(package, manifest, write=True, completed_gates=completed)
    print(json.dumps({"status": "passed", "package": str(package), "pages": len(normalized["pages"])}, ensure_ascii=False))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

#!/usr/bin/env python3
"""User-global, evidence-backed terms for every compatible ASR invocation."""

from __future__ import annotations

import json
import os
import re
import tempfile
from datetime import datetime
from pathlib import Path
from typing import Any

ASR_LEXICON_ENV = "CREATOR_INGEST_ASR_LEXICON"
PROMOTABLE_CATEGORIES = {"named_entity", "program_or_work", "brand_or_product", "english_term"}
MAX_PROMPT_TERMS = 32
MAX_PROMPT_CHARS = 420
SEED_PATH = Path(__file__).resolve().parents[1] / "assets" / "asr-lexicon.seed.json"


def now_iso() -> str:
    return datetime.now().astimezone().isoformat(timespec="seconds")


def default_lexicon_path() -> Path:
    configured = os.getenv(ASR_LEXICON_ENV)
    if configured:
        return Path(configured).expanduser().resolve()
    xdg_config_home = os.getenv("XDG_CONFIG_HOME")
    if xdg_config_home:
        return Path(xdg_config_home).expanduser().resolve() / "creator-content-ingest" / "asr-lexicon.json"
    return Path.home() / ".config" / "creator-content-ingest" / "asr-lexicon.json"


def _clean(value: Any) -> str:
    return re.sub(r"\s+", " ", str(value or "")).strip()


def _key(value: Any) -> str:
    return re.sub(r"[\s._-]+", "", _clean(value)).casefold()


def _empty() -> dict:
    return {"schema": "creator-asr-lexicon/v1", "version": 1, "terms": []}


def _load_file(path: Path) -> dict:
    try:
        value = json.loads(path.read_text("utf-8"))
    except (OSError, json.JSONDecodeError):
        return _empty()
    if not isinstance(value, dict) or value.get("schema") != "creator-asr-lexicon/v1":
        return _empty()
    return value


def load_asr_lexicon(path: Path | None = None) -> dict:
    """Load the user-global lexicon, falling back to the shipped seed on first use."""
    source = (path or default_lexicon_path()).expanduser().resolve()
    value = _load_file(source) if source.is_file() else _load_file(SEED_PATH)
    terms = []
    for item in value.get("terms") or []:
        if not isinstance(item, dict):
            continue
        term = _clean(item.get("term"))
        if not term:
            continue
        aliases = [_clean(alias) for alias in item.get("aliases") or []]
        terms.append({**item, "term": term, "aliases": [alias for alias in aliases if alias]})
    return {**_empty(), **value, "terms": terms}


def prompt_terms(lexicon: dict | None, manual_terms: list[str] | None = None) -> list[str]:
    """Return bounded verified spellings for an ASR initial prompt."""
    values = [
        _clean(entry.get("term"))
        for entry in (lexicon or {}).get("terms") or []
        if isinstance(entry, dict) and entry.get("status", "verified") == "verified"
    ]
    values.extend(_clean(value) for value in manual_terms or [])
    output: list[str] = []
    seen: set[str] = set()
    size = 0
    for term in values:
        key = _key(term)
        if not term or not key or key in seen:
            continue
        projected = size + len(term) + (1 if output else 0)
        if len(output) >= MAX_PROMPT_TERMS or projected > MAX_PROMPT_CHARS:
            break
        output.append(term)
        seen.add(key)
        size = projected
    return output


def _atomic_json(path: Path, value: Any) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    descriptor, temporary = tempfile.mkstemp(prefix=f".{path.name}.", dir=path.parent)
    try:
        with os.fdopen(descriptor, "w", encoding="utf-8") as handle:
            json.dump(value, handle, ensure_ascii=False, indent=2)
            handle.write("\n")
            handle.flush()
            os.fsync(handle.fileno())
        os.replace(temporary, path)
    finally:
        if os.path.exists(temporary):
            os.unlink(temporary)


def record_verified_term(
    term: str,
    *,
    category: str,
    mistaken_form: str = "",
    source_package: str = "",
    reviewer: str = "",
    path: Path | None = None,
) -> dict | None:
    """Promote a confirmed correction for use by every future ASR run."""
    canonical, mistaken = _clean(term), _clean(mistaken_form)
    if category not in PROMOTABLE_CATEGORIES or not (2 <= len(canonical) <= 64):
        return None
    target = (path or default_lexicon_path()).expanduser().resolve()
    lexicon = load_asr_lexicon(target)
    entries = lexicon.setdefault("terms", [])
    canonical_key = _key(canonical)
    entry = next((item for item in entries if _key(item.get("term")) == canonical_key), None)
    if entry is None:
        entry = {
            "term": canonical,
            "aliases": [],
            "category": category,
            "status": "verified",
            "source": "confirmed_transcript_correction",
            "first_confirmed_at": now_iso(),
            "confirmations": [],
        }
        entries.append(entry)
    aliases = [_clean(value) for value in entry.get("aliases") or []]
    if mistaken and _key(mistaken) != canonical_key and mistaken not in aliases:
        aliases.append(mistaken)
    entry["aliases"] = aliases[:24]
    entry["category"] = entry.get("category") or category
    entry["status"] = "verified"
    confirmations = [item for item in entry.get("confirmations") or [] if isinstance(item, dict)]
    confirmations.append({
        "mistaken_form": mistaken or None,
        "source_package": source_package or None,
        "reviewer": reviewer or None,
        "confirmed_at": now_iso(),
    })
    entry["confirmations"] = confirmations[-20:]
    lexicon["updated_at"] = now_iso()
    _atomic_json(target, lexicon)
    return {"term": entry["term"], "aliases": entry["aliases"], "path": str(target)}

#!/usr/bin/env python3
"""Archive a publisher-authored timestamped transcript linked by a source package."""

from __future__ import annotations

import argparse
import hashlib
import json
import re
import subprocess
from datetime import datetime
from pathlib import Path

from manifest_changes import mark_manifest_change


def now_iso() -> str:
    return datetime.now().astimezone().isoformat(timespec="seconds")


def sha256(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def atomic_json(path: Path, value: object) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    temporary = path.with_suffix(path.suffix + ".tmp")
    temporary.write_text(json.dumps(value, ensure_ascii=False, indent=2) + "\n", "utf-8")
    temporary.replace(path)


def decode_next_transcript(html: str) -> list[dict]:
    """Decode timestamp/speaker/content rows from the page's Next.js payload."""
    rows: list[dict] = []
    for match in re.finditer(r'content\\":\\"((?:\\\\.|[^"\\\\])*)\\"', html):
        prefix = html[max(0, match.start() - 1800):match.start()]
        times = re.findall(r'children\\":\\"(\d\d:\d\d:\d\d)\\"', prefix)
        values = re.findall(r'children\\":\\"([^"\\]+)\\"', prefix)
        if not times:
            continue
        timestamp = times[-1]
        speaker = next((
            value for value in reversed(values)
            if value != timestamp and not re.fullmatch(r"\d\d:\d\d:\d\d", value)
        ), "")
        try:
            content = json.loads('"' + match.group(1) + '"')
        except json.JSONDecodeError:
            continue
        row = {"timestamp": timestamp, "speaker": speaker, "content": content.strip()}
        if row["content"] and (not rows or row != rows[-1]):
            rows.append(row)
    return rows


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--package", required=True, type=Path)
    parser.add_argument("--url", required=True)
    args = parser.parse_args()

    package = args.package.expanduser().resolve()
    manifest_path = package / "manifest.json"
    manifest = json.loads(manifest_path.read_text("utf-8"))
    fetched = subprocess.run(
        ["curl", "-sSL", "--fail", args.url], check=True, text=True, capture_output=True,
    ).stdout
    rows = decode_next_transcript(fetched)
    if len(rows) < 5:
        raise ValueError("publisher page did not expose a usable timestamped transcript")

    output = package / "evidence/source/publisher-transcript.json"
    record = {
        "schema_version": "publisher-transcript/v1",
        "source_url": args.url,
        "captured_at": now_iso(),
        "authorship": "publisher_linked_transcript",
        "fidelity_notice": "publisher_transcript_may_contain_typos_or_recognition_errors",
        "review_policy": "use_with_bounded_audio_for_spoken_quote_verification",
        "segments": rows,
    }
    atomic_json(output, record)

    source = manifest.setdefault("source", {})
    source["publisher_transcript"] = {
        "url": args.url,
        "path": str(output.relative_to(package)),
        "segments": len(rows),
        "captured_at": record["captured_at"],
        "status": "archived_review_evidence",
    }
    artifacts = manifest.setdefault("artifacts", [])
    relative = str(output.relative_to(package))
    artifacts[:] = [item for item in artifacts if not isinstance(item, dict) or item.get("path") != relative]
    artifacts.append({"path": relative, "kind": "publisher_transcript", "sha256": sha256(output)})
    mark_manifest_change(manifest, "agent_analysis", now_iso())
    atomic_json(manifest_path, manifest)
    print(json.dumps({"status": "archived", "package": str(package), "path": relative, "segments": len(rows)}, ensure_ascii=False))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

#!/usr/bin/env python3
"""Acquire supported social/video/article sources into canonical evidence packages."""

from __future__ import annotations

import argparse
import fcntl
import hashlib
import html
import importlib.util
import json
import mimetypes
import os
import re
import shutil
import subprocess
import sys
import tempfile
import time
from contextlib import contextmanager
from datetime import datetime
from pathlib import Path
from typing import Any
from urllib.parse import parse_qsl, urlencode, unquote, urlparse, urlunparse

import requests

SCRIPT_DIR = Path(__file__).resolve().parent
if str(SCRIPT_DIR) not in sys.path:
    sys.path.insert(0, str(SCRIPT_DIR))
from social_author_tags import normalize_social_author_tags
from ocr_organization import organize_ocr_records
from comment_selection import (
    comment_enrichment_decision,
    deferred_comment_result,
    select_comments,
    selection_policy,
)
from title_normalization import douyin_display_title
from package_document import assign_package_document
from bs4 import BeautifulSoup, NavigableString, Tag
from PIL import Image

UA = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/124 Safari/537.36"
HEADERS = {"User-Agent": UA, "Accept-Language": "zh-CN,zh;q=0.9,en;q=0.7"}
XHS_HOSTS = ("xiaohongshu.com", "xhslink.cn", "xhslink.com")
DOUYIN_HOSTS = ("douyin.com", "iesdouyin.com")
BILIBILI_HOSTS = ("bilibili.com", "b23.tv")
XIAOYUZHOU_HOSTS = ("xiaoyuzhoufm.com",)
TRACKING_QUERY_KEYS = {
    "fbclid", "gclid", "msclkid", "mc_cid", "mc_eid", "_hsenc", "_hsmi",
    "yclid", "igshid", "ref", "referrer",
}
TRANSIENT_QUERY_KEYS = {
    "access_token", "auth", "auth_token", "credential", "expires", "expiry", "key", "key-pair-id",
    "nonce", "pass_ticket", "policy", "session", "session_id", "sessionid", "share_token", "sid",
    "signature", "sig", "timestamp", "token", "ts", "web_session", "wx_header",
}
TRANSIENT_QUERY_PREFIXES = ("x-amz-", "x-goog-", "x-oss-", "x-cos-", "x-ms-")
WECHAT_IDENTITY_QUERY_KEYS = {"__biz", "mid", "idx", "sn", "chksm"}


def now_iso() -> str:
    return datetime.now().astimezone().isoformat(timespec="seconds")


def sha256_file(path: Path) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        for chunk in iter(lambda: handle.read(1024 * 1024), b""):
            digest.update(chunk)
    return digest.hexdigest()


def atomic_text(path: Path, text: str) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    fd, temp = tempfile.mkstemp(prefix=f".{path.name}.", dir=path.parent)
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as handle:
            handle.write(text)
            handle.flush()
            os.fsync(handle.fileno())
        os.replace(temp, path)
    finally:
        if os.path.exists(temp):
            os.unlink(temp)


def atomic_json(path: Path, value: Any) -> None:
    atomic_text(path, json.dumps(value, ensure_ascii=False, indent=2) + "\n")


@contextmanager
def package_capture_lock(output_root: Path, platform: str, stable_id: str):
    """Serialize the first writer for one canonical package across processes."""
    package_id = f"S-{platform}-{safe_component(stable_id, limit=48)}"
    lock_root = output_root.expanduser().resolve() / ".capture-locks"
    lock_root.mkdir(parents=True, exist_ok=True)
    lock_path = lock_root / f"{package_id}.lock"
    with lock_path.open("a+", encoding="utf-8") as handle:
        fcntl.flock(handle.fileno(), fcntl.LOCK_EX)
        try:
            yield
        finally:
            fcntl.flock(handle.fileno(), fcntl.LOCK_UN)


def safe_component(value: str, fallback: str = "unknown", limit: int = 80) -> str:
    clean = re.sub(r"[\\/:*?\"<>|\x00-\x1f]", "-", value or "")
    clean = re.sub(r"\s+", " ", clean).strip(" .-")
    return (clean or fallback)[:limit]


def canonical_without_query(url: str) -> str:
    parsed = urlparse(url)
    return urlunparse((parsed.scheme or "https", parsed.netloc, parsed.path, "", "", ""))


def redact_transient_error(value: object) -> str:
    """Keep signed XHS carrier values out of durable stderr capture."""
    return re.sub(
        r"(?i)(xsec_token(?:=|%3d))[^&\s\"'<>]+",
        r"\1[redacted]",
        str(value or ""),
    )


def canonical_public_url(url: str) -> str:
    """Preserve query fields that select content while dropping trackers and signed access material."""
    parsed = urlparse(url)
    hostname = (parsed.hostname or "").lower()
    try:
        port = parsed.port
    except ValueError:
        port = None
    netloc = f"[{hostname}]" if ":" in hostname and not hostname.startswith("[") else hostname
    if port:
        netloc = f"{netloc}:{port}"
    wechat_article = hostname == "mp.weixin.qq.com" and parsed.path.rstrip("/") == "/s"
    wechat_slug = hostname == "mp.weixin.qq.com" and parsed.path.startswith("/s/")
    query_items = []
    for key, value in parse_qsl(parsed.query, keep_blank_values=True):
        lowered = key.lower()
        if wechat_slug:
            continue
        if wechat_article and lowered not in WECHAT_IDENTITY_QUERY_KEYS:
            continue
        if lowered.startswith(("utm_", *TRANSIENT_QUERY_PREFIXES)) or lowered in TRACKING_QUERY_KEYS or lowered in TRANSIENT_QUERY_KEYS:
            continue
        query_items.append((key, value))
    return urlunparse(((parsed.scheme or "https").lower(), netloc, parsed.path, "", urlencode(sorted(query_items), doseq=True), ""))


SHARE_URL_RE = re.compile(r"https?://[^\s<>\[\]{}\"']+")
XHS_REQUEST_QUERY_KEYS = {
    "app_platform",
    "app_version",
    "source",
    "share_from_user_hidden",
    "xsec_source",
    "type",
    "xsec_token",
    "author_share",
    "xhsshare",
    "shareRedId",
    "apptime",
    "share_id",
}


def clean_shared_url(value: str) -> str:
    return html.unescape(str(value or "")).rstrip(".,;:!?，。；：！？、）)]}>'\"")


def is_xhs_host(host: str) -> bool:
    clean = (host or "").split(":", 1)[0].lower().rstrip(".")
    return any(clean == name or clean.endswith(f".{name}") for name in XHS_HOSTS)


def xhs_acquisition_url(explicit_url: str, raw_input: str, ephemeral_share_url: str | None = None) -> tuple[str, str | None]:
    """Return a public detail carrier; logged state is never used by this process."""
    if ephemeral_share_url:
        expected_note_id = xhs_note_id(explicit_url)
        if not expected_note_id:
            raise ValueError("an ephemeral share carrier requires a stable child note URL")
        candidate = clean_shared_url(ephemeral_share_url)
        if not is_xhs_host(urlparse(candidate).netloc):
            raise ValueError("ephemeral share carrier must use an official Xiaohongshu host")
        if xhs_note_id(candidate) != expected_note_id:
            raise ValueError("ephemeral share carrier must identify the same Xiaohongshu note")
        if "xsec_token=" not in urlparse(candidate).query:
            raise ValueError("ephemeral share carrier must contain a collection-issued xsec token")
        return candidate, expected_note_id
    return xhs_url_from_raw_input(explicit_url, raw_input), None


def xhs_url_from_raw_input(explicit_url: str, raw_input: str) -> str:
    """Prefer the user's original XHS share URL over a token-stripped derivative."""
    explicit = clean_shared_url(explicit_url)
    parsed_explicit = urlparse(explicit)
    if "xsec_token=" in parsed_explicit.query:
        return explicit
    explicit_note = re.search(r"/(?:explore|discovery/item)/([0-9a-f]{24})(?:/|$)", parsed_explicit.path, re.I)
    candidates = []
    for match in SHARE_URL_RE.findall(raw_input or ""):
        candidate = clean_shared_url(match)
        host = urlparse(candidate).netloc.lower()
        if any(name in host for name in XHS_HOSTS):
            candidates.append(candidate)
    if not candidates:
        return explicit

    # A collection child is captured with an explicit note URL while raw_input
    # still contains the parent board URL. Never let that parent carrier replace
    # the child identity: doing so routes image/video notes through the generic
    # board-page path. A token-bearing URL is useful only when it identifies the
    # same child note.
    if explicit_note:
        note_id = explicit_note.group(1).lower()
        matching_notes = []
        for candidate in candidates:
            parsed = urlparse(candidate)
            match = re.search(r"/(?:explore|discovery/item)/([0-9a-f]{24})(?:/|$)", parsed.path, re.I)
            if match and match.group(1).lower() == note_id:
                matching_notes.append(candidate)
        signed_match = next((item for item in matching_notes if "xsec_token=" in urlparse(item).query), None)
        return signed_match or explicit

    def rank(candidate: str) -> tuple[int, int]:
        parsed = urlparse(candidate)
        host = parsed.netloc.lower()
        if "xsec_token=" in parsed.query:
            priority = 4
        elif "xhslink." in host:
            priority = 3
        elif "/explore/" in parsed.path or "/discovery/item/" in parsed.path:
            priority = 2
        else:
            priority = 1
        return priority, len(candidate)

    return max(candidates, key=rank)


def recover_xhs_target_from_security_url(value: str) -> str | None:
    """Recover the token-bearing note URL nested inside an XHS security redirect."""
    parsed = urlparse(value)
    query_items = parse_qsl(parsed.query, keep_blank_values=True)
    query = dict(query_items)
    nested = unquote(query.get("redirectPath") or "")
    if not nested.startswith(("http://", "https://")):
        return None
    nested_parsed = urlparse(nested)
    if not any(name in nested_parsed.netloc.lower() for name in XHS_HOSTS):
        return None
    merged = dict(parse_qsl(nested_parsed.query, keep_blank_values=True))
    for key, item in query_items:
        if key in XHS_REQUEST_QUERY_KEYS and item:
            merged.setdefault(key, item)
    return urlunparse(
        (
            nested_parsed.scheme or "https",
            nested_parsed.netloc,
            nested_parsed.path,
            "",
            urlencode(merged),
            "",
        )
    )


def parse_count(value: Any) -> int | None:
    if value is None:
        return None
    raw = str(value).strip().lower().replace(",", "")
    if not raw:
        return None
    multiplier = 1
    if raw.endswith("万") or raw.endswith("w"):
        multiplier, raw = 10000, raw[:-1]
    elif raw.endswith("k"):
        multiplier, raw = 1000, raw[:-1]
    try:
        return int(float(raw) * multiplier)
    except ValueError:
        return None


def epoch_iso(value: Any) -> str | None:
    try:
        number = float(value)
        if number > 10_000_000_000:
            number /= 1000
        return datetime.fromtimestamp(number).astimezone().isoformat(timespec="seconds")
    except (TypeError, ValueError, OSError):
        return None


def xhs_display_title(title: Any, description: Any, fallback: str) -> str:
    """Keep an XHS card title single-line when the platform title is absent."""
    explicit = re.sub(r"\s+", " ", str(title or "")).strip()
    if explicit:
        return explicit[:120]
    lines = [re.sub(r"\s+", " ", line).strip() for line in str(description or "").splitlines()]
    first = next((line for line in lines if line), "")
    if not first:
        return fallback
    first = re.split(r"\s*#", first, maxsplit=1)[0].strip() or first
    if len(first) <= 80:
        return first
    sentence = re.match(r"^(.{4,80}?[。！？!?])", first)
    return sentence.group(1).strip() if sentence else first[:79].rstrip() + "…"


def douyin_display_author(info: dict) -> str:
    """Prefer a visible nickname and never expose a numeric account ID as the author."""
    identity_values = {
        str(info.get("uploader_id") or "").strip(),
        str(info.get("channel_id") or "").strip(),
    }
    for key in ("channel", "creator", "artist", "uploader"):
        candidate = re.sub(r"\s+", " ", str(info.get(key) or "")).strip()
        if not candidate or candidate in identity_values or re.fullmatch(r"[\d_\-]+", candidate):
            continue
        return candidate
    return ""


def douyin_body_capture_status(body: str) -> str:
    # Treat the creator-authored Douyin description as the source body.
    # A platform warning shown beside it does not make the captured body partial.
    return "passed"


def request(session: requests.Session, url: str, *, referer: str | None = None, stream: bool = False) -> requests.Response:
    headers = dict(HEADERS)
    if referer:
        headers["Referer"] = referer
    response = session.get(url, headers=headers, timeout=30, allow_redirects=True, stream=stream)
    response.raise_for_status()
    return response


def initial_state(page: str) -> dict:
    marker = "window.__INITIAL_STATE__="
    start = page.find(marker)
    if start < 0:
        raise ValueError("XHS initial state not found")
    start += len(marker)
    end = page.find("</script>", start)
    if end < 0:
        raise ValueError("XHS initial-state script is not terminated")
    raw = page[start:end].strip().rstrip(";")
    raw = re.sub(r"\bundefined\b", "null", raw)
    value = json.loads(raw)
    if not isinstance(value, dict):
        raise ValueError("XHS initial state is not an object")
    return value


def normalize_xhs_comments(entry: dict) -> dict:
    container = entry.get("comments") or {}
    items = container.get("list") if isinstance(container, dict) else container
    if not isinstance(items, list) or not items:
        return {
            "status": "unavailable",
            "ranking_scope": "not_in_public_payload",
            "sample_size": 0,
            "has_more": bool(container.get("hasMore")) if isinstance(container, dict) else None,
            "acquired_at": now_iso(),
            "method": "xhs_initial_state",
            "comments": [],
            "reason": "Public initial state did not include comments; the public comment scan is complete and no authenticated retry is required.",
            "topic_signal_status": "no_meaningful_signal",
            "topic_signals": [],
        }
    normalized = []
    for item in items:
        if not isinstance(item, dict):
            continue
        user = item.get("userInfo") or item.get("user") or {}
        raw_likes = item.get("likeCount") or item.get("likedCount") or item.get("likes")
        text = str(item.get("content") or item.get("text") or "").strip()
        if not text:
            continue
        cid = str(item.get("id") or item.get("commentId") or "")
        normalized.append({
            "comment_id": cid or None,
            "author_display": user.get("nickname") or user.get("name") or "",
            "text": text,
            "like_count": parse_count(raw_likes),
            "like_count_raw": str(raw_likes or "0"),
            "reply_count": parse_count(item.get("subCommentCount") or item.get("replyCount")),
            "created_at": epoch_iso(item.get("createTime") or item.get("time")),
            "locator": f"comment:{cid}" if cid else None,
        })
    top, discarded = select_comments(normalized)
    return {
        "status": "passed",
        "ranking_scope": "initial_state_sample_sorted_by_likes",
        "sample_size": len(normalized),
        "has_more": bool(container.get("hasMore")) if isinstance(container, dict) else None,
        "acquired_at": now_iso(),
        "method": "xhs_initial_state",
        "comments": top,
        "selection_policy": selection_policy(),
        "discarded": discarded,
        "topic_signal_status": "pending_analysis" if top else "no_meaningful_signal",
        "topic_signals": [],
        **({"reason": "The public sample contained no comment meeting the like and information-quality thresholds."} if not top else {}),
    }


def xhs_image_urls(image: dict) -> list[str]:
    """Return ordered public image routes without persisting signed URLs."""
    info = [item for item in image.get("infoList") or [] if isinstance(item, dict)]
    values = [image.get("urlDefault")]
    values.extend(
        item.get("url")
        for item in info
        if item.get("imageScene") in ("WB_DFT", "CRD_DFT")
    )
    values.extend([image.get("urlPre"), image.get("url")])
    values.extend(
        item.get("url")
        for item in info
        if item.get("imageScene") in ("WB_PRV", "CRD_PRV")
    )
    values.extend(item.get("url") for item in info)
    urls = []
    for value in values:
        if not value:
            continue
        normalized = str(value).replace("http://", "https://", 1)
        if normalized not in urls:
            urls.append(normalized)
    return urls


def xhs_image_url(image: dict) -> str | None:
    """Compatibility accessor for the highest-priority public image route."""
    return next(iter(xhs_image_urls(image)), None)


def xhs_stream_candidates(streams: dict, source: str) -> list[dict]:
    candidates = []
    for codec in ("h264", "h265", "h266", "av1"):
        for stream in streams.get(codec) or []:
            routes = [("master", stream.get("masterUrl"))]
            routes.extend(
                (f"backup_{index}", value)
                for index, value in enumerate(stream.get("backupUrls") or [], 1)
            )
            for route, url in routes:
                if not url:
                    continue
                candidates.append(
                    {
                        "codec": codec,
                        "source": f"{source}.{codec}",
                        "quality_type": stream.get("qualityType"),
                        "selection_priority": 1,
                        "transport_priority": 2 if route == "master" else 1,
                        "transport_route": route,
                        "ephemeral_url": str(url),
                        **{
                            key: stream.get(key)
                            for key in (
                                "width",
                                "height",
                                "size",
                                "duration",
                                "fps",
                                "videoBitrate",
                                "weight",
                            )
                        },
                    }
                )
    candidates.sort(
        key=lambda item: (
            int(item.get("width") or 0) * int(item.get("height") or 0),
            int(item.get("videoBitrate") or 0),
            int(item.get("size") or 0),
            int(item.get("weight") or 0),
            int(item.get("transport_priority") or 0),
        ),
        reverse=True,
    )
    unique = []
    seen = set()
    for candidate in candidates:
        url = candidate["ephemeral_url"]
        if url in seen:
            continue
        seen.add(url)
        unique.append(candidate)
    return unique


def first_public_image_url(value: Any) -> str | None:
    if isinstance(value, str) and value:
        return value
    if not isinstance(value, dict):
        return None
    for key in ("urlDefault", "url", "url_list"):
        candidate = value.get(key)
        if isinstance(candidate, str) and candidate:
            return candidate
        if isinstance(candidate, list):
            for item in candidate:
                if isinstance(item, str) and item:
                    return item
    return None


def xhs_note_id(value: str) -> str | None:
    match = re.search(r"/(?:explore|discovery/item)/([0-9a-f]{24})(?:/|$)", urlparse(value).path, re.I)
    return match.group(1).lower() if match else None


def xhs_public_route_candidates(source_url: str) -> list[str]:
    """Build a bounded, no-login route ladder for transient public-page failures."""
    candidates = [source_url]
    note_id = xhs_note_id(source_url)
    if note_id:
        parsed = urlparse(source_url)
        acquisition_query = [
            (key, value)
            for key, value in parse_qsl(parsed.query, keep_blank_values=True)
            if key in XHS_REQUEST_QUERY_KEYS and value
        ]
        if any(key == "xsec_token" for key, _ in acquisition_query):
            alternate_path = (
                f"/discovery/item/{note_id}"
                if "/explore/" in parsed.path
                else f"/explore/{note_id}"
            )
            candidates.append(urlunparse((
                "https",
                "www.xiaohongshu.com",
                alternate_path,
                "",
                urlencode(acquisition_query),
                "",
            )))
        candidates.extend([
            f"https://www.xiaohongshu.com/explore/{note_id}",
            f"https://www.xiaohongshu.com/discovery/item/{note_id}",
        ])
    return list(dict.fromkeys(candidates))


def extract_xhs(session: requests.Session, source_url: str) -> dict:
    expected_note_id = xhs_note_id(source_url)
    last_error: Exception | None = None
    response = None
    state = None
    # Two short rounds cover transient security redirects without an unbounded
    # retry loop. The session is anonymous and never imports browser cookies.
    for attempt in range(2):
        for candidate in xhs_public_route_candidates(source_url):
            try:
                response = request(session, candidate, referer=source_url if candidate != source_url else None)
                recovered_url = recover_xhs_target_from_security_url(response.url)
                if recovered_url and recovered_url != candidate:
                    response = request(session, recovered_url, referer=source_url)
                state = initial_state(response.text)
                break
            except (ValueError, json.JSONDecodeError, requests.RequestException) as exc:
                last_error = exc
        if state is not None:
            break
        if attempt == 0:
            time.sleep(0.35)
    if state is None or response is None:
        raise ValueError(
            "XHS public no-login acquisition did not expose note detail after bounded retries; "
            "no browser or account login fallback was used."
        ) from last_error
    note_store = state.get("note") or {}
    detail_map = note_store.get("noteDetailMap") or {}
    note_id = note_store.get("currentNoteId") or note_store.get("firstNoteId")
    entry = detail_map.get(note_id) if note_id else None
    if not entry and detail_map:
        note_id, entry = next(iter(detail_map.items()))
    note = (entry or {}).get("note") or {}
    note_id = str(note.get("noteId") or note_id or "")
    if not note_id or not note:
        raise ValueError("XHS note payload is empty; the share link may be expired or unavailable")
    if expected_note_id and note_id.lower() != expected_note_id:
        raise ValueError("XHS public response identity did not match the requested collection child")
    user = note.get("user") or {}
    avatar_url = (
        first_public_image_url(user.get("avatar"))
        or first_public_image_url(user.get("image"))
        or first_public_image_url(user.get("avatarUrl"))
    )
    images = []
    for index, image in enumerate(note.get("imageList") or [], 1):
        media_urls = xhs_image_urls(image)
        if not media_urls:
            continue
        is_live_photo = bool(image.get("livePhoto"))
        images.append({
            "index": index,
            "role": "cover" if index == 1 else "content",
            "ephemeral_url": media_urls[0],
            "ephemeral_urls": media_urls,
            "expected_width": image.get("width"),
            "expected_height": image.get("height"),
            "live_photo": is_live_photo,
            "motion_candidates": (
                xhs_stream_candidates(image.get("stream") or {}, "imageList.stream")
                if is_live_photo
                else []
            ),
        })
    video = note.get("video") or {}
    consumer_origin = (((video.get("consumer") or {}).get("originVideo") or {}).get("url"))
    streams = ((video.get("media") or {}).get("stream") or {})
    candidates = []
    if consumer_origin:
        candidates.append({
            "codec": "origin",
            "source": "video.consumer.originVideo",
            "selection_priority": 2,
            "transport_priority": 2,
            "transport_route": "origin",
            "ephemeral_url": str(consumer_origin),
        })
    candidates.extend(xhs_stream_candidates(streams, "video.media.stream"))
    candidates.sort(key=lambda x: (
        int(x.get("selection_priority") or 0),
        int(x.get("width") or 0) * int(x.get("height") or 0),
        int(x.get("videoBitrate") or 0),
        int(x.get("size") or 0),
        int(x.get("transport_priority") or 0),
    ), reverse=True)
    is_video = note.get("type") == "video" or bool(candidates)
    canonical = f"https://www.xiaohongshu.com/discovery/item/{note_id}"
    interact = note.get("interactInfo") or {}
    return {
        "platform": "xhs",
        "platform_display_name": "小红书",
        "content_form": "video" if is_video else "image_post",
        "stable_id": note_id,
        "canonical_url": canonical,
        "resolved_url_redacted": canonical_without_query(response.url),
        "title": xhs_display_title(note.get("title"), note.get("desc"), note_id),
        "author": user.get("nickname") or "",
        "author_profile": {
            "display_name": user.get("nickname") or "",
            "platform_user_id": user.get("userId") or user.get("userId64") or None,
            "public_handle": user.get("redId") or None,
            "avatar_candidate": avatar_url,
        },
        "published_at": epoch_iso(note.get("time")),
        "updated_at": epoch_iso(note.get("lastUpdateTime")),
        "body": note.get("desc") or "",
        "tags": [tag.get("name") for tag in note.get("tagList") or [] if isinstance(tag, dict) and tag.get("name")],
        "metrics": {
            "likes": parse_count(interact.get("likedCount")),
            "collects": parse_count(interact.get("collectedCount")),
            "comments": parse_count(interact.get("commentCount")),
            "shares": parse_count(interact.get("shareCount")),
        },
        "metrics_captured_at": now_iso(),
        "images": images,
        "live_photo_count": sum(1 for image in images if image.get("live_photo")),
        "video_candidates": candidates,
        "comments": normalize_xhs_comments(entry or {}),
        "capture_method": "public_share_initial_state_no_login",
    }


def ytdlp_command(platform: str) -> list[str]:
    """Resolve yt-dlp in both shell and embedded desktop runtimes."""
    executable = shutil.which("yt-dlp")
    if executable:
        return [executable]
    if importlib.util.find_spec("yt_dlp") is not None:
        return [sys.executable, "-m", "yt_dlp"]
    raise ValueError(f"yt-dlp is required for {platform} capture")


def run_ytdlp_json(source_url: str, browser: str, platform: str) -> dict:
    command = [
        *ytdlp_command(platform),
        "--simulate",
        "--no-playlist",
        "--no-warnings",
        "--socket-timeout",
        "15",
        "--extractor-retries",
        "1",
        "--print",
        "%(.{id,display_id,title,description,uploader,uploader_id,channel,channel_id,creator,artist,timestamp,duration,thumbnail,like_count,comment_count,repost_count,tags,webpage_url})j",
        source_url,
    ]
    if platform.lower() != "douyin":
        command[8:8] = ["--cookies-from-browser", browser]
    result = subprocess.run(command, text=True, capture_output=True, timeout=120)
    if result.returncode != 0 or not result.stdout.strip():
        detail = (result.stderr or result.stdout).strip().splitlines()
        message = detail[-1] if detail else "unknown yt-dlp error"
        raise ValueError(f"{platform} metadata acquisition failed: {message}")
    value = json.loads(result.stdout)
    if not isinstance(value, dict) or not value.get("id"):
        raise ValueError(f"{platform} metadata payload is incomplete")
    return value


def extract_douyin(source_url: str, browser: str, mode: str = "full") -> dict:
    resolved = source_url
    if not re.search(r"/(?:note|video)/\d+", urlparse(source_url).path):
        resolver = requests.Session()
        resolved = resolver.get(source_url, headers=HEADERS, timeout=30, allow_redirects=True).url
    note_match = re.search(r"/note/(\d+)", resolved)
    if note_match:
        from douyin_web_capture import capture_image_note

        return capture_image_note(note_match.group(1), browser, mode, source_url)
    video_match = re.search(r"/video/(\d+)", resolved)
    if video_match:
        from douyin_web_capture import capture_video_note

        return capture_video_note(video_match.group(1), browser, mode, source_url)
    raise ValueError("Anonymous Douyin redirect did not resolve to a supported public video or image-note detail URL")


def bilibili_content_key(*urls: str) -> tuple[str, str]:
    for url in urls:
        bvid = re.search(r"/video/(BV[0-9A-Za-z]+)", str(url or ""), re.I)
        if bvid:
            return "bvid", bvid.group(1)
        avid = re.search(r"/video/(av\d+)", str(url or ""), re.I)
        if avid:
            return "aid", avid.group(1)[2:]
    raise ValueError("Bilibili metadata did not expose a canonical video ID")


def bilibili_public_video_metadata(session: requests.Session, key: str, value: str, referer: str) -> dict:
    response = request(
        session,
        f"https://api.bilibili.com/x/web-interface/view?{key}={value}",
        referer=referer,
    )
    payload = response.json()
    data = (payload.get("data") or {}) if isinstance(payload, dict) else {}
    owner = data.get("owner") or {}
    avatar = first_public_image_url(owner.get("face"))
    return {
        **data,
        "author_profile": {
            "display_name": str(owner.get("name") or "").strip(),
            "platform_user_id": str(owner.get("mid") or "").strip() or None,
            "avatar_candidate": avatar or None,
        },
    }


def bilibili_public_tags(session: requests.Session, key: str, value: str, referer: str) -> list[str] | None:
    """Return public archive tags, or None so the existing yt-dlp fallback can run."""
    try:
        response = request(
            session,
            f"https://api.bilibili.com/x/tag/archive/tags?{key}={value}",
            referer=referer,
        )
        payload = response.json()
        data = payload.get("data") if isinstance(payload, dict) else None
        if not isinstance(data, list):
            return None
        return [str(item.get("tag_name") or "").strip() for item in data if isinstance(item, dict) and item.get("tag_name")]
    except (ValueError, requests.RequestException):
        return None


def bilibili_public_comments(session: requests.Session, aid: Any, referer: str) -> dict:
    if not aid:
        return {
            "status": "review",
            "ranking_scope": "missing_video_aid",
            "sample_size": 0,
            "has_more": None,
            "acquired_at": now_iso(),
            "method": "bilibili_anonymous_guest_hot_comments_api",
            "comments": [],
            "reason": "Bilibili metadata did not expose the numeric video ID required for comments.",
        }
    try:
        response = session.get(
            f"https://api.bilibili.com/x/v2/reply/main?type=1&oid={aid}&mode=3&next=0&ps=20",
            headers={"User-Agent": "Mozilla/5.0", "Referer": referer, "Accept-Language": HEADERS["Accept-Language"]},
            timeout=30,
            allow_redirects=True,
        )
        response.raise_for_status()
        payload = response.json()
        if not isinstance(payload, dict) or payload.get("code") != 0:
            raise ValueError(str(payload.get("message") or payload.get("code") or "invalid response"))
        data = payload.get("data") or {}
        replies = data.get("replies") or []
        normalized = []
        for item in replies if isinstance(replies, list) else []:
            if not isinstance(item, dict):
                continue
            content = item.get("content") or {}
            member = item.get("member") or {}
            text = str(content.get("message") or "").strip()
            if not text:
                continue
            comment_id = str(item.get("rpid_str") or item.get("rpid") or "")
            normalized.append({
                "comment_id": comment_id or None,
                "author_display": str(member.get("uname") or ""),
                "text": text,
                "like_count": parse_count(item.get("like")),
                "like_count_raw": str(item.get("like") if item.get("like") is not None else "unknown"),
                "like_count_precision": "exact" if item.get("like") is not None else "unknown",
                "reply_count": parse_count(item.get("rcount")),
                "created_at": epoch_iso(item.get("ctime")),
                "locator": f"comment:{comment_id}" if comment_id else None,
            })
        top, discarded = select_comments(normalized)
        cursor = data.get("cursor") or {}
        all_count = parse_count(cursor.get("all_count")) if isinstance(cursor, dict) else None
        has_more = (
            all_count > len(normalized)
            if all_count is not None
            else not bool(cursor.get("is_end")) if isinstance(cursor, dict) else None
        )
        return {
            "status": "passed",
            "ranking_scope": "public_hot_page_1_sorted_by_likes",
            "sample_size": len(normalized),
            "has_more": has_more,
            "acquired_at": now_iso(),
            "method": "bilibili_anonymous_guest_hot_comments_api",
            "comments": top,
            "selection_policy": selection_policy(),
            "discarded": discarded,
            "topic_signal_status": "pending_analysis" if top else "no_meaningful_signal",
            "topic_signals": [],
            **({"reason": "The public hot sample contained no comment meeting the like and information-quality thresholds."} if not top else {}),
        }
    except Exception as exc:
        return {
            "status": "review",
            "ranking_scope": "public_hot_page_1_unavailable",
            "sample_size": 0,
            "has_more": None,
            "acquired_at": now_iso(),
            "method": "bilibili_anonymous_guest_hot_comments_api",
            "comments": [],
            "reason": f"Bilibili public hot-comment capture failed: {exc}",
        }


def extract_bilibili(session: requests.Session, source_url: str, browser: str, mode: str = "full") -> dict:
    info: dict = {}
    try:
        key, stable_id = bilibili_content_key(source_url)
        webpage = source_url
    except ValueError:
        info = run_ytdlp_json(source_url, browser, "Bilibili")
        webpage = str(info.get("webpage_url") or source_url)
        key, stable_id = bilibili_content_key(webpage, source_url)
    canonical = f"https://www.bilibili.com/video/{stable_id if key == 'bvid' else f'av{stable_id}'}"
    public = bilibili_public_video_metadata(session, key, stable_id, canonical)
    tags = bilibili_public_tags(session, key, stable_id, canonical)
    if tags is None:
        if not info:
            info = run_ytdlp_json(source_url, browser, "Bilibili")
        tags = [str(tag) for tag in info.get("tags") or [] if tag]
    profile = public["author_profile"]
    stat = public.get("stat") or {}
    author = profile["display_name"] or str(info.get("uploader") or info.get("channel") or "")
    images = []
    thumbnail = public.get("pic") or info.get("thumbnail")
    if thumbnail:
        images.append({
            "index": 1,
            "role": "cover",
            "ephemeral_url": str(thumbnail),
            "expected_width": None,
            "expected_height": None,
        })
    metrics = {
        "likes": parse_count(stat.get("like") if "like" in stat else info.get("like_count")),
        "collects": parse_count(stat.get("favorite")),
        "comments": parse_count(stat.get("reply") if "reply" in stat else info.get("comment_count")),
        "shares": parse_count(stat.get("share")),
    }
    comment_decision = comment_enrichment_decision("bilibili", mode, metrics)
    comments = (
        bilibili_public_comments(session, public.get("aid"), canonical)
        if comment_decision["eligible"]
        else deferred_comment_result(comment_decision)
    )
    if comments.get("status") == "passed":
        comments["enrichment_decision"] = comment_decision
    return {
        "platform": "bilibili",
        "platform_display_name": "Bilibili",
        "content_form": "video",
        "stable_id": stable_id,
        "canonical_url": canonical,
        "resolved_url_redacted": canonical_without_query(webpage),
        "title": str(public.get("title") or info.get("title") or stable_id),
        "author": author,
        "author_profile": profile,
        "published_at": epoch_iso(public.get("pubdate") or info.get("timestamp")),
        "updated_at": None,
        "body": str(public.get("desc") or info.get("description") or ""),
        "tags": tags,
        "metrics": metrics,
        "metrics_captured_at": now_iso(),
        "duration_seconds": public.get("duration") or info.get("duration"),
        "images": images,
        "video_candidates": [{"method": "yt_dlp_browser_session", "browser": browser}],
        "comments": comments,
        "capture_method": "bilibili_public_apis_with_ytdlp_media_download",
        "download_source_url": source_url,
    }


def xiaoyuzhou_next_data(page: str) -> dict:
    soup = BeautifulSoup(page, "html.parser")
    node = soup.select_one("#__NEXT_DATA__")
    if not node or not node.string:
        raise ValueError("Xiaoyuzhou Next.js episode payload not found")
    value = json.loads(node.string)
    page_props = ((value.get("props") or {}).get("pageProps") or {})
    if not isinstance(page_props, dict):
        raise ValueError("Xiaoyuzhou episode payload is invalid")
    return page_props


def xiaoyuzhou_shownotes_markdown(shownotes: str) -> tuple[str, list[dict], list[dict]]:
    """Keep publisher-authored shownote order while exposing links and images."""
    soup = BeautifulSoup(str(shownotes or ""), "html.parser")
    links: list[dict] = []
    images: list[dict] = []
    blocks: list[str] = []
    pending_image_caption = ""

    def explicit_image_caption(node: Any) -> str:
        if not isinstance(node, Tag) or node.name not in {"p", "div", "figcaption"}:
            return ""
        plain = re.sub(r"[*_`\[\]]", "", node.get_text(" ", strip=True))
        match = re.search(r"(?:相关图片|配图|图片)[：:]\s*(.+)$", plain)
        return match.group(1).strip() if match else ""

    def image_markdown(node: Tag, caption: str = "") -> str:
        source_url = html.unescape(str(node.get("src") or node.get("data-src") or "")).strip()
        if not source_url.startswith(("http://", "https://")):
            return ""
        placeholder = f"[[XIAOYUZHOU_IMAGE_{len(images) + 1:03d}]]"
        images.append({
            "placeholder": placeholder,
            "ephemeral_url": source_url,
            "alt": str(node.get("alt") or "").strip(),
            "caption": caption,
        })
        return placeholder

    def inline_markdown(node: Any, image_caption: str = "") -> str:
        if isinstance(node, NavigableString):
            return str(node)
        if not isinstance(node, Tag):
            return ""
        if node.name == "br":
            return "\n"
        if node.name == "img":
            return image_markdown(node, image_caption)
        child_caption = image_caption
        if node.name == "figure":
            figcaption = node.find("figcaption")
            if figcaption:
                child_caption = figcaption.get_text(" ", strip=True) or image_caption
        child_text = []
        local_caption = child_caption
        for child in node.children:
            explicit_caption = explicit_image_caption(child)
            image_count_before = len(images)
            child_text.append(inline_markdown(child, explicit_caption or local_caption))
            if len(images) > image_count_before:
                local_caption = ""
            elif explicit_caption:
                local_caption = explicit_caption
        if node.name in {"ul", "ol"}:
            return "\n".join(part.strip() for part in child_text if part.strip())
        text = re.sub(r"[ \t]+", " ", "".join(child_text))
        clean_text = text.strip()
        if node.name == "a":
            href = html.unescape(str(node.get("href") or "")).strip()
            if href.startswith(("http://", "https://")) and clean_text:
                record = {"title": clean_text, "url": href}
                if record not in links:
                    links.append(record)
                return f"[{clean_text}]({href})"
        if node.name in {"strong", "b"} and clean_text:
            return f"**{clean_text}**"
        if node.name in {"em", "i"} and clean_text:
            return f"*{clean_text}*"
        if node.name == "li" and clean_text:
            return f"- {clean_text}"
        return text

    container = soup.body or soup
    for node in container.children:
        if isinstance(node, NavigableString):
            text = re.sub(r"\s+", " ", str(node)).strip()
            if text:
                blocks.append(text)
            continue
        if not isinstance(node, Tag):
            continue
        explicit_caption = explicit_image_caption(node)
        block_caption = explicit_caption or pending_image_caption
        image_count_before = len(images)
        text = inline_markdown(node, block_caption)
        text = re.sub(r"\n{3,}", "\n\n", text).strip()
        if not text:
            continue
        if node.name in {"h1", "h2", "h3", "h4", "h5", "h6"}:
            level = min(4, int(node.name[1]) + 1)
            text = f"{'#' * level} {text}"
        blocks.append(text)
        pending_image_caption = "" if len(images) > image_count_before else explicit_caption
    markdown = "\n\n".join(blocks).strip()
    return normalize_xiaoyuzhou_shownote_layout(markdown), links, images


def normalize_xiaoyuzhou_shownote_layout(markdown: str) -> str:
    """Restore timestamp rows collapsed by nested rich-text containers."""
    timeline_labels = ("时间轴", "时间线", "本期你会听到", "本期答疑清单", "节目时间轴", "本期内容")
    end_labels = ("猜你想看", "相关链接", "参考链接", "节目链接", "资料链接", "延伸阅读", "本期相关内容", "相关内容", "本期主播", "主播", "联系我们")
    timestamp = r"(?:(?:\d{1,2})[:：])?\d{2}[:：]\d{2}"
    output: list[str] = []
    in_timeline = False
    for raw_line in str(markdown or "").splitlines():
        line = raw_line.strip()
        if not line:
            output.append("")
            continue
        plain = re.sub(r"[*_`#\[\]]", "", line)
        normalized = re.sub(r"^[^\w\u4e00-\u9fff]+", "", plain).strip("【】 ").rstrip("：:")
        if any(normalized.startswith(label) for label in timeline_labels):
            in_timeline = True
        elif any(normalized.startswith(label) for label in end_labels):
            in_timeline = False
        timestamp_matches = list(re.finditer(rf"(?<![\d:]){timestamp}(?=[ \t]*\S)", line))
        collapsed_timeline = bool(re.match(rf"^{timestamp}", line)) and len(timestamp_matches) > 1
        if in_timeline or collapsed_timeline:
            # Rich-text containers can collapse adjacent <p> nodes, but never
            # split from the middle of a timestamp (for example ``00:02:13``).
            line = re.sub(rf"(?<!^)(?<![\d:\n])(?={timestamp})", "\n\n", line)
            # Publisher HTML is inconsistent about inserting whitespace after
            # the time anchor. Store one canonical separator so every consumer
            # sees the same ``timestamp title`` row shape.
            line = "\n".join(
                re.sub(
                    rf"^({timestamp})[ \t]*(?=\S)",
                    lambda match: f"{match.group(1).replace('：', ':')} ",
                    part.strip(),
                )
                for part in line.splitlines()
            )
        output.extend(line.splitlines())
    return re.sub(r"\n{3,}", "\n\n", "\n".join(output)).strip()


def xiaoyuzhou_shownote_times(description: str) -> dict:
    """Separate navigational timeline entries from timestamped reference notes."""
    timeline: list[dict] = []
    timestamp_notes: list[dict] = []
    pattern = re.compile(r"^(?:(\d{1,2})[:：])?(\d{1,2})[:：](\d{2})\s*(.+)$")
    timeline_labels = ("时间轴", "时间线", "本期你会听到", "本期答疑清单", "本期节目清单", "节目时间轴", "本期内容")
    highlight_reference_labels = ("猜你想看",)
    link_reference_labels = ("相关链接", "参考链接", "节目链接", "资料链接", "延伸阅读")
    reference_labels = highlight_reference_labels + link_reference_labels
    source_lines = [line.strip() for line in str(description or "").splitlines() if line.strip()]
    normalized_lines = [re.sub(r"^[^\w\u4e00-\u9fff]+", "", line).strip("【】[] ") for line in source_lines]
    has_explicit_timeline = any(any(line.startswith(label) for label in timeline_labels) for line in normalized_lines)
    section = "timeline" if not has_explicit_timeline else "other"
    reference_kind = ""
    pending_group: dict | None = None
    timed_group_markers: list[dict] = []
    pending_reference: dict | None = None
    group_sequence = 0

    def new_group(title: str, *, number: str | None = None, start: float | None = None, evidence: str) -> dict:
        nonlocal group_sequence
        group_sequence += 1
        group = {
            "group_title": title.strip(),
            "group_number": number,
            "group_evidence": evidence,
            "_group_key": f"shownote-group-{group_sequence}",
        }
        if start is not None:
            group["group_start"] = float(start)
        return group

    def untimed_group_marker(raw_line: str) -> dict | None:
        """Recognize source-authored parent syntax, never plain prose alone."""
        clean = raw_line.strip().lstrip("-–—• ")
        heading = re.match(r"^#{1,6}\s+(.+)$", clean)
        if heading:
            title = re.sub(r"^\*\*|\*\*$", "", heading.group(1).strip()).strip()
            return new_group(title, evidence="markdown_heading") if title else None
        bold = re.fullmatch(r"\*\*\s*(.+?)\s*\*\*", clean)
        if bold:
            return new_group(bold.group(1), evidence="bold_heading")
        numbered = re.match(
            r"^(?:(\d{1,2})(?:[️⃣①-⑳❶-❿]+|[.、．|｜]\s*|\s+))(.+)$",
            clean,
        )
        if numbered and not pattern.match(clean):
            return new_group(numbered.group(2), number=numbered.group(1), evidence="numbered_heading")
        chinese = re.match(r"^(?:([一二三四五六七八九十]+)[、.．]|第([一二三四五六七八九十\d]+)(?:章|部分|节)[：:]?)\s*(.+)$", clean)
        if chinese:
            return new_group(chinese.group(3), number=chinese.group(1) or chinese.group(2), evidence="numbered_heading")
        bracketed = re.fullmatch(r"[【\[]\s*([^\]】]+?)\s*[\]】]", clean)
        if bracketed:
            return new_group(bracketed.group(1), evidence="bracket_heading")
        return None

    def flush_pending_reference() -> None:
        nonlocal pending_reference
        if pending_reference:
            timestamp_notes.append(pending_reference)
            pending_reference = None

    for raw_line, normalized in zip(source_lines, normalized_lines):
        if any(normalized.startswith(label) for label in timeline_labels):
            flush_pending_reference()
            section, pending_group, reference_kind = "timeline", None, ""
            continue
        if any(normalized.startswith(label) for label in reference_labels):
            flush_pending_reference()
            section, pending_group = "reference", None
            reference_kind = "links" if any(normalized.startswith(label) for label in link_reference_labels) else "highlights"
            continue
        if section == "timeline":
            group_marker = untimed_group_marker(raw_line)
            if group_marker:
                pending_group = group_marker
                continue
        if section == "reference" and re.match(r"^(?:#{1,6}\s*)?[🎧🌐🎺🍻🔗]", raw_line):
            flush_pending_reference()
            section = "other"
            reference_kind = ""
            continue
        clean = raw_line.lstrip("-–—• ")
        match = pattern.match(clean)
        if not match:
            if pending_reference:
                flush_pending_reference()
            continue
        hours, minutes, seconds, title = match.groups()
        start = int(hours or 0) * 3600 + int(minutes) * 60 + int(seconds)
        if not title.strip():
            continue
        timed_parent = re.fullmatch(r"\*\*\s*(.+?)\s*\*\*", title.strip())
        if section == "timeline" and timed_parent:
            parent_title = timed_parent.group(1).strip()
            marker = {
                "start": float(start),
                "title": parent_title,
                "source": "xiaoyuzhou_shownotes",
                "key": f"{float(start):.3f}:{parent_title}",
            }
            timed_group_markers.append(marker)
            pending_group = new_group(parent_title, start=float(start), evidence="timed_bold_heading")
            pending_group["_timed_group_key"] = marker["key"]
            continue
        # Some publishers put link/cover metadata before an otherwise plain
        # timeline without adding a new timeline heading. A zero-based run is
        # unambiguous navigation, not a timestamped reference list.
        if section == "reference" and not has_explicit_timeline and reference_kind == "links" and start == 0:
            section = "timeline"
            pending_reference = None
        if section == "reference":
            item = {"start": float(start), "title": title.strip(), "section": "猜你想看", "kind": "reference"}
            if not has_explicit_timeline and reference_kind == "links":
                if pending_reference and start > pending_reference["start"]:
                    section = "timeline"
                    timeline.extend([
                        {"start": pending_reference["start"], "title": pending_reference["title"], "source": "xiaoyuzhou_shownotes"},
                        {"start": float(start), "title": title.strip(), "source": "xiaoyuzhou_shownotes"},
                    ])
                    pending_reference = None
                else:
                    flush_pending_reference()
                    pending_reference = item
            else:
                timestamp_notes.append(item)
        elif section == "timeline" and (not timeline or start > timeline[-1]["start"]):
            item = {"start": float(start), "title": title.strip(), "source": "xiaoyuzhou_shownotes"}
            if pending_group:
                item.update(pending_group)
            timeline.append(item)
    flush_pending_reference()
    grouped_counts: dict[str, int] = {}
    grouped_first_start: dict[str, float] = {}
    for item in timeline:
        key = str(item.get("_group_key") or "")
        if key:
            grouped_counts[key] = grouped_counts.get(key, 0) + 1
            grouped_first_start.setdefault(key, float(item["start"]))
    for item in timeline:
        key = str(item.get("_group_key") or "")
        if key and grouped_counts.get(key, 0) >= 2:
            item["outline_level"] = 2
            item.setdefault("group_start", grouped_first_start[key])
        else:
            item.pop("group_number", None)
            item.pop("group_title", None)
            item.pop("group_start", None)
            item.pop("group_evidence", None)
            item["outline_level"] = 1
        item.pop("_group_key", None)
        item.pop("_timed_group_key", None)
    retained_timed_groups = {
        f"{float(item.get('group_start')):.3f}:{item.get('group_title')}"
        for item in timeline if item.get("group_title") and item.get("group_start") is not None
    }
    for marker in timed_group_markers:
        if marker["key"] not in retained_timed_groups:
            timeline.append({**{key: marker[key] for key in ("start", "title", "source")}, "outline_level": 1})
    timeline.sort(key=lambda item: float(item.get("start") or 0))
    return {"timeline": timeline, "timestamp_notes": timestamp_notes}


def xiaoyuzhou_timeline(description: str) -> list[dict]:
    return xiaoyuzhou_shownote_times(description)["timeline"]


def normalize_xiaoyuzhou_comments(items: Any) -> dict:
    normalized = []
    for item in items if isinstance(items, list) else []:
        if not isinstance(item, dict) or not str(item.get("text") or "").strip():
            continue
        author = item.get("author") or {}
        comment_id = str(item.get("id") or "")
        normalized.append({
            "comment_id": comment_id or None,
            "author_display": author.get("nickname") or "",
            "text": str(item.get("text") or "").strip(),
            "like_count": parse_count(item.get("likeCount")),
            "like_count_raw": str(item.get("likeCount") if item.get("likeCount") is not None else "unknown"),
            "like_count_precision": "exact" if item.get("likeCount") is not None else "unknown",
            "reply_count": parse_count(item.get("replyCount") or item.get("threadReplyCount")),
            "created_at": item.get("createdAt") or None,
            "locator": f"comment:{comment_id}" if comment_id else None,
        })
    top, discarded = select_comments(normalized)
    return {
        "status": "passed",
        "ranking_scope": "public_page_initial_sample_sorted_by_likes",
        "sample_size": len(normalized),
        "has_more": len(normalized) >= 20,
        "acquired_at": now_iso(),
        "method": "xiaoyuzhou_next_data",
        "comments": top,
        "selection_policy": selection_policy(),
        "discarded": discarded,
        "topic_signal_status": "pending_analysis" if top else "no_meaningful_signal",
        "topic_signals": [],
        **({"reason": "The public sample contained no comment meeting the like and information-quality thresholds."} if normalized and not top else {"reason": "Public page did not expose a comment sample."} if not normalized else {}),
    }


def extract_xiaoyuzhou(session: requests.Session, source_url: str) -> dict:
    response = request(session, source_url)
    page_props = xiaoyuzhou_next_data(response.text)
    episode = page_props.get("episode") or {}
    episode_id = str(episode.get("eid") or page_props.get("id") or "")
    if not episode_id or not isinstance(episode, dict):
        raise ValueError("Xiaoyuzhou episode payload is empty")
    enclosure = episode.get("enclosure") or {}
    audio_url = enclosure.get("url") if isinstance(enclosure, dict) else enclosure
    if not audio_url:
        raise ValueError("Xiaoyuzhou episode has no public audio enclosure")
    podcast = episode.get("podcast") or {}
    podcast_image = podcast.get("image") or {}
    episode_image = episode.get("image") or {}
    cover_url = episode_image.get("largePicUrl") or episode_image.get("picUrl") or podcast_image.get("largePicUrl") or podcast_image.get("picUrl")
    podcast_title = str(podcast.get("title") or "")
    raw_shownotes = str(episode.get("shownotes") or "")
    shownote_markdown, shownote_links, shownote_images = xiaoyuzhou_shownotes_markdown(raw_shownotes)
    description = str(episode.get("description") or BeautifulSoup(raw_shownotes, "html.parser").get_text("\n", strip=True)).strip()
    body = shownote_markdown or description
    shownote_times = xiaoyuzhou_shownote_times(body)
    canonical = f"https://www.xiaoyuzhoufm.com/episode/{episode_id}"
    images = []
    if cover_url:
        images.append({
            "index": 1,
            "role": "cover",
            "ephemeral_url": str(cover_url),
            "expected_width": episode_image.get("width") or podcast_image.get("width"),
            "expected_height": episode_image.get("height") or podcast_image.get("height"),
            "alt": f"{episode.get('title') or episode_id} 封面",
        })
    for shownote_image in shownote_images:
        index = len(images) + 1
        source_placeholder = f"[[SOURCE_IMAGE_{index:03d}]]"
        body = body.replace(str(shownote_image["placeholder"]), source_placeholder)
        images.append({
            "index": index,
            "role": "shownote",
            "ephemeral_url": shownote_image["ephemeral_url"],
            "alt": shownote_image["alt"],
            "caption": shownote_image["caption"],
            "placeholder": source_placeholder,
        })
    return {
        "platform": "xiaoyuzhou",
        "platform_display_name": "小宇宙",
        "content_track": "播客节目",
        "content_form": "podcast_episode",
        "stable_id": episode_id,
        "canonical_url": canonical,
        "resolved_url_redacted": canonical_without_query(response.url),
        "title": str(episode.get("title") or episode_id),
        "author": podcast_title,
        "author_profile": {
            "display_name": podcast_title,
            "platform_user_id": podcast.get("pid") or episode.get("pid") or None,
            "public_handle": None,
            "avatar_candidate": podcast_image.get("largePicUrl") or podcast_image.get("picUrl") or None,
        },
        "published_at": episode.get("pubDate") or None,
        "updated_at": None,
        "description": description,
        "body": body,
        "shownote_links": shownote_links,
        "timeline": shownote_times["timeline"],
        "timestamp_notes": shownote_times["timestamp_notes"],
        "tags": [str(item) for item in episode.get("labels") or [] if isinstance(item, str)],
        "metrics": {
            "likes": parse_count(episode.get("clapCount")),
            "collects": parse_count(episode.get("favoriteCount")),
            "comments": parse_count(episode.get("commentCount")),
            "shares": None,
            "plays": parse_count(episode.get("playCount")),
        },
        "metrics_captured_at": now_iso(),
        "duration_seconds": episode.get("duration"),
        "images": images,
        "audio_candidates": [{
            "ephemeral_url": str(audio_url),
            "mime_type": (episode.get("media") or {}).get("mimeType"),
            "declared_bytes": (episode.get("media") or {}).get("size"),
            "source": "public_episode_enclosure",
        }],
        "comments": normalize_xiaoyuzhou_comments(page_props.get("comments")),
        "capture_method": "xiaoyuzhou_public_next_data",
    }


def meta_content(soup: BeautifulSoup, *selectors: str) -> str:
    for selector in selectors:
        node = soup.select_one(selector)
        if not node:
            continue
        value = node.get("content") if node.name == "meta" else node.get_text(" ", strip=True)
        if value:
            return str(value).strip()
    return ""


ARTICLE_CHROME = re.compile(
    r"nav|menu|footer|header|sidebar|aside|share|tool|related|recommend|"
    r"comment|subscribe|advert|social|login|breadcrumb|metadata",
    re.I,
)


def _node_identity(node: Tag) -> str:
    if not getattr(node, "attrs", None):
        return ""
    return " ".join([str(node.get("id") or ""), *(str(item) for item in (node.get("class") or []))])


def _article_score(node: Tag) -> float:
    text = re.sub(r"\s+", " ", node.get_text(" ", strip=True)).strip()
    if len(text) < 100:
        return float("-inf")
    paragraph_count = len(node.find_all(["p", "blockquote"], recursive=True))
    heading_count = len(node.find_all(["h1", "h2"

#!/usr/bin/env python3
"""Enumerate a public Xiaohongshu board without browser or account state."""

from __future__ import annotations

import argparse
import json
import re
import sys
import time
from datetime import datetime
from typing import Any
from urllib.parse import urljoin, urlparse

import requests

MOBILE_UA = (
    "Mozilla/5.0 (iPhone; CPU iPhone OS 18_0 like Mac OS X) "
    "AppleWebKit/605.1.15 Version/18.0 Mobile/15E148 Safari/604.1"
)
ALLOWED_HOSTS = {"xiaohongshu.com", "www.xiaohongshu.com", "m.xiaohongshu.com", "xhslink.cn", "www.xhslink.cn"}


def valid_url(value: str) -> str:
    parsed = urlparse(value)
    if parsed.scheme not in {"http", "https"} or (parsed.hostname or "").lower() not in ALLOWED_HOSTS:
        raise ValueError("unsupported Xiaohongshu board URL")
    if parsed.username or parsed.password:
        raise ValueError("URL must not contain credentials")
    return value


def fetch(url: str, timeout: float) -> tuple[str, str, int]:
    current = valid_url(url)
    headers = {"User-Agent": MOBILE_UA, "Accept": "text/html,application/xhtml+xml", "Accept-Language": "zh-CN,zh;q=0.9"}
    for _ in range(4):
        response = requests.get(current, headers=headers, timeout=timeout, allow_redirects=False, stream=True)
        if 300 <= response.status_code < 400:
            current = valid_url(urljoin(current, response.headers.get("location") or ""))
            continue
        response.raise_for_status()
        chunks, size = [], 0
        for chunk in response.iter_content(64 * 1024):
            size += len(chunk)
            if size > 5 * 1024 * 1024:
                raise ValueError("public board HTML exceeded 5 MiB")
            chunks.append(chunk)
        return b"".join(chunks).decode(response.encoding or "utf-8", errors="replace"), current, size
    raise ValueError("too many redirects")


def initial_state(page: str) -> dict[str, Any]:
    marker = re.search(r"window\.__INITIAL_STATE__\s*=\s*", page)
    if not marker or marker.end() >= len(page) or page[marker.end()] != "{":
        raise ValueError("XHS public board state not found")
    start, depth, quoted, escaped = marker.end(), 0, False, False
    for index in range(start, len(page)):
        char = page[index]
        if quoted:
            if escaped:
                escaped = False
            elif char == "\\":
                escaped = True
            elif char == '"':
                quoted = False
            continue
        if char == '"':
            quoted = True
        elif char == "{":
            depth += 1
        elif char == "}":
            depth -= 1
            if depth == 0:
                raw = re.sub(r"(?<![\w$])undefined(?![\w$])", "null", page[start:index + 1])
                value = json.loads(raw)
                if isinstance(value, dict):
                    return value
                break
    raise ValueError("XHS public board state was incomplete")


def fetch_public_board_items(board_id: str, board_url: str, timeout: float) -> list[dict[str, Any]]:
    """Use Xiaohongshu's own H5 board endpoint with anonymous-only signing."""
    try:
        from xhshow import Xhshow
    except ImportError:
        return []
    client = Xhshow()
    a1 = client.generate_a1()
    cookies = {
        "a1": a1,
        "webId": client.generate_web_id(a1),
        "webBuild": "4.83.0",
        "xsecappid": "xhs-pc-web",
    }
    if "web_session" in cookies:
        raise ValueError("account session must never enter public board enumeration")
    path = "/api/sns/h5/v1/board/note"
    params = {"board_id": board_id}
    signed = client.sign_headers_get(uri=path, cookies=cookies, params=params)
    response = requests.get(
        "https://edith.xiaohongshu.com" + path,
        params=params,
        headers={
            "User-Agent": MOBILE_UA,
            "Accept": "application/json, text/plain, */*",
            "Origin": "https://www.xiaohongshu.com",
            "Referer": board_url,
            **signed,
        },
        cookies=cookies,
        timeout=timeout,
    )
    response.raise_for_status()
    payload = response.json()
    items = (payload.get("data") or {}).get("items") if isinstance(payload, dict) else None
    return items if payload.get("success") is True and isinstance(items, list) else []


def text(value: Any, *keys: str) -> str:
    if not isinstance(value, dict):
        return ""
    return next((str(value[key]).strip() for key in keys if isinstance(value.get(key), str) and value[key].strip()), "")


def count(value: Any, *keys: str) -> int | None:
    if not isinstance(value, dict):
        return None
    for key in keys:
        raw = value.get(key)
        if isinstance(raw, int) and raw >= 0:
            return raw
        if isinstance(raw, str) and raw.isdigit():
            return int(raw)
    return None


def normalize(state: dict[str, Any], resolved_url: str, response_bytes: int, elapsed: float) -> dict[str, Any]:
    board = state.get("board") or {}
    info = board.get("boardInfo") or board.get("boardDetails") or {}
    notes = board.get("notes")
    cursor, has_more = None, None
    if not isinstance(notes, list):
        feed_map = board.get("boardFeedsMap") or {}
        feed = next((item for item in feed_map.values() if isinstance(item, dict)), {}) if isinstance(feed_map, dict) else {}
        notes = feed.get("notes") or []
        cursor = text(feed, "cursor") or None
        has_more = feed.get("hasMore") if isinstance(feed.get("hasMore"), bool) else None
    match = re.fullmatch(r"/board/([A-Za-z0-9_-]+)/*", urlparse(resolved_url).path)
    if not match:
        raise ValueError("resolved page is not a Xiaohongshu board")
    board_id = text(info, "id", "boardId", "board_id") or match.group(1)
    if board_id != match.group(1):
        raise ValueError("board identity mismatch")
    items, seen = [], set()
    for candidate in notes if isinstance(notes, list) else []:
        note_id = text(candidate, "noteId", "note_id", "id")
        if not note_id or note_id in seen:
            continue
        seen.add(note_id)
        token = text(candidate, "xsecToken", "xsec_token")
        source = "pc_board" if token else "yamcha_boardpage"
        query = f"?xsec_token={token}&xsec_source={source}" if token else f"?source={source}"
        items.append({
            "position": len(items) + 1,
            "note_id": note_id,
            "canonical_url": f"https://www.xiaohongshu.com/discovery/item/{note_id}",
            "acquisition_url": f"https://www.xiaohongshu.com/explore/{note_id}{query}",
            "title": text(candidate, "displayTitle", "title", "name"),
            "content_form_hint": "video" if text(candidate, "type") == "video" else "image_post",
            "author": {"display_name": text(candidate.get("user") or {}, "nickname", "displayName", "name")},
            "detail_access": "public_board_token" if token else "public_board_navigation_generated",
        })
    declared = count(info, "noteCount", "totalCount", "total", "count")
    declared = len(items) if declared is None else declared
    complete = len(items) == declared and has_more is not True
    enumeration_method = "anonymous_signed_h5_board_api" if board.get("anonymousH5Enumeration") else "public_mobile_ssr"
    return {
        "schema_version": "1.0",
        "kind": "xhs_public_collection",
        "status": "complete" if complete else "partial",
        "capture": {"method": enumeration_method, "authenticated": False, "browser_profile_used": False, "account_session_used": False, "retrieved_at": datetime.now().astimezone().isoformat(timespec="seconds"), "elapsed_seconds": round(elapsed, 3), "response_bytes": response_bytes},
        "collection": {"platform": "xhs", "board_id": board_id, "canonical_url": f"https://www.xiaohongshu.com/board/{board_id}", "title": text(info, "name", "title", "boardName"), "declared_count": declared},
        "coverage": {"declared_count": declared, "observed_count": len(items), "cursor": cursor, "has_more": False if complete else has_more, "complete": complete, "enumeration_method": enumeration_method, "reason": "declared_count_matched" if complete else "public_board_endpoint_returned_subset"},
        "items": items,
    }


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--url", required=True)
    parser.add_argument("--timeout", type=float, default=20)
    args = parser.parse_args()
    started = time.perf_counter()
    try:
        page, resolved, size = fetch(args.url, args.timeout)
        state = initial_state(page)
        board_match = re.fullmatch(r"/board/([A-Za-z0-9_-]+)/*", urlparse(resolved).path)
        if board_match:
            api_items = fetch_public_board_items(board_match.group(1), resolved, args.timeout)
            board = state.setdefault("board", {})
            if len(api_items) >= len(board.get("notes") or []):
                board["notes"] = api_items
                board["anonymousH5Enumeration"] = True
        print(json.dumps(normalize(state, resolved, size, time.perf_counter() - started), ensure_ascii=False))
        return 0
    except (ValueError, OSError, requests.RequestException, json.JSONDecodeError) as exc:
        print(json.dumps({"status": "failed", "kind": "xhs_public_collection", "message": str(exc)}, ensure_ascii=False), file=sys.stderr)
        return 2


if __name__ == "__main__":
    raise SystemExit(main())

#!/usr/bin/env python3
"""Create an honest partial source package from a public XHS board listing.

This is a bounded fallback for child notes whose detail page cannot be read
anonymously. It preserves only metadata and cover evidence actually present in
the collection listing. Expiring image URLs are used in memory and never stored.
"""

from __future__ import annotations

import argparse
import hashlib
import json
import mimetypes
import os
import re
import tempfile
from datetime import datetime
from pathlib import Path
from typing import Any
from urllib.parse import urlparse

import requests

MOBILE_UA = (
    "Mozilla/5.0 (iPhone; CPU iPhone OS 18_0 like Mac OS X) "
    "AppleWebKit/605.1.15 Version/18.0 Mobile/15E148 Safari/604.1"
)


def now_iso() -> str:
    return datetime.now().astimezone().isoformat(timespec="seconds")


def fetch_public_html(source_url: str, timeout: float) -> tuple[str, str, int]:
    response = requests.get(
        source_url,
        headers={"User-Agent": MOBILE_UA, "Accept": "text/html,application/xhtml+xml", "Accept-Language": "zh-CN,zh;q=0.9"},
        timeout=timeout,
    )
    response.raise_for_status()
    if "text/html" not in response.headers.get("content-type", "").lower():
        raise ValueError("XHS collection did not return HTML")
    if len(response.content) > 5 * 1024 * 1024:
        raise ValueError("XHS collection HTML exceeded 5 MiB")
    return response.text, response.url, len(response.content)


def replace_undefined(source: str) -> str:
    output: list[str] = []
    index = 0
    in_string = False
    escaped = False
    while index < len(source):
        char = source[index]
        if in_string:
            output.append(char)
            if escaped:
                escaped = False
            elif char == "\\":
                escaped = True
            elif char == '"':
                in_string = False
            index += 1
            continue
        if char == '"':
            in_string = True
            output.append(char)
            index += 1
            continue
        if source.startswith("undefined", index):
            output.append("null")
            index += len("undefined")
            continue
        output.append(char)
        index += 1
    return "".join(output)


def initial_state(page: str) -> dict[str, Any]:
    marker = re.search(r"window\.__INITIAL_STATE__\s*=\s*", page)
    if not marker or marker.end() >= len(page) or page[marker.end()] != "{":
        raise ValueError("XHS collection initial state was not found")
    start = marker.end()
    depth = 0
    in_string = False
    escaped = False
    for index in range(start, len(page)):
        char = page[index]
        if in_string:
            if escaped:
                escaped = False
            elif char == "\\":
                escaped = True
            elif char == '"':
                in_string = False
            continue
        if char == '"':
            in_string = True
        elif char == "{":
            depth += 1
        elif char == "}":
            depth -= 1
            if depth == 0:
                return json.loads(replace_undefined(page[start:index + 1]))
    raise ValueError("XHS collection initial state was incomplete")


def atomic_text(path: Path, value: str) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    fd, temporary = tempfile.mkstemp(prefix=f".{path.name}.", dir=path.parent)
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as handle:
            handle.write(value)
            handle.flush()
            os.fsync(handle.fileno())
        os.replace(temporary, path)
    finally:
        if os.path.exists(temporary):
            os.unlink(temporary)


def atomic_json(path: Path, value: Any) -> None:
    atomic_text(path, json.dumps(value, ensure_ascii=False, indent=2) + "\n")


def note_id_from_url(value: str) -> str:
    parsed = urlparse(value)
    match = re.search(r"/(?:explore|discovery/item)/([A-Za-z0-9_-]{6,64})(?:/|$)", parsed.path)
    if not match:
        raise ValueError("XHS collection child URL has no stable note ID")
    return match.group(1).lower()


def board_notes(state: dict[str, Any]) -> list[dict[str, Any]]:
    board = state.get("board") or {}
    notes = board.get("notes")
    if isinstance(notes, list):
        return [item for item in notes if isinstance(item, dict)]
    feed_map = board.get("boardFeedsMap") or {}
    for feed in feed_map.values() if isinstance(feed_map, dict) else []:
        if isinstance(feed, dict) and isinstance(feed.get("notes"), list):
            return [item for item in feed["notes"] if isinstance(item, dict)]
    return []


def timestamp(value: Any) -> str | None:
    try:
        number = float(value)
        if number > 10_000_000_000:
            number /= 1000
        return datetime.fromtimestamp(number).astimezone().isoformat(timespec="seconds")
    except (TypeError, ValueError, OSError):
        return None


def cover_candidate(item: dict[str, Any]) -> dict[str, Any] | None:
    images = item.get("imagesList") or item.get("imageList") or []
    if not isinstance(images, list):
        return None
    image = next((entry for entry in images if isinstance(entry, dict)), None)
    if not image:
        return None
    url = next((image.get(key) for key in ("original", "urlSizeLarge", "url") if image.get(key)), None)
    return {"url": str(url or ""), "width": image.get("width"), "height": image.get("height")}


def download_cover(session: requests.Session, item: dict[str, Any], package: Path, referer: str) -> tuple[dict[str, Any] | None, str | None]:
    candidate = cover_candidate(item)
    if not candidate or not candidate["url"]:
        return None, "Collection listing did not expose a cover image."
    try:
        response = session.get(candidate["url"], headers={"User-Agent": MOBILE_UA, "Referer": referer}, timeout=25)
        response.raise_for_status()
        content_type = response.headers.get("content-type", "image/jpeg").split(";", 1)[0].lower()
        if not content_type.startswith("image/") or len(response.content) < 512:
            raise ValueError("collection cover response was not a usable image")
        extension = mimetypes.guess_extension(content_type) or ".jpg"
        if extension == ".jpe":
            extension = ".jpg"
        relative = f"evidence/cover/cover{extension}"
        destination = package / relative
        destination.parent.mkdir(parents=True, exist_ok=True)
        destination.write_bytes(response.content)
        return {
            "path": relative,
            "mime": content_type,
            "bytes": len(response.content),
            "width": candidate.get("width"),
            "height": candidate.get("height"),
            "sha256": hashlib.sha256(response.content).hexdigest(),
            "acquisition": "public_collection_listing",
        }, None
    except (requests.RequestException, OSError, ValueError) as exc:
        return None, str(exc)


def parse_args() -> argparse.Namespace:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--url", required=True)
    parser.add_argument("--collection-url", required=True)
    parser.add_argument("--raw-input", required=True)
    parser.add_argument("--instruction", default="")
    parser.add_argument("--output-root", required=True, type=Path)
    parser.add_argument("--job-id", default="manual")
    parser.add_argument("--mode", default="full")
    parser.add_argument("--html-file", type=Path, help=argparse.SUPPRESS)
    return parser.parse_args()


def main() -> int:
    args = parse_args()
    try:
        note_id = note_id_from_url(args.url)
        if args.html_file:
            html = args.html_file.read_text("utf-8")
            resolved = args.collection_url
        else:
            html, resolved, _ = fetch_public_html(args.collection_url, 20)
        item = next((entry for entry in board_notes(initial_state(html)) if str(entry.get("id") or entry.get("noteId") or "").lower() == note_id), None)
        if not item:
            raise ValueError("XHS collection listing no longer contains the requested child note")
        user = item.get("user") or item.get("author") or {}
        timing = item.get("noteTime") or {}
        content_form = "video" if str(item.get("type") or "").lower() == "video" else "image_post"
        package = args.output_root.expanduser().resolve() / f"S-xhs-{note_id}"
        package.mkdir(parents=True, exist_ok=True)
        session = requests.Session()
        cover, cover_error = download_cover(session, item, package, resolved)
        canonical = f"https://www.xiaohongshu.com/discovery/item/{note_id}"
        title = str(item.get("displayTitle") or item.get("title") or note_id).strip()
        author = str(user.get("nickname") or user.get("displayName") or "").strip()
        source = {
            "platform": "xhs", "platform_display_name": "小红书", "content_form": content_form, "stable_id": note_id,
            "canonical_url": canonical, "resolved_url_redacted": canonical,
            "title": title, "author": author,
            "published_at": timestamp(timing.get("createTime")),
            "updated_at": timestamp(timing.get("updateTime")),
            "body": "", "tags": [], "metrics": {"likes": None, "collects": None, "comments": None, "shares": None},
            "capture_method": "public_collection_listing_fallback",
            "detail_access": "public_no_login_detail_unavailable",
        }
        artifacts = [{"path": cover["path"], "kind": "cover", "sha256": cover["sha256"]}] if cover else []
        errors = [{
            "at": now_iso(), "stage": "acquisition", "code": "XHS_DETAIL_UNAVAILABLE_COLLECTION_FALLBACK",
            "message": "The collection listed this note, but every no-login detail route was unavailable; the item capture stopped after archiving only collection metadata and cover evidence.",
            "recoverable": True, "artifact": cover["path"] if cover else "输入上下文.md",
        }]
        if cover_error:
            errors.append({"at": now_iso(), "stage": "cover", "code": "COLLECTION_COVER_UNAVAILABLE", "message": cover_error, "recoverable": True, "artifact": "evidence/cover"})
        manifest = {
            "package_schema": "1.1", "package_id": package.name, "job_id": args.job_id, "mode": args.mode,
            "content_form": content_form, "source": source, "artifacts": artifacts,
            "gates": {"identity": "passed", "acquisition": "failed", "extraction": "failed", "fidelity": "failed", "analysis": "not_applicable", "comments": "not_applicable", "cover": "passed" if cover else "failed", "cards": "not_applicable", "archive": "review"},
            "cover": cover, "errors": errors, "created_at": now_iso(),
        }
        atomic_json(package / "manifest.json", manifest)
        atomic_text(package / "输入上下文.md", f"# 输入上下文\n\n{args.raw_input}\n\n## 用户要求\n\n{args.instruction or '保存收藏夹内容。'}\n")
        atomic_text(package / "原始内容.md", "\n".join([f"# {title}", "", f"- 作者：{author or '未知'}", f"- 来源：{canonical}", "- 获取方式：小红书收藏夹公开目录回退", "- 完整性：未完成。无登录详情路径不可读，本次已停止；仅保留目录元数据与封面。", ""]))
        cover_md = f"![收藏夹目录封面]({cover['path']})\n\n" if cover else ""
        atomic_text(package / "标准内容.md", f"# {title}\n\n{cover_md}> 未完成：当前仅有收藏夹目录证据。无登录详情抓取未取得正文、完整图片或视频，本次已停止，不得视为审核通过。\n")
        print(json.dumps({"status": "captured", "package": str(package), "platform": "xhs", "content_form": content_form, "stable_id": note_id, "fallback": "collection_listing"}, ensure_ascii=False))
        return 0
    except (OSError, ValueError, requests.RequestException, json.JSONDecodeError) as exc:
        print(json.dumps({"status": "failed", "error": str(exc)}, ensure_ascii=False), file=os.sys.stderr)
        return 2


if __name__ == "__main__":
    raise SystemExit(main())

#!/usr/bin/env python3
"""Deterministically select useful, sufficiently-liked top-level comments."""

from __future__ import annotations

import os
import re
from datetime import datetime
from typing import Any

MIN_COMMENT_LIKES = 10
MAX_RETAINED_COMMENTS = 10
COMMENT_ENRICHMENT_MODES = {"analyze", "cards", "full"}
DEFAULT_MIN_SOURCE_COMMENTS = 20

_LOW_INFORMATION = re.compile(
    r"^(?:哈{2,}|呵{2,}|嘿{2,}|[好棒牛绝妙强赞]+|支持|喜欢|确实|同意|来了|学到了|收藏了|转发了|沙发|第一|蹲|码住|求链接|求教程|\d{2,}|[a-z]{1,4})[!！。\.～~…]*$",
    re.I,
)
_SHORT_SIGNAL = re.compile(
    r"为什么|怎么|什么|哪里|哪种|哪一个|能否|能不能|有没有|是否|因为|但是|其实|建议|请问|想问|我用|我试|我觉得|相比|区别|价格|链接|教程|方法|问题|需求|经历|体验|结果|原因"
)


def meaningful_comment(text: Any) -> bool:
    """Keep complete information; reject emoji-only, cheering, and fragmentary remarks."""
    normalized = re.sub(r"\s+", " ", str(text or "")).strip()
    if not normalized:
        return False
    semantic = re.sub(r"https?://\S+|@[\w\-.一-龥]+", "", normalized)
    semantic = "".join(re.findall(r"[A-Za-z0-9一-龥]", semantic))
    if len(semantic) < 6 or _LOW_INFORMATION.fullmatch(normalized):
        return False
    if len(semantic) >= 10:
        return True
    if _SHORT_SIGNAL.search(normalized):
        return True
    return False


def select_comments(
    items: list[dict],
    *,
    min_likes: int = MIN_COMMENT_LIKES,
    limit: int = MAX_RETAINED_COMMENTS,
) -> tuple[list[dict], dict[str, int]]:
    """Rank eligible comments by verified likes and return selection diagnostics."""
    counts = {"below_min_likes": 0, "unknown_likes": 0, "low_information": 0}
    eligible = []
    for item in items:
        likes = item.get("like_count")
        if likes is None:
            counts["unknown_likes"] += 1
            continue
        if likes < min_likes:
            counts["below_min_likes"] += 1
            continue
        if not meaningful_comment(item.get("text")):
            counts["low_information"] += 1
            continue
        eligible.append(item)
    eligible.sort(key=lambda item: item["like_count"], reverse=True)
    retained = eligible[:limit]
    for rank, item in enumerate(retained, 1):
        item["rank"] = rank
    return retained, counts


def selection_policy() -> dict[str, int | str]:
    return {
        "ranking": "normalized_like_count_desc",
        "min_like_count": MIN_COMMENT_LIKES,
        "max_retained": MAX_RETAINED_COMMENTS,
        "text_quality": "complete_information_not_fragmentary_or_generic_reaction",
    }


def comment_enrichment_decision(
    platform: str,
    mode: str,
    metrics: dict[str, Any] | None,
    *,
    visible_sample_available: bool = False,
) -> dict[str, Any]:
    """Decide whether a bounded public comment sample is worth acquiring."""
    try:
        min_total = max(0, int(os.getenv("CREATOR_INGEST_COMMENT_MIN_TOTAL", DEFAULT_MIN_SOURCE_COMMENTS)))
    except ValueError:
        min_total = DEFAULT_MIN_SOURCE_COMMENTS
    enabled = os.getenv("CREATOR_INGEST_ENABLE_COMMENTS", "1").lower() not in {"0", "false", "no"}
    total = (metrics or {}).get("comments")
    try:
        total = int(total) if total is not None else None
    except (TypeError, ValueError):
        total = None
    eligible = bool(
        enabled
        and platform in {"douyin", "bilibili"}
        and mode in COMMENT_ENRICHMENT_MODES
        and ((total is not None and total >= min_total) or (total is None and visible_sample_available))
    )
    if not enabled:
        reason = "Comment enrichment is disabled by runtime policy."
    elif platform not in {"douyin", "bilibili"}:
        reason = "This adaptive public-comment policy applies only to Douyin and Bilibili."
    elif mode not in COMMENT_ENRICHMENT_MODES:
        reason = f"Mode {mode} does not require comment enrichment."
    elif total is not None and total < min_total:
        reason = f"Visible source comment count {total} is below the enrichment threshold {min_total}."
    elif total is None and not visible_sample_available:
        reason = "No public total or already-rendered anonymous comment sample was available."
    else:
        reason = "Eligible for one bounded public hot/visible comment sample."
    return {
        "eligible": eligible,
        "enabled": enabled,
        "mode": mode,
        "min_source_comments": min_total,
        "source_comment_count": total,
        "visible_sample_available": visible_sample_available,
        "reason": reason,
    }


def deferred_comment_result(decision: dict[str, Any]) -> dict[str, Any]:
    return {
        "status": "not_applicable",
        "ranking_scope": "skipped_by_policy",
        "sample_size": 0,
        "has_more": None,
        "acquired_at": datetime.now().astimezone().isoformat(timespec="seconds"),
        "method": "adaptive_comment_enrichment_policy",
        "comments": [],
        "selection_policy": selection_policy(),
        "enrichment_decision": decision,
        "topic_signal_status": "no_evidence_available",
        "topic_signals": [],
        "comment_capture_status": "skipped_by_policy",
        "reason": decision.get("reason"),
    }

#!/usr/bin/env python3
"""Portable, dependency-free queue manager for creator content jobs."""

from __future__ import annotations

import argparse
import hashlib
import json
import os
import re
import sys
import tempfile
from datetime import datetime
from pathlib import Path
from urllib.parse import urlparse

STATUSES = ("pending", "running", "completed", "partial", "review", "failed")
FORMS = ("unknown", "video", "image_post", "article", "audio", "document")
MODES = ("capture", "standard", "analyze", "cards", "full")
PLATFORMS = ("unknown", "xhs", "douyin", "wechat", "web", "youtube", "bilibili", "podcast", "local")
URL_RE = re.compile(r"https?://[^\s<>\]\[\)（），,]+")


def now_iso() -> str:
    return datetime.now().astimezone().isoformat(timespec="seconds")


def atomic_json(path: Path, data: dict) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    fd, tmp_name = tempfile.mkstemp(prefix=f".{path.name}.", suffix=".tmp", dir=path.parent)
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as handle:
            json.dump(data, handle, ensure_ascii=False, indent=2)
            handle.write("\n")
            handle.flush()
            os.fsync(handle.fileno())
        os.replace(tmp_name, path)
    finally:
        if os.path.exists(tmp_name):
            os.unlink(tmp_name)


def read_json(path: Path) -> dict:
    with path.open(encoding="utf-8") as handle:
        value = json.load(handle)
    if not isinstance(value, dict):
        raise ValueError("job root must be an object")
    return value


def redact_url(url: str) -> str:
    """Keep an intake URL intact; logging/archiving code must canonicalize later."""
    return url.strip()


def platform_hint(url: str) -> str:
    host = urlparse(url).netloc.lower()
    if "xiaohongshu" in host or "xhslink" in host:
        return "xhs"
    if "douyin" in host:
        return "douyin"
    if "weixin.qq.com" in host:
        return "wechat"
    if "youtube" in host or "youtu.be" in host:
        return "youtube"
    if "bilibili" in host or "b23.tv" in host:
        return "bilibili"
    return "web" if host else "unknown"


def job_id(seed: str, ordinal: int) -> str:
    stamp = datetime.now().astimezone().strftime("%Y%m%d-%H%M%S")
    digest = hashlib.sha256(f"{seed}\0{ordinal}\0{now_iso()}".encode()).hexdigest()[:8]
    return f"CCI-{stamp}-{digest}"


def ensure_queue(root: Path) -> None:
    for status in STATUSES:
        (root / status).mkdir(parents=True, exist_ok=True)


def validate_job(job: dict) -> list[str]:
    errors: list[str] = []
    required = ("schema_version", "job_id", "created_at", "source_agent", "raw_input", "source", "instruction", "routing", "state")
    for key in required:
        if key not in job:
            errors.append(f"missing field: {key}")
    if errors:
        return errors
    if job["schema_version"] != "1.0":
        errors.append("unsupported schema_version")
    if not isinstance(job["raw_input"], str) or not job["raw_input"].strip():
        errors.append("raw_input must be non-empty text")
    if job["instruction"].get("mode") not in MODES:
        errors.append("invalid instruction.mode")
    if job["routing"].get("content_form") not in FORMS:
        errors.append("invalid routing.content_form")
    if job["routing"].get("platform") not in PLATFORMS:
        errors.append("invalid routing.platform")
    if job["state"].get("status") not in STATUSES:
        errors.append("invalid state.status")
    if not job["source"].get("url") and not job["source"].get("local_files"):
        errors.append("source needs url or local_files")
    return errors


def cmd_init(args: argparse.Namespace) -> int:
    root = Path(args.queue).expanduser().resolve()
    ensure_queue(root)
    print(root)
    return 0


def cmd_enqueue(args: argparse.Namespace) -> int:
    root = Path(args.queue).expanduser().resolve()
    ensure_queue(root)
    raw = args.raw_input
    urls = args.url or URL_RE.findall(raw)
    urls = list(dict.fromkeys(redact_url(url) for url in urls))
    local_files = [str(Path(item).expanduser().resolve()) for item in (args.file or [])]
    sources: list[tuple[str | None, list[str]]] = [(url, []) for url in urls]
    sources.extend((None, [path]) for path in local_files)
    if not sources:
        raise ValueError("no URL or local file found")

    created: list[str] = []
    for index, (url, files) in enumerate(sources, start=1):
        jid = job_id(url or files[0], index)
        platform = args.platform if args.platform != "auto" else (platform_hint(url) if url else "local")
        timestamp = now_iso()
        job = {
            "schema_version": "1.0",
            "job_id": jid,
            "created_at": timestamp,
            "source_agent": args.agent,
            "input_channel": "user-direct",
            "raw_input": raw,
            "source": {
                "url": url,
                "local_files": files,
                "platform_hint": platform,
                "content_form_hint": args.form,
            },
            "instruction": {
                "raw": args.instruction or raw,
                "mode": args.mode,
                "requested_outputs": args.output or [],
            },
            "routing": {
                "platform": platform,
                "content_form": args.form,
                "priority": args.priority,
            },
            "state": {
                "status": "pending",
                "stage": "resolve",
                "attempts": 0,
                "claimed_by": None,
                "claimed_at": None,
                "updated_at": timestamp,
            },
            "resolution": {},
            "outputs": {},
            "errors": [],
        }
        errors = validate_job(job)
        if errors:
            raise ValueError("; ".join(errors))
        path = root / "pending" / f"{jid}.json"
        atomic_json(path, job)
        created.append(str(path))
    print("\n".join(created))
    return 0


def cmd_claim(args: argparse.Namespace) -> int:
    root = Path(args.queue).expanduser().resolve()
    ensure_queue(root)
    candidates = sorted((root / "pending").glob("*.json"), key=lambda p: p.stat().st_mtime)
    if args.job_id:
        candidates = [root / "pending" / f"{args.job_id}.json"]
    for source in candidates:
        if not source.exists():
            continue
        target = root / "running" / source.name
        try:
            os.replace(source, target)
        except FileNotFoundError:
            continue
        job = read_json(target)
        timestamp = now_iso()
        job["state"].update({
            "status": "running",
            "claimed_by": args.worker,
            "claimed_at": timestamp,
            "updated_at": timestamp,
            "attempts": int(job["state"].get("attempts", 0)) + 1,
        })
        atomic_json(target, job)
        print(target)
        return 0
    print("no pending job", file=sys.stderr)
    return 3


def locate_job(root: Path, jid: str) -> Path:
    matches = [root / status / f"{jid}.json" for status in STATUSES]
    existing = [path for path in matches if path.exists()]
    if len(existing) != 1:
        raise ValueError(f"expected one job named {jid}, found {len(existing)}")
    return existing[0]


def cmd_transition(args: argparse.Namespace) -> int:
    root = Path(args.queue).expanduser().resolve()
    ensure_queue(root)
    source = locate_job(root, args.job_id)
    job = read_json(source)
    if args.status == "completed":
        if not args.manifest or not args.output_path:
            raise ValueError("completed transition requires --manifest and --output-path")
        package = Path(args.output_path).expanduser().resolve()
        manifest_path = Path(args.manifest).expanduser().resolve()
        if manifest_path != package / "manifest.json":
            raise ValueError("--manifest must be <output-path>/manifest.json")
        sys.path.insert(0, str(Path(__file__).resolve().parent))
        from validate_package import validate as validate_package
        package_errors = validate_package(package, allow_incomplete=False)
        if package_errors:
            raise ValueError("package validation failed: " + "; ".join(package_errors))
    timestamp = now_iso()
    job["state"].update({"status": args.status, "stage": args.stage, "updated_at": timestamp})
    if args.output_path:
        job["outputs"]["package_path"] = str(Path(args.output_path).expanduser().resolve())
    if args.manifest:
        job["outputs"]["manifest"] = str(Path(args.manifest).expanduser().resolve())
    if args.message:
        job["errors"].append({
            "at": timestamp,
            "stage": args.stage,
            "code": args.code,
            "message": args.message,
            "recoverable": args.status in ("partial", "review"),
        })
    target = root / args.status / source.name
    atomic_json(source, job)
    if target != source:
        os.replace(source, target)
    print(target)
    return 0


def cmd_validate(args: argparse.Namespace) -> int:
    path = Path(args.job).expanduser().resolve()
    errors = validate_job(read_json(path))
    if errors:
        print("\n".join(errors), file=sys.stderr)
        return 2
    print(f"valid: {path}")
    return 0


def parser() -> argparse.ArgumentParser:
    result = argparse.ArgumentParser(description=__doc__)
    sub = result.add_subparsers(dest="command", required=True)

    init = sub.add_parser("init")
    init.add_argument("--queue", required=True)
    init.set_defaults(func=cmd_init)

    enqueue = sub.add_parser("enqueue")
    enqueue.add_argument("--queue", required=True)
    enqueue.add_argument("--raw-input", required=True)
    enqueue.add_argument("--instruction")
    enqueue.add_argument("--url", action="append")
    enqueue.add_argument("--file", action="append")
    enqueue.add_argument("--agent", default="unknown")
    enqueue.add_argument("--mode", choices=MODES, default="full")
    enqueue.add_argument("--form", choices=FORMS, default="unknown")
    enqueue.add_argument("--platform", choices=("auto",) + PLATFORMS, default="auto")
    enqueue.add_argument("--priority", choices=("high", "normal", "low"), default="high")
    enqueue.add_argument("--output", action="append")
    enqueue.set_defaults(func=cmd_enqueue)

    claim = sub.add_parser("claim")
    claim.add_argument("--queue", required=True)
    claim.add_argument("--worker", required=True)
    claim.add_argument("--job-id")
    claim.set_defaults(func=cmd_claim)

    transition = sub.add_parser("transition")
    transition.add_argument("--queue", required=True)
    transition.add_argument("--job-id", required=True)
    transition.add_argument("--status", choices=STATUSES, required=True)
    transition.add_argument("--stage", required=True)
    transition.add_argument("--output-path")
    transition.add_argument("--manifest")
    transition.add_argument("--code", default="WORKER_NOTE")
    transition.add_argument("--message")
    transition.set_defaults(func=cmd_transition)

    validate = sub.add_parser("validate")
    validate.add_argument("--job", required=True)
    validate.set_defaults(func=cmd_validate)
    return result


def main() -> int:
    try:
        args = parser().parse_args()
        return args.func(args)
    except (OSError, ValueError, json.JSONDecodeError) as exc:
        print(f"error: {exc}", file=sys.stderr)
        return 2


if __name__ == "__main__":
    raise SystemExit(main())

#!/usr/bin/env python3
"""Anonymous Douyin public-detail adapter.

The adapter launches a disposable headless browser profile, reads only the
public detail page, and destroys the profile after the snapshot. It never reads
the user's browser profile, login cookies, or account state.
"""

from __future__ import annotations

import html
import json
import os
import re
import signal
import shutil
import subprocess
import sys
from datetime import datetime
from pathlib import Path
from typing import Any

from bs4 import BeautifulSoup

from comment_selection import (
    comment_enrichment_decision,
    deferred_comment_result,
    select_comments,
    selection_policy,
)
from title_normalization import douyin_display_title


SNAPSHOT_SCRIPT = Path(__file__).with_name("public_page_snapshot.mjs")


def now_iso() -> str:
    return datetime.now().astimezone().isoformat(timespec="seconds")


def _display_title(description: str, fallback: str) -> str:
    return douyin_display_title(description, fallback)


def _first_url(value: Any) -> str | None:
    if isinstance(value, str) and value:
        return value
    if not isinstance(value, dict):
        return None
    for key in ("download_url_list", "url_list"):
        for url in value.get(key) or []:
            if isinstance(url, str) and url:
                return url
    for key in ("download_url", "url"):
        url = value.get(key)
        if isinstance(url, str) and url:
            return url
    return None


def _count(value: Any) -> int | None:
    try:
        return int(value) if value is not None else None
    except (TypeError, ValueError):
        return None


def _public_count(value: str) -> int | None:
    text = str(value or "").strip().replace(",", "")
    match = re.fullmatch(r"([\d.]+)([万亿]?)", text)
    if not match:
        return None
    scale = {"": 1, "万": 10_000, "亿": 100_000_000}[match.group(2)]
    return round(float(match.group(1)) * scale)


def public_comment_result(reason: str = "The no-login public page did not expose a stable top-level comment sample.") -> dict[str, Any]:
    return {
        "status": "passed",
        "ranking_scope": "public_page_no_login_no_comment_sample",
        "sample_size": 0,
        "has_more": None,
        "acquired_at": now_iso(),
        "method": "douyin_public_page_no_login",
        "comments": [],
        "selection_policy": selection_policy(),
        "discarded": {"below_min_likes": 0, "unknown_likes": 0, "low_information": 0},
        "topic_signal_status": "no_meaningful_signal",
        "topic_signals": [],
        "comment_capture_status": "confirmed_no_public_source",
        "reason": reason,
    }


def _douyin_comment_items(page_html: str) -> list[Any]:
    soup = BeautifulSoup(page_html, "html.parser")
    direct = soup.select('[data-e2e="comment-list"] > [data-e2e="comment-item"]')
    if direct:
        return direct[:20]
    return [
        item for item in soup.select('[data-e2e="comment-item"]')
        if item.find_parent(attrs={"data-e2e": "comment-item"}) is None
    ][:20]


def _node_text(node: Any) -> str:
    if node is None:
        return ""
    for image in node.select("img[alt]"):
        image.replace_with(str(image.get("alt") or ""))
    return re.sub(r"\s+", " ", node.get_text(" ", strip=True)).strip()


def douyin_public_comments(page_html: str, mode: str, metrics: dict[str, Any] | None) -> dict[str, Any]:
    has_visible_sample = "comment-item-stats-container" in page_html
    decision = comment_enrichment_decision(
        "douyin", mode, metrics, visible_sample_available=has_visible_sample,
    )
    if not decision["eligible"]:
        return deferred_comment_result(decision)
    items = _douyin_comment_items(page_html)
    if not items:
        result = public_comment_result()
        result["enrichment_decision"] = decision
        return result
    normalized = []
    for item in items:
        info = item.select_one(".comment-item-info-wrap")
        author_link = info.select_one('a[href*="/user/"]') if info else None
        content = info.find_next_sibling("div") if info else None
        stats = item.select_one(".comment-item-stats-container")
        like_node = stats.select_one("p span") if stats else None
        raw_likes = _node_text(like_node) or "0"
        like_count = _public_count(raw_likes)
        reply_node = item.select_one('button[class*="comment-reply-expand"], [class*="comment-reply-expand"]')
        raw_replies = _node_text(reply_node)
        reply_match = re.search(r"(\d+)", raw_replies)
        tooltip = item.select_one('[id^="tooltip_"]')
        comment_id = None
        if tooltip:
            match = re.fullmatch(r"tooltip_(\d{10,})", str(tooltip.get("id") or ""))
            comment_id = match.group(1) if match else None
        text = _node_text(content)
        if not text:
            continue
        normalized.append({
            "comment_id": comment_id,
            "author_display": _node_text(author_link),
            "text": text,
            "like_count": like_count,
            "like_count_raw": raw_likes,
            "like_count_precision": "exact" if like_count is not None else "unknown",
            "reply_count": int(reply_match.group(1)) if reply_match else 0,
            "created_at": None,
            "locator": f"comment:{comment_id}" if comment_id else f"anonymous-dom-comment:{len(normalized) + 1}",
        })
    top, discarded = select_comments(normalized)
    return {
        "status": "passed",
        "ranking_scope": "visible_rendered_sample_sorted_by_likes",
        "sample_size": len(normalized),
        "has_more": None,
        "acquired_at": now_iso(),
        "method": "douyin_disposable_anonymous_visible_dom",
        "comments": top,
        "selection_policy": selection_policy(),
        "enrichment_decision": decision,
        "discarded": discarded,
        "topic_signal_status": "pending_analysis" if top else "no_meaningful_signal",
        "topic_signals": [],
        **({"reason": "The bounded visible sample contained no qualifying comment."} if not top else {}),
    }


def _stable_snapshot_error(detail: str) -> str:
    text = str(detail or "")
    if "DOUYIN_PUBLIC_CHALLENGE" in text:
        return "DOUYIN_PUBLIC_CHALLENGE: 公开页面触发安全验证"
    if "CDP" in text or "timed out" in text or "debugging port" in text:
        return "DOUYIN_BROWSER_RUNTIME_TIMEOUT: 匿名浏览器运行时未及时就绪"
    if "DOUYIN_ANONYMOUS_DETAIL_NOT_READY" in text or "did not become ready" in text:
        return "DOUYIN_ANONYMOUS_DETAIL_NOT_READY: 公开页面未提供所需详情证据"
    return "DOUYIN_ANONYMOUS_CAPTURE_FAILED: 匿名公开页面抓取失败"


def _snapshot_once(url: str, *, image_note: bool = False, media_only: bool = False) -> dict[str, Any]:
    node = shutil.which("node")
    if not node or not SNAPSHOT_SCRIPT.is_file():
        raise ValueError("Anonymous Douyin public-page runtime is unavailable")
    command = [node, str(SNAPSHOT_SCRIPT), url]
    if image_note:
        command.append("--image-note")
    if media_only:
        command.append("--media-only")
    process = subprocess.Popen(
        command,
        text=True,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        start_new_session=True,
    )
    try:
        stdout, stderr = process.communicate(timeout=50)
    except subprocess.TimeoutExpired as exc:
        try:
            os.killpg(process.pid, signal.SIGTERM)
            process.communicate(timeout=2)
        except (ProcessLookupError, subprocess.TimeoutExpired):
            try:
                os.killpg(process.pid, signal.SIGKILL)
            except ProcessLookupError:
                pass
            process.communicate()
        raise ValueError("DOUYIN_BROWSER_RUNTIME_TIMEOUT: 匿名浏览器运行超时，已终止本次策略") from exc
    if process.returncode != 0 or not stdout.strip():
        detail = (stderr or stdout).strip().splitlines()
        error_line = next((line.strip() for line in reversed(detail) if line.strip().startswith(("Error:", "DOMException"))), None)
        raw_error = error_line or (detail[-1] if detail else "empty response")
        print(json.dumps({"event": "douyin_snapshot_failure", "url": url, "detail": raw_error[-8_000:]}, ensure_ascii=False), file=sys.stderr)
        raise ValueError(_stable_snapshot_error(raw_error))
    value = json.loads(stdout)
    if not isinstance(value, dict) or not value.get("html"):
        raise ValueError("Anonymous Douyin public page returned no readable detail")
    return value


def _snapshot(
    url: str,
    *,
    fallback_urls: list[str] | None = None,
    max_attempts: int | None = None,
    image_note: bool = False,
) -> dict[str, Any]:
    """Try each distinct public route once; never repeat a failed strategy."""
    routes = []
    for candidate in [url, *(fallback_urls or [])]:
        if candidate and candidate not in routes:
            routes.append(candidate)
    if not routes:
        raise ValueError("Anonymous Douyin public page has no acquisition route")
    if max_attempts is not None:
        routes = routes[:max_attempts]

    errors = []
    for candidate in routes:
        try:
            return _snapshot_once(candidate, image_note=True) if image_note else _snapshot_once(candidate)
        except ValueError as exc:
            errors.append(str(exc))
    raise ValueError(
        f"DOUYIN_PUBLIC_STRATEGIES_EXHAUSTED: {len(errors)} 条不同公开路径均不可用；{'; '.join(errors)}"
    )


def _authentication() -> dict[str, Any]:
    return {"login_used": False, "user_cookies_used": False, "session_scope": "disposable_anonymous_browser"}


def _detail_common(detail: dict[str, Any], aweme_id: str, content_form: str, page_html: str, mode: str) -> dict[str, Any]:
    author = detail.get("author") or {}
    stats = detail.get("statistics") or {}
    description = str(detail.get("desc") or "").strip()
    avatar_url = _first_url(author.get("avatar_larger")) or _first_url(author.get("avatar_medium")) or _first_url(author.get("avatar_thumb"))
    route = "note" if content_form == "image_post" else "video"
    result = {
        "platform": "douyin",
        "platform_display_name": "抖音",
        "content_form": content_form,
        "stable_id": aweme_id,
        "canonical_url": f"https://www.douyin.com/{route}/{aweme_id}",
        "resolved_url_redacted": f"https://www.douyin.com/{route}/{aweme_id}",
        "title": _display_title(description, aweme_id),
        "author": str(author.get("nickname") or ""),
        "author_profile": {
            "display_name": str(author.get("nickname") or ""),
            "platform_user_id": str(author.get("sec_uid") or "") or None,
            "public_handle": str(author.get("unique_id") or author.get("short_id") or "") or None,
            "avatar_candidate": avatar_url,
        },
        "published_at": datetime.fromtimestamp(detail["create_time"]).astimezone().isoformat(timespec="seconds") if detail.get("create_time") else None,
        "updated_at": None,
        "body": description,
        "body_capture_status": "passed",
        "tags": [],
        "metrics": {
            "likes": _count(stats.get("digg_count")),
            "collects": _count(stats.get("collect_count")),
            "comments": _count(stats.get("comment_count")),
            "shares": _count(stats.get("share_count")),
        },
        "metrics_captured_at": now_iso(),
        "capture_method": "douyin_public_detail_disposable_anonymous_browser",
        "authentication": _authentication(),
    }
    result["comments"] = douyin_public_comments(page_html, mode, result["metrics"])
    return result


def capture_video_note(aweme_id: str, _browser: str = "chrome", mode: str = "full", acquisition_url: str | None = None) -> dict[str, Any]:
    canonical_url = f"https://www.douyin.com/video/{aweme_id}"
    primary_url = acquisition_url or canonical_url
    try:
        value = _snapshot_once(primary_url)
    except ValueError as detail_error:
        try:
            value = _snapshot_once(canonical_url, media_only=True)
        except ValueError as media_error:
            raise ValueError(
                f"DOUYIN_PUBLIC_STRATEGIES_EXHAUSTED: 详情接口与渲染媒体两条公开路径均不可用；{detail_error}; {media_error}"
            ) from media_error
        return _video_dom_fallback(value, aweme_id, mode, detail_error)
    page_html = str(value.get("html") or "")
    detail = (value.get("detail") or {}).get("aweme_detail") or {}
    if str(detail.get("aweme_id") or "") != aweme_id:
        raise ValueError("Anonymous Douyin video detail did not match the requested ID")
    video = detail.get("video") or {}
    media_url = _first_url(video.get("play_addr_h264")) or _first_url(video.get("play_addr"))
    cover_url = _first_url(video.get("cover")) or _first_url(video.get("origin_cover"))
    if not media_url or not cover_url:
        raise ValueError("Anonymous Douyin video detail contains no downloadable public media")
    result = _detail_common(detail, aweme_id, "video", page_html, mode)
    result.update({
        "duration_seconds": (float(video["duration"]) / 1000) if video.get("duration") else None,
        "images": [{"index": 1, "role": "cover", "ephemeral_url": cover_url, "expected_width": video.get("width"), "expected_height": video.get("height")}],
        "video_candidates": [{"method": "douyin_public_detail_direct_media", "ephemeral_url": media_url, "width": video.get("width"), "height": video.get("height")}],
    })
    return result


def _meta_property(page_html: str, name: str) -> str:
    match = re.search(rf'<meta[^>]+property="{re.escape(name)}"[^>]+content="([^"]*)"', page_html, re.I)
    return html.unescape(match.group(1)).strip() if match else ""


def _ld_json_author(page_html: str) -> str:
    for payload in re.findall(r'<script[^>]+type="application/ld\+json"[^>]*>(.*?)</script>', page_html, re.S | re.I):
        try:
            schema = json.loads(html.unescape(payload))
        except json.JSONDecodeError:
            continue
        author = schema.get("author") if isinstance(schema, dict) else None
        if isinstance(author, dict) and author.get("name"):
            return str(author["name"]).strip()
    return ""


def _video_dom_fallback(value: dict[str, Any], aweme_id: str, mode: str, prior_error: Exception) -> dict[str, Any]:
    page_html = str(value.get("html") or "")
    media = value.get("media") or {}
    media_url = str(media.get("url") or "")
    cover_url = str(media.get("poster") or _meta_property(page_html, "og:image") or "")
    if not re.match(r"^https?://", media_url):
        raise ValueError("DOUYIN_RENDERED_MEDIA_UNUSABLE: 页面播放器没有可下载的公开媒体地址")
    raw_title = re.sub(r"\s*-\s*抖音\s*$", "", str(value.get("title") or "")).strip()
    description = _meta_content(page_html, "description") or _meta_property(page_html, "og:description") or raw_title
    author = _ld_json_author(page_html)
    result = {
        "platform": "douyin",
        "platform_display_name": "抖音",
        "content_form": "video",
        "stable_id": aweme_id,
        "canonical_url": f"https://www.douyin.com/video/{aweme_id}",
        "resolved_url_redacted": f"https://www.douyin.com/video/{aweme_id}",
        "title": _display_title(description, raw_title or aweme_id),
        "author": author,
        "author_profile": {"display_name": author, "platform_user_id": None, "public_handle": None, "avatar_candidate": None},
        "published_at": None,
        "updated_at": None,
        "body": description,
        "body_capture_status": "passed" if description else "review",
        "tags": [],
        "metrics": {"likes": None, "collects": None, "comments": None, "shares": None},
        "metrics_captured_at": now_iso(),
        "duration_seconds": float(media["duration"]) if media.get("duration") else None,
        "images": ([{"index": 1, "role": "cover", "ephemeral_url": cover_url, "expected_width": None, "expected_height": None}] if re.match(r"^https?://", cover_url) else []),
        "video_candidates": [{"method": "douyin_public_rendered_media_dom", "ephemeral_url": media_url, "width": None, "height": None}],
        "capture_method": "douyin_public_rendered_media_dom_fallback",
        "capture_fallback": {"from": "aweme_detail", "reason": str(prior_error), "to": "rendered_media_dom"},
        "authentication": _authentication(),
    }
    result["comments"] = douyin_public_comments(page_html, mode, result["metrics"])
    return result


def _meta_content(page_html: str, name: str) -> str:
    match = re.search(rf'<meta[^>]+name="{re.escape(name)}"[^>]+content="([^"]*)"', page_html, re.I)
    return html.unescape(match.group(1)).strip() if match else ""


def _embedded_aweme_detail(page_html: str, aweme_id: str) -> dict[str, Any]:
    """Recover the public image-note payload embedded in the rendered page.

    Douyin image notes do not consistently issue the video-detail request that
    the ordinary video route uses.  The same public detail is nevertheless
    serialized into React's streamed page data, including each image's optional
    motion carrier.  Decode only the matching aweme and never persist the page
    payload or its expiring URLs.
    """
    pattern = re.compile(
        r'self\.__pace_f\.push\(\[1,("(?:\\.|[^"\\])*")\]\)',
        re.S,
    )
    decoder = json.JSONDecoder()
    for match in pattern.finditer(page_html):
        try:
            payload = json.loads(match.group(1))
        except json.JSONDecodeError:
            continue
        for aweme_match in re.finditer(r'"aweme"\s*:\s*', payload):
            cursor = aweme_match.end()
            try:
                aweme, _ = decoder.raw_decode(payload[cursor:])
            except json.JSONDecodeError:
                continue
            detail = aweme.get("detail") if isinstance(aweme, dict) else None
            if not isinstance(detail, dict):
                continue
            embedded_id = str(detail.get("awemeId") or detail.get("aweme_id") or "")
            if embedded_id == aweme_id:
                return detail
    return {}


def _ordered_public_urls(value: Any) -> list[str]:
    """Flatten the small URL container variants used by Douyin page data."""
    values: list[Any]
    if isinstance(value, list):
        values = value
    else:
        values = [value]
    urls = []
    for item in values:
        if isinstance(item, str):
            candidates = [item]
        elif isinstance(item, dict):
            candidates = [item.get("src"), item.get("url")]
            candidates.extend(item.get("urlList") or item.get("url_list") or [])
        else:
            candidates = []
        for candidate in candidates:
            if not isinstance(candidate, str) or not candidate.startswith(("http://", "https://")):
                continue
            normalized = candidate.replace("http://", "https://", 1)
            if normalized not in urls:
                urls.append(normalized)
    return urls


def _structured_image_items(page_html: str, aweme_id: str) -> list[dict[str, Any]]:
    detail = _embedded_aweme_detail(page_html, aweme_id)
    raw_images = detail.get("images") if isinstance(detail, dict) else None
    if not isinstance(raw_images, list):
        return []
    images = []
    for index, image in enumerate(raw_images, 1):
        if not isinstance(image, dict):
            continue
        image_urls = _ordered_public_urls(image.get("urlList") or image.get("url_list"))
        image_urls.extend(
            url for url in _ordered_public_urls(image.get("downloadUrlList") or image.get("download_url_list"))
            if url not in image_urls
        )
        if not image_urls:
            continue
        video = image.get("video") if isinstance(image.get("video"), dict) else {}
        motion_urls = _ordered_public_urls(video.get("playAddr") or video.get("play_addr"))
        motion_urls.extend(
            url for url in _ordered_public_urls(video.get("playAddrH265") or video.get("play_addr_h265"))
            if url not in motion_urls
        )
        motion_candidates = [
            {
                "method": "douyin_embedded_image_motion",
                "ephemeral_url": url,
                "width": video.get("width"),
                "height": video.get("height"),
                "duration": video.get("duration"),
                "selection_priority": 2 if position == 0 else 1,
                "transport_priority": len(motion_urls) - position,
            }
            for position, url in enumerate(motion_urls)
        ]
        images.append({
            "index": index,
            "role": "cover" if index == 1 else "content",
            "ephemeral_url": image_urls[0],
            "ephemeral_urls": image_urls,
            "expected_width": image.get("width"),
            "expected_height": image.get("height"),
            "live_photo": bool(motion_candidates),
            "motion_candidates": motion_candidates,
        })
    return images


def capture_image_note(aweme_id: str, _browser: str = "chrome", mode: str = "full", acquisition_url: str | None = None) -> dict[str, Any]:
    canonical_url = f"https://www.douyin.com/note/{aweme_id}"
    value = _snapshot(
        acquisition_url or canonical_url,
        fallback_urls=[canonical_url, f"https://www.iesdouyin.com/share/slides/{aweme_id}/"],
        image_note=True,
    )
    page_html = str(value.get("html") or "")
    title = re.sub(r"\s*-\s*抖音\s*$", "", str(value.get("title") or "")).strip()
    description_meta = _meta_content(page_html, "description")
    description = description_meta.split(" - ", 1)[0].strip() or title
    author = ""
    author_url = ""
    for payload in re.findall(r'<script[^>]+type="application/ld\+json"[^>]*>(.*?)</script>', page_html, re.S | re.I):
        try:
            schema = json.loads(html.unescape(payload))
        except json.JSONDecodeError:
            continue
        entries = schema.get("itemListElement") if isinstance(schema, dict) else None
        if isinstance(entries, list) and len(entries) > 1 and isinstance(entries[1], dict):
            author = str(entries[1].get("name") or "")
            author_url = str(entries[1].get("item") or "")
            break
    meta_match = re.search(r"\s-\s(.+?)于(\d{8})发布在抖音，已经收获了([\d.,万亿]+)个喜欢", description_meta)
    published_at = None
    likes = None
    if meta_match:
        author = author or meta_match.group(1).strip()
        published_at = datetime.strptime(meta_match.group(2), "%Y%m%d").astimezone().isoformat(timespec="seconds")
        likes = _public_count(meta_match.group(3))
    images = _structured_image_items(page_html, aweme_id)
    if not images:
        image_urls = []
        for url in re.findall(r'<img[^>]+src="([^"]+)"', page_html, re.I):
            candidate = html.unescape(url)
            if "biz_tag=aweme_images" not in candidate or candidate in image_urls:
                continue
            image_urls.append(candidate)
        images = [
            {
                "index": index,
                "role": "cover" if index == 1 else "content",
                "ephemeral_url": url,
                "ephemeral_urls": [url],
                "expected_width": None,
                "expected_height": None,
                "live_photo": False,
                "motion_candidates": [],
            }
            for index, url in enumerate(image_urls, 1)
        ]
    if not images:
        raise ValueError("Anonymous Douyin image note contains no downloadable public images")
    profile_match = re.search(r"/user/([^/?#]+)", author_url)
    public_handle = profile_match.group(1) if profile_match else None
    result = {
        "platform": "douyin",
        "platform_display_name": "抖音",
        "content_form": "image_post",
        "stable_id": aweme_id,
        "canonical_url": f"https://www.douyin.com/note/{aweme_id}",
        "resolved_url_redacted": f"https://www.douyin.com/note/{aweme_id}",
        "title": _display_title(description, aweme_id),
        "author": author,
        "author_profile": {"display_name": author, "platform_user_id": public_handle, "public_handle": None, "avatar_candidate": None},
        "published_at": published_at,
        "updated_at": None,
        "body": description,
        "body_capture_status": "passed",
        "tags": [],
        "metrics": {"likes": likes, "collects": None, "comments": None, "shares": None},
        "metrics_captured_at": now_iso(),
        "images": images,
        "live_photo_count": sum(1 for image in images if image.get("live_photo")),
        "video_candidates": [],
        "capture_method": "douyin_public_detail_disposable_anonymous_browser",
        "authentication": _authentication(),
    }
    result["comments"] = douyin_public_comments(page_html, mode, result["metrics"])
    return result

#!/usr/bin/env python3
"""Finalize the deterministic archive gate after every substantive gate is settled."""

from __future__ import annotations

import argparse
import copy
import json
import os
import re
import tempfile
from pathlib import Path

from gate_ledger import sync_gate_ledger
from package_document import package_document_path
from validate_package import validate


def sync_favorite_status(package: Path, status: str = "complete") -> None:
    """Keep the human-facing record aligned with the finalized gate state."""
    manifest = json.loads((package / "manifest.json").read_text("utf-8"))
    path = package_document_path(package, manifest)
    if not path.is_file():
        return
    current = path.read_text("utf-8")
    updated = re.sub(
        r'(?m)^(capture_status:\s*)(?:"[^"]*"|\S+)(\s*)$',
        rf'\1"{status}"\2',
        current,
        count=1,
    )
    updated = re.sub(
        r'(?m)^(>\s+[^\n]*?\s+·\s+[^\n]*?\s+·\s+)(?:failed|review|partial|captured|complete)(\s*)$',
        rf'\1{status}\2',
        updated,
        count=1,
    )
    if updated == current:
        return
    descriptor, temporary = tempfile.mkstemp(prefix=f".{path.name}.", suffix=".tmp", dir=path.parent)
    try:
        with os.fdopen(descriptor, "w", encoding="utf-8") as handle:
            handle.write(updated)
        os.replace(temporary, path)
    finally:
        if os.path.exists(temporary):
            os.unlink(temporary)


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--package", required=True, type=Path)
    args = parser.parse_args()
    package = args.package.expanduser().resolve()
    manifest_path = package / "manifest.json"
    manifest = json.loads(manifest_path.read_text("utf-8"))
    # No-speech/visual-text branches may legitimately finish without derived
    # transcript chapters. Older runs sometimes retained declarations for
    # those absent optional files, producing a false manual archive blocker.
    transcript = manifest.get("transcript") if isinstance(manifest.get("transcript"), dict) else {}
    chapter_paths = {
        "evidence/transcript/逐字稿-章节版.json",
        "evidence/transcript/逐字稿-章节版.md",
    }
    no_speech_projection = (
        transcript.get("mode") in {"no_reliable_speech", "visual_text"}
        and transcript.get("lexical_review_status") == "not_applicable_no_reliable_speech"
        and not any((package / path).is_file() for path in chapter_paths)
    )
    if no_speech_projection:
        manifest["artifacts"] = [
            item for item in (manifest.get("artifacts") or [])
            if not (isinstance(item, dict) and str(item.get("path") or "") in chapter_paths)
        ]
        if str(transcript.get("chapters_path") or "") in chapter_paths:
            transcript.pop("chapters_path", None)
    # Some older Agent apply paths wrote a verified passed ledger and later
    # downgraded only the summary `gates` map back to review.  Reconcile that
    # split-brain state only when a dry-run with the prior passed summaries
    # still verifies every current input and output fingerprint.
    candidate = copy.deepcopy(manifest)
    prior_ledger = candidate.get("gate_ledger") if isinstance(candidate.get("gate_ledger"), dict) else {}
    candidate_summary = candidate.setdefault("gates", {})
    repaired = []
    for gate, entry in prior_ledger.items():
        if gate == "archive" or not isinstance(entry, dict):
            continue
        if entry.get("status") == "passed" and entry.get("summary_status") == "passed" \
                and candidate_summary.get(gate) in {"review", "partial"}:
            candidate_summary[gate] = "passed"
            repaired.append(gate)
    if repaired:
        candidate_ledger, _ = sync_gate_ledger(package, candidate, write=False)
        if all(entry.get("status") in {"passed", "not_applicable"} for gate, entry in candidate_ledger.items() if gate != "archive"):
            manifest = candidate
    ledger, _ = sync_gate_ledger(package, manifest, write=False)
    unresolved = [
        gate for gate, entry in ledger.items()
        if gate != "archive" and entry.get("status") not in {"passed", "not_applicable"}
    ]
    if unresolved:
        print(json.dumps({"status": "blocked", "unresolved": unresolved}, ensure_ascii=False))
        return 2
    # The ledger is authoritative for applicability. Legacy manifests may omit
    # summary keys for carrier-inapplicable gates (for example podcast cover),
    # while strict package validation requires an explicit result.
    for gate, entry in ledger.items():
        if entry.get("status") == "not_applicable" and not manifest.setdefault("gates", {}).get(gate):
            manifest["gates"][gate] = "not_applicable"
    manifest.setdefault("gates", {})["archive"] = "passed"
    # Validate the candidate state in memory. Older or interrupted packages may
    # legitimately omit the archive summary until this deterministic finalizer
    # commits it; rereading the pre-commit manifest here would reject that
    # recoverable state as `archive: None` before it can be repaired.
    structural_errors = validate(package, allow_incomplete=True, manifest_override=manifest)
    if structural_errors:
        print("\n".join(structural_errors))
        return 2
    # The reader-facing status is part of the archive output fingerprint. Write
    # it before certifying the archive so finalization does not immediately
    # invalidate itself by changing the canonical Markdown afterwards.
    sync_favorite_status(package)
    sync_gate_ledger(package, manifest, write=True, completed_gates={"archive"})
    strict_errors = validate(package, allow_incomplete=False)
    if strict_errors:
        failed_manifest = json.loads((package / "manifest.json").read_text("utf-8"))
        failed_manifest.setdefault("gates", {})["archive"] = "review"
        sync_gate_ledger(package, failed_manifest, write=True)
        sync_favorite_status(package, "review")
        print("\n".join(strict_errors))
        return 2
    print(json.dumps({"status": "passed", "package": str(package)}, ensure_ascii=False))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

#!/usr/bin/env python3
"""Revalidate existing bounded comment evidence without fetching the source again."""

from __future__ import annotations

import argparse
import json
from datetime import datetime
from pathlib import Path

from gate_ledger import atomic_manifest, sync_gate_ledger
from manifest_changes import mark_manifest_change
from validate_social_analysis import validate_comments


def finalize_comments(package: Path) -> dict:
    package = package.expanduser().resolve()
    manifest_path = package / "manifest.json"
    comments_path = package / "evidence" / "comments" / "top-comments.json"
    manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
    comments = json.loads(comments_path.read_text(encoding="utf-8"))

    errors = validate_comments(comments)
    if errors:
        raise ValueError("; ".join(errors))

    evidence_status = str(comments.get("status") or "")
    retained = comments.get("comments") or []
    if evidence_status == "passed":
        capture_status = "passed"
    elif evidence_status == "unavailable" and not retained:
        capture_status = "confirmed_no_public_source"
    else:
        raise ValueError(f"comments evidence remains {evidence_status or 'missing'}")

    manifest.setdefault("gates", {})["comments"] = "passed"
    manifest["comment_capture_status"] = capture_status
    mark_manifest_change(manifest, "system_maintenance", datetime.now().astimezone().isoformat(timespec="seconds"))
    atomic_manifest(manifest_path, manifest)
    ledger, _ = sync_gate_ledger(package, manifest, write=True, completed_gates={"comments"})
    return {
        "status": "passed",
        "package": str(package),
        "comment_capture_status": capture_status,
        "retained_comments": len(retained),
        "gate_status": ledger["comments"]["status"],
        "source_refetched": False,
    }


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--package", required=True, type=Path)
    args = parser.parse_args()
    try:
        result = finalize_comments(args.package)
    except (OSError, json.JSONDecodeError, ValueError) as error:
        print(json.dumps({"status": "failed", "error": str(error)}, ensure_ascii=False))
        return 2
    print(json.dumps(result, ensure_ascii=False))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

#!/usr/bin/env python3
"""Re-fingerprint an already-settled, strictly valid asset bundle without semantic reruns."""

from __future__ import annotations

import argparse
import json
from pathlib import Path

from gate_ledger import sync_gate_ledger
from validate_source_assets import validate_package


SETTLED_TYPE_STATUSES = {"admitted", "no_asset", "not_applicable"}


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--package", required=True, type=Path)
    args = parser.parse_args()
    package = args.package.expanduser().resolve()
    manifest_path = package / "manifest.json"
    manifest = json.loads(manifest_path.read_text("utf-8"))
    review = manifest.get("asset_review") if isinstance(manifest.get("asset_review"), dict) else {}
    applicable = [str(item) for item in review.get("applicable_types") or ["quote", "viewpoint", "hook", "structure"]]
    type_status = review.get("type_status") if isinstance(review.get("type_status"), dict) else {}
    unsettled = [item for item in applicable if str(type_status.get(item) or "") not in SETTLED_TYPE_STATUSES]
    if review.get("understanding_pending") is not False or review.get("pending_types") or unsettled:
        print(json.dumps({
            "status": "blocked",
            "reason": "semantic_asset_units_not_settled",
            "understanding_pending": review.get("understanding_pending"),
            "pending_types": review.get("pending_types") or [],
            "unsettled_types": unsettled,
        }, ensure_ascii=False))
        return 2
    ledger, _ = sync_gate_ledger(package, manifest, write=False)
    if str((ledger.get("analysis") or {}).get("status") or "") not in {"passed", "not_applicable"}:
        print(json.dumps({"status": "blocked", "reason": "analysis_gate_not_settled"}, ensure_ascii=False))
        return 2
    errors = validate_package(package, allow_review=False)
    if errors:
        print("\n".join(errors))
        return 2
    manifest.setdefault("gates", {})["cards"] = "passed"
    sync_gate_ledger(package, manifest, write=True, completed_gates={"cards"})
    print(json.dumps({"status": "passed", "package": str(package), "gate": "cards"}, ensure_ascii=False))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

#!/usr/bin/env python3
"""Validate a corrected transcript review and make quotation readiness explicit."""

from __future__ import annotations

import argparse
import hashlib
import json
import os
import re
import tempfile
from datetime import datetime
from pathlib import Path
from typing import Any

from manifest_changes import mark_manifest_change, refresh_artifact_hashes

CHECKS = ("names_and_products", "numbers_and_units", "english_terms", "quote_candidates", "automated_low_confidence_ranges")


def now_iso() -> str:
    return datetime.now().astimezone().isoformat(timespec="seconds")


def atomic_json(path: Path, value: Any) -> None:
    fd, temporary = tempfile.mkstemp(prefix=f".{path.name}.", dir=path.parent)
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as handle:
            json.dump(value, handle, ensure_ascii=False, indent=2)
            handle.write("\n")
            handle.flush()
            os.fsync(handle.fileno())
        os.replace(temporary, path)
    finally:
        if os.path.exists(temporary):
            os.unlink(temporary)


def sha256_file(path: Path) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        for chunk in iter(lambda: handle.read(1024 * 1024), b""):
            digest.update(chunk)
    return digest.hexdigest()


def compact_transcript(text: str) -> str:
    text = re.sub(r"^---.*?---", "", text, flags=re.S)
    text = re.sub(r"^#.*$|^>.*$|^- 内容：.*$|^- 模型：.*$|^- 自动 QA：.*$|^- 说明：.*$", "", text, flags=re.M)
    text = re.sub(r"`\d\d:\d\d(?:–|-)\d\d:\d\d`", "", text)
    return re.sub(r"\s+|[-*_#]", "", text)


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--package", required=True, type=Path)
    args = parser.parse_args()
    package = args.package.expanduser().resolve()
    evidence = package / "evidence" / "transcript"
    corrected_path = evidence / "逐字稿-校正版.md"
    raw_path = evidence / "逐字稿-机器原稿.md"
    review_path = evidence / "transcript-review.json"
    manifest_path = package / "manifest.json"
    corrected = corrected_path.read_text(encoding="utf-8")
    raw = raw_path.read_text(encoding="utf-8")
    review = json.loads(review_path.read_text(encoding="utf-8"))
    errors = []
    if "pending_human_or_agent_audio_review" in corrected or len(compact_transcript(corrected)) < 20:
        errors.append("corrected transcript is still pending or empty")
    raw_length = len(compact_transcript(raw))
    corrected_length = len(compact_transcript(corrected))
    ratio = corrected_length / max(raw_length, 1)
    if ratio < 0.65 or ratio > 1.45:
        errors.append(f"corrected/raw compact-length ratio is suspicious: {ratio:.3f}")
    if review.get("status") != "passed" or not review.get("reviewer") or not review.get("reviewed_at"):
        errors.append("review status, reviewer, and reviewed_at must be completed")
    if review.get("coverage") != "all_speech_and_quote_candidates":
        errors.append("coverage must be all_speech_and_quote_candidates")
    checks = review.get("checks") or {}
    for check in CHECKS:
        if checks.get(check) not in ("passed", "not_applicable"):
            errors.append(f"review check not complete: {check}")
    if review.get("unresolved"):
        errors.append("unresolved transcript items remain")
    if not review.get("evidence_method"):
        errors.append("review needs at least one evidence_method")
    if errors:
        print("\n".join(errors))
        return 2
    manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
    transcript = manifest.setdefault("transcript", {})
    transcript["lexical_review_status"] = "passed"
    transcript["quotation_ready"] = True
    transcript["corrected_transcript"] = "evidence/transcript/逐字稿-校正版.md"
    transcript["review_log"] = "evidence/transcript/transcript-review.json"
    transcript["corrected_sha256"] = sha256_file(corrected_path)
    manifest.setdefault("gates", {})["extraction"] = "passed"
    manifest["gates"]["fidelity"] = "passed"
    mark_manifest_change(manifest, "agent_analysis", now_iso())
    known = {item.get("path") for item in manifest.get("artifacts") or [] if isinstance(item, dict)}
    if "evidence/transcript/逐字稿-校正版.md" in known:
        for item in manifest["artifacts"]:
            if item.get("path") == "evidence/transcript/逐字稿-校正版.md":
                item["kind"] = "transcript_corrected"
                item["sha256"] = transcript["corrected_sha256"]
    refresh_artifact_hashes(package, manifest, {
        "evidence/transcript/逐字稿-校正版.md",
        "evidence/transcript/transcript-review.json",
    })
    atomic_json(manifest_path, manifest)
    from gate_ledger import sync_gate_ledger
    sync_gate_ledger(package, manifest, write=True, completed_gates={"extraction", "fidelity"})
    print(json.dumps({"package": str(package), "quotation_ready": True, "length_ratio": round(ratio, 3)}, ensure_ascii=False))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

#!/usr/bin/env python3
"""Versioned, fingerprinted quality-gate ledger for incremental content ingest."""

from __future__ import annotations

import argparse
import hashlib
import json
import os
import re
import tempfile
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
from typing import Any

from package_document import package_document_path


@dataclass(frozen=True)
class GateSpec:
    rule_version: str
    depends_on: tuple[str, ...]
    executor: str
    context_policy: str


# Bump only the gate whose acceptance rule or deterministic implementation changed.
# Its transitive dependants become stale automatically; unrelated gates stay reusable.
GATE_SPECS: dict[str, GateSpec] = {
    "identity": GateSpec("identity-v1", (), "deterministic", "manifest_only"),
    "acquisition": GateSpec("acquisition-v1", ("identity",), "deterministic", "manifest_only"),
    "extraction": GateSpec("extraction-v4", ("acquisition",), "local_compute", "compact_review_packet"),
    "fidelity": GateSpec("fidelity-v6", ("extraction",), "agent_review", "targeted_evidence"),
    "comments": GateSpec("comments-v3", ("identity",), "no_login_public_adapter", "adaptive_bounded_comment_sample_plus_independent_topic_signals"),
    "cover": GateSpec("cover-v2", ("acquisition",), "agent_review", "cover_plus_bounded_text"),
    "analysis": GateSpec("analysis-v2", (), "agent_review", "carrier_evidence_manifest_plus_type_revisions"),
    "cards": GateSpec("cards-v2", ("analysis",), "agent_review", "type_scoped_candidates_plus_locators"),
    "archive": GateSpec("archive-v2-package-signature", ("identity", "acquisition", "extraction", "fidelity", "analysis", "comments", "cover", "cards"), "deterministic", "package_signature"),
}

LEDGER_STATUSES = {"passed", "review", "partial", "failed", "not_applicable", "stale", "missing"}
_FILE_DIGEST_CACHE: dict[str, tuple[tuple[int, int, int, int, int], str]] = {}
FINGERPRINT_CACHE_SCHEMA = "creator-gate-fingerprint-cache/v1"
FINGERPRINT_CACHE_RELATIVE = Path("evidence/.gate-fingerprint-cache.json")
INPUT_FINGERPRINT_SCHEMA = "semantic-evidence-v3"
CARDS_OUTPUT_FINGERPRINT_SCHEMA = "semantic-assets-v2-user-visuals-excluded"
FIDELITY_OUTPUT_FINGERPRINT_SCHEMA = "lexical-fidelity-v2"
IMAGE_POST_OCR_POLICY_VERSION = "image-ocr-mandatory-v1"


def effective_rule_version(manifest: dict, gate: str) -> str:
    """Version the mandatory OCR rule only for image-post dependent gates."""
    version = GATE_SPECS[gate].rule_version
    if manifest.get("content_form") == "image_post" and gate in {"extraction", "fidelity", "analysis"}:
        return f"{version}+{IMAGE_POST_OCR_POLICY_VERSION}"
    return version


def effective_dependencies(manifest: dict, gate: str) -> tuple[str, ...]:
    dependencies = GATE_SPECS[gate].depends_on
    if manifest.get("content_form") == "image_post" and gate == "analysis":
        return ("fidelity",)
    return dependencies


def now_iso() -> str:
    return datetime.now().astimezone().isoformat(timespec="seconds")


def digest(value: Any) -> str:
    payload = json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":"), default=str)
    return hashlib.sha256(payload.encode("utf-8")).hexdigest()


def load_fingerprint_cache(package: Path) -> dict:
    path = package / FINGERPRINT_CACHE_RELATIVE
    try:
        value = json.loads(path.read_text("utf-8"))
    except (OSError, json.JSONDecodeError):
        return {"schema": FINGERPRINT_CACHE_SCHEMA, "files": {}, "dirty": False}
    if value.get("schema") != FINGERPRINT_CACHE_SCHEMA or not isinstance(value.get("files"), dict):
        return {"schema": FINGERPRINT_CACHE_SCHEMA, "files": {}, "dirty": False}
    files = {
        relative: metadata
        for relative, metadata in value["files"].items()
        if isinstance(relative, str) and (package / relative).is_file()
    }
    return {
        "schema": FINGERPRINT_CACHE_SCHEMA,
        "files": files,
        "dirty": len(files) != len(value["files"]),
    }


def persist_fingerprint_cache(package: Path, cache: dict) -> None:
    if not cache.get("dirty"):
        return
    path = package / FINGERPRINT_CACHE_RELATIVE
    path.parent.mkdir(parents=True, exist_ok=True)
    atomic_manifest(path, {"schema": FINGERPRINT_CACHE_SCHEMA, "files": cache.get("files") or {}})
    cache["dirty"] = False


def file_digest(path: Path, package: Path | None = None, persistent_cache: dict | None = None) -> str | None:
    try:
        stat = path.stat()
    except OSError:
        return None
    if not path.is_file():
        return None
    cache_key = str(path.resolve())
    signature = (stat.st_dev, stat.st_ino, stat.st_size, stat.st_mtime_ns, stat.st_ctime_ns)
    cached = _FILE_DIGEST_CACHE.get(cache_key)
    if cached and cached[0] == signature:
        return cached[1]
    relative = None
    if package is not None and persistent_cache is not None:
        try:
            relative = str(path.resolve().relative_to(package.resolve()))
        except ValueError:
            relative = None
        persisted = (persistent_cache.get("files") or {}).get(relative) if relative else None
        if isinstance(persisted, dict) and tuple(persisted.get("signature") or ()) == signature and persisted.get("sha256"):
            result = str(persisted["sha256"])
            _FILE_DIGEST_CACHE[cache_key] = (signature, result)
            return result
    value = hashlib.sha256()
    with path.open("rb") as handle:
        for chunk in iter(lambda: handle.read(1024 * 1024), b""):
            value.update(chunk)
    result = value.hexdigest()
    _FILE_DIGEST_CACHE[cache_key] = (signature, result)
    if relative is not None and persistent_cache is not None:
        persistent_cache.setdefault("files", {})[relative] = {"signature": list(signature), "sha256": result}
        persistent_cache["dirty"] = True
    return result


def markdown_section(path: Path, heading: str) -> str:
    if not path.is_file():
        return ""
    text = path.read_text("utf-8")
    match = re.search(rf"^##\s+{re.escape(heading)}\s*$\n(.*?)(?=^##\s+|\Z)", text, re.MULTILINE | re.DOTALL)
    return match.group(1).strip() if match else ""


def markdown_sections(path: Path, headings: tuple[str, ...]) -> dict[str, str]:
    if not path.is_file():
        return {heading: "" for heading in headings}
    text = path.read_text("utf-8")
    return {
        heading: (match.group(1).strip() if (match := re.search(
            rf"^##\s+{re.escape(heading)}\s*$\n(.*?)(?=^##\s+|\Z)", text, re.MULTILINE | re.DOTALL,
        )) else "")
        for heading in headings
    }


def selected_artifacts(manifest: dict, kinds: set[str]) -> list[dict]:
    selected = []
    for item in manifest.get("artifacts") or []:
        if isinstance(item, dict) and item.get("kind") in kinds:
            selected.append({key: item.get(key) for key in ("path", "kind", "sha256")})
    return sorted(selected, key=lambda item: (str(item.get("kind")), str(item.get("path"))))


def artifact_file_states(package: Path, artifacts: list[dict]) -> tuple[list[dict], bool]:
    """Fingerprint artifact presence/size cheaply without rereading large media.

    Timestamps are deliberately excluded: iCloud may touch them without
    changing bytes, which must not reopen extraction and every downstream gate.
    """
    states: list[dict] = []
    complete = True
    package_root = package.resolve()
    for artifact in artifacts:
        relative = str(artifact.get("path") or "")
        state = {
            "path": relative,
            "kind": artifact.get("kind"),
            "sha256": artifact.get("sha256"),
            "exists": False,
            "bytes": 0,
        }
        try:
            target = (package / relative).resolve()
            target.relative_to(package_root)
            stat = target.stat()
            if target.is_file() and stat.st_size > 0:
                state.update({
                    "exists": True,
                    "bytes": stat.st_size,
                })
            else:
                complete = False
        except (OSError, ValueError):
            complete = False
        states.append(state)
    return states, complete


def registered_package_signature(package: Path, manifest: dict) -> tuple[dict, bool]:
    artifacts = [
        item for item in (manifest.get("artifacts") or [])
        if isinstance(item, dict) and str(item.get("path") or "").strip()
    ]
    states, complete = artifact_file_states(package, artifacts)
    registered = {str(item.get("path") or "") for item in artifacts}
    declared = []
    for label in ("media", "cover"):
        record = manifest.get(label) if isinstance(manifest.get(label), dict) else {}
        relative = str(record.get("path") or "").strip()
        if not relative:
            continue
        entry_states, entry_complete = artifact_file_states(package, [{"path": relative, "kind": label}])
        declared.extend(entry_states)
        complete = complete and entry_complete and (label != "media" or relative in registered)
    return {"registered": states, "declared": declared}, complete


def semantic_asset_bundle_fingerprint(package: Path) -> str | None:
    """Fingerprint Agent semantics without user visual-library interactions."""
    path = package / "evidence/assets/assets.json"
    try:
        bundle = json.loads(path.read_text("utf-8"))
    except (OSError, json.JSONDecodeError):
        return None
    if not isinstance(bundle, dict):
        return None
    projected = dict(bundle)
    projected["assets"] = [
        asset for asset in bundle.get("assets") or []
        if not (
            isinstance(asset, dict)
            and asset.get("origin") == "user_injected"
            and asset.get("type") in {"image", "shot", "frame"}
        )
    ]
    return digest(projected)


def transcript_fidelity_fingerprint(package: Path) -> str | None:
    """Fingerprint transcript words/timing without semantic chapter navigation.

    Chapter titles, summaries, grouping, and review explanations are downstream
    reading aids. Editing them must not reopen an already settled lexical review.
    """
    path = package / "evidence/transcript/逐字稿-章节版.json"
    try:
        transcript = json.loads(path.read_text("utf-8"))
    except (OSError, json.JSONDecodeError):
        return None
    if not isinstance(transcript, dict):
        return None

    chapter_keys = (
        "id", "start", "end", "paragraphs", "paragraph_blocks",
        "boundary_source", "punctuation_source", "source_boundary_start",
        "source_segment_count", "source_segment_ids", "source_segment_locators",
        "source_segment_range", "text_fidelity",
    )
    chapters = []
    for chapter in transcript.get("chapters") or []:
        if isinstance(chapter, dict):
            chapters.append({key: chapter.get(key) for key in chapter_keys})
    projected = {
        "schema_version": transcript.get("schema_version"),
        "source": transcript.get("source"),
        "boundary_source": transcript.get("boundary_source"),
        "speaker_policy": transcript.get("speaker_policy"),
        "speaker_profiles": transcript.get("speaker_profiles"),
        "text_fidelity_status": transcript.get("text_fidelity_status"),
        "lexical_correction_count": transcript.get("lexical_correction_count"),
        "chapters": chapters,
    }
    return digest(projected)


def fidelity_receipt_settled(manifest: dict) -> bool:
    """Return whether canonical transcript receipts settle lexical fidelity."""
    transcript = manifest.get("transcript") or {}
    status = transcript.get("lexical_review_status")
    if status in {"passed", "limited_dialect", "not_applicable_no_reliable_speech"}:
        return True
    if status not in {"targeted_passed", "targeted_limited"}:
        return False
    ranges = transcript.get("lexical_issue_ranges") or []
    receipts = {
        int(item.get("index")): item.get("status")
        for item in transcript.get("lexical_range_reviews") or []
        if isinstance(item, dict) and str(item.get("index", "")).lstrip("-").isdigit()
    }
    return all(receipts.get(index) in {"passed", "limited"} for index in range(len(ranges)))


def paths_fingerprint(package: Path, paths: tuple[str, ...], persistent_cache: dict | None = None) -> tuple[str, int]:
    values = []
    present = 0
    for relative in paths:
        value = file_digest(package / relative, package, persistent_cache)
        values.append((relative, value))
        present += int(value is not None)
    return digest(values), present


def tree_fingerprint(package: Path, relative: str, patterns: tuple[str, ...], persistent_cache: dict | None = None) -> tuple[str, int]:
    """Fingerprint matching files in a canonical evidence subtree."""
    root = package / relative
    paths: set[Path] = set()
    if root.is_dir():
        for pattern in patterns:
            paths.update(path for path in root.rglob(pattern) if path.is_file())
    values = [
        (str(path.relative_to(package)), file_digest(path, package, persistent_cache))
        for path in sorted(paths, key=lambda item: str(item))
    ]
    return digest(values), len(values)


def output_payload(
    package: Path,
    manifest: dict,
    gate: str,
    persistent_cache: dict | None = None,
    favorite_sections: dict[str, str] | None = None,
) -> tuple[Any, bool]:
    source = manifest.get("source") or {}
    if gate == "identity":
        value = {key: source.get(key) for key in ("platform", "content_form", "stable_id", "canonical_url")}
        return value, bool(value.get("platform") and (value.get("stable_id") or value.get("canonical_url")))
    if gate == "acquisition":
        artifacts = selected_artifacts(manifest, {"image", "video", "audio", "live_photo_motion", "author_avatar"})
        states, files_complete = artifact_file_states(package, artifacts)
        media = manifest.get("media") if isinstance(manifest.get("media"), dict) else {}
        media_path = str(media.get("path") or "").strip()
        registered = {str(item.get("path") or "") for item in artifacts}
        media_registered = not media_path or media_path in registered
        has_primary = bool(artifacts or source.get("body"))
        return {
            "artifacts": artifacts,
            "artifact_files": states,
            "media": media,
            "images": manifest.get("image_inventory"),
        }, has_primary and files_complete and media_registered
    if gate == "extraction":
        primary, primary_count = paths_fingerprint(package, (
            "evidence/transcript/transcript-raw.json",
            "evidence/ocr/ocr-raw.json",
            "evidence/ocr/decision.json",
            "evidence/article/body.json",
            "evidence/article/正文-章节版.json",
            "evidence/lyrics/lyrics.json",
        ), persistent_cache)
        ocr, ocr_count = tree_fingerprint(package, "evidence/ocr", ("*.raw.txt", "ocr-raw.json"), persistent_cache)
        return {"primary": primary, "ocr": ocr}, primary_count + ocr_count > 0
    if gate == "fidelity":
        primary, primary_count = paths_fingerprint(package, (
            "evidence/transcript/逐字稿-校正版.md",
            "evidence/transcript/transcript-review.json",
            "evidence/article/正文-章节版.json",
            "evidence/ocr/逐图文字稿-校正版.md",
            "evidence/ocr/decision.json",
            "evidence/lyrics/original.lrc",
        ), persistent_cache)
        transcript = transcript_fidelity_fingerprint(package)
        corrected_ocr, corrected_count = tree_fingerprint(package, "evidence/ocr", ("*.corrected.md",), persistent_cache)
        return {
            "primary": primary,
            "transcript": transcript,
            "corrected_ocr": corrected_ocr,
        }, primary_count + corrected_count > 0 or transcript is not None
    if gate == "comments":
        value = file_digest(package / "evidence/comments/top-comments.json", package, persistent_cache)
        return value, value is not None
    if gate == "cover":
        value, count = paths_fingerprint(package, ("evidence/cover/analysis.json",), persistent_cache)
        return value, count > 0
    if gate == "analysis":
        value = (favorite_sections or {}).get("内容拆解", "")
        return digest({"section": value, "classification": manifest.get("classification")}), bool(value)
    if gate == "cards":
        value = semantic_asset_bundle_fingerprint(package)
        return digest({"assets": value, "section": (favorite_sections or {}).get("素材卡片", "")}), value is not None
    if gate == "archive":
        document = file_digest(package_document_path(package, manifest), package, persistent_cache)
        package_signature, artifacts_complete = registered_package_signature(package, manifest)
        return {"document": document, "package": package_signature}, document is not None and artifacts_complete
    return None, False


def input_fingerprint(manifest: dict, gate: str, ledger: dict) -> str:
    spec = GATE_SPECS[gate]
    dependencies = {}
    for dependency in effective_dependencies(manifest, gate):
        entry = ledger.get(dependency) or {}
        if entry.get("status") == "not_applicable":
            continue
        dependencies[dependency] = {
            "rule_version": entry.get("rule_version"),
            "output_fingerprint": entry.get("output_fingerprint"),
        }
    context = {
        "mode": manifest.get("mode"),
        "content_form": manifest.get("content_form"),
        "platform": (manifest.get("source") or {}).get("platform"),
    }
    if gate == "analysis":
        asset_review = manifest.get("asset_review") or {}
        context["semantic_evidence"] = {
            "understanding_revision": asset_review.get("understanding_revision"),
            "type_revisions": asset_review.get("type_revisions") or {},
            "source_changed_after_analysis": bool(asset_review.get("source_changed_after_analysis")),
            "analysis_profile": asset_review.get("analysis_profile"),
        }
    return digest({"rule_version": effective_rule_version(manifest, gate), "dependencies": dependencies, "context": context})


def applicable(manifest: dict, gate: str, summary_status: str) -> bool:
    if summary_status == "not_applicable":
        return False
    mode = manifest.get("mode", "full")
    platform = str((manifest.get("source") or {}).get("platform") or "").lower()
    if gate in {"analysis", "cover"} and mode not in {"analyze", "cards", "full"}:
        return False
    if gate == "cards" and mode not in {"cards", "full"}:
        return False
    if gate == "cover" and platform not in {"xhs", "douyin"}:
        return False
    return True


def sync_gate_ledger(package: Path, manifest: dict, *, write: bool = False, completed_gates: set[str] | None = None) -> tuple[dict, list[str]]:
    """Reconcile summary gates with versions/fingerprints and return stale gates."""
    package = package.expanduser().resolve()
    previous = manifest.get("gate_ledger") if isinstance(manifest.get("gate_ledger"), dict) else {}
    summary = manifest.setdefault("gates", {})
    comments_path = package / "evidence/comments/top-comments.json"
    try:
        comments_evidence = json.loads(comments_path.read_text("utf-8"))
    except (OSError, json.JSONDecodeError):
        comments_evidence = {}
    comment_topic_status = str(comments_evidence.get("topic_signal_status") or "")
    if comment_topic_status:
        manifest["comment_topic_signal_status"] = comment_topic_status
    if comment_topic_status == "pending_analysis":
        # A successful bounded capture is not the same checkpoint as topic
        # interpretation. Never let an old summary pass hide that Agent unit.
        summary["comments"] = "review"
    ledger: dict[str, dict] = {}
    invalidated: list[str] = []
    timestamp = now_iso()
    completed_gates = completed_gates or set()
    persistent_cache = load_fingerprint_cache(package)
    favorite_sections = markdown_sections(package_document_path(package, manifest), ("内容拆解", "素材卡片"))

    for gate, spec in GATE_SPECS.items():
        rule_version = effective_rule_version(manifest, gate)
        dependencies = effective_dependencies(manifest, gate)
        summary_status = str(summary.get(gate) or "missing")
        if summary_status not in LEDGER_STATUSES:
            summary_status = "failed"
        prior = previous.get(gate) if isinstance(previous.get(gate), dict) else None
        output, has_output = output_payload(package, manifest, gate, persistent_cache, favorite_sections)
        output_fp = digest(output) if output is not None else None
        current_input = input_fingerprint(manifest, gate, ledger)
        status = summary_status
        reason = None

        if not applicable(manifest, gate, summary_status):
            status = "not_applicable"
        elif any((ledger.get(dependency) or {}).get("status") not in {"passed", "not_applicable"} for dependency in dependencies):
            status, reason = "stale", "dependency_not_settled"
        elif gate in completed_gates:
            status = summary_status
        elif prior:
            if prior.get("rule_version") != rule_version:
                status, reason = "stale", "rule_version_changed"
            elif gate == "cards" and prior.get("output_fingerprint_schema") != CARDS_OUTPUT_FINGERPRINT_SCHEMA:
                # Visual favorites share the canonical bundle but are not Agent
                # semantic output. Adopt the semantic-only fingerprint without
                # reopening an already settled type gate.
                if summary_status in {"passed", "review", "partial"} and has_output:
                    status = summary_status
                else:
                    status, reason = "stale", "output_fingerprint_schema_changed"
            elif gate == "fidelity" and prior.get("output_fingerprint_schema") != FIDELITY_OUTPUT_FINGERPRINT_SCHEMA:
                # v1 hashed the entire chapter document, including titles,
                # summaries, grouping, and navigation metadata. Adopt the
                # lexical-only projection only when canonical review receipts
                # prove the transcript decision was already settled. Packages
                # without a chapter transcript are unaffected by this schema.
                has_chapter_transcript = (package / "evidence/transcript/逐字稿-章节版.json").is_file()
                can_migrate = not has_chapter_transcript or fidelity_receipt_settled(manifest)
                if summary_status in {"passed", "review", "partial"} and has_output and can_migrate:
                    status = summary_status
                else:
                    status, reason = "stale", "output_fingerprint_schema_changed"
            elif prior.get("input_fingerprint_schema") != INPUT_FINGERPRINT_SCHEMA:
                # v1 included the dependency's transient ledger status in the
                # fingerprint. Merely settling stale -> passed therefore
                # invalidated every downstream gate although no evidence had
                # changed. Migrate a verifiable passed output in place; actual
                # output changes and non-passed summaries still require work.
                if summary_status in {"passed", "review", "partial"} and has_output and prior.get("output_fingerprint") == output_fp:
                    # A fingerprint-schema migration changes how we cache a
                    # decision; it is not new source evidence. Preserve the
                    # prior honest capability state instead of converting a
                    # review/partial checkpoint into an automatic semantic
                    # rerun. Strict completion remains unchanged.
                    status = summary_status
                else:
                    status, reason = "stale", "input_fingerprint_schema_changed"
            elif prior.get("status") == "stale" \
                    and summary_status == "passed" \
                    and has_output \
                    and prior.get("input_fingerprint") == current_input \
                    and prior.get("output_fingerprint") == output_fp \
                    and (spec.executor == "deterministic" or prior.get("invalidation_reason") == "dependency_not_settled"):
                # A deterministic gate may be revalidated from its current
                # verifiable output. Downstream gates that were stale only
                # because that dependency had not settled can then reuse an
                # unchanged certified output. Do not revive semantic gates
                # invalidated by an actual input/output or rule change.
                status = "passed"
            elif prior.get("input_fingerprint") != current_input:
                status, reason = "stale", "dependency_or_input_changed"
            elif prior.get("status") == "passed" and (not has_output or prior.get("output_fingerprint") != output_fp):
                status, reason = "stale", "output_missing_or_changed"
            elif summary_status != prior.get("summary_status"):
                status = summary_status
            else:
                status = str(prior.get("status") or summary_status)
        elif summary_status == "passed" and not has_output:
            status, reason = "stale", "legacy_pass_without_verifiable_output"
        elif summary_status in {"review", "failed"} and not has_output:
            status, reason = "stale", "required_output_missing"

        entry = {
            "status": status,
            "summary_status": summary_status,
            "rule_version": rule_version,
            "input_fingerprint": current_input,
            "input_fingerprint_schema": INPUT_FINGERPRINT_SCHEMA,
            "output_fingerprint": output_fp,
            "depends_on": list(dependencies),
            "executor": spec.executor,
            "context_policy": spec.context_policy,
            "checked_at": timestamp,
        }
        if gate == "cards":
            entry["output_fingerprint_schema"] = CARDS_OUTPUT_FINGERPRINT_SCHEMA
        elif gate == "fidelity":
            entry["output_fingerprint_schema"] = FIDELITY_OUTPUT_FINGERPRINT_SCHEMA
        if gate in completed_gates and status == "passed":
            entry["completed_at"] = timestamp
        elif prior and prior.get("completed_at"):
            entry["completed_at"] = prior["completed_at"]
        elif status == "passed":
            entry["completed_at"] = timestamp
        if reason:
            entry["invalidation_reason"] = reason
            entry["invalidated_at"] = timestamp
        elif status == "stale" and prior and prior.get("invalidation_reason"):
            # Keep the cause while a stale checkpoint remains unresolved. The
            # next reconciliation must not lose the distinction between a
            # dependency-only cascade and a real evidence/rule invalidation.
            entry["invalidation_reason"] = prior["invalidation_reason"]
            entry["invalidated_at"] = prior.get("invalidated_at") or timestamp
        if status == "stale":
            invalidated.append(gate)
        ledger[gate] = entry

    manifest["gate_ledger_schema"] = "creator-gates/v1"
    manifest["gate_ledger"] = ledger
    if write:
        atomic_manifest(package / "manifest.json", manifest)
        persist_fingerprint_cache(package, persistent_cache)
    return ledger, invalidated


def execution_plan_from_ledger(ledger: dict, invalidated: list[str] | None = None) -> dict:
    """Build a run/review/reuse plan from an already reconciled ledger."""
    invalidated = invalidated or []
    run = []
    review = []
    reuse = []
    for gate in GATE_SPECS:
        entry = ledger[gate]
        item = {
            "gate": gate,
            "status": entry["status"],
            "rule_version": entry["rule_version"],
            "executor": entry["executor"],
            "context_policy": entry["context_policy"],
            "reason": entry.get("invalidation_reason"),
        }
        if entry["status"] in {"stale", "missing", "failed"}:
            run.append(item)
        elif entry["status"] in {"review", "partial"}:
            review.append(item)
        else:
            reuse.append(item)
    return {"schema": "creator-gate-plan/v1", "run": run, "review": review, "reuse": reuse, "invalidated": invalidated}


def execution_plan(package: Path, manifest: dict) -> dict:
    ledger, invalidated = sync_gate_ledger(package, manifest, write=False)
    return execution_plan_from_ledger(ledger, invalidated)


def atomic_manifest(path: Path, manifest: dict) -> None:
    fd, temporary = tempfile.mkstemp(prefix=f".{path.name}.", suffix=".tmp", dir=path.parent)
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as handle:
            json.dump(manifest, handle, ensure_ascii=False, indent=2)
            handle.write("\n")
            handle.flush()
            os.fsync(handle.fileno())
        os.replace(temporary, path)
    finally:
        if os.path.exists(temporary):
            os.unlink(temporary)


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("command", choices=("sync", "plan"))
    parser.add_argument("--package", required=True, type=Path)
    args = parser.parse_args()
    package = args.package.expanduser().resolve()
    manifest = json.loads((package / "manifest.json").read_text("utf-8"))
    if args.command == "sync":
        ledger, invalidated = sync_gate_ledger(package, manifest, write=True)
        result = {"status": "synced", "invalidated": invalidated, "ledger": ledger}
    else:
        result = execution_plan(package, manifest)
    print(json.dumps(result, ensure_ascii=False, indent=2))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

#!/usr/bin/env python3
"""Keep user edits, Agent analysis, and maintenance timestamps independent."""

from __future__ import annotations

import hashlib
from pathlib import Path
from typing import Any


CHANGE_FIELDS = {
    "user_edit": "user_edited_at",
    "agent_analysis": "analysis_updated_at",
    "system_maintenance": "system_updated_at",
}


def mark_manifest_change(manifest: dict[str, Any], kind: str, changed_at: str) -> None:
    normalized_kind = kind if kind in CHANGE_FIELDS else "system_maintenance"
    tracking = manifest.setdefault("change_tracking", {})
    tracking[CHANGE_FIELDS[normalized_kind]] = changed_at
    tracking["last_change_kind"] = normalized_kind
    tracking["last_changed_at"] = changed_at
    # Legacy consumers still use this broad timestamp. New UI code must use the
    # domain-specific fields above instead of inferring an edit from this value.
    manifest["updated_at"] = changed_at


def refresh_artifact_hashes(package: Path, manifest: dict[str, Any], relative_paths: list[str] | tuple[str, ...] | set[str]) -> int:
    """Refresh only explicitly mutated, already-registered artifacts."""
    wanted = {str(value) for value in relative_paths}
    changed = 0
    for artifact in manifest.get("artifacts") or []:
        if not isinstance(artifact, dict) or str(artifact.get("path") or "") not in wanted:
            continue
        target = package / str(artifact["path"])
        if not target.is_file():
            continue
        digest = hashlib.sha256()
        with target.open("rb") as handle:
            for chunk in iter(lambda: handle.read(1024 * 1024), b""):
                digest.update(chunk)
        value = digest.hexdigest()
        if artifact.get("sha256") != value:
            artifact["sha256"] = value
            changed += 1
    return changed

#!/usr/bin/env python3
"""Normalize a rendered social-comment sample and retain up to ten useful comments."""

from __future__ import annotations

import argparse
import json
import os
import re
import tempfile
from datetime import datetime
from pathlib import Path
from typing import Any

from comment_selection import select_comments, selection_policy
from manifest_changes import mark_manifest_change


FORBIDDEN_LOGIN_METHOD = re.compile(r"authorized|authenticated|logged[_ -]?in|cookie|browser[_ -]?profile", re.I)


def reject_login_comment_input(method: Any) -> None:
    if FORBIDDEN_LOGIN_METHOD.search(str(method or "")):
        raise ValueError("logged-in comment evidence is disabled; use a no-login public adapter")


def now_iso() -> str:
    return datetime.now().astimezone().isoformat(timespec="seconds")


def atomic_json(path: Path, value: Any) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    fd, temporary = tempfile.mkstemp(prefix=f".{path.name}.", dir=path.parent)
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as handle:
            json.dump(value, handle, ensure_ascii=False, indent=2)
            handle.write("\n")
            handle.flush()
            os.fsync(handle.fileno())
        os.replace(temporary, path)
    finally:
        if os.path.exists(temporary):
            os.unlink(temporary)


def parse_count_detail(value: Any) -> tuple[int | None, str]:
    if value is None:
        return None, "unknown"
    visible = str(value).strip()
    raw = visible.lower().replace(",", "").replace(" ", "")
    if not raw or raw in {"赞", "回复", "点赞", "unknown"}:
        return None, "unknown"
    match = re.fullmatch(r"(\d+(?:\.\d+)?)(万|w|k)?(\+)?", raw)
    if not match:
        return None, "unknown"
    multiplier = {"万": 10000, "w": 10000, "k": 1000}.get(match.group(2), 1)
    try:
        value = int(float(match.group(1)) * multiplier)
    except ValueError:
        return None, "unknown"
    return value, "lower_bound" if match.group(3) else "exact"


def parse_count(value: Any) -> int | None:
    return parse_count_detail(value)[0]


def normalize(item: dict, index: int) -> dict | None:
    text = str(item.get("text") or item.get("content") or item.get("comment") or "").strip()
    if not text:
        return None
    comment_type = str(item.get("comment_type") or item.get("type") or "").strip().lower()
    if (
        item.get("is_reply") is True
        or item.get("top_level") is False
        or item.get("parent_comment_id")
        or comment_type in {"reply", "subcomment", "child"}
    ):
        return None
    raw_likes = item.get("like_count_raw", item.get("like_count", item.get("likes", item.get("likedCount"))))
    comment_id = str(item.get("comment_id") or item.get("id") or "").strip() or None
    like_count, like_precision = parse_count_detail(raw_likes)
    if like_count is None and item.get("like_count") is not None:
        like_count, like_precision = parse_count_detail(item.get("like_count"))
    raw_replies = item.get("reply_count_raw", item.get("reply_count", item.get("replies")))
    reply_count, reply_precision = parse_count_detail(raw_replies)
    if reply_count is None and item.get("reply_count") is not None:
        reply_count, reply_precision = parse_count_detail(item.get("reply_count"))
    return {
        "comment_id": comment_id,
        "author_display": str(item.get("author_display") or item.get("author") or item.get("nickname") or ""),
        "text": re.sub(r"\s+", " ", text),
        "like_count": like_count,
        "like_count_raw": str(raw_likes if raw_likes is not None else "unknown"),
        "like_count_precision": str(item.get("like_count_precision") or like_precision),
        "reply_count": reply_count,
        "reply_count_raw": str(raw_replies if raw_replies is not None else "unknown"),
        "reply_count_precision": str(item.get("reply_count_precision") or reply_precision),
        "created_at": item.get("created_at"),
        "locator": item.get("locator") or (f"comment:{comment_id}" if comment_id else f"rendered-sample:{index}"),
        "is_reply": False,
    }


def deduplicate(items: list[dict]) -> list[dict]:
    seen: set[tuple[str, str]] = set()
    result = []
    for item in items:
        if item.get("comment_id"):
            key = ("id", str(item["comment_id"]))
        else:
            key = ("text", re.sub(r"\s+", " ", str(item.get("text") or "")).strip().casefold())
        if key in seen:
            continue
        seen.add(key)
        result.append(item)
    return result


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--package", required=True, type=Path)
    parser.add_argument("--input", required=True, type=Path, help="JSON array or object containing comments/items/list")
    parser.add_argument("--platform-hot-order", action="store_true", help="Input order is the platform's declared hot ranking")
    parser.add_argument("--ranking-scope", required=True, choices=("platform_hot_loaded_sample", "all_loaded_top_level_comments", "visible_rendered_sample"))
    parser.add_argument("--method", default="anonymous_public_page_dom")
    parser.add_argument("--has-more", choices=("true", "false", "unknown"), default="unknown")
    args = parser.parse_args()

    package = args.package.expanduser().resolve()
    value = json.loads(args.input.expanduser().read_text(encoding="utf-8"))
    input_metadata = value if isinstance(value, dict) else {}
    reject_login_comment_input(input_metadata.get("method") or args.method)
    if isinstance(value, dict):
        items = value.get("comments") or value.get("items") or value.get("list") or []
    else:
        items = value
    if not isinstance(items, list):
        raise ValueError("comment input must be an array")
    raw_sample_size = len(items)
    normalized = [record for index, item in enumerate(items, 1) if isinstance(item, dict) and (record := normalize(item, index))]
    normalized = deduplicate(normalized)
    ranking_review = any(item["like_count"] is None for item in normalized)
    ranking_method = "normalized_like_count_desc"
    top, discarded = select_comments(normalized)
    input_status = input_metadata.get("status")
    if input_status in {"review", "partial", "unavailable"}:
        output_status = input_status
    elif ranking_review:
        output_status = "review"
    else:
        output_status = "passed"
    has_more = None
    if args.has_more != "unknown":
        has_more = args.has_more == "true"
    elif "has_more" in input_metadata:
        input_has_more = input_metadata.get("has_more")
        if isinstance(input_has_more, bool) or input_has_more is None:
            has_more = input_has_more
        elif str(input_has_more).lower() in {"true", "false"}:
            has_more = str(input_has_more).lower() == "true"
    output = {
        "status": output_status,
        "ranking_scope": args.ranking_scope,
        "ranking_method": ranking_method,
        "sample_size": len(normalized),
        "raw_sample_size": raw_sample_size,
        "has_more": has_more,
        "acquired_at": now_iso(),
        "method": input_metadata.get("method") if args.method == "anonymous_public_page_dom" and input_metadata.get("method") else args.method,
        "comments": top,
        "selection_policy": selection_policy(),
        "discarded": discarded,
        "topic_signal_status": "pending_analysis" if top else "no_meaningful_signal",
        "topic_signals": [],
    }
    if normalized and not top:
        output["reason"] = "The captured sample contained no comment meeting the like and information-quality thresholds."
    elif not normalized:
        output["reason"] = "The captured sample contained no usable top-level comment."
        output["status"] = "passed"
    output_status = output["status"]
    if isinstance(input_metadata.get("platform"), str):
        output["platform"] = input_metadata["platform"]
    source = input_metadata.get("source")
    if isinstance(source, dict):
        output["source"] = {
            key: source[key]
            for key in ("stable_id", "title")
            if key in source and isinstance(source[key], (str, int, float, type(None)))
        }
    for key in ("source_verification", "access_boundary", "selectors", "reason"):
        if key in input_metadata:
            output[key] = input_metadata[key]
    target = package / "evidence" / "comments" / "top-comments.json"
    atomic_json(target, output)
    manifest_path = package / "manifest.json"
    manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
    gate_status = "review" if output_status == "unavailable" else output_status
    manifest.setdefault("gates", {})["comments"] = gate_status
    manifest["errors"] = [
        item
        for item in manifest.get("errors") or []
        if item.get("code") not in {"COMMENTS_REQUIRE_BROWSER", "COMMENTS_BROWSER_BOUNDARY", "COMMENTS_PUBLIC_SOURCE_INCOMPLETE"}
    ]
    if output_status != "passed":
        manifest["errors"].append({
            "at": now_iso(),
            "stage": "comments",
            "code": "COMMENTS_PUBLIC_SOURCE_INCOMPLETE",
            "message": output.get("reason") or "The no-login public comment source remains incomplete",
            "recoverable": True,
            "artifact": "evidence/comments/top-comments.json",
        })
    mark_manifest_change(manifest, "system_maintenance", now_iso())
    atomic_json(manifest_path, manifest)
    from gate_ledger import sync_gate_ledger
    sync_gate_ledger(package, manifest, write=True, completed_gates={"comments"})
    print(json.dumps({
        "package": str(package),
        "captured": len(normalized),
        "raw_sample_size": raw_sample_size,
        "retained": len(top),
        "status": output_status,
        "ranking_scope": args.ranking_scope,
        "next": "analyze topic signals or explicitly mark no_meaningful_signal",
    }, ensure_ascii=False, indent=2))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

#!/usr/bin/env python3
"""Build a readable per-image OCR projection without replacing raw evidence."""

from __future__ import annotations

import argparse
import json
import os
import re
import tempfile
from pathlib import Path
from typing import Any


WATERMARK_RE = re.compile(
    r"(?:抖音(?:号)?|小红书(?:号)?|快手(?:号)?|微博(?:号)?|B站(?:号)?|公众号|"
    r"(?:扫描|扫码).{0,8}(?:二维码|码)|二维码|watermark|douyin|xiaohongshu)",
    re.IGNORECASE,
)
HANDLE_RE = re.compile(r"^@[\w.\-]{2,}$")
TERMINAL_RE = re.compile(r"[。！？!?；;：:]$")
BULLET_RE = re.compile(r"^(?:[-*•·]|[①-⑳]|\d+[.)、])\s*")
ASCII_FRAGMENT_RE = re.compile(r"^[A-Za-z0-9_./|©®™+:#-]{1,10}$")


def atomic_json(path: Path, value: Any) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    fd, temporary = tempfile.mkstemp(prefix=f".{path.name}.", dir=path.parent)
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as handle:
            json.dump(value, handle, ensure_ascii=False, indent=2)
            handle.write("\n")
            handle.flush()
            os.fsync(handle.fileno())
        os.replace(temporary, path)
    finally:
        if os.path.exists(temporary):
            os.unlink(temporary)


def clean_line(value: Any) -> str:
    return re.sub(r"\s+", " ", str(value or "")).strip()


def is_fragment(text: str) -> bool:
    compact = re.sub(r"\s+", "", text)
    if not compact:
        return True
    if WATERMARK_RE.search(compact) or HANDLE_RE.fullmatch(compact):
        return True
    if len(compact) <= 1:
        return True
    return bool(ASCII_FRAGMENT_RE.fullmatch(compact))


def join_lines(lines: list[str]) -> str:
    output = ""
    for line in lines:
        if not output:
            output = line
            continue
        if BULLET_RE.match(line) or TERMINAL_RE.search(output):
            output += "\n" + line
        elif re.search(r"[A-Za-z0-9]$", output) and re.match(r"^[A-Za-z0-9]", line):
            output += " " + line
        else:
            output += line
    return output.strip()


def source_lines(record: dict) -> list[dict]:
    lines = [
        {
            "text": clean_line(line.get("text")),
            "y": float(line.get("y")) if line.get("y") is not None else None,
            "height": float(line.get("height")) if line.get("height") is not None else None,
        }
        for line in (record.get("lines") or [])
        if clean_line(line.get("text"))
    ]
    if lines:
        return lines
    return [{"text": clean_line(value), "y": None, "height": None} for value in str(record.get("text") or "").splitlines() if clean_line(value)]


def has_visual_break(previous: dict, current: dict) -> bool:
    if previous["y"] is None or current["y"] is None:
        return False
    previous_height = previous["height"] or 0
    current_height = current["height"] or 0
    gap = previous["y"] - (current["y"] + current_height)
    return gap > max(previous_height, current_height, 0.012) * 1.45


def organize_ocr_record(record: dict) -> dict:
    """Put continuous readable prose before watermarks and isolated fragments."""
    paragraphs: list[str] = []
    fragments: list[str] = []
    active: list[str] = []
    previous: dict | None = None

    def flush() -> None:
        nonlocal active
        value = join_lines(active)
        if value:
            paragraphs.append(value)
        active = []

    for line in source_lines(record):
        text = line["text"]
        if is_fragment(text):
            flush()
            fragments.append(text)
            previous = line
            continue
        if active and (has_visual_break(previous or line, line) or BULLET_RE.match(text)):
            flush()
        active.append(text)
        if TERMINAL_RE.search(text):
            flush()
        previous = line
    flush()

    paragraphs = list(dict.fromkeys(paragraphs))
    fragments = list(dict.fromkeys(fragment for fragment in fragments if fragment not in paragraphs))
    reading_blocks = (
        [{"type": "paragraph", "text": text} for text in paragraphs]
        + [{"type": "fragment", "text": text} for text in fragments]
    )
    substantial_character_count = len(re.findall(
        r"[\u3400-\u9fffA-Za-z0-9]",
        "".join(paragraphs),
    ))
    if substantial_character_count >= 32:
        text_class = "substantial_text"
        text_class_label = "大段文字"
    elif paragraphs or fragments:
        text_class = "sparse_text"
        text_class_label = "零星文字"
    else:
        text_class = "no_text"
        text_class_label = "无文字"
    return {
        **record,
        "paragraphs": paragraphs,
        "fragments": fragments,
        "reading_blocks": reading_blocks,
        "readable_text": "\n\n".join(paragraphs),
        "fragment_text": "\n".join(fragments),
        "text_class": text_class,
        "text_class_label": text_class_label,
        "substantial_character_count": substantial_character_count,
    }


def organize_ocr_records(records: list[dict]) -> list[dict]:
    return [organize_ocr_record(record if isinstance(record, dict) else {}) for record in records]


def refresh_package(package: Path) -> dict:
    raw_path = package / "evidence" / "ocr" / "ocr-raw.json"
    try:
        raw = json.loads(raw_path.read_text("utf-8"))
    except (OSError, json.JSONDecodeError) as error:
        raise ValueError(f"无法读取 OCR 原始证据：{error}") from error
    if not isinstance(raw, list):
        raise ValueError("OCR 原始证据格式不是逐图列表")
    organized = organize_ocr_records(raw)
    output = package / "evidence" / "ocr" / "organized.json"
    atomic_json(output, organized)
    return {"status": "refreshed", "package": str(package), "records": len(organized), "artifact": str(output)}


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--package", type=Path, required=True)
    args = parser.parse_args()
    print(json.dumps(refresh_package(args.package.resolve()), ensure_ascii=False))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

#!/usr/bin/env python3
"""Resolve reusable local OCR backends and normalize their evidence output."""

from __future__ import annotations

import argparse
import importlib.util
import json
import os
import platform
import shutil
import subprocess
import sys
from pathlib import Path
from typing import Any

SCRIPT_DIR = Path(__file__).resolve().parent
if str(SCRIPT_DIR) not in sys.path:
    sys.path.insert(0, str(SCRIPT_DIR))

from resource_guard import local_inference_slot

OCR_COMMAND_ENV = "CREATOR_INGEST_OCR_COMMAND"
OCR_PYTHON_ENV = "CREATOR_INGEST_OCR_PYTHON"
OCR_BACKEND_ENV = "CREATOR_INGEST_OCR_BACKEND"
OCR_CHILD_ENV = "CREATOR_INGEST_OCR_CHILD"

_RAPID_ENGINE: Any | None = None


def native_ocr_command() -> list[str] | None:
    configured = os.getenv(OCR_COMMAND_ENV, "").strip()
    if configured:
        executable = Path(configured).expanduser()
        if executable.is_file():
            return [str(executable)]

    if platform.system() != "Darwin":
        return None
    machine = "arm64" if platform.machine().lower() in {"arm64", "aarch64"} else "x64"
    candidates = [
        Path.home() / ".cache/creator-content-ingest/ocr-runtime/ocr-images",
        Path(__file__).resolve().parents[1] / "bin" / f"ocr-images-darwin-{machine}",
    ]
    for executable in candidates:
        if executable.is_file() and os.access(executable, os.X_OK):
            return [str(executable)]

    swift = shutil.which("swift")
    source = Path(__file__).with_name("ocr_images.swift")
    if swift and source.is_file():
        return [swift, str(source)]
    return None


def rapidocr_available() -> bool:
    return importlib.util.find_spec("rapidocr") is not None and importlib.util.find_spec("onnxruntime") is not None


def requested_backend(value: str | None = None) -> str:
    backend = (value or os.getenv(OCR_BACKEND_ENV, "auto")).strip().lower().replace("-", "_")
    aliases = {
        "vision": "apple_vision",
        "native": "apple_vision",
        "rapid": "rapidocr",
        "onnx": "rapidocr",
        "rapidocr_onnx": "rapidocr",
    }
    backend = aliases.get(backend, backend)
    return backend if backend in {"apple_vision", "rapidocr"} else "auto"


def runtime_details(backend: str | None = None, *, allow_external: bool = False) -> dict:
    selected = requested_backend(backend)
    command = native_ocr_command() if selected in {"auto", "apple_vision"} else None
    if command:
        return {
            "available": True,
            "backend": "Apple Vision",
            "backend_id": "apple_vision",
            "command": command[0],
            "mode": "swift_source" if len(command) > 1 else "native_helper",
            "architecture": platform.machine(),
        }
    if selected == "apple_vision":
        raise RuntimeError("Apple Vision OCR is unavailable in this runtime.")
    configured_python = os.getenv(OCR_PYTHON_ENV, "").strip()
    if allow_external and configured_python and Path(configured_python).expanduser().is_file():
        return {
            "available": True,
            "backend": "RapidOCR ONNX",
            "backend_id": "rapidocr",
            "mode": "python_runtime",
            "python": configured_python,
            "architecture": platform.machine(),
        }
    if rapidocr_available():
        return {
            "available": True,
            "backend": "RapidOCR ONNX",
            "backend_id": "rapidocr",
            "mode": "python_runtime",
            "python": sys.executable,
            "architecture": platform.machine(),
        }
    raise RuntimeError(
        "No compatible local OCR runtime was found. Use Apple Vision on macOS, install "
        "RapidOCR with ONNX Runtime once from Pickoo settings, or configure a compatible "
        f"helper with {OCR_COMMAND_ENV}."
    )


def _native_run(command: list[str], paths: list[Path], timeout: int) -> list[dict]:
    completed = subprocess.run(
        [*command, *map(str, paths)],
        text=True,
        capture_output=True,
        timeout=timeout,
    )
    if not completed.stdout.strip():
        raise RuntimeError(f"OCR produced no JSON: {completed.stderr.strip()}")
    records = json.loads(completed.stdout)
    if completed.returncode not in {0, 1}:
        raise RuntimeError(completed.stderr.strip() or "OCR helper failed")
    return records


def _rapid_engine() -> Any:
    global _RAPID_ENGINE
    if _RAPID_ENGINE is None:
        from rapidocr import RapidOCR

        _RAPID_ENGINE = RapidOCR()
    return _RAPID_ENGINE


def _rapid_record(path: Path) -> dict:
    from PIL import Image

    try:
        with Image.open(path) as image:
            width, height = image.size
        result = _rapid_engine()(path)
        boxes = [] if result.boxes is None else list(result.boxes)
        texts = [] if result.txts is None else list(result.txts)
        scores = [] if result.scores is None else list(result.scores)
        lines = []
        for box, text, score in zip(boxes, texts, scores):
            points = list(box)
            xs = [float(point[0]) for point in points]
            ys = [float(point[1]) for point in points]
            left, right = min(xs), max(xs)
            top, bottom = min(ys), max(ys)
            lines.append({
                "text": str(text),
                "confidence": float(score),
                "x": left / max(width, 1),
                "y": 1.0 - (bottom / max(height, 1)),
                "width": (right - left) / max(width, 1),
                "height": (bottom - top) / max(height, 1),
                "_top": top,
            })
        lines.sort(key=lambda line: (round(float(line["_top"]) / max(height, 1), 3), float(line["x"])))
        for line in lines:
            line.pop("_top", None)
        mean = sum(float(line["confidence"]) for line in lines) / len(lines) if lines else 0.0
        low = sum(1 for line in lines if float(line["confidence"]) < 0.55)
        status = "no_text" if not lines else ("pass" if mean >= 0.72 and low == 0 else "review")
        return {
            "path": str(path),
            "status": status,
            "text": "\n".join(str(line["text"]) for line in lines),
            "mean_confidence": mean,
            "low_confidence_count": low,
            "lines": lines,
            "error": None,
        }
    except Exception as error:
        return {
            "path": str(path),
            "status": "fail",
            "text": "",
            "mean_confidence": 0.0,
            "low_confidence_count": 0,
            "lines": [],
            "error": str(error),
        }


def _rapid_run(paths: list[Path], timeout: int) -> list[dict]:
    configured = os.getenv(OCR_PYTHON_ENV, "").strip()
    if configured and os.getenv(OCR_CHILD_ENV) != "1":
        completed = subprocess.run(
            [configured, str(Path(__file__).resolve()), "--run", *map(str, paths), "--backend", "rapidocr"],
            text=True,
            capture_output=True,
            timeout=timeout,
            env={**os.environ, OCR_PYTHON_ENV: configured, OCR_BACKEND_ENV: "rapidocr", OCR_CHILD_ENV: "1"},
        )
        if completed.returncode != 0 or not completed.stdout.strip():
            raise RuntimeError(completed.stderr.strip() or "RapidOCR runtime failed")
        return json.loads(completed.stdout.strip().splitlines()[-1])
    if not rapidocr_available():
        raise RuntimeError(f"{sys.executable} does not provide RapidOCR with ONNX Runtime")
    return [_rapid_record(path) for path in paths]


def run_ocr(paths: list[Path], *, timeout: int = 300) -> list[dict]:
    if not paths:
        return []
    details = runtime_details(allow_external=True)
    if details["backend_id"] == "apple_vision":
        command = native_ocr_command()
        if not command:
            raise RuntimeError("Apple Vision OCR command disappeared after validation")
        return _native_run(command, paths, timeout)
    with local_inference_slot("cpu-inference"):
        return _rapid_run(paths, timeout)


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--runtime-check", action="store_true")
    parser.add_argument("--run", nargs="*")
    parser.add_argument("--backend", default=None)
    args = parser.parse_args()
    if args.backend:
        os.environ[OCR_BACKEND_ENV] = requested_backend(args.backend)
    if args.run is not None:
        print(json.dumps(run_ocr([Path(value) for value in args.run]), ensure_ascii=False))
        return 0
    print(json.dumps(runtime_details(args.backend), ensure_ascii=False))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

"""Resolve and assign the one canonical, system-named Markdown document per package."""

from __future__ import annotations

import json
import os
import re
from datetime import datetime
from pathlib import Path
from zoneinfo import ZoneInfo


DOCUMENT_NAMING_VERSION = 4
REFERENCE_MIGRATION_VERSION = 1


def _component(value: object, fallback: str, limit: int) -> str:
    clean = re.sub(r"[\x00-\x1f/:*?\"<>|\\]", " ", str(value or ""))
    clean = re.sub(r"\s+", " ", clean).strip() or fallback
    return clean[:limit]


def _date(value: object) -> str:
    text = str(value or "").strip()
    try:
        parsed = datetime.fromisoformat(text.replace("Z", "+00:00")) if text else datetime.now().astimezone()
        if parsed.tzinfo is None:
            parsed = parsed.replace(tzinfo=ZoneInfo("Asia/Shanghai"))
        return parsed.astimezone(ZoneInfo("Asia/Shanghai")).strftime("%Y%m%d")
    except (ValueError, OSError):
        return datetime.now(ZoneInfo("Asia/Shanghai")).strftime("%Y%m%d")


def system_document_name(manifest: dict, title: str = "", author: str = "") -> str:
    source = manifest.get("source") if isinstance(manifest.get("source"), dict) else {}
    content_form = str(manifest.get("content_form") or source.get("content_form") or "").lower()
    platform = str(source.get("platform") or "").lower()
    form_name = {
        "image_post": "图文", "video": "视频", "article": "文章", "document": "文章",
        "audio": "音频", "podcast": "播客", "podcast_episode": "播客", "music": "音乐",
    }.get(content_form, "其他")
    platform_name = {
        "xhs": "小红书", "xiaohongshu": "小红书", "douyin": "抖音", "tiktok": "TikTok",
        "bilibili": "哔哩哔哩", "b23": "哔哩哔哩", "xiaoyuzhou": "小宇宙",
        "netease": "网易云音乐", "netease_music": "网易云音乐", "wechat": "微信公众号",
        "weixin": "微信公众号", "youtube": "YouTube", "web": "网页",
    }.get(platform, _component(source.get("platform_display_name") or platform, "未知渠道", 12))
    publisher = source.get("publisher") if isinstance(source.get("publisher"), dict) else {}
    author_profile = source.get("author_profile") if isinstance(source.get("author_profile"), dict) else {}
    short_title = _component(title or source.get("title"), "未命名", 10)
    short_author = _component(author or source.get("author") or author_profile.get("display_name") or publisher.get("display_name"), "未知作者", 10)
    stamp = _date(source.get("published_at") or source.get("created_at") or manifest.get("created_at") or manifest.get("captured_at"))
    return f"{form_name}-{platform_name}-{short_title}-{short_author}-{stamp}.md"


def _assigned_document_name(manifest: dict | None) -> str:
    manifest = manifest if isinstance(manifest, dict) else {}
    document = manifest.get("document") if isinstance(manifest.get("document"), dict) else {}
    relative = str(document.get("path") or "").strip()
    if relative and Path(relative).name == relative and relative.lower().endswith(".md"):
        return relative
    return ""


def package_document_path(package: Path, manifest: dict | None = None) -> Path:
    """Return the assigned canonical document; ordinary callers never use a legacy fallback."""
    package = Path(package)
    relative = _assigned_document_name(manifest)
    if relative:
        return package / relative
    raise ValueError(f"manifest.document.path is missing or invalid for {package}")


def migration_source_document_path(package: Path, manifest: dict | None = None) -> Path | None:
    """Find a pre-assignment document only while assigning or migrating a package."""
    package = Path(package)
    relative = _assigned_document_name(manifest)
    if relative:
        return package / relative
    legacy = package / "收藏.md"
    if legacy.exists():
        return legacy
    manifest = manifest if isinstance(manifest, dict) else {}
    package_id = str(manifest.get("package_id") or package.name)
    for candidate in sorted(package.glob("*.md")):
        try:
            head = candidate.read_text("utf-8")[:1200]
        except OSError:
            continue
        if re.search(rf"(?m)^favorite_id:\s*[\"']?{re.escape(package_id)}[\"']?\s*$", head):
            return candidate
    return None


def _replace_legacy_reference(value: object, document_name: str) -> tuple[object, int]:
    if isinstance(value, dict):
        changed = 0
        result = {}
        for key, item in value.items():
            next_key, key_changed = _replace_legacy_reference(key, document_name)
            next_item, item_changed = _replace_legacy_reference(item, document_name)
            result[next_key] = next_item
            changed += key_changed + item_changed
        return result, changed
    if isinstance(value, list):
        changed = 0
        result = []
        for item in value:
            next_item, item_changed = _replace_legacy_reference(item, document_name)
            result.append(next_item)
            changed += item_changed
        return result, changed
    if isinstance(value, str) and (value == "收藏.md" or value.startswith("收藏.md#") or value.startswith("收藏.md｜")):
        return document_name + value[len("收藏.md"):], 1
    return value, 0


def _atomic_json(path: Path, value: object) -> None:
    temporary = path.with_name(f".{path.name}.tmp-{os.getpid()}")
    temporary.write_text(json.dumps(value, ensure_ascii=False, indent=2) + "\n", "utf-8")
    os.replace(temporary, path)


def migrate_package_document_references(package: Path, manifest: dict) -> dict:
    """Retarget live legacy locators/caches without rewriting source prose or audit backups."""
    package = Path(package)
    document_path = package_document_path(package, manifest)
    document_name = document_path.name
    report = {"manifest": 0, "asset_json": 0, "markdown": 0, "cache": 0}

    migrated_manifest, manifest_changes = _replace_legacy_reference(manifest, document_name)
    if manifest_changes:
        manifest.clear()
        manifest.update(migrated_manifest)
        report["manifest"] = manifest_changes

    assets_dir = package / "evidence/assets"
    if assets_dir.is_dir():
        for path in sorted(assets_dir.glob("*.json")):
            try:
                payload = json.loads(path.read_text("utf-8"))
            except (OSError, json.JSONDecodeError):
                continue
            migrated, changes = _replace_legacy_reference(payload, document_name)
            if changes:
                _atomic_json(path, migrated)
                report["asset_json"] += changes

    if document_path.is_file():
        text = document_path.read_text("utf-8")
        lines = []
        changes = 0
        for line in text.splitlines(keepends=True):
            if "**证据定位：**" in line and "收藏.md" in line:
                next_line = line.replace("收藏.md｜", f"{document_name}｜").replace("收藏.md#", f"{document_name}#")
                changes += int(next_line != line)
                line = next_line
            lines.append(line)
        if changes:
            temporary = document_path.with_name(f".{document_path.name}.tmp-{os.getpid()}")
            temporary.write_text("".join(lines), "utf-8")
            os.replace(temporary, document_path)
            report["markdown"] = changes

    cache_path = package / "evidence/.gate-fingerprint-cache.json"
    try:
        cache = json.loads(cache_path.read_text("utf-8"))
    except (OSError, json.JSONDecodeError):
        cache = None
    if isinstance(cache, dict) and isinstance(cache.get("files"), dict):
        files = cache["files"]
        legacy_entry = files.pop("收藏.md", None)
        if legacy_entry is not None and document_name not in files:
            files[document_name] = legacy_entry
        stale = [relative for relative in files if not (package / relative).is_file()]
        for relative in stale:
            files.pop(relative, None)
        changes = int(legacy_entry is not None) + len(stale)
        if changes:
            _atomic_json(cache_path, cache)
            report["cache"] = changes

    document = manifest.get("document") if isinstance(manifest.get("document"), dict) else {}
    if document.get("reference_migration_version") != REFERENCE_MIGRATION_VERSION:
        manifest["document"] = {**document, "reference_migration_version": REFERENCE_MIGRATION_VERSION}
        report["manifest"] += 1
    return report


def assign_package_document(package: Path, manifest: dict, *, title: str = "", author: str = "") -> Path:
    package = Path(package)
    document = manifest.get("document") if isinstance(manifest.get("document"), dict) else {}
    if document.get("naming_version") == DOCUMENT_NAMING_VERSION and _assigned_document_name(manifest):
        assigned = package_document_path(package, manifest)
        report = migrate_package_document_references(package, manifest)
        if report["manifest"] and (package / "manifest.json").exists():
            _atomic_json(package / "manifest.json", manifest)
        return assigned
    current = migration_source_document_path(package, manifest)
    preferred = system_document_name(manifest, title, author)
    destination = package / preferred
    counter = 2
    while destination.exists() and destination != current:
        destination = package / f"{Path(preferred).stem}-{counter}.md"
        counter += 1
    if current is not None and current.exists() and current != destination:
        os.replace(current, destination)
    manifest["document"] = {**document, "path": destination.name, "naming_version": DOCUMENT_NAMING_VERSION}
    migrate_package_document_references(package, manifest)
    manifest_path = package / "manifest.json"
    if manifest_path.exists():
        _atomic_json(manifest_path, manifest)
    return destination

#!/usr/bin/env python3
"""Plan evidence-backed transcript issue windows; never sweep a whole recording."""

from __future__ import annotations

import argparse
import json
import os
import tempfile
from pathlib import Path

from manifest_changes import refresh_artifact_hashes


def issue_ranges(qa: dict, review: dict, duration: float, extra_intervals: list[tuple[float, float, str]] | None = None, *, padding: float = 8.0, max_span: float = 120.0) -> list[dict]:
    intervals: list[tuple[float, float, str]] = []
    excluded_segments = {
        int(item["segment"])
        for item in qa.get("reader_exclusions") or []
        if isinstance(item, dict) and item.get("segment") is not None
    }
    for item in qa.get("low_confidence_ranges") or []:
        if not isinstance(item, dict):
            continue
        if item.get("segment") is not None and int(item["segment"]) in excluded_segments:
            continue
        start, end = float(item.get("start") or 0), float(item.get("end") or item.get("start") or 0)
        reasons = {str(value) for value in item.get("reasons") or []}
        text = str(item.get("text") or "").strip()
        # Whisper occasionally assigns a perfectly ordinary short phrase to a
        # sub-second segment boundary. That is a timing-alignment warning, not
        # evidence that the wording is incomplete. Keep it out of lexical
        # review unless another QA reason or visibly corrupt text is present.
        timing_only = reasons == {"implausibly_dense_text"}
        short_clean_phrase = end - start <= 1.0 and 0 < len(text) <= 18 and not any(mark in text for mark in ("�", "＝", "□"))
        if timing_only and short_clean_phrase:
            continue
        intervals.append((max(0.0, start - padding), min(duration, end + padding), "automated_low_confidence"))
    for item in review.get("unresolved") or []:
        if not isinstance(item, dict) or item.get("start") is None:
            continue
        start, end = float(item["start"]), float(item.get("end") or item["start"])
        intervals.append((max(0.0, start - padding), min(duration, end + padding), "unresolved"))
    intervals.extend(extra_intervals or [])
    merged: list[dict] = []
    for start, end, reason in sorted(intervals):
        if end <= start:
            continue
        if merged and start - merged[-1]["end"] <= 20 and max(end, merged[-1]["end"]) - merged[-1]["start"] <= max_span:
            merged[-1]["end"] = max(merged[-1]["end"], end)
            merged[-1]["reasons"] = sorted(set([*merged[-1]["reasons"], reason]))
        else:
            while end - start > max_span:
                merged.append({"start": round(start, 3), "end": round(start + max_span, 3), "reasons": [reason]})
                start += max_span
            merged.append({"start": round(start, 3), "end": round(end, 3), "reasons": [reason]})
    for index, item in enumerate(merged):
        item.update({"index": index, "total": len(merged)})
    return merged


def quote_candidate_intervals(package: Path, duration: float) -> list[tuple[float, float, str]]:
    assets_path = package / "evidence/assets/assets.json"
    chapters_path = package / "evidence/transcript/逐字稿-章节版.json"
    if not assets_path.is_file() or not chapters_path.is_file():
        return []
    assets = json.loads(assets_path.read_text("utf-8"))
    chapters = json.loads(chapters_path.read_text("utf-8"))
    chapter_ranges = {
        str(item.get("id") or ""): (float(item.get("start") or 0), float(item.get("end") or 0))
        for item in chapters.get("chapters") or [] if isinstance(item, dict)
    }
    intervals: list[tuple[float, float, str]] = []
    for candidate in (assets.get("review") or {}).get("candidates") or []:
        if not isinstance(candidate, dict) or candidate.get("type") != "quote" or candidate.get("verdict") not in {"review", "partial"}:
            continue
        for locator in candidate.get("evidence") or []:
            if isinstance(locator, dict) and locator.get("kind") == "time_range":
                start, end = float(locator.get("start") or 0), float(locator.get("end") or 0)
                if end > start:
                    intervals.append((max(0.0, start), min(duration, end), "quote_candidate"))
            elif isinstance(locator, str) and "#chapter-" in locator:
                chapter_id = locator.rsplit("#", 1)[-1]
                start, end = chapter_ranges.get(chapter_id, (0.0, 0.0))
                if end > start:
                    intervals.append((max(0.0, start), min(duration, end), "quote_candidate_chapter"))
    return intervals


def atomic_json(path: Path, value: dict) -> None:
    fd, temporary = tempfile.mkstemp(prefix=f".{path.name}.", dir=path.parent)
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as handle:
            json.dump(value, handle, ensure_ascii=False, indent=2)
            handle.write("\n")
            handle.flush()
            os.fsync(handle.fileno())
        os.replace(temporary, path)
    finally:
        if os.path.exists(temporary):
            os.unlink(temporary)


def plan_package(package: Path) -> dict:
    package = package.expanduser().resolve()
    manifest_path = package / "manifest.json"
    qa_path = package / "evidence/transcript/transcript-qa.json"
    review_path = package / "evidence/transcript/transcript-review.json"
    manifest = json.loads(manifest_path.read_text("utf-8"))
    qa = json.loads(qa_path.read_text("utf-8"))
    review = json.loads(review_path.read_text("utf-8")) if review_path.is_file() else {}
    transcript = manifest.setdefault("transcript", {})
    duration = max(0.0, float(transcript.get("duration_seconds") or manifest.get("source", {}).get("duration_seconds") or 0))
    ranges = issue_ranges(qa, review, duration, quote_candidate_intervals(package, duration))
    migrating_legacy_plan = transcript.get("lexical_review_policy") != "targeted_issues_v1"
    if migrating_legacy_plan:
        legacy_manifest_receipts = transcript.get("lexical_range_reviews") or []
        legacy_review_receipts = review.get("range_reviews") or []
        if legacy_manifest_receipts:
            transcript["legacy_lexical_range_reviews"] = legacy_manifest_receipts
        if legacy_review_receipts:
            review["legacy_range_reviews"] = legacy_review_receipts
        # Legacy receipts describe the old 120-second full sweep and therefore
        # cannot satisfy a newly planned targeted range merely by sharing an
        # index.  Preserve them as evidence, but reset canonical coverage.
        transcript["lexical_range_reviews"] = []
        review["range_reviews"] = []
    transcript["lexical_review_policy"] = "targeted_issues_v1"
    transcript["lexical_issue_ranges"] = ranges
    if review_path.is_file() or review:
        atomic_json(review_path, review)
        refresh_artifact_hashes(package, manifest, {"evidence/transcript/transcript-review.json"})
    atomic_json(manifest_path, manifest)
    return {"status": "planned", "package": str(package), "issue_ranges": ranges, "count": len(ranges)}


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--package", required=True, type=Path)
    args = parser.parse_args()
    print(json.dumps(plan_package(args.package), ensure_ascii=False))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

#!/usr/bin/env python3
"""Persist bounded transcript review receipts and finalize only complete coverage."""

from __future__ import annotations

import argparse
import json
import os
import subprocess
import sys
import tempfile
from datetime import datetime
from pathlib import Path
from typing import Any

from manifest_changes import refresh_artifact_hashes


CHECKS = (
    "names_and_products",
    "numbers_and_units",
    "english_terms",
    "quote_candidates",
    "automated_low_confidence_ranges",
)
CATEGORY_ALIASES = {
    "name_or_title": "named_entity",
    "proper_noun": "named_entity",
    "title_or_work": "program_or_work",
    "product_or_brand": "brand_or_product",
    "clear_asr_error": "other_clear_asr_error",
}
ALLOWED_CATEGORIES = {
    "named_entity", "program_or_work", "brand_or_product", "english_term",
    "number_or_unit", "other_clear_asr_error",
}


def now_iso() -> str:
    return datetime.now().astimezone().isoformat(timespec="seconds")


def load_json(path: Path) -> dict[str, Any]:
    value = json.loads(path.read_text("utf-8"))
    if not isinstance(value, dict):
        raise ValueError(f"{path} must contain a JSON object")
    return value


def atomic_json(path: Path, value: dict[str, Any]) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    fd, temporary = tempfile.mkstemp(prefix=f".{path.name}.", dir=path.parent)
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as handle:
            json.dump(value, handle, ensure_ascii=False, indent=2)
            handle.write("\n")
            handle.flush()
            os.fsync(handle.fileno())
        os.replace(temporary, path)
    finally:
        if os.path.exists(temporary):
            os.unlink(temporary)


def validated_receipt(args: argparse.Namespace, value: dict[str, Any]) -> dict[str, Any]:
    if args.index < 0 or args.total <= 0 or args.index >= args.total:
        raise ValueError("range index must be within total")
    if args.end <= args.start or args.end - args.start > 180.001:
        raise ValueError("range must be positive and at most 180 seconds")
    # The CLI arguments come from the signed execution manifest and are the
    # authoritative schedule. Agents sometimes omit these duplicated fields
    # from an otherwise valid receipt. Fill omissions from the schedule, but
    # continue to reject any explicit conflict so a receipt cannot escape its
    # assigned range.
    for key, expected in (("index", args.index), ("total", args.total)):
        if key in value and int(value[key]) != expected:
            raise ValueError(f"receipt {key} does not match the scheduled range")
    for key, expected in (("start", args.start), ("end", args.end)):
        if key in value and abs(float(value[key]) - expected) > 0.05:
            raise ValueError(f"receipt {key} does not match the scheduled range")
    raw_status = str(value.get("status") or "")
    status = "limited" if raw_status == "blocked" else raw_status
    if status not in {"passed", "limited"}:
        raise ValueError("range receipt status must be passed or limited")
    if raw_status == "blocked" and not value.get("evidence_method"):
        value["evidence_method"] = "bounded_audio_retranscription_inconclusive"
    if not str(value.get("reviewer") or "").strip() or not str(value.get("evidence_method") or "").strip():
        raise ValueError("range receipt needs reviewer and evidence_method")
    checks = value.get("checks") if isinstance(value.get("checks"), dict) else {}
    if status == "passed":
        incomplete = [key for key in CHECKS if checks.get(key) != "passed"]
        if incomplete:
            raise ValueError(f"range receipt has incomplete checks: {', '.join(incomplete)}")
    unresolved = value.get("unresolved") or value.get("uncertain_spans") or []
    if status == "limited" and not unresolved:
        raise ValueError("limited range receipt must declare unresolved spans")
    corrections = []
    for correction in value.get("corrections") or []:
        if not isinstance(correction, dict):
            raise ValueError("every range correction must be an object")
        category = CATEGORY_ALIASES.get(str(correction.get("category") or ""), str(correction.get("category") or ""))
        if category not in ALLOWED_CATEGORIES:
            raise ValueError(f"unsupported range correction category: {category or '(empty)'}")
        corrections.append({**correction, "category": category})
    return {
        **value,
        "status": status,
        "unresolved": unresolved,
        "corrections": corrections,
        "index": args.index,
        "total": args.total,
        "start": round(args.start, 3),
        "end": round(args.end, 3),
        "reviewed_at": str(value.get("reviewed_at") or now_iso()),
    }


def apply_receipt(args: argparse.Namespace) -> dict[str, Any]:
    package = args.package.expanduser().resolve()
    review_path = package / "evidence/transcript/transcript-review.json"
    manifest_path = package / "manifest.json"
    receipt = validated_receipt(args, load_json(args.input.expanduser().resolve()))
    corrections = [item for item in receipt.get("corrections") or [] if isinstance(item, dict)]
    correction_script = Path(__file__).resolve().parents[2] / "save-to-favorites/scripts/save_link.py"
    for correction in corrections:
        command = [
            sys.executable, str(correction_script), "correct-transcript",
            "--package", str(package),
            "--from-text", str(correction.get("from") or ""),
            "--to-text", str(correction.get("to") or ""),
            "--category", str(correction.get("category") or "other_clear_asr_error"),
            "--reviewer", str(receipt.get("reviewer") or "bounded-range-review"),
            "--evidence-method", str(correction.get("evidence_method") or receipt.get("evidence_method") or "bounded_range_review"),
            "--range-start", str(args.start),
            "--range-end", str(args.end),
        ]
        completed = subprocess.run(command, text=True, capture_output=True, check=False)
        if completed.returncode != 0:
            raise RuntimeError((completed.stderr or completed.stdout).strip()[-1000:])
    review = load_json(review_path)
    manifest = load_json(manifest_path)
    current = [item for item in review.get("range_reviews") or [] if int(item.get("index", -1)) != args.index]
    review["range_reviews"] = sorted([*current, receipt], key=lambda item: int(item["index"]))
    if review.get("status") != "passed":
        review["status"] = "in_progress"
        review["coverage"] = "bounded_ranges_in_progress"
    transcript = manifest.setdefault("transcript", {})
    manifest_receipts = [item for item in transcript.get("lexical_range_reviews") or [] if int(item.get("index", -1)) != args.index]
    transcript["lexical_range_reviews"] = sorted([*manifest_receipts, receipt], key=lambda item: int(item["index"]))
    transcript["lexical_review_status"] = "in_progress"
    manifest["updated_at"] = now_iso()
    receipt_path = package / "evidence/transcript/range-reviews" / f"span-{args.index + 1:03d}-of-{args.total:03d}.json"
    atomic_json(receipt_path, receipt)
    atomic_json(review_path, review)
    refresh_artifact_hashes(package, manifest, {"evidence/transcript/transcript-review.json"})
    atomic_json(manifest_path, manifest)
    return {"status": "recorded", "range": receipt, "receipt_path": str(receipt_path)}


def finalize_coverage(args: argparse.Namespace) -> dict[str, Any]:
    package = args.package.expanduser().resolve()
    review_path = package / "evidence/transcript/transcript-review.json"
    manifest_path = package / "manifest.json"
    review = load_json(review_path)
    manifest = load_json(manifest_path)
    receipts = sorted(review.get("range_reviews") or [], key=lambda item: int(item.get("index", -1)))
    transcript = manifest.setdefault("transcript", {})
    if transcript.get("lexical_review_policy") == "targeted_issues_v1":
        targets = sorted(transcript.get("lexical_issue_ranges") or [], key=lambda item: int(item.get("index", -1)))
        if len(receipts) != len(targets):
            raise ValueError("targeted review receipts do not cover every declared issue range")
        for receipt, target in zip(receipts, targets):
            if int(receipt.get("index", -1)) != int(target.get("index", -2)):
                raise ValueError("targeted review receipt index mismatch")
            if abs(float(receipt.get("start", -1)) - float(target.get("start", -2))) > 0.05 or abs(float(receipt.get("end", -1)) - float(target.get("end", -2))) > 0.05:
                raise ValueError("targeted review receipt range mismatch")
            if receipt.get("status") not in {"passed", "limited"}:
                raise ValueError("every targeted issue range must be passed or explicitly limited")
        limited = [item for item in receipts if item.get("status") == "limited"]
        unresolved = [
            {**item, "range_index": receipt.get("index"), "range_start": receipt.get("start"), "range_end": receipt.get("end")}
            for receipt in limited for item in (receipt.get("unresolved") or []) if isinstance(item, dict)
        ]
        settled_status = "targeted_limited" if limited else "targeted_passed"
        review.update({
            "status": settled_status,
            "reviewer": "bounded-targeted-review",
            "reviewed_at": now_iso(),
            "coverage": "automated_qa_plus_targeted_issues",
            "evidence_method": ["automated_asr_qa", "bounded_targeted_issue_review"],
            "checks": {
                "names_and_products": "candidate_local_only",
                "numbers_and_units": settled_status,
                "english_terms": settled_status,
                "quote_candidates": "candidate_local_only",
                "automated_low_confidence_ranges": settled_status,
            },
            "unresolved": unresolved,
        })
        transcript["lexical_review_status"] = settled_status
        transcript["quotation_ready"] = False
        transcript["lexical_review_scope"] = "automated_qa_plus_targeted_issues"
        transcript["lexical_limited_ranges"] = [
            {key: item.get(key) for key in ("index", "start", "end", "unresolved")}
            for item in limited
        ]
        manifest.setdefault("gates", {})["extraction"] = "passed"
        manifest["gates"]["fidelity"] = "passed"
        manifest["updated_at"] = now_iso()
        atomic_json(review_path, review)
        refresh_artifact_hashes(package, manifest, {"evidence/transcript/transcript-review.json"})
        atomic_json(manifest_path, manifest)
        sys.path.insert(0, str(Path(__file__).resolve().parent))
        from gate_ledger import sync_gate_ledger
        sync_gate_ledger(package, manifest, write=True, completed_gates={"extraction", "fidelity"})
        return {"status": "targeted_finalized", "review_status": settled_status, "ranges": len(targets), "limited_ranges": len(limited), "quotation_ready": False}
    totals = {int(item.get("total", 0)) for item in receipts}
    if len(totals) != 1 or not totals:
        raise ValueError("range receipts do not declare one consistent total")
    total = totals.pop()
    if [int(item.get("index", -1)) for item in receipts] != list(range(total)):
        raise ValueError("range receipts do not cover every scheduled index")
    if any(item.get("status") != "passed" for item in receipts):
        raise ValueError("all range receipts must be passed")
    review.update({
        "status": "passed",
        "reviewer": "bounded-range-review",
        "reviewed_at": now_iso(),
        "coverage": "all_speech_and_quote_candidates",
        "checks": {key: "passed" for key in CHECKS},
    })
    manifest.setdefault("transcript", {})["lexical_review_status"] = "in_progress"
    manifest["updated_at"] = now_iso()
    atomic_json(review_path, review)
    refresh_artifact_hashes(package, manifest, {"evidence/transcript/transcript-review.json"})
    atomic_json(manifest_path, manifest)
    finalizer = Path(__file__).with_name("finalize_transcript.py")
    completed = subprocess.run(
        [sys.executable, str(finalizer), "--package", str(package)],
        text=True,
        capture_output=True,
        check=False,
    )
    if completed.returncode != 0:
        raise RuntimeError((completed.stderr or completed.stdout).strip()[-1000:])
    return {"status": "finalized", "ranges": total, "finalizer": completed.stdout.strip()}


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    subparsers = parser.add_subparsers(dest="command", required=True)
    apply_parser = subparsers.add_parser("apply")
    apply_parser.add_argument("--package", required=True, type=Path)
    apply_parser.add_argument("--input", required=True, type=Path)
    apply_parser.add_argument("--index", required=True, type=int)
    apply_parser.add_argument("--total", required=True, type=int)
    apply_parser.add_argument("--start", required=True, type=float)
    apply_parser.add_argument("--end", required=True, type=float)
    finalize_parser = subparsers.add_parser("finalize-coverage")
    finalize_parser.add_argument("--package", required=True, type=Path)
    args = parser.parse_args()
    try:
        result = apply_receipt(args) if args.command == "apply" else finalize_coverage(args)
    except (OSError, ValueError, RuntimeError, json.JSONDecodeError) as error:
        print(str(error), file=sys.stderr)
        return 2
    print(json.dumps(result, ensure_ascii=False))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

#!/usr/bin/env python3
"""Refresh only unsettled comment evidence for Douyin, Bilibili, and Xiaoyuzhou packages."""

from __future__ import annotations

import argparse
import json
from pathlib import Path
from typing import Any

import requests

from capture_source import (
    atomic_json,
    bilibili_content_key,
    bilibili_public_comments,
    bilibili_public_video_metadata,
    normalize_xiaoyuzhou_comments,
    now_iso,
    request,
    xiaoyuzhou_next_data,
)
from douyin_web_capture import _snapshot as douyin_snapshot, douyin_public_comments
from comment_selection import COMMENT_ENRICHMENT_MODES, comment_enrichment_decision, deferred_comment_result
from gate_ledger import sync_gate_ledger
from manifest_changes import mark_manifest_change


PLATFORMS = {"douyin", "bilibili", "xiaoyuzhou"}


def refresh_payload(manifest: dict[str, Any], *, force: bool = False) -> dict[str, Any]:
    source = manifest.get("source") or {}
    platform = str(source.get("platform") or "")
    canonical = str(source.get("canonical_url") or source.get("resolved_url_redacted") or "")
    stable_id = str(source.get("stable_id") or "")
    mode = str(manifest.get("mode") or "full")
    metrics = source.get("metrics") or {}
    if platform == "douyin":
        if not stable_id:
            raise ValueError("Douyin package has no stable ID")
        preliminary = comment_enrichment_decision(platform, mode, metrics)
        if not force and not preliminary["eligible"] and (
            not preliminary["enabled"]
            or mode not in COMMENT_ENRICHMENT_MODES
            or preliminary["source_comment_count"] is not None
        ):
            return deferred_comment_result(preliminary)
        route = "note" if source.get("content_form") == "image_post" else "video"
        snapshot = douyin_snapshot(f"https://www.douyin.com/{route}/{stable_id}")
        page_html = str(snapshot.get("html") or "")
        if not force:
            decision = comment_enrichment_decision(
                platform, mode, metrics,
                visible_sample_available="comment-item-stats-container" in page_html,
            )
            if not decision["eligible"]:
                return deferred_comment_result(decision)
        return douyin_public_comments(page_html, "full" if force else mode, {} if force else metrics)
    session = requests.Session()
    if platform == "bilibili":
        decision = comment_enrichment_decision(platform, mode, metrics)
        if not force and not decision["eligible"]:
            return deferred_comment_result(decision)
        request(session, canonical)
        key, value = bilibili_content_key(canonical)
        public = bilibili_public_video_metadata(session, key, value, canonical)
        result = bilibili_public_comments(session, public.get("aid"), canonical)
        result["enrichment_decision"] = {**decision, "forced": force}
        return result
    if platform == "xiaoyuzhou":
        page = request(session, canonical)
        props = xiaoyuzhou_next_data(page.text)
        return normalize_xiaoyuzhou_comments(props.get("comments"))
    raise ValueError(f"unsupported comment platform: {platform}")


def package_title(manifest: dict[str, Any], package: Path) -> str:
    source = manifest.get("source") or {}
    return str(source.get("title") or manifest.get("title") or package.name)


def commit(package: Path, manifest: dict[str, Any], payload: dict[str, Any]) -> None:
    target = package / "evidence/comments/top-comments.json"
    atomic_json(target, payload)
    payload_status = payload.get("status")
    status = payload_status if payload_status in {"passed", "not_applicable"} else "review"
    manifest.setdefault("gates", {})["comments"] = status
    manifest["comment_capture_status"] = payload.get("comment_capture_status") or status
    manifest["errors"] = [
        item for item in manifest.get("errors") or []
        if not (isinstance(item, dict) and item.get("stage") == "comments")
    ]
    if status == "review":
        manifest["errors"].append({
            "at": now_iso(),
            "stage": "comments",
            "code": "COMMENTS_REFRESH_INCOMPLETE",
            "message": payload.get("reason") or "Comment refresh remains incomplete",
            "recoverable": True,
            "artifact": "evidence/comments/top-comments.json",
        })
    mark_manifest_change(manifest, "system_maintenance", now_iso())
    atomic_json(package / "manifest.json", manifest)
    sync_gate_ledger(package, manifest, write=True, completed_gates={"comments"} if status in {"passed", "not_applicable"} else set())


def candidates(root: Path, explicit: list[Path], refresh_all: bool, platforms: set[str]) -> list[Path]:
    if explicit:
        return [item.expanduser().resolve() for item in explicit]
    result = []
    for manifest_path in root.rglob("manifest.json"):
        try:
            manifest = json.loads(manifest_path.read_text("utf-8"))
        except (OSError, json.JSONDecodeError):
            continue
        platform = str((manifest.get("source") or {}).get("platform") or "")
        current = (manifest.get("gates") or {}).get("comments")
        if platform in platforms and (refresh_all or current not in {"passed", "not_applicable"}):
            result.append(manifest_path.parent)
    return sorted(result, key=lambda item: str(item))


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--root", type=Path)
    parser.add_argument("--package", action="append", type=Path, default=[])
    parser.add_argument("--all", action="store_true", help="Refresh passed packages too")
    parser.add_argument("--platform", action="append", choices=sorted(PLATFORMS), default=[])
    parser.add_argument("--dry-run", action="store_true")
    parser.add_argument("--force", action="store_true", help="Bypass mode/count eligibility for explicitly requested recovery")
    args = parser.parse_args()
    if not args.root and not args.package:
        parser.error("provide --root or at least one --package")
    selected_platforms = set(args.platform) or PLATFORMS
    packages = candidates(args.root.expanduser().resolve() if args.root else Path.cwd(), args.package, args.all, selected_platforms)
    results = []
    for package in packages:
        manifest_path = package / "manifest.json"
        manifest = json.loads(manifest_path.read_text("utf-8"))
        source = manifest.get("source") or {}
        platform = str(source.get("platform") or "")
        title = package_title(manifest, package)
        try:
            payload = refresh_payload(manifest, force=args.force)
        except Exception as exc:
            payload = {
                "status": "review",
                "ranking_scope": "refresh_unavailable",
                "sample_size": 0,
                "has_more": None,
                "acquired_at": now_iso(),
                "method": f"{platform}_comment_refresh",
                "comments": [],
                "topic_signal_status": "pending_analysis",
                "topic_signals": [],
                "reason": f"{type(exc).__name__}: {exc}",
            }
        if not args.dry_run:
            commit(package, manifest, payload)
        results.append({
            "package": package.name,
            "platform": platform,
            "title": title,
            "status": payload.get("status"),
            "sample_size": payload.get("sample_size", 0),
            "retained": len(payload.get("comments") or []),
            "likes": [item.get("like_count") for item in payload.get("comments") or []],
            "reason": payload.get("reason"),
        })
    print(json.dumps({"dry_run": args.dry_run, "count": len(results), "results": results}, ensure_ascii=False, indent=2))
    return 0 if all(item["status"] in {"passed", "not_applicable"} for item in results) else 2


if __name__ == "__main__":
    raise SystemExit(main())

#!/usr/bin/env python3
"""Backfill Xiaoyuzhou rich shownotes without re-downloading episode audio."""

from __future__ import annotations

import argparse
import json
import re
from pathlib import Path

import requests

from capture_source import (
    HEADERS,
    article_image_markdown,
    atomic_json,
    atomic_text,
    download,
    extract_xiaoyuzhou,
    image_record,
    normalize_xiaoyuzhou_shownote_layout,
    now_iso,
    xiaoyuzhou_shownote_times,
)
from gate_ledger import sync_gate_ledger
from package_document import package_document_path


def replace_section(markdown: str, heading: str, body: str) -> str:
    prefix, title = heading.split(" ", 1)
    pattern = re.compile(
        rf"(^{re.escape(prefix)}\s+{re.escape(title)}\s*$\n)[\s\S]*?(?=^#{{1,{len(prefix)}}}\s+|\Z)",
        re.MULTILINE,
    )
    return pattern.sub(lambda match: f"{match.group(1)}\n{body.strip()}\n\n", markdown)


def repair_package_layout(package: Path) -> dict:
    """Repair shownote line structure and timeline without reacquiring evidence."""
    manifest_path = package / "manifest.json"
    manifest = json.loads(manifest_path.read_text("utf-8"))
    source = manifest.get("source") or {}
    if source.get("platform") != "xiaoyuzhou":
        return {"package": str(package), "status": "skipped"}
    original_body = str(source.get("body") or source.get("description") or "")
    body = normalize_xiaoyuzhou_shownote_layout(original_body)
    parsed = xiaoyuzhou_shownote_times(body)
    source.update({
        "body": body,
        "timeline": parsed["timeline"],
        "timestamp_notes": parsed["timestamp_notes"],
    })
    manifest["source"] = source
    atomic_json(manifest_path, manifest)

    metadata_path = package / "evidence/source/metadata.json"
    if metadata_path.is_file():
        metadata = json.loads(metadata_path.read_text("utf-8"))
        metadata.update({
            "body": body,
            "timeline": source["timeline"],
            "timestamp_notes": source["timestamp_notes"],
        })
        atomic_json(metadata_path, metadata)

    for name, heading in (("原始内容.md", "## 正文/描述"), ("标准内容.md", "## 正文/描述")):
        path = package / name
        if path.is_file():
            atomic_text(path, replace_section(path.read_text("utf-8"), heading, body))
    favorite = package_document_path(package, manifest)
    if favorite.is_file():
        atomic_text(favorite, replace_section(favorite.read_text("utf-8"), "### 正文/描述", body))

    manifest = json.loads(manifest_path.read_text("utf-8"))
    completed = {gate for gate, status in (manifest.get("gates") or {}).items() if status in {"passed", "not_applicable"}}
    sync_gate_ledger(package, manifest, write=True, completed_gates=completed)
    return {
        "package": str(package),
        "status": "repaired" if body != original_body else "refreshed",
        "timeline_entries": len(source["timeline"]),
        "groups": len({str(item.get("group_title")) for item in source["timeline"] if item.get("group_title")}),
        "network_used": False,
    }


def refresh_package(package: Path, session: requests.Session) -> dict:
    manifest_path = package / "manifest.json"
    manifest = json.loads(manifest_path.read_text("utf-8"))
    source = manifest.get("source") or {}
    if source.get("platform") != "xiaoyuzhou":
        return {"package": str(package), "status": "skipped"}
    fresh = extract_xiaoyuzhou(session, str(source.get("canonical_url") or ""))
    candidates = [item for item in fresh.get("images") or [] if item.get("role") == "shownote"]
    records = []
    image_errors = []
    body = str(fresh.get("body") or fresh.get("description") or "")
    for ordinal, candidate in enumerate(candidates, 1):
        marker = f"[[SOURCE_IMAGE_{int(candidate['index']):03d}]]"
        try:
            path, _ = download(
                session,
                str(candidate["ephemeral_url"]),
                package / "evidence/images" / f"shownote-{ordinal:03d}",
                referer=str(source.get("canonical_url") or ""),
                image=True,
            )
            record = image_record(path, int(candidate["index"]), candidate)
        except Exception as exc:
            image_errors.append({
                "at": now_iso(),
                "index": int(candidate["index"]),
                "stage": "acquire",
                "code": "XIAOYUZHOU_SHOWNOTE_IMAGE_DOWNLOAD_FAILED",
                "message": str(exc),
                "recoverable": True,
                "artifact": f"evidence/images/shownote-{ordinal:03d}",
            })
            body = body.replace(marker, "_来源图片暂未取得。_")
            continue
        record["role"] = "shownote"
        record["relative_path"] = str(path.relative_to(package))
        record.pop("path", None)
        records.append(record)
        body = body.replace(marker, article_image_markdown(record))

    source.update({
        "description": fresh.get("description") or source.get("description") or "",
        "body": body,
        "shownote_links": fresh.get("shownote_links") or [],
        "shownote_image_errors": image_errors,
        "timeline": fresh.get("timeline") or [],
        "timestamp_notes": fresh.get("timestamp_notes") or [],
    })
    manifest["source"] = source
    retained_images = [item for item in manifest.get("image_inventory") or [] if item.get("role") != "shownote"]
    manifest["image_inventory"] = sorted([*retained_images, *records], key=lambda item: int(item.get("index") or 0))
    retained_artifacts = [
        item for item in manifest.get("artifacts") or []
        if not (item.get("kind") == "image" and item.get("role") == "shownote")
    ]
    manifest["artifacts"] = [*retained_artifacts, *[
        {"path": item["relative_path"], "kind": "image", "role": "shownote", "sha256": item["sha256"]}
        for item in records
    ]]
    if image_errors:
        manifest.setdefault("gates", {})["acquisition"] = "review"
        existing_errors = [item for item in manifest.get("errors") or [] if item.get("code") != "XIAOYUZHOU_SHOWNOTE_IMAGE_DOWNLOAD_FAILED"]
        manifest["errors"] = [*existing_errors, *image_errors]
    atomic_json(manifest_path, manifest)

    metadata_path = package / "evidence/source/metadata.json"
    if metadata_path.is_file():
        metadata = json.loads(metadata_path.read_text("utf-8"))
        metadata.update({
            "description": source["description"],
            "body": body,
            "shownote_links": source["shownote_links"],
            "shownote_image_errors": image_errors,
            "timeline": source["timeline"],
            "timestamp_notes": source["timestamp_notes"],
            "image_count_expected": len(retained_images) + len(candidates),
            "image_count_acquired": len(manifest["image_inventory"]),
        })
        atomic_json(metadata_path, metadata)

    for name, heading in (("原始内容.md", "## 正文/描述"), ("标准内容.md", "## 正文/描述")):
        path = package / name
        if path.is_file():
            atomic_text(path, replace_section(path.read_text("utf-8"), heading, body))
    favorite = package_document_path(package, manifest)
    if favorite.is_file():
        atomic_text(favorite, replace_section(favorite.read_text("utf-8"), "### 正文/描述", body))

    manifest = json.loads(manifest_path.read_text("utf-8"))
    completed = {gate for gate, status in (manifest.get("gates") or {}).items() if status in {"passed", "not_applicable"}}
    sync_gate_ledger(package, manifest, write=True, completed_gates=completed)
    return {
        "package": str(package),
        "status": "partial" if image_errors else "refreshed",
        "images": len(records),
        "failed_images": len(image_errors),
        "links": len(source["shownote_links"]),
    }


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--package", type=Path)
    parser.add_argument("--library-root", type=Path)
    parser.add_argument("--local-only", action="store_true", help="Repair stored shownote layout without network or image downloads.")
    args = parser.parse_args()
    packages = [args.package.expanduser().resolve()] if args.package else [
        path.parent for path in args.library_root.expanduser().resolve().glob("S-xiaoyuzhou-*/manifest.json")
    ]
    session = requests.Session()
    session.headers.update(HEADERS)
    results = []
    for package in packages:
        try:
            results.append(repair_package_layout(package) if args.local_only else refresh_package(package, session))
        except Exception as exc:
            results.append({"package": str(package), "status": "failed", "message": str(exc)})
    print(json.dumps({"status": "completed", "results": results}, ensure_ascii=False))
    return 2 if any(item["status"] == "failed" for item in results) else 0


if __name__ == "__main__":
    raise SystemExit(main())

#!/usr/bin/env python3
"""Bound local inference concurrency without keeping model processes alive."""

from __future__ import annotations

import json
import os
import re
import shutil
import subprocess
import sys
import time
import uuid
from contextlib import contextmanager
from pathlib import Path
from typing import Iterator


def _float_env(name: str, default: float, minimum: float = 0.0) -> float:
    try:
        return max(minimum, float(os.getenv(name, str(default))))
    except ValueError:
        return default


def _system_memory() -> tuple[int | None, int | None]:
    """Return available and total bytes without requiring psutil."""
    try:
        import psutil  # type: ignore
        memory = psutil.virtual_memory()
        return int(memory.available), int(memory.total)
    except (ImportError, AttributeError, OSError):
        pass
    if sys.platform.startswith("linux"):
        try:
            values = {}
            for line in Path("/proc/meminfo").read_text("utf-8").splitlines():
                key, _, raw = line.partition(":")
                match = re.search(r"\d+", raw)
                if match:
                    values[key] = int(match.group()) * 1024
            return values.get("MemAvailable"), values.get("MemTotal")
        except OSError:
            return None, None
    if sys.platform == "darwin":
        try:
            completed = subprocess.run(
                ["/usr/bin/vm_stat"], text=True, capture_output=True, timeout=2, check=True,
            )
            page_match = re.search(r"page size of (\d+) bytes", completed.stdout)
            page_size = int(page_match.group(1)) if page_match else 4096
            reclaimable = 0
            for label in ("Pages free", "Pages inactive", "Pages speculative", "Pages purgeable"):
                match = re.search(rf"^{re.escape(label)}:\s+(\d+)\.", completed.stdout, re.MULTILINE)
                if match:
                    reclaimable += int(match.group(1))
            total = int(os.sysconf("SC_PAGE_SIZE")) * int(os.sysconf("SC_PHYS_PAGES"))
            return reclaimable * page_size, total
        except (OSError, ValueError, subprocess.SubprocessError):
            return None, None
    if sys.platform == "win32":
        try:
            import ctypes

            class MemoryStatus(ctypes.Structure):
                _fields_ = [
                    ("length", ctypes.c_ulong),
                    ("memory_load", ctypes.c_ulong),
                    ("total_physical", ctypes.c_ulonglong),
                    ("available_physical", ctypes.c_ulonglong),
                    ("total_page_file", ctypes.c_ulonglong),
                    ("available_page_file", ctypes.c_ulonglong),
                    ("total_virtual", ctypes.c_ulonglong),
                    ("available_virtual", ctypes.c_ulonglong),
                    ("available_extended_virtual", ctypes.c_ulonglong),
                ]

            status = MemoryStatus()
            status.length = ctypes.sizeof(MemoryStatus)
            if ctypes.windll.kernel32.GlobalMemoryStatusEx(ctypes.byref(status)):
                return int(status.available_physical), int(status.total_physical)
        except (AttributeError, OSError, ValueError):
            pass
    return None, None


def _capacity_snapshot() -> dict:
    available, total = _system_memory()
    cores = max(1, os.cpu_count() or 1)
    try:
        load_per_core = float(os.getloadavg()[0]) / cores
    except (AttributeError, OSError):
        load_per_core = None
    return {
        "available_memory_mb": round(available / (1024 ** 2), 1) if available is not None else None,
        "available_memory_ratio": round(available / total, 4) if available is not None and total else None,
        "load_per_core": round(load_per_core, 3) if load_per_core is not None else None,
    }


def _capacity_blockers(snapshot: dict) -> list[str]:
    if os.getenv("CREATOR_INGEST_DISABLE_CAPACITY_GUARD", "").lower() in {"1", "true", "yes"}:
        return []
    blockers = []
    available_mb = snapshot.get("available_memory_mb")
    available_ratio = snapshot.get("available_memory_ratio")
    load_per_core = snapshot.get("load_per_core")
    if available_mb is not None and available_ratio is not None:
        if available_mb < _float_env("CREATOR_INGEST_MIN_AVAILABLE_MEMORY_MB", 512.0, 128.0) \
                and available_ratio < _float_env("CREATOR_INGEST_MIN_AVAILABLE_MEMORY_RATIO", 0.05, 0.01):
            blockers.append("memory_pressure")
    if load_per_core is not None and load_per_core > _float_env("CREATOR_INGEST_MAX_LOAD_PER_CORE", 2.0, 1.0):
        blockers.append("cpu_pressure")
    return blockers


def _trace(kind: str, event: str, **details) -> None:
    configured = os.getenv("CREATOR_INGEST_RESOURCE_TRACE", "").strip()
    if not configured:
        return
    path = Path(configured).expanduser()
    record = {
        "at": time.time(),
        "pid": os.getpid(),
        "kind": kind,
        "event": event,
        **details,
    }
    try:
        path.parent.mkdir(parents=True, exist_ok=True)
        descriptor = os.open(path, os.O_APPEND | os.O_CREAT | os.O_WRONLY, 0o600)
        try:
            os.write(descriptor, (json.dumps(record, ensure_ascii=False, separators=(",", ":")) + "\n").encode("utf-8"))
        finally:
            os.close(descriptor)
    except OSError:
        pass


def _wait_for_capacity(kind: str, deadline: float) -> tuple[float, dict]:
    started = time.monotonic()
    waiting = False
    while True:
        snapshot = _capacity_snapshot()
        blockers = _capacity_blockers(snapshot)
        if not blockers:
            return (time.monotonic() - started) * 1000, snapshot
        if not waiting:
            _trace(kind, "capacity_wait_started", blockers=blockers, snapshot=snapshot)
            waiting = True
        if time.monotonic() >= deadline:
            _trace(kind, "capacity_wait_timed_out", blockers=blockers, snapshot=snapshot)
            raise TimeoutError(f"timed out waiting for local {kind} capacity: {','.join(blockers)}")
        time.sleep(_float_env("CREATOR_INGEST_CAPACITY_POLL_SECONDS", 1.0, 0.1))


def _process_alive(pid: int) -> bool:
    if pid <= 0:
        return False
    try:
        os.kill(pid, 0)
        return True
    except PermissionError:
        return True
    except OSError:
        return False


def _lock_root() -> Path:
    configured = os.getenv("CREATOR_INGEST_RUNTIME_LOCK_DIR", "").strip()
    return Path(configured).expanduser() if configured else Path.home() / ".cache/creator-content-ingest/runtime-locks"


def _stale_lock(path: Path) -> bool:
    try:
        owner = json.loads((path / "owner.json").read_text("utf-8"))
        pid = int(owner.get("pid") or 0)
        return not _process_alive(pid)
    except (FileNotFoundError, ValueError, json.JSONDecodeError, OSError):
        try:
            return time.time() - path.stat().st_mtime > 60
        except OSError:
            return False


def _release_owned_lock(path: Path, token: str) -> None:
    try:
        owner = json.loads((path / "owner.json").read_text("utf-8"))
        if owner.get("token") == token:
            shutil.rmtree(path, ignore_errors=True)
    except (FileNotFoundError, json.JSONDecodeError, OSError):
        pass


def _lock_timeout(value: float | None) -> float:
    if value is not None:
        return max(1.0, float(value))
    try:
        return max(1.0, float(os.getenv("CREATOR_INGEST_INFERENCE_LOCK_TIMEOUT_SECONDS", "28800")))
    except ValueError:
        return 28800.0


@contextmanager
def local_inference_slot(kind: str, *, timeout_seconds: float | None = None) -> Iterator[dict]:
    """Serialize one heavyweight backend class; release immediately at task exit."""
    if os.getenv("CREATOR_INGEST_DISABLE_RESOURCE_GUARD", "").lower() in {"1", "true", "yes"}:
        yield {"guarded": False}
        return
    safe_kind = "".join(character for character in str(kind) if character.isalnum() or character in {"-", "_"}) or "local"
    root = _lock_root()
    root.mkdir(parents=True, exist_ok=True)
    lock = root / f"{safe_kind}.lock"
    token = uuid.uuid4().hex
    wait_started = time.monotonic()
    deadline = time.monotonic() + _lock_timeout(timeout_seconds)
    capacity_wait_ms, snapshot = _wait_for_capacity(safe_kind, deadline)
    lock_wait_started = time.monotonic()
    while True:
        try:
            lock.mkdir()
            (lock / "owner.json").write_text(json.dumps({
                "pid": os.getpid(),
                "token": token,
                "created_at": time.time(),
            }), "utf-8")
            break
        except FileExistsError:
            if _stale_lock(lock):
                shutil.rmtree(lock, ignore_errors=True)
                continue
            if time.monotonic() >= deadline:
                _trace(safe_kind, "slot_wait_timed_out", snapshot=snapshot)
                raise TimeoutError(f"timed out waiting for local {safe_kind} inference slot")
            time.sleep(0.25)
    lock_wait_ms = (time.monotonic() - lock_wait_started) * 1000
    try:
        post_lock_wait_ms, snapshot = _wait_for_capacity(safe_kind, deadline)
    except BaseException:
        _release_owned_lock(lock, token)
        raise
    capacity_wait_ms += post_lock_wait_ms
    acquired_at = time.monotonic()
    metrics = {
        "guarded": True,
        "wait_ms": round((acquired_at - wait_started) * 1000),
        "capacity_wait_ms": round(capacity_wait_ms),
        "lock_wait_ms": round(lock_wait_ms),
        "snapshot": snapshot,
    }
    _trace(safe_kind, "acquired", **metrics)
    try:
        yield metrics
    finally:
        _release_owned_lock(lock, token)
        _trace(safe_kind, "released", held_ms=round((time.monotonic() - acquired_at) * 1000))

#!/usr/bin/env python3
"""Settle legacy Xiaohongshu packages whose public payload exposed no comments."""

from __future__ import annotations

import argparse
import json
import os
import tempfile
from datetime import datetime
from pathlib import Path

from manifest_changes import mark_manifest_change


TERMINAL_REASON = (
    "Public initial state did not include comments; the public comment scan is "
    "complete and no authenticated retry is required."
)


def now_iso() -> str:
    return datetime.now().astimezone().isoformat(timespec="seconds")


def atomic_json(path: Path, value: dict) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    fd, temporary = tempfile.mkstemp(prefix=f".{path.name}.", dir=path.parent)
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as handle:
            json.dump(value, handle, ensure_ascii=False, indent=2)
            handle.write("\n")
            handle.flush()
            os.fsync(handle.fileno())
        os.replace(temporary, path)
    finally:
        if os.path.exists(temporary):
            os.unlink(temporary)


def settle_package(package: Path, *, commit: bool) -> str:
    manifest_path = package / "manifest.json"
    if not manifest_path.is_file():
        return "missing_manifest"
    manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
    if manifest.get("source", {}).get("platform") != "xhs":
        return "not_xhs"
    comments_path = package / "evidence" / "comments" / "top-comments.json"
    comments = json.loads(comments_path.read_text(encoding="utf-8")) if comments_path.is_file() else {}
    if comments.get("comments"):
        return "has_comments"
    already_settled = (
        manifest.get("comment_capture_status") == "confirmed_no_public_source"
        and manifest.get("gates", {}).get("comments") == "passed"
        and comments.get("status") == "unavailable"
        and not any(error.get("stage") == "comments" for error in manifest.get("errors") or [])
    )
    if already_settled:
        return "unchanged"
    comments.update({
        "status": "unavailable",
        "ranking_scope": comments.get("ranking_scope") or "not_in_public_payload",
        "sample_size": 0,
        "has_more": comments.get("has_more"),
        "acquired_at": comments.get("acquired_at") or now_iso(),
        "method": comments.get("method") or "xhs_initial_state",
        "comments": [],
        "reason": TERMINAL_REASON,
        "topic_signal_status": "no_meaningful_signal",
        "topic_signals": [],
    })
    manifest.setdefault("gates", {})["comments"] = "passed"
    manifest["comment_capture_status"] = "confirmed_no_public_source"
    manifest["errors"] = [
        error for error in manifest.get("errors") or []
        if error.get("stage") != "comments"
    ]
    mark_manifest_change(manifest, "system_maintenance", now_iso())
    if commit:
        atomic_json(comments_path, comments)
        atomic_json(manifest_path, manifest)
        from gate_ledger import sync_gate_ledger
        sync_gate_ledger(package, manifest, write=True, completed_gates={"comments"})
    return "settled"


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    group = parser.add_mutually_exclusive_group(required=True)
    group.add_argument("--package", action="append", type=Path)
    group.add_argument("--root", type=Path)
    parser.add_argument("--commit", action="store_true")
    args = parser.parse_args()
    packages = [path.expanduser().resolve() for path in args.package or []]
    if args.root:
        root = args.root.expanduser().resolve()
        packages = sorted(path.parent for path in root.glob("S-xhs-*/manifest.json"))
    results: dict[str, int] = {}
    for package in packages:
        result = settle_package(package, commit=args.commit)
        results[result] = results.get(result, 0) + 1
    print(json.dumps({"commit": args.commit, "packages": len(packages), "results": results}, ensure_ascii=False))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

"""Normalize creator-authored social topic tags without damaging prose."""

from __future__ import annotations

import re
from typing import Any


XHS_TOPIC_RE = re.compile(r"#\s*([^#\r\n]{1,80}?)\s*\[话题\]\s*#", re.UNICODE)
HASHTAG_RE = re.compile(
    r"#([^\s#\r\n]{1,80}?)(?=\s|$|[，。！？、,.!?;；:：])",
    re.UNICODE,
)
XHS_TOPIC_TOKEN = r"#\s*[^#\r\n]{1,80}?\s*\[话题\]\s*#"
PLAIN_HASHTAG_TOKEN = r"#[^\s#\r\n]{1,80}?(?=\s|$|[，。！？、,.!?;；:：])"
TRAILING_TAG_BLOCK_RE = re.compile(
    rf"^(?P<prefix>.*?[。！？!?；;:：])(?P<tags>(?:{XHS_TOPIC_TOKEN}|{PLAIN_HASHTAG_TOKEN})"
    rf"(?:\s*(?:{XHS_TOPIC_TOKEN}|{PLAIN_HASHTAG_TOKEN}))*)\s*$",
    re.UNICODE,
)
TAG_LINE_SEPARATORS_RE = re.compile(r"[\s|｜、,，;；·/／]+", re.UNICODE)
XHS_DECORATION_TOKEN_RE = re.compile(
    r"\[(?:回招呼|打招呼|笑哭|偷笑|开心|哭惹|赞|红色心形|比心|抱拳|猫猫|狗头|[^\[\]\r\n]{1,20}R)\]",
    re.UNICODE,
)


def clean_tag(value: Any) -> str:
    """Return one usable tag label, without platform markup."""
    text = re.sub(r"\s+", " ", str(value or "")).strip()
    text = text.strip("#").strip()
    text = re.sub(r"\s*\[话题\]\s*$", "", text).strip()
    return text[:80]


def unique_tags(values: list[Any]) -> list[str]:
    seen: set[str] = set()
    tags: list[str] = []
    for value in values:
        tag = clean_tag(value)
        key = tag.casefold()
        if not tag or key in seen:
            continue
        seen.add(key)
        tags.append(tag)
    return tags


def tags_in_line(line: str) -> tuple[list[str], str]:
    """Extract XHS and regular hashtag forms and leave only non-tag content."""
    tags = [clean_tag(match.group(1)) for match in XHS_TOPIC_RE.finditer(line)]
    remainder = XHS_TOPIC_RE.sub("", line)
    tags.extend(clean_tag(match.group(1)) for match in HASHTAG_RE.finditer(remainder))
    remainder = HASHTAG_RE.sub("", remainder)
    remainder = re.sub(r"[ \t]{2,}", " ", remainder)
    return unique_tags(tags), remainder


def strip_trailing_tag_block(line: str) -> tuple[str, list[str]] | None:
    """Remove a contiguous hashtag block that follows completed prose."""
    match = TRAILING_TAG_BLOCK_RE.match(line.strip())
    if not match:
        return None
    tag_text = match.group("tags")
    tags, remainder = tags_in_line(tag_text)
    if not tags or TAG_LINE_SEPARATORS_RE.sub("", remainder):
        return None
    return match.group("prefix").rstrip(), tags


def split_social_author_tags(body: Any) -> tuple[str, list[str]]:
    """Remove only standalone tag lines; preserve hashtags embedded in prose."""
    raw = str(body or "").replace("\r\n", "\n").replace("\r", "\n")
    author_tags: list[str] = []
    readable_lines: list[str] = []
    for line in raw.split("\n"):
        line = XHS_DECORATION_TOKEN_RE.sub("", line)
        line_tags, remainder = tags_in_line(line)
        trailing = strip_trailing_tag_block(line)
        if trailing:
            readable_lines.append(tags_in_line(trailing[0])[1])
            author_tags.extend(line_tags)
            continue
        author_tags.extend(line_tags)
        if line_tags and not TAG_LINE_SEPARATORS_RE.sub("", remainder):
            continue
        readable_lines.append(remainder)
    readable = re.sub(r"\n{3,}", "\n\n", "\n".join(readable_lines)).strip()
    return readable, unique_tags(author_tags)


def normalize_social_author_tags(source: dict[str, Any]) -> dict[str, Any]:
    """Store creator tags and create a clean reading-body projection for XHS/Douyin."""
    if str(source.get("platform") or "").lower() not in {"xhs", "douyin"}:
        return source
    raw_body = str(source.get("body_raw") or source.get("body") or "")
    readable_body, extracted = split_social_author_tags(raw_body)
    prior_author_tags = source.get("author_tags") if isinstance(source.get("author_tags"), list) else []
    platform_tags = source.get("tags") if isinstance(source.get("tags"), list) else []
    source["body_raw"] = raw_body
    source["body"] = readable_body
    source["author_tags"] = unique_tags([*prior_author_tags, *extracted])
    source["tags"] = unique_tags([*platform_tags, *source["author_tags"]])
    return source

#!/usr/bin/env python3
"""Restore mandatory per-image OCR in an existing image-post package."""

from __future__ import annotations

import argparse
import json
import re
from pathlib import Path

from capture_source import append_image_ocr_reading, atomic_json, atomic_text, reader_ocr_text, run_ocr
from ocr_organization import organize_ocr_records
from package_document import package_document_path


def selected_paths(package: Path, pages: list[int] | None = None) -> tuple[list[Path], list[int]]:
    manifest = json.loads((package / "manifest.json").read_text("utf-8"))
    inventory = {
        int(item.get("index") or position): package / str(item.get("relative_path") or "")
        for position, item in enumerate(manifest.get("image_inventory") or [], 1)
        if isinstance(item, dict)
    }
    all_pages = sorted(inventory)
    missing = [page for page in all_pages if not inventory[page].is_file()]
    if missing:
        raise ValueError(f"source OCR pages are unavailable: {missing}")
    if not all_pages:
        raise ValueError("image post has no source image inventory")
    return [inventory[page] for page in all_pages], all_pages


def refresh_standard_content(package: Path, manifest: dict, records: list[dict]) -> None:
    standard_path = package / "标准内容.md"
    existing = standard_path.read_text("utf-8") if standard_path.is_file() else "# 标准内容\n"
    prefix = existing.split("## 逐图内容", 1)[0].rstrip()
    prefix = prefix.replace(
        "> 原配文与图片已按顺序整理；因原配文足以组织内容，未默认运行 OCR。",
        "> 每张原图均已逐张扫描并按原图顺序整理；疑字可直接回看对应原图。",
    ).replace(
        "> 图片与文字已按原图顺序整理；疑字可直接回看对应原图。",
        "> 每张原图均已逐张扫描并按原图顺序整理；疑字可直接回看对应原图。",
    )
    lines = [prefix, ""]
    append_image_ocr_reading(lines, records, manifest.get("image_inventory") or [])
    atomic_text(standard_path, "\n".join(lines).rstrip() + "\n")
    favorite_path = package_document_path(package, manifest)
    if favorite_path.is_file():
        favorite = favorite_path.read_text("utf-8").replace("\r\n", "\n")
        projection = standard_path.read_text("utf-8").split("\n", 1)[1].strip()
        projection = re.sub(r"(?m)^(#{2,})\s", lambda match: "#" + match.group(1) + " ", projection)
        replacement = f"## 标准内容\n\n{projection}\n\n"
        refreshed, count = re.subn(
            r"(?ms)^## 标准内容\s*\n.*?(?=^## \S|\Z)",
            replacement,
            favorite,
            count=1,
        )
        if count == 1:
            atomic_text(favorite_path, refreshed.rstrip() + "\n")


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--package", type=Path, required=True)
    parser.add_argument("--page", type=int, action="append", help="optional priority hint; all source images are still scanned")
    parser.add_argument("--reuse-existing", action="store_true", help="rebuild projections from existing raw OCR without rescanning images")
    args = parser.parse_args()
    package = args.package.expanduser().resolve()
    priority_pages = sorted({page for page in args.page or [] if page > 0})
    manifest = json.loads((package / "manifest.json").read_text("utf-8"))
    paths, pages = selected_paths(package, priority_pages)
    evidence = package / "evidence"
    if args.reuse_existing:
        raw = json.loads((evidence / "ocr/ocr-raw.json").read_text("utf-8"))
        if not isinstance(raw, list) or len(raw) != len(pages):
            raise ValueError("existing raw OCR does not cover every source image")
        records = organize_ocr_records(raw)
        atomic_json(evidence / "ocr/organized.json", records)
        for record, page in zip(records, pages):
            atomic_text(evidence / "ocr" / f"image-{page:03d}.corrected.md", reader_ocr_text(record) + "\n")
    else:
        records = run_ocr(paths, evidence, source_indices=pages)
    atomic_json(evidence / "ocr" / "decision.json", {
        "status": "all_pages_ocr",
        "policy": "mandatory_per_image_v1",
        "reason": "Every source image was scanned once for auditable image text evidence.",
        "priority_pages": priority_pages,
    })
    refresh_standard_content(package, manifest, records)
    print(json.dumps({"status": "all_pages_ocr", "package": str(package), "pages": pages, "records": len(records)}, ensure_ascii=False))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

#!/usr/bin/env python3
"""Shared display-title normalization for social-source adapters."""

from __future__ import annotations

import html
import re
from typing import Any


def douyin_display_title(value: Any, fallback: str) -> str:
    """Return a compact title while leaving the captured caption untouched."""
    text = html.unescape(str(value or ""))
    text = re.sub(r"\s+", " ", text).strip()
    text = re.sub(r"(?:……|\.\.\.)?版本过低，升级后可展示全部信息.*$", "", text).strip()
    without_tags = re.split(r"\s*#", text, maxsplit=1)[0].strip()
    if without_tags:
        text = without_tags

    # Chinese prose normally starts the next sentence immediately after `。`.
    # The punctuation itself is therefore the boundary; whitespace is not.
    sentence = re.match(r"^(.{4,48}?[。！？!?])", text)
    if sentence:
        return sentence.group(1).strip()
    if len(text) <= 48:
        return text or fallback
    clause = re.split(r"[，,；;：:]", text, maxsplit=1)[0].strip()
    if 4 <= len(clause) <= 48:
        return clause
    return text[:47].rstrip() + "…"

#!/usr/bin/env python3
"""Add traceable local Whisper evidence to a canonical video or audio package."""

from __future__ import annotations

import argparse
import difflib
import hashlib
import importlib.util
import json
import math
import os
import platform
import re
import shutil
import subprocess
import sys
import tempfile
import wave
from datetime import datetime
from pathlib import Path
from typing import Any

SCRIPT_DIR = Path(__file__).resolve().parent
if str(SCRIPT_DIR) not in sys.path:
    sys.path.insert(0, str(SCRIPT_DIR))

from manifest_changes import mark_manifest_change, refresh_artifact_hashes
from asr_lexicon import default_lexicon_path, load_asr_lexicon, prompt_terms
from plan_transcript_issues import issue_ranges
from resource_guard import local_inference_slot

DEFAULT_MODEL = os.getenv("CREATOR_INGEST_ASR_MODEL", "mlx-community/whisper-large-v3-turbo")
ASR_PYTHON_ENV = "CREATOR_INGEST_ASR_PYTHON"
ASR_BACKEND_ENV = "CREATOR_INGEST_ASR_BACKEND"

_FASTER_WHISPER_MODELS: dict[tuple[str, str, int], Any] = {}

DIALECT_PROFILES = {
    "cantonese": {
        "cues": ("粤语", "粵語", "广东话", "廣東話", "cantonese"),
        # Current Whisper performs better on Cantonese with the Chinese decoder
        # plus an original-wording prior than with a forced yue token.  This is
        # transcription, not translation into Mandarin.
        "decoder_language": "zh",
        "prompt": "粤语口语原文逐字稿，保留粤语词汇和句式。",
    },
}


def venv_python(root: Path) -> Path:
    return root / ("Scripts/python.exe" if os.name == "nt" else "bin/python")


def asr_python_candidates() -> list[Path]:
    """Return reusable local ASR runtimes without assuming which agent launched us."""
    home = Path.home()
    values = [
        os.getenv(ASR_PYTHON_ENV),
        sys.executable,
        shutil.which("python3"),
        shutil.which("python"),
        venv_python(home / ".cache/creator-content-ingest/asr-runtime/venv"),
        venv_python(Path(__file__).resolve().parents[1] / ".venv"),
        venv_python(home / ".codex/skills/creator-content-ingest/.venv"),
        venv_python(home / ".hermes/skills/social-media/creator-content-ingest/.venv"),
        venv_python(home / ".workbuddy/skills/creator-content-ingest/.venv"),
    ]
    candidates: list[Path] = []
    seen: set[str] = set()
    for value in values:
        if not value:
            continue
        # Preserve virtual-environment symlinks: resolving them selects the base
        # interpreter and silently drops that environment's site-packages.
        path = Path(os.path.abspath(Path(value).expanduser()))
        key = str(path)
        if key not in seen and path.is_file():
            seen.add(key)
            candidates.append(path)
    return candidates


def preferred_asr_backends() -> tuple[str, ...]:
    requested = os.getenv(ASR_BACKEND_ENV, "").strip().lower().replace("-", "_")
    aliases = {"mlx": "mlx_whisper", "faster": "faster_whisper", "ctranslate2": "faster_whisper"}
    requested = aliases.get(requested, requested)
    if requested in {"mlx_whisper", "faster_whisper"}:
        return (requested,)
    if platform.system() == "Darwin" and platform.machine().lower() == "arm64":
        return ("mlx_whisper", "faster_whisper")
    return ("faster_whisper", "mlx_whisper")


def socks_proxy_transport_ready() -> bool:
    """Reject a runtime that cannot honor the process's active SOCKS proxy.

    Importing Whisper alone is not a sufficient health check: Hugging Face's
    HTTP client initializes its proxy transport before recognition starts.  A
    reused environment created before ``socksio`` became part of the managed
    requirements otherwise looks healthy and then fails every transcription.
    """
    all_proxy = str(os.getenv("ALL_PROXY") or os.getenv("all_proxy") or "").strip().lower()
    return not all_proxy.startswith(("socks://", "socks4://", "socks4a://", "socks5://", "socks5h://")) \
        or importlib.util.find_spec("socksio") is not None


def probe_asr_runtime(python: Path) -> dict | None:
    for backend in preferred_asr_backends():
        probe = (
            "import importlib.util, json, os, platform, sys; "
            "p=str(os.getenv('ALL_PROXY') or os.getenv('all_proxy') or '').strip().lower(); "
            "assert not p.startswith(('socks://','socks4://','socks4a://','socks5://','socks5h://')) or importlib.util.find_spec('socksio') is not None, 'SOCKS proxy transport is unavailable'; "
            f"import {backend}; "
            "print(json.dumps({'python': sys.executable, 'architecture': platform.machine(), "
            "'backend': " + repr(backend) + "}))"
        )
        completed = subprocess.run(
            [str(python), "-c", probe], text=True, capture_output=True, timeout=20,
        )
        if completed.returncode != 0:
            continue
        try:
            return json.loads(completed.stdout.strip().splitlines()[-1])
        except (json.JSONDecodeError, IndexError):
            continue
    return None


def resolve_asr_runtime() -> tuple[Path, dict]:
    for python in asr_python_candidates():
        details = probe_asr_runtime(python)
        if details:
            return python, details
    raise RuntimeError(
        "No compatible local Whisper runtime passed the complete health check. Do not install "
        "or repair ASR during an ingest job and do not substitute keyframes for speech. Prepare "
        f"the local ASR capability in Pickoo Settings, or connect {ASR_PYTHON_ENV} to a runtime "
        "that includes its active proxy transport dependencies."
    )


def enter_asr_runtime() -> dict:
    """Re-exec under the best local Whisper backend for this architecture."""
    for backend in preferred_asr_backends():
        if importlib.util.find_spec(backend) is not None and socks_proxy_transport_ready():
            os.environ[ASR_BACKEND_ENV] = backend
            return {"python": sys.executable, "architecture": platform.machine(), "backend": backend}
    python, details = resolve_asr_runtime()
    environment = os.environ.copy()
    environment[ASR_PYTHON_ENV] = str(python)
    environment[ASR_BACKEND_ENV] = str(details["backend"])
    os.execve(str(python), [str(python), str(Path(__file__).resolve()), *sys.argv[1:]], environment)
    return details


def now_iso() -> str:
    return datetime.now().astimezone().isoformat(timespec="seconds")


def atomic_text(path: Path, text: str) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    fd, temporary = tempfile.mkstemp(prefix=f".{path.name}.", dir=path.parent)
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as handle:
            handle.write(text)
            handle.flush()
            os.fsync(handle.fileno())
        os.replace(temporary, path)
    finally:
        if os.path.exists(temporary):
            os.unlink(temporary)


def atomic_json(path: Path, value: Any) -> None:
    atomic_text(path, json.dumps(value, ensure_ascii=False, indent=2) + "\n")


def sha256_file(path: Path) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        for chunk in iter(lambda: handle.read(1024 * 1024), b""):
            digest.update(chunk)
    return digest.hexdigest()


def executable(name: str) -> str:
    found = shutil.which(name)
    if not found:
        raise ValueError(f"missing dependency: {name}")
    return found


def faster_whisper_model(model: str) -> str:
    """Map the shared MLX setting to the equivalent CTranslate2 model name."""
    if model.startswith("mlx-community/"):
        model = model.split("/", 1)[1]
    return model.removeprefix("whisper-")


def resolve_model(model: str, backend: str = "mlx_whisper") -> str:
    candidate = Path(model).expanduser()
    if candidate.exists():
        return str(candidate.resolve())
    if backend == "faster_whisper":
        return faster_whisper_model(model)
    if "/" not in model:
        return model
    owner, repo = model.split("/", 1)
    snapshots = Path.home() / ".cache" / "huggingface" / "hub" / f"models--{owner}--{repo}" / "snapshots"
    existing = sorted((path for path in snapshots.glob("*") if path.is_dir()), key=lambda path: path.stat().st_mtime, reverse=True)
    return str(existing[0]) if existing else model


def model_repository(model: str, backend: str) -> str | None:
    if Path(model).expanduser().exists():
        return None
    if backend == "faster_whisper":
        mapped = faster_whisper_model(model)
        if "/" in mapped:
            return mapped
        if mapped in {"large-v3-turbo", "turbo"}:
            return "mobiuslabsgmbh/faster-whisper-large-v3-turbo"
        return f"Systran/faster-whisper-{mapped}"
    return model if "/" in model else None


def model_snapshot_ready(path: Path) -> bool:
    """Require both metadata and actual weights, not a partial HF directory."""
    if not path.is_dir() or not (path / "config.json").is_file():
        return False
    return any(path.glob("*.safetensors")) or any(path.glob("*.npz")) or (path / "model.bin").is_file()


def model_is_cached(model: str, backend: str) -> bool:
    candidate = Path(model).expanduser()
    if candidate.is_dir():
        return model_snapshot_ready(candidate)
    if backend == "faster_whisper":
        mapped = faster_whisper_model(model)
        if "/" in mapped:
            owner, repo = mapped.split("/", 1)
        elif mapped in {"large-v3-turbo", "turbo"}:
            owner, repo = "mobiuslabsgmbh", "faster-whisper-large-v3-turbo"
        else:
            owner, repo = "Systran", f"faster-whisper-{mapped}"
    elif "/" in model:
        owner, repo = model.split("/", 1)
    else:
        return False
    snapshots = Path.home() / ".cache" / "huggingface" / "hub" / f"models--{owner}--{repo}" / "snapshots"
    return any(model_snapshot_ready(path) for path in snapshots.glob("*"))


def _faster_segment(segment: Any) -> dict:
    words = []
    for word in segment.words or []:
        words.append({
            "word": word.word,
            "start": word.start,
            "end": word.end,
            "probability": word.probability,
        })
    return {
        "id": segment.id,
        "seek": segment.seek,
        "start": segment.start,
        "end": segment.end,
        "text": segment.text,
        "tokens": list(segment.tokens),
        "temperature": segment.temperature,
        "avg_logprob": segment.avg_logprob,
        "compression_ratio": segment.compression_ratio,
        "no_speech_prob": segment.no_speech_prob,
        "words": words,
    }


def transcribe_local(
    media: str,
    model: str,
    backend: str,
    *,
    word_timestamps: bool,
    condition_on_previous_text: bool,
    initial_prompt: str,
    language: str | None,
) -> dict:
    if backend == "mlx_whisper":
        from mlx_whisper import transcribe
        result = transcribe(
            media,
            path_or_hf_repo=model,
            word_timestamps=word_timestamps,
            condition_on_previous_text=condition_on_previous_text,
            initial_prompt=initial_prompt,
            language=language,
            verbose=False,
        )
        return {**result, "backend": backend}
    if backend == "faster_whisper":
        from faster_whisper import WhisperModel
        compute_type = os.getenv("CREATOR_INGEST_ASR_COMPUTE_TYPE", "int8")
        configured_threads = str(os.getenv("CREATOR_INGEST_ASR_CPU_THREADS") or "").strip()
        cpu_threads = int(configured_threads) if configured_threads.isdigit() else max(1, min(4, (os.cpu_count() or 2) - 1))
        model_key = (model, compute_type, cpu_threads)
        runtime = _FASTER_WHISPER_MODELS.get(model_key)
        if runtime is None:
            runtime = WhisperModel(
                model,
                device="cpu",
                compute_type=compute_type,
                cpu_threads=cpu_threads,
                num_workers=1,
            )
            _FASTER_WHISPER_MODELS[model_key] = runtime
        segment_stream, info = runtime.transcribe(
            media,
            word_timestamps=word_timestamps,
            condition_on_previous_text=condition_on_previous_text,
            initial_prompt=initial_prompt,
            language=language,
            vad_filter=False,
        )
        segments = [_faster_segment(segment) for segment in segment_stream]
        return {
            "text": "".join(str(segment.get("text") or "") for segment in segments),
            "segments": segments,
            "language": info.language,
            "language_probability": info.language_probability,
            "duration": info.duration,
            "backend": backend,
        }
    raise RuntimeError(f"Unsupported local ASR backend: {backend}")


def probe(media: Path) -> dict:
    result = subprocess.run([
        executable("ffprobe"), "-v", "error", "-show_entries",
        "format=duration:stream=index,codec_type,width,height,sample_rate,channels",
        "-of", "json", str(media),
    ], check=True, text=True, capture_output=True)
    value = json.loads(result.stdout)
    duration = float((value.get("format") or {}).get("duration") or 0)
    streams = value.get("streams") or []
    if duration <= 0 or not any(item.get("codec_type") == "audio" for item in streams):
        raise ValueError("media needs a positive duration and an audio stream")
    return {
        "duration": duration,
        "video": next((item for item in streams if item.get("codec_type") == "video"), None),
        "audio": next((item for item in streams if item.get("codec_type") == "audio"), None),
    }


def compact(value: Any) -> str:
    return re.sub(r"\s+", "", str(value or ""))


def source_review_context(manifest: dict, limit: int = 1200) -> str:
    """Return bounded source-authored text for spelling review, never as transcript truth."""
    source = manifest.get("source") or {}
    values = [source.get("title"), source.get("body"), source.get("description"), source.get("caption")]
    output: list[str] = []
    seen: set[str] = set()
    for value in values:
        cleaned = re.sub(r"\s+", " ", str(value or "")).strip()
        if cleaned and cleaned not in seen:
            seen.add(cleaned)
            output.append(cleaned)
    return " ｜ ".join(output)[:limit]


def source_dialect_profile(manifest: dict) -> tuple[str | None, list[str]]:
    """Use explicit source metadata only; never guess a whole clip's dialect."""
    source = manifest.get("source") or {}
    values = [source.get("title"), source.get("body"), source.get("description"), source.get("caption")]
    # Platform topic tags are source-authored, structured metadata.  They are
    # safer dialect cues than guessing from a short audio fragment and must not
    # be lost merely because capture normalized hashtags out of source.body.
    tags = source.get("tags") or []
    if isinstance(tags, (list, tuple, set)):
        values.extend(tags)
    haystack = "\n".join(str(value or "") for value in values).lower()
    for name, profile in DIALECT_PROFILES.items():
        matched = [cue for cue in profile["cues"] if cue.lower() in haystack]
        if matched:
            return name, matched
    return None, []


def asr_prompt(dialect_profile: str | None, terms: str = "") -> str:
    if dialect_profile:
        prompt = str(DIALECT_PROFILES[dialect_profile]["prompt"])
    else:
        prompt = "以下是中文播客或视频的逐字稿，请保留原话并使用自然的中文标点。"
    if terms:
        prompt += f"已确认专有名词：{terms}。"
    return prompt


def dialect_review_markdown(title: str, profile: str, qa: dict) -> str:
    label = "粤语" if profile == "cantonese" else profile
    return "\n".join([
        "# 逐字稿（方言识别版）", "", "> status: limited_dialect_released", "",
        f"- 内容：{title}",
        f"- 语言路线：{label}原话转写（不翻译为普通话）",
        f"- 自动 QA：{qa.get('status') or 'unknown'}",
        "- 处理：机器稿与时间轴可供浏览和内容理解；识别不足已按方言限制正常放行，不再循环复核。",
        "- 边界：未经对应音频或同期字幕核验的机器文字，不得作为精确金句或事实引文。", "",
    ])


SPACE_DELIMITED_LANGUAGE_PREFIXES = {
    "en", "fr", "de", "es", "it", "pt", "nl", "sv", "no", "da", "fi",
}
ASR_QA_POLICY_VERSION = "speech-qa-v10-repeated-generic-settlement"
ASR_STANDARD_PROJECTION_VERSION = ASR_QA_POLICY_VERSION
KNOWN_DECODER_WATERMARKS = (
    re.compile(r"^\s*(?:中文)?字幕志愿者(?:\s+|[：:])?[^\n]{0,24}\s*$"),
    re.compile(r"^\s*优优独播剧场(?:\s*[-—–｜|:：]+\s*|\s+)?YoYo\s+Television\s+Series\s+Exclusive\s*$", re.I),
)
GENERIC_DECODER_PHRASES = {
    "thank you",
    "thanks",
    "thanks for watching",
    "please subscribe",
    "subscribe",
}


def known_decoder_watermark(text: str) -> bool:
    """Match only stable decoder hallucinations, never generic subtitle prose."""
    return any(pattern.fullmatch(str(text or "").strip()) for pattern in KNOWN_DECODER_WATERMARKS)


def normalized_latin_phrase(text: str) -> str:
    return " ".join(re.findall(r"[a-z]+(?:['’][a-z]+)?", str(text or "").lower()))


def repeated_low_confidence_generic_indices(segments: list[dict], duration: float) -> set[int]:
    """Identify spaced, repeated generic decoder phrases before QA scoring.

    Whisper often emits a low-confidence first word followed by a high-confidence
    punctuation/ending token, which makes a phrase such as ``Thank you`` look
    superficially coherent.  Requiring three occurrences plus weak word evidence
    keeps an isolated real utterance untouched while preventing music-only clips
    from manufacturing several seconds of "reliable" speech.
    """
    candidates: dict[str, list[tuple[int, float, float]]] = {}
    for index, segment in enumerate(segments):
        phrase = normalized_latin_phrase(str(segment.get("text") or ""))
        if phrase not in GENERIC_DECODER_PHRASES:
            continue
        probabilities = [
            float(word.get("probability"))
            for word in (segment.get("words") or [])
            if isinstance(word, dict) and word.get("probability") is not None
        ]
        weak_word_evidence = bool(probabilities and min(probabilities) < 0.35)
        weak_segment_evidence = segment.get("avg_logprob") is not None and float(segment["avg_logprob"]) < -0.30
        if not (weak_word_evidence or weak_segment_evidence):
            continue
        candidates.setdefault(phrase, []).append((
            index,
            float(segment.get("start") or 0),
            float(segment.get("end") or segment.get("start") or 0),
        ))
    candidate_phrases = set(candidates)
    coherent_remainder_characters = 0
    coherent_remainder_seconds = 0.0
    for segment in segments:
        text = str(segment.get("text") or "").strip()
        if not text or normalized_latin_phrase(text) in candidate_phrases or known_decoder_watermark(text):
            continue
        if float(segment.get("compression_ratio") or 0) > 10.0:
            continue
        coherent_remainder_characters += len(compact(text))
        coherent_remainder_seconds += max(
            0.0,
            float(segment.get("end") or 0) - float(segment.get("start") or 0),
        )
    # This exclusion is a sparse/no-speech settlement rule, not a transcript
    # cleanup rule.  Substantial surrounding speech keeps every phrase intact.
    if coherent_remainder_characters > 48 or coherent_remainder_seconds > 5.0:
        return set()

    repeated: set[int] = set()
    for occurrences in candidates.values():
        # Two weak, identical decoder pleasantries that are far apart in a
        # long clip and have no coherent remainder are also a stable
        # music-only hallucination. Keep the old three-occurrence threshold
        # for shorter clips so two genuine nearby utterances remain intact.
        sparse_pair = bool(
            duration >= 45.0
            and len(occurrences) >= 2
            and max(start for _, start, _ in occurrences)
                - min(start for _, start, _ in occurrences) >= 15.0
        )
        if len(occurrences) < 3 and not sparse_pair:
            continue
        starts = [start for _, start, _ in occurrences]
        if max(starts) - min(starts) < 10.0:
            continue
        repeated.update(index for index, _, _ in occurrences)
    return repeated


def isolated_low_confidence_latin_remainder_indices(
    segments: list[dict],
    duration: float,
    repeated_generic_indices: set[int],
) -> set[int]:
    """Reject one weak lyric-like crumb when the rest is repeated decoder noise."""
    if duration < 30.0 or len(repeated_generic_indices) < 2:
        return set()
    remaining: list[tuple[int, dict]] = []
    for index, segment in enumerate(segments):
        text = str(segment.get("text") or "").strip()
        if not text or index in repeated_generic_indices or known_decoder_watermark(text):
            continue
        if float(segment.get("compression_ratio") or 0) > 10.0:
            continue
        remaining.append((index, segment))
    if len(remaining) != 1:
        return set()
    index, segment = remaining[0]
    text = str(segment.get("text") or "").strip()
    words = re.findall(r"[A-Za-z]+(?:['’][A-Za-z]+)?", text)
    span = max(0.0, float(segment.get("end") or 0) - float(segment.get("start") or 0))
    if not words or len(words) > 5 or re.search(r"[\u3400-\u9fff]", text) or span > 4.0:
        return set()
    probabilities = [
        float(word.get("probability"))
        for word in (segment.get("words") or [])
        if isinstance(word, dict) and word.get("probability") is not None
    ]
    weak_words = sum(probability < 0.25 for probability in probabilities)
    weak_segment = segment.get("avg_logprob") is not None and float(segment["avg_logprob"]) < -0.60
    weak_terminal_suffix = bool(
        len(words) == 1
        and span <= 0.5
        and probabilities
        and min(probabilities) < 0.5
        and weak_segment
    )
    return {index} if probabilities and (weak_words >= 2 and weak_segment or weak_terminal_suffix) else set()


def score_segments(segments: list[dict], duration: float, language: str | None = None) -> dict:
    language_prefix = str(language or "").lower().split("-", 1)[0]
    use_word_density = language_prefix in SPACE_DELIMITED_LANGUAGE_PREFIXES
    flagged = []
    catastrophic_ranges = []
    ignored_ranges = []
    reader_exclusions = []
    speech_seconds = 0.0
    characters = 0
    reliable_characters = 0
    reliable_speech_seconds = 0.0
    previous_start: float | None = None
    repeated_generic_indices = repeated_low_confidence_generic_indices(segments, duration)
    isolated_weak_latin_indices = isolated_low_confidence_latin_remainder_indices(
        segments, duration, repeated_generic_indices,
    )
    for index, segment in enumerate(segments):
        start = float(segment.get("start") or 0)
        end = float(segment.get("end") or start)
        text = str(segment.get("text") or "").strip()
        span = max(0.0, end - start)
        chars = len(compact(text))
        if text and not re.search(r"[0-9A-Za-z\u3400-\u9fff]", text):
            ignored_ranges.append({
                "segment": index,
                "start": start,
                "end": end,
                "text": text,
                "reasons": ["ignored_symbol_only_decoder_artifact"],
            })
            previous_start = start
            continue
        if known_decoder_watermark(text):
            exclusion = {
                "segment": index,
                "start": start,
                "end": end,
                "text": text,
                "reasons": ["known_music_watermark_hallucination"],
            }
            ignored_ranges.append(exclusion)
            reader_exclusions.append(exclusion)
            previous_start = start
            continue
        if index in repeated_generic_indices:
            exclusion = {
                "segment": index,
                "start": start,
                "end": end,
                "text": text,
                "reasons": ["repeated_low_confidence_generic_decoder_phrase"],
            }
            ignored_ranges.append(exclusion)
            reader_exclusions.append(exclusion)
            previous_start = start
            continue
        if index in isolated_weak_latin_indices:
            exclusion = {
                "segment": index,
                "start": start,
                "end": end,
                "text": text,
                "reasons": ["isolated_low_confidence_latin_remainder"],
            }
            ignored_ranges.append(exclusion)
            reader_exclusions.append(exclusion)
            previous_start = start
            continue
        characters += chars
        if chars:
            speech_seconds += span
        reasons = []
        catastrophic_reasons = []
        if previous_start is not None and start + 0.05 < previous_start:
            reasons.append("timestamp_out_of_order")
            catastrophic_reasons.append("timestamp_out_of_order")
        previous_start = start
        if segment.get("avg_logprob") is not None and float(segment["avg_logprob"]) < -1.0:
            reasons.append("low_log_probability")
        if segment.get("no_speech_prob") is not None and text and float(segment["no_speech_prob"]) > 0.60:
            reasons.append("high_no_speech_probability")
        if segment.get("compression_ratio") is not None and float(segment["compression_ratio"]) > 2.40:
            reasons.append("high_compression_ratio")
            if float(segment["compression_ratio"]) > 10.0:
                catastrophic_reasons.append("extreme_compression_loop")
        if "\ufffd" in text:
            reasons.append("replacement_character")
        words = re.findall(r"[A-Za-z]+(?:['’][A-Za-z]+)?", text)
        if use_word_density:
            density_units = len(words)
        else:
            # Counting every Latin letter as one spoken unit makes ordinary
            # mixed Chinese/English technical phrases look impossibly dense
            # whenever Whisper emits a short segment. Count a Latin word or a
            # number group as one unit, while preserving one unit per CJK
            # character. Pure Chinese behavior is unchanged.
            without_latin_words = re.sub(r"[A-Za-z]+(?:['’][A-Za-z]+)?", "", text)
            without_latin_or_numbers = re.sub(r"\d+(?:[.,]\d+)?", "", without_latin_words)
            cjk_and_other_units = len(compact(without_latin_or_numbers))
            number_units = len(re.findall(r"\d+(?:[.,]\d+)?", without_latin_words))
            density_units = cjk_and_other_units + len(words) + number_units
        density = density_units / span if span else 0
        dense_threshold = 7 if use_word_density else 14
        extreme_threshold = 12 if use_word_density else 25
        if span and density > dense_threshold:
            reasons.append("implausibly_dense_text")
            if density > extreme_threshold:
                catastrophic_reasons.append("extreme_text_density")
        terminal_latin_crumb = bool(
            catastrophic_reasons
            and index >= max(0, len(segments) - 2)
            and span <= 0.5
            and chars <= 3
            and re.fullmatch(r"[A-Za-z]+", compact(text) or "") is not None
        )
        if terminal_latin_crumb:
            ignored_ranges.append({
                "segment": index,
                "start": start,
                "end": end,
                "text": text,
                "reasons": ["ignored_terminal_latin_decoder_crumb", *catastrophic_reasons],
            })
            continue
        # Whisper occasionally assigns a coherent, very short utterance to a
        # near-zero timestamp span. That is an alignment warning, not evidence
        # that the surrounding transcript is hallucinated. Keep the source
        # text, but do not promote this timing-only defect into a catastrophic
        # lexical range.
        subsecond_alignment_warning = bool(
            catastrophic_reasons == ["extreme_text_density"]
            and 0 < span <= 0.5
            and 0 < density_units <= 8
            and re.search(r"[0-9A-Za-z\u3400-\u9fff]", text)
        )
        if subsecond_alignment_warning:
            ignored_ranges.append({
                "segment": index,
                "start": start,
                "end": end,
                "text": text,
                "reasons": ["ignored_subsecond_alignment_density", *catastrophic_reasons],
            })
            reliable_characters += chars
            reliable_speech_seconds += span
            continue
        if catastrophic_reasons:
            catastrophic = {
                "segment": index,
                "start": start,
                "end": end,
                "text": text,
                "reasons": catastrophic_reasons,
            }
            catastrophic_ranges.append(catastrophic)
            reader_exclusions.append({**catastrophic, "reasons": ["catastrophic_decoder_artifact", *catastrophic_reasons]})
        elif chars:
            reliable_characters += chars
            reliable_speech_seconds += span
        if reasons:
            flagged.append({"segment": index, "start": start, "end": end, "text": text, "reasons": reasons})
    issue_ratio = len(flagged) / max(len(segments), 1)
    cps = characters / duration
    catastrophic_ratio = len(catastrophic_ranges) / max(len(segments), 1)
    catastrophic_clusters: list[dict] = []
    for item in sorted(catastrophic_ranges, key=lambda value: (float(value["start"]), float(value["end"]))):
        start, end = float(item["start"]), float(item["end"])
        if catastrophic_clusters and start <= catastrophic_clusters[-1]["end"] + 30.0:
            catastrophic_clusters[-1]["end"] = max(catastrophic_clusters[-1]["end"], end)
            catastrophic_clusters[-1]["range_count"] += 1
        else:
            catastrophic_clusters.append({"start": start, "end": end, "range_count": 1})
    catastrophic_seconds = sum(max(0.0, item["end"] - item["start"]) for item in catastrophic_clusters)
    catastrophic_characters = sum(len(compact(str(item.get("text") or ""))) for item in catastrophic_ranges)
    catastrophic_duration_ratio = catastrophic_seconds / max(duration, 0.001)
    catastrophic_systemic = bool(
        catastrophic_ranges
        and (
            catastrophic_ratio > 0.10
            or catastrophic_duration_ratio > 0.10
            or len(segments) <= 3
            or len(catastrophic_clusters) >= 3
        )
    )
    repetitive_systemic = bool(
        len(segments) >= 3
        and issue_ratio > 0.80
        and all("high_compression_ratio" in item.get("reasons", []) for item in flagged)
    )
    decoder_loop_only = bool(
        catastrophic_ranges
        and reliable_characters == 0
        and all("extreme_compression_loop" in item.get("reasons", []) for item in catastrophic_ranges)
    )
    # A music-heavy clip can contain a long decoder loop plus only one or two
    # incidental phrases. That is a valid sparse/no-transcript carrier, not a
    # systemic backend failure. Keep the bound narrow and require the loop to
    # dominate both duration and the small coherent remainder.
    decoder_loop_with_sparse_remainder = bool(
        catastrophic_ranges
        and reliable_characters <= 48
        and reliable_speech_seconds <= 5.0
        and (
            (
                (
                    catastrophic_duration_ratio >= 0.20
                    or (
                        catastrophic_duration_ratio >= 0.10
                        and catastrophic_characters >= max(200, reliable_characters * 10)
                    )
                )
                and catastrophic_seconds >= reliable_speech_seconds * 2
            )
            or (
                duration >= 20.0
                and reliable_speech_seconds / max(duration, 0.001) <= 0.12
                and len(segments) <= 6
                and len(catastrophic_clusters) == 1
                and catastrophic_characters >= max(200, reliable_characters * 4)
            )
        )
        and all("extreme_compression_loop" in item.get("reasons", []) for item in catastrophic_ranges)
    )
    catastrophic_indices = {int(item["segment"]) for item in catastrophic_ranges}
    noncatastrophic_flagged = [
        item for item in flagged if int(item.get("segment", -1)) not in catastrophic_indices
    ]
    # A single decoder loop often explodes into dozens of zero-duration
    # segments. Segment ratio therefore measures decoder fragmentation, not
    # source-wide damage. When one bounded loop is surrounded by substantial
    # coherent speech, omit that loop from the reader and keep the remainder
    # eligible for targeted review.
    localized_decoder_loop_with_reliable_remainder = bool(
        duration >= 30.0
        and catastrophic_ranges
        and len(catastrophic_clusters) == 1
        and catastrophic_duration_ratio <= 0.35
        and reliable_characters >= 80
        and reliable_speech_seconds >= 10.0
        and len(noncatastrophic_flagged) <= max(2, int(len(segments) * 0.10))
        and all("extreme_compression_loop" in item.get("reasons", []) for item in catastrophic_ranges)
    )
    if localized_decoder_loop_with_reliable_remainder:
        catastrophic_systemic = False
    watermark_only = bool(segments and not reliable_characters and reader_exclusions and not catastrophic_ranges)
    if not segments or not characters or (repetitive_systemic and not localized_decoder_loop_with_reliable_remainder) or decoder_loop_only or decoder_loop_with_sparse_remainder or watermark_only:
        status = "rejected"
    elif (catastrophic_systemic or issue_ratio > 0.30) and not localized_decoder_loop_with_reliable_remainder:
        status = "failed"
    elif flagged or catastrophic_ranges:
        status = "review"
    else:
        status = "passed"
    failure_reason = None
    if (repetitive_systemic and not localized_decoder_loop_with_reliable_remainder) or decoder_loop_only or decoder_loop_with_sparse_remainder or watermark_only:
        failure_reason = "music_or_no_reliable_speech"
    elif not segments or not characters:
        failure_reason = "no_recognized_speech"
    elif catastrophic_systemic:
        failure_reason = "non_monotonic_or_hallucinated_asr"
    elif issue_ratio > 0.30 and not localized_decoder_loop_with_reliable_remainder:
        failure_reason = "unreliable_or_hallucinated_asr"
    total_words = len(re.findall(
        r"[A-Za-z]+(?:['’][A-Za-z]+)?",
        " ".join(str(segment.get("text") or "") for segment in segments),
    )) if use_word_density else None
    return {
        "policy_version": ASR_QA_POLICY_VERSION,
        "status": status,
        "reader_eligible": status in {"passed", "review"},
        "fallback": None,
        "failure_reason": failure_reason,
        "segment_count": len(segments),
        "flagged_segment_count": len(flagged),
        "issue_ratio": round(issue_ratio, 4),
        "speech_coverage": round(min(1.0, speech_seconds / duration), 4),
        "characters_per_second": round(cps, 4),
        "reliable_character_count": reliable_characters,
        "reliable_speech_seconds": round(reliable_speech_seconds, 4),
        "density_metric": "words_per_second" if use_word_density else "characters_per_second",
        "words_per_minute": round(total_words / duration * 60, 2) if total_words is not None and duration > 0 else None,
        "low_confidence_ranges": flagged,
        "catastrophic_ranges": catastrophic_ranges,
        "ignored_ranges": ignored_ranges,
        "reader_exclusions": reader_exclusions,
        "catastrophic_impact": {
            "systemic": catastrophic_systemic,
            "range_count": len(catastrophic_ranges),
            "cluster_count": len(catastrophic_clusters),
            "segment_ratio": round(catastrophic_ratio, 4),
            "duration_ratio": round(catastrophic_duration_ratio, 4),
        },
        "sparse_speech": {
            "coherent_sparse_speech_allowed": True,
            "decoder_loop_dominates_remainder": decoder_loop_with_sparse_remainder,
            "localized_decoder_loop_omitted": localized_decoder_loop_with_reliable_remainder,
        },
    }


def rejected_asr_markdown(title: str, qa: dict) -> str:
    return "\n".join([
        "# 音轨识别结论", "", "> status: no_reliable_speech", "",
        f"- 内容：{title}",
        f"- 判定：{qa.get('failure_reason') or 'unreliable_asr'}",
        "- 处理：逐字稿仅显示‘本视频无可识别人声’，不生成画面文字章节或逐帧 OCR 兜底。",
        "- 说明：被拒收的机器识别仅留在机器证据中，不进入全文、章节或金句资产。", "",
    ])


def settle_no_reliable_speech_review(evidence: Path, qa: dict) -> None:
    """Close lexical review when deterministic QA found no usable speech text."""
    review_path = evidence / "transcript-review.json"
    review = json.loads(review_path.read_text(encoding="utf-8")) if review_path.is_file() else {}
    already_settled = bool(
        review.get("status") == "not_applicable_no_reliable_speech"
        and review.get("failure_reason") == qa.get("failure_reason")
    )
    review.update({
        "status": "not_applicable_no_reliable_speech",
        "reviewer": "system-asr-qa",
        "reviewed_at": review.get("reviewed_at") if already_settled and review.get("reviewed_at") else now_iso(),
        "coverage": "not_applicable_no_reliable_speech",
        "checks": {
            "names_and_products": "not_applicable",
            "numbers_and_units": "not_applicable",
            "english_terms": "not_applicable",
            "quote_candidates": "not_applicable",
            "automated_low_confidence_ranges": "settled_no_reliable_speech",
        },
        "unresolved": [],
        "range_reviews": [],
        "evidence_method": ["deterministic_asr_rejection"],
        "failure_reason": qa.get("failure_reason"),
    })
    review.setdefault("corrections", [])
    atomic_json(review_path, review)


def clear_transcript_chapter_projection(package: Path, manifest: dict) -> None:
    """Remove stale reader chapters after ASR settles as non-reader evidence."""
    relatives = {
        "evidence/transcript/逐字稿-章节版.json",
        "evidence/transcript/逐字稿-章节版.md",
    }
    for relative in relatives:
        (package / relative).unlink(missing_ok=True)
    manifest["artifacts"] = [
        item for item in (manifest.get("artifacts") or [])
        if not (isinstance(item, dict) and str(item.get("path") or "") in relatives)
    ]
    transcript = manifest.setdefault("transcript", {})
    for key in (
        "chapters_path", "chapter_count", "chapter_status", "chapter_quality",
        "chapter_boundary_source", "chapter_semantic_review", "chapter_rebuild_status",
    ):
        transcript.pop(key, None)


def invalidate_recovered_speech_projection(package: Path, manifest: dict) -> None:
    """Reopen only evidence derived from a formerly reader-blocked ASR."""
    clear_transcript_chapter_projection(package, manifest)
    transcript = manifest.setdefault("transcript", {})
    transcript["chapter_rebuild_status"] = "pending_recovered_speech"
    review = manifest.get("asset_review")
    if isinstance(review, dict):
        review["status"] = "review"
        review["understanding_pending"] = True
        review["source_changed_after_analysis"] = True
    gates = manifest.setdefault("gates", {})
    for gate in ("analysis", "cards", "archive"):
        if gate in gates:
            gates[gate] = "review"


def stamp(seconds: float) -> str:
    value = max(0, int(seconds))
    return f"{value // 60:02d}:{value % 60:02d}"


def parse_timecode(value: str) -> float:
    parts = value.strip().split(":")
    if not parts or len(parts) > 3:
        raise ValueError(f"invalid timecode: {value}")
    try:
        numbers = [float(part) for part in parts]
    except ValueError as exc:
        raise ValueError(f"invalid timecode: {value}") from exc
    seconds = 0.0
    for number in numbers:
        if number < 0:
            raise ValueError(f"invalid negative timecode: {value}")
        seconds = seconds * 60 + number
    return seconds


def parse_retry_range(value: str, duration: float = math.inf) -> tuple[float, float]:
    match = re.fullmatch(r"\s*([^\s-]+)\s*-\s*([^\s-]+)\s*", value)
    if not match:
        raise ValueError(f"retry range must be START-END: {value}")
    start, end = parse_timecode(match.group(1)), parse_timecode(match.group(2))
    if end <= start or start >= duration:
        raise ValueError(f"retry range is outside media or empty: {value}")
    end = min(end, duration)
    if end - start > 180:
        raise ValueError("one retry range may not exceed 180 seconds; split it into bounded spans")
    return start, end


def replace_segments_in_range(
    segments: list[dict],
    replacements: list[dict],
    start: float,
    end: float,
) -> tuple[list[dict], int]:
    """Replace only segments intersecting one explicitly requested repair span."""
    kept = [
        dict(item) for item in segments
        if float(item.get("end") or item.get("start") or 0) <= start
        or float(item.get("start") or 0) >= end
    ]
    removed = len(segments) - len(kept)
    merged = sorted([*kept, *(dict(item) for item in replacements)], key=lambda item: (float(item.get("start") or 0), float(item.get("end") or 0)))
    for index, item in enumerate(merged):
        item["id"] = index
    return merged, removed


def retry_audio_ranges(
    package: Path,
    media: Path,
    ranges: list[str],
    duration: float,
    model: str,
    language: str | None,
    condition_on_previous_text: bool,
    prompt: str,
    backend: str,
) -> dict:
    """Transcribe bounded spans as additive review evidence without replacing canonical ASR."""
    output_dir = package / "evidence" / "transcript" / "range-reviews"
    output_dir.mkdir(parents=True, exist_ok=True)
    records = []
    with tempfile.TemporaryDirectory(prefix="cci-range-review-") as temporary:
        temp_dir = Path(temporary)
        for raw_range in ranges:
            start, end = parse_retry_range(raw_range, duration)
            clip = temp_dir / f"{int(start * 1000)}-{int(end * 1000)}.wav"
            subprocess.run([
                executable("ffmpeg"), "-loglevel", "error", "-ss", f"{start:.3f}",
                "-t", f"{end - start:.3f}", "-i", str(media), "-vn", "-ac", "1",
                "-ar", "16000", str(clip),
            ], check=True)
            result = transcribe_local(
                str(clip), model, backend, word_timestamps=True,
                condition_on_previous_text=condition_on_previous_text,
                initial_prompt=prompt, language=language,
            )
            segments = []
            for segment in result.get("segments") or []:
                adjusted = dict(segment)
                adjusted["start"] = round(start + float(segment.get("start") or 0), 3)
                adjusted["end"] = round(start + float(segment.get("end") or 0), 3)
                segments.append(adjusted)
            record = {
                "schema_version": "transcript-range-review/v1",
                "requested_range": raw_range,
                "start": start,
                "end": end,
                "reviewed_at": now_iso(),
                "evidence_method": "bounded_audio_retranscription",
                "media_path": str(media.relative_to(package)),
                "media_sha256": sha256_file(media),
                "model": model,
                "backend": backend,
                "language": result.get("language"),
                "segments": segments,
            }
            path = output_dir / f"{int(start * 1000):010d}-{int(end * 1000):010d}.json"
            atomic_json(path, record)
            records.append({
                "range": raw_range,
                "path": str(path.relative_to(package)),
                "text": "".join(str(item.get("text") or "").strip() for item in segments),
                "segments": len(segments),
            })
    return {"status": "reviewed_ranges", "package": str(package), "ranges": records}


def transcript_markdown(title: str, model: str, qa: dict, segments: list[dict]) -> str:
    lines = [
        "# 逐字稿（机器原稿）", "", "> status: pending_lexical_and_audio_review", "",
        f"- 内容：{title}", f"- 模型：`{model}`", f"- 自动 QA：`{qa['status']}`",
        "- 说明：这是时间轴机器稿；专名、数字、英文、引语在视觉/音频核验前不得作为准确引文。", "",
        "## 时间轴", "",
    ]
    for segment in segments:
        lines.append(f"- `{stamp(segment.get('start', 0))}–{stamp(segment.get('end', 0))}` {str(segment.get('text') or '').strip()}")
    return "\n".join(lines) + "\n"


def review_packet(title: str, segments: list[dict], captions: dict, source_context: str = "", asr_terms: list[str] | None = None) -> str:
    track = captions.get("track") or []
    lines = [
        "# 逐字稿复核包", "", "> status: pending_lexical_and_audio_review", "",
        f"- 内容：{title}",
        "- 用法：逐段对照 ASR、同期画面 OCR 和原音；OCR 只是第二证据，不能自动覆盖 ASR。", "",
        "## 来源正文校词参考", "",
        "> 仅用于核对专名和明显错词，不能覆盖原音或补写音频中没有说过的话。", "",
        source_context or "_来源未提供可用正文/描述_", "",
        "## 本次 ASR 词典", "",
        f"- 自动注入：{', '.join(asr_terms or []) or '无'}",
        "- 词典只提供术语拼写提示；所有词仍须由原音、来源正文或同期画面复核。", "",
        "## 高风险词候选", "",
    ]
    asr_text = " ".join(str(item.get("text") or "") for item in segments)
    entity_pattern = re.compile(r"[A-Za-z][A-Za-z0-9._+-]{1,}|\d+(?:\.\d+)?(?:%|[kKwW万码片秒分小时])?")
    lines.append(f"- ASR 中的英文/数字：{', '.join(dict.fromkeys(entity_pattern.findall(asr_text))) or '无'}")
    lines.append("- OCR 不做全局词表展开；仅在下方同期候选中显示，避免把整段画面文字送入模型。")
    lines.extend(["", "## 分段对照", ""])
    for segment in segments:
        start = float(segment.get("start") or 0)
        end = float(segment.get("end") or start)
        asr = str(segment.get("text") or "").strip()
        candidates = []
        for item in track:
            if float(item.get("end") or 0) < start - 1 or float(item.get("start") or 0) > end + 1:
                continue
            for candidate in str(item.get("text") or "").splitlines():
                candidate = candidate.strip()
                if len(compact(candidate)) < 2:
                    continue
                score = difflib.SequenceMatcher(None, compact(asr), compact(candidate)).ratio()
                if score >= 0.28:
                    candidates.append((score, candidate))
        candidates.sort(reverse=True)
        best = list(dict.fromkeys(candidate for _, candidate in candidates[:3]))
        lines.extend([
            f"### {stamp(start)}–{stamp(end)}", "",
            f"- ASR：{asr or '_empty_'}",
            f"- 同期 OCR 候选：{' / '.join(best) if best else '_无可靠候选，必须回听_'}",
            "- 复核结论：_pending_", "",
        ])
    lines.extend(["## 准入规则", "", "只有把复核结论写入 `逐字稿-校正版.md`，并把 `quotation_ready` 改为 true 后，才可进入高质量语料库或作为逐字引文。", ""])
    return "\n".join(lines)


def write_standard_transcript(package: Path, qa: dict, captions: dict, segments: list[dict], dialect_profile: str | None = None) -> None:
    """Replace the video placeholder with a navigable, explicitly non-final ASR timeline."""
    standard_path = package / "标准内容.md"
    standard = standard_path.read_text(encoding="utf-8")
    standard = standard.replace(
        "> 完整度：待高质量逐字稿",
        "> 完整度：机器时间轴已提取；校正版与可引用状态待音频/画面复核。",
    )
    standard = re.sub(r"\n## 逐字稿\n\n_待运行高质量转写管线。_\n?", "\n", standard)
    standard = re.sub(r"\n## (?:高质量逐字稿证据|逐字稿（机器原稿）)\n.*\Z", "", standard, flags=re.S)
    state_line = (
        "> 状态：方言识别结果已有限放行；可用于浏览和概要理解，未经音频或同期字幕核验不得精确引用。"
        if dialect_profile else
        "> 状态：待词汇与音频复核。专名、数字、英文和引语在复核前不得作为准确引文。"
    )
    lines = [
        "## 逐字稿（机器原稿）", "",
        state_line, "",
        f"- ASR QA：`{qa['status']}`",
        f"- 字幕 OCR：`{captions['status']}`",
        "- 原始 ASR：`evidence/transcript/transcript-raw.json`",
        "- 复核包：`evidence/transcript/逐字稿-复核包.md`",
        "- 校正版：`evidence/transcript/逐字稿-校正版.md`", "",
        "### 时间轴", "",
    ]
    excluded_segments = {
        int(item["segment"])
        for item in qa.get("reader_exclusions") or []
        if isinstance(item, dict) and item.get("segment") is not None
    }
    for index, segment in enumerate(segments):
        if index in excluded_segments:
            continue
        text = str(segment.get("text") or "").strip() or "_empty_"
        lines.append(f"- `{stamp(segment.get('start', 0))}–{stamp(segment.get('end', 0))}` {text}")
    atomic_text(standard_path, standard.rstrip() + "\n\n" + "\n".join(lines) + "\n")


def write_no_speech_standard(package: Path, qa: dict) -> None:
    """Keep failed/hallucinated ASR out of the reader-facing document."""
    standard_path = package / "标准内容.md"
    standard = standard_path.read_text(encoding="utf-8")
    standard = re.sub(r"\n## (?:高质量逐字稿证据|逐字稿（机器原稿）|逐字稿|音轨识别结论)\n.*\Z", "", standard, flags=re.S)
    lines = [
        "## 音轨识别结论", "",
        "> 未发现可可靠转写的人声；已拒收机器识别结果。", "",
        "- 处理路径：保留逐字稿入口并显示‘本视频无可识别人声’；不运行逐帧 OCR 兜底。",
        "- 被拒收的 ASR 只保留为机器审计证据，不进入正文、章节或金句。", "",
    ]
    atomic_text(standard_path, standard.rstrip() + "\n\n" + "\n".join(lines))


def extract_keyframes(media: Path, output: Path, duration: float, maximum: int) -> list[dict]:
    output.mkdir(parents=True, exist_ok=True)
    count = max(1, min(maximum, math.ceil(duration / 3)))
    interval = max(duration / count, 1.0)
    command = [
        executable("ffmpeg"), "-loglevel", "error", "-i", str(media),
        "-vf", f"fps=1/{interval:.4f},scale='min(1280,iw)':-2",
        "-q:v", "2", str(output / "frame-%03d.jpg"),
    ]
    subprocess.run(command, check=True)
    records = []
    for index, path in enumerate(sorted(output.glob("frame-*.jpg"))[:maximum], 1):
        records.append({
            "index": index,
            "approx_second": round((index - 0.5) * interval, 2),
            "path": str(path),
            "bytes": path.stat().st_size,
            "sha256": sha256_file(path),
        })
    return records


def run_caption_ocr(media: Path, output: Path, duration: float) -> dict:
    from ocr_runtime import run_ocr as recognize_images

    sample_rate = max(1.0, duration / 120.0)
    with tempfile.TemporaryDirectory(prefix="cci-caption-ocr-") as temporary:
        frames = Path(temporary)
        subprocess.run([
            executable("ffmpeg"), "-loglevel", "error", "-i", str(media),
            "-vf", f"fps=1/{sample_rate:.4f},crop=iw:ih*0.35:0:ih*0.65,scale=iw*2:-2",
            "-q:v", "3", str(frames / "%06d.jpg"),
        ], check=True)
        paths = sorted(frames.glob("*.jpg"))
        if not paths:
            return {"status": "no_frames", "sample_interval_seconds": sample_rate, "frames": []}
        records = recognize_images(paths)
    compact_track = []
    for index, record in enumerate(records):
        text = str(record.get("text") or "").strip()
        if not text:
            continue
        second = round(index * sample_rate, 2)
        if compact_track and compact_track[-1]["text"] == text:
            compact_track[-1]["end"] = second
        else:
            compact_track.append({"start": second, "end": second, "text": text, "ocr_status": record.get("status")})
    value = {"status": "available" if compact_track else "no_text", "sample_interval_seconds": sample_rate, "track": compact_track, "raw_frame_count": len(records)}
    atomic_json(output / "caption-track.json", value)
    return value


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--package", type=Path)
    parser.add_argument("--runtime-check", action="store_true", help="Resolve and report the reusable ASR runtime without transcribing")
    parser.add_argument("--prepare-model", action="store_true", help="Download the configured model only from the Settings capability installer")
    parser.add_argument("--health-check", action="store_true", help="Run one tiny local inference sample from the Settings capability installer")
    parser.add_argument("--media", type=Path)
    parser.add_argument("--model", default=DEFAULT_MODEL)
    parser.add_argument("--prompt-term", action="append", default=[])
    parser.add_argument("--language", help="Force the speech language, e.g. zh or en, when auto-detection is unreliable")
    parser.add_argument("--condition-on-previous-text", action=argparse.BooleanOptionalAction, default=False, help="Reuse prior decoding-window text. Disabled by default to prevent cross-window loops and timestamp corruption; punctuation is restored downstream.")
    parser.add_argument("--skip-caption-ocr", action="store_true")
    parser.add_argument("--caption-ocr", action="store_true", help="Run bounded lower-frame OCR only after a named visual/text conflict requires it")
    parser.add_argument("--max-keyframes", type=int, default=24)
    parser.add_argument("--rescore-existing", action="store_true", help="Re-run deterministic QA against the archived raw ASR without retranscribing")
    parser.add_argument("--refresh-standard", action="store_true", help="Rebuild standard content from existing transcript evidence without retranscribing")
    parser.add_argument("--retry-range", action="append", default=[], metavar="START-END", help="Re-transcribe one bounded audio span as additive review evidence; repeat for multiple spans")
    args = parser.parse_args()

    if args.runtime_check or args.prepare_model or args.health_check:
        details = enter_asr_runtime()
        if args.prepare_model and not model_is_cached(args.model, details["backend"]):
            repository = model_repository(args.model, details["backend"])
            if not repository:
                raise RuntimeError("The configured ASR model is neither a readable local directory nor a downloadable repository.")
            from huggingface_hub import snapshot_download
            snapshot_download(repo_id=repository)
        if args.health_check:
            if not model_is_cached(args.model, details["backend"]):
                raise RuntimeError("The configured ASR model is incomplete; the minimum sample health check cannot run.")
            with tempfile.TemporaryDirectory(prefix="pickoo-asr-health-") as temporary:
                sample = Path(temporary) / "silence.wav"
                with wave.open(str(sample), "wb") as output:
                    output.setnchannels(1)
                    output.setsampwidth(2)
                    output.setframerate(16_000)
                    output.writeframes(b"\x00\x00" * 16_000)
                transcribe_local(
                    str(sample), resolve_model(args.model, details["backend"]), details["backend"],
                    word_timestamps=True, condition_on_previous_text=False,
                    initial_prompt="", language="zh",
                )
            details["minimum_sample_passed"] = True
        details["model"] = resolve_model(args.model, details["backend"])
        details["model_cached"] = model_is_cached(args.model, details["backend"])
        print(json.dumps(details, ensure_ascii=False))
        return 0
    if args.package is None:
        parser.error("--package is required unless --runtime-check is used")

    package = args.package.expanduser().resolve()
    manifest_path = package / "manifest.json"
    manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
    content_form = manifest.get("content_form")
    if content_form not in {"video", "audio", "podcast", "podcast_episode"}:
        raise ValueError("package content_form must be video or audio")
    evidence = package / "evidence" / "transcript"
    media = args.media.expanduser().resolve() if args.media else package / str((manifest.get("media") or {}).get("path") or "")
    if not media.is_file() and not args.refresh_standard:
        raise ValueError(f"media file not found: {media}")
    info = probe(media) if media.is_file() else {
        "duration": float((manifest.get("transcript") or {}).get("duration_seconds") or 0),
        "video": content_form == "video",
        "audio": True,
    }
    if args.rescore_existing:
        raw_path = evidence / "transcript-raw.json"
        if not raw_path.is_file():
            raise ValueError("existing transcript-raw.json is required for --rescore-existing")
        raw = json.loads(raw_path.read_text(encoding="utf-8"))
        qa = score_segments(raw.get("segments") or [], info["duration"], raw.get("language"))
        atomic_json(evidence / "transcript-qa.json", qa)
        transcript = manifest.setdefault("transcript", {})
        transcript["asr_qa_status"] = qa["status"]
        transcript["asr_qa_policy_version"] = qa.get("policy_version")
        transcript["reader_exclusion_count"] = len(qa.get("reader_exclusions") or [])
        previous_mode = str(transcript.get("mode") or "")
        no_reliable_speech = bool(
            qa.get("status") == "rejected"
            and qa.get("failure_reason") in {"music_or_no_reliable_speech", "no_recognized_speech"}
            and qa.get("reader_eligible") is False
        )
        asr_failed = qa.get("status") == "failed"
        next_mode = "no_reliable_speech" if no_reliable_speech else "asr_failed" if asr_failed else "speech"
        transcript["mode"] = next_mode
        transcript["foundation_status"] = (
            "settled_no_reliable_speech" if no_reliable_speech
            else "blocked_asr_failed" if asr_failed
            else "speech_evidence_ready"
        )
        transcript["lexical_review_status"] = (
            "not_applicable_no_reliable_speech" if no_reliable_speech
            else "blocked_asr_failed" if asr_failed
            else "pending" if transcript.get("lexical_review_status") in {None, "", "not_applicable_no_reliable_speech", "blocked_asr_failed"}
            else transcript.get("lexical_review_status")
        )
        transcript["asr_recovery_status"] = (
            "settled_no_reliable_speech" if no_reliable_speech
            else "failed_systemic" if asr_failed
            else "qa_reopened"
        )
        transcript["lexical_issue_ranges"] = (
            [] if no_reliable_speech
            else issue_ranges(qa, {"unresolved": []}, info["duration"]) if content_form == "video"
            else []
        )
        if no_reliable_speech:
            clear_transcript_chapter_projectio

#!/usr/bin/env python3
"""Validate a canonical creator-content package manifest and required files."""

from __future__ import annotations

import argparse
import hashlib
import json
import re
import sys
from pathlib import Path

from package_document import package_document_path

GATES = ("identity", "acquisition", "extraction", "fidelity", "analysis", "comments", "cover", "cards", "archive")
RESULTS = ("passed", "failed", "review", "partial", "not_applicable")
REQUIRED_FILES = ("manifest.json", "输入上下文.md", "原始内容.md", "标准内容.md")
HASH_CHECK_SUFFIXES = {".json", ".md", ".txt", ".html", ".lrc", ".vtt", ".srt"}
MAX_HASH_CHECK_BYTES = 64 * 1024 * 1024


def sha256_file(path: Path) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        for chunk in iter(lambda: handle.read(1024 * 1024), b""):
            digest.update(chunk)
    return digest.hexdigest()


def manifest_file(package: Path, relative: object, label: str, errors: list[str]) -> Path | None:
    if not isinstance(relative, str) or not relative.strip():
        return None
    target = (package / relative).resolve()
    try:
        target.relative_to(package.resolve())
    except ValueError:
        errors.append(f"{label} escapes package: {relative}")
        return None
    if not target.is_file() or target.stat().st_size == 0:
        errors.append(f"missing or empty {label}: {relative}")
        return None
    return target


def validate(package: Path, allow_incomplete: bool, *, manifest_override: dict | None = None) -> list[str]:
    errors: list[str] = []
    manifest_path = package / "manifest.json"
    if manifest_override is not None:
        manifest = manifest_override
    else:
        try:
            manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
        except (OSError, json.JSONDecodeError) as exc:
            manifest = {}
            errors.append(f"invalid manifest.json: {exc}")
    for name in REQUIRED_FILES:
        path = package / name
        if not path.is_file() or path.stat().st_size == 0:
            errors.append(f"missing or empty: {name}")
    organized_path = package / "梳理后内容.md"
    document_relative = (manifest.get("document") or {}).get("path") if isinstance(manifest.get("document"), dict) else None
    if document_relative:
        manifest_file(package, document_relative, "manifest.document.path", errors)
    else:
        errors.append("manifest.document.path is missing or invalid")
    try:
        favorite_path = package_document_path(package, manifest)
    except ValueError:
        favorite_path = package / ".missing-canonical-document"
    favorite_text = favorite_path.read_text("utf-8") if favorite_path.is_file() else ""
    has_canonical_analysis = bool(re.search(r"(?m)^## 内容拆解\s*$[\s\S]*?(?=^##\s|\Z)", favorite_text))
    if (not organized_path.is_file() or organized_path.stat().st_size == 0) and not has_canonical_analysis:
        errors.append("missing or empty: 梳理后内容.md (or package document #内容拆解)")
    if errors and not manifest_path.is_file():
        return errors
    for field in ("package_schema", "package_id", "job_id", "content_form", "source", "artifacts", "gates", "errors"):
        if field not in manifest:
            errors.append(f"manifest missing field: {field}")
    gates = manifest.get("gates", {})
    for gate in GATES:
        value = gates.get(gate)
        if value not in RESULTS:
            errors.append(f"gate {gate} has invalid result: {value!r}")
        if not allow_incomplete and value in ("failed", "review", "partial"):
            errors.append(f"gate {gate} is not complete: {value}")
    ledger = manifest.get("gate_ledger")
    if ledger is not None:
        if manifest.get("gate_ledger_schema") != "creator-gates/v1" or not isinstance(ledger, dict):
            errors.append("invalid gate ledger schema")
        else:
            for gate in GATES:
                entry = ledger.get(gate)
                if not isinstance(entry, dict):
                    errors.append(f"gate ledger missing entry: {gate}")
                    continue
                if not entry.get("rule_version") or not entry.get("input_fingerprint"):
                    errors.append(f"gate ledger {gate} lacks version or input fingerprint")
                if not allow_incomplete and entry.get("status") in {"stale", "missing", "failed", "review", "partial"}:
                    errors.append(f"gate ledger {gate} is not complete: {entry.get('status')}")
    platform = ((manifest.get("source") or {}).get("platform") or "").lower()
    mode = manifest.get("mode", "full")
    if manifest.get("content_form") == "image_post" and (
        gates.get("extraction") == "passed" or gates.get("fidelity") == "passed"
    ):
        inventory_indices = {
            int(item.get("index"))
            for item in manifest.get("image_inventory") or []
            if isinstance(item, dict) and str(item.get("index") or "").isdigit()
        }
        decision_path = package / "evidence/ocr/decision.json"
        try:
            decision = json.loads(decision_path.read_text("utf-8"))
        except (OSError, json.JSONDecodeError):
            decision = {}
        if decision.get("status") != "all_pages_ocr" or decision.get("policy") != "mandatory_per_image_v1":
            errors.append("passed image_post requires mandatory_per_image_v1 OCR decision")
        raw_indices = {
            int(match.group(1))
            for path in (package / "evidence/ocr").glob("image-*.raw.txt")
            if (match := re.fullmatch(r"image-(\d+)\.raw\.txt", path.name))
        }
        corrected_indices = {
            int(match.group(1))
            for path in (package / "evidence/ocr").glob("image-*.corrected.md")
            if (match := re.fullmatch(r"image-(\d+)\.corrected\.md", path.name))
        }
        if not inventory_indices or raw_indices != inventory_indices or corrected_indices != inventory_indices:
            errors.append("passed image_post requires raw and corrected OCR for every inventory image")
        try:
            organized = json.loads((package / "evidence/ocr/organized.json").read_text("utf-8"))
        except (OSError, json.JSONDecodeError):
            organized = []
        classified_indices = {
            int(row.get("source_index"))
            for row in organized
            if isinstance(row, dict)
            and str(row.get("source_index") or "").isdigit()
            and row.get("text_class") in {"substantial_text", "sparse_text", "no_text"}
        }
        if classified_indices != inventory_indices:
            errors.append("passed image_post requires one controlled text classification per source image")
    social_analysis = platform in ("xhs", "douyin") and mode in ("analyze", "cards", "full")
    if social_analysis:
        cover_file = package / "封面分析.md"
        structured_cover = package / "evidence" / "cover" / "analysis.json"
        if (not cover_file.is_file() or cover_file.stat().st_size == 0) \
                and (not structured_cover.is_file() or structured_cover.stat().st_size == 0):
            errors.append("missing or empty: 封面分析.md (or evidence/cover/analysis.json)")
        comments_file = package / "evidence" / "comments" / "top-comments.json"
        if not comments_file.is_file() or comments_file.stat().st_size == 0:
            errors.append("missing or empty: evidence/comments/top-comments.json")
    package_id = manifest.get("package_id")
    if package_id and package.name != package_id:
        errors.append(f"package directory {package.name!r} != package_id {package_id!r}")
    artifacts = manifest.get("artifacts", [])
    if not isinstance(artifacts, list):
        errors.append("artifacts must be an array")
    else:
        artifact_paths: set[str] = set()
        for index, artifact in enumerate(artifacts):
            rel = artifact.get("path") if isinstance(artifact, dict) else None
            if not rel:
                errors.append(f"artifact {index} missing path")
                continue
            target = (package / rel).resolve()
            try:
                target.relative_to(package.resolve())
            except ValueError:
                errors.append(f"artifact escapes package: {rel}")
                continue
            if not target.is_file():
                errors.append(f"artifact does not exist: {rel}")
                continue
            artifact_paths.add(str(rel))
            expected_hash = str(artifact.get("sha256") or "")
            if expected_hash and target.suffix.lower() in HASH_CHECK_SUFFIXES and target.stat().st_size <= MAX_HASH_CHECK_BYTES:
                actual_hash = sha256_file(target)
                if actual_hash != expected_hash:
                    errors.append(f"artifact sha256 mismatch: {rel}")
        media_relative = (manifest.get("media") or {}).get("path") if isinstance(manifest.get("media"), dict) else None
        if media_relative:
            media_file = manifest_file(package, media_relative, "manifest.media.path", errors)
            if media_file and media_relative not in artifact_paths:
                errors.append(f"manifest.media.path is not registered as an artifact: {media_relative}")
            declared_bytes = (manifest.get("media") or {}).get("bytes")
            if media_file and isinstance(declared_bytes, (int, float)) and int(declared_bytes) != media_file.stat().st_size:
                errors.append(f"manifest.media.bytes mismatch: {media_relative}")
        cover_relative = (manifest.get("cover") or {}).get("path") if isinstance(manifest.get("cover"), dict) else None
        if cover_relative:
            manifest_file(package, cover_relative, "manifest.cover.path", errors)
    return errors


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("package")
    parser.add_argument("--allow-incomplete", action="store_true")
    args = parser.parse_args()
    package = Path(args.package).expanduser().resolve()
    errors = validate(package, args.allow_incomplete)
    if errors:
        print("\n".join(errors), file=sys.stderr)
        return 2
    print(f"valid package: {package}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

#!/usr/bin/env python3
"""Validate and optionally commit social comment signals and cover analysis gates."""

from __future__ import annotations

import argparse
import json
import os
import tempfile
from datetime import datetime
from pathlib import Path
from typing import Any

from comment_selection import MIN_COMMENT_LIKES, meaningful_comment
from manifest_changes import mark_manifest_change

VOCAB = {
    "promise_mechanism": {"how_to", "result_reveal", "problem_solution", "counterintuitive_claim", "question_gap", "numbered_value", "identity_relevance", "experience_confession", "comparison_choice", "novel_artifact", "emotional_scene", "spectacle_curiosity"},
    "visual_archetype": {"talking_head_big_text", "talking_head_subtitle_frame", "person_plus_product", "person_plus_result_overlay", "product_hero", "artifact_or_demo_hero", "interface_or_screenshot", "before_after_split", "collage_evidence", "poster_typography", "diagram_map_infographic", "documentary_scene", "cinematic_still", "lifestyle_scene", "tutorial_action"},
    "text_structure": {"no_added_text", "single_keyword", "short_proposition", "question", "number_plus_benefit", "problem_plus_solution", "main_title_plus_subtitle", "multi_label_annotation", "quote_or_confession", "series_episode_label"},
    "subject_role": {"creator_identity", "demonstrator", "scale_reference", "emotion_carrier", "object_is_hero", "space_is_hero", "screen_or_document_is_hero", "no_dominant_subject"},
    "information_density": {"low", "medium", "high"},
    "attention_device": {"oversized_type", "high_contrast_block", "bright_accent", "outlined_cutout", "arrow_or_circle", "hand_gesture", "facial_expression", "scale_surprise", "split_screen", "dense_collage", "negative_space", "cinematic_light", "unusual_object", "UI_proof", "before_after_boundary"},
    "trust_device": {"face_identity", "real_usage_scene", "finished_result", "before_after", "screen_or_data_proof", "credential_or_experience", "specific_number", "product_detail", "social_proof", "none_visible"},
    "tone": {"expert", "practical", "intimate", "aspirational", "playful", "urgent", "provocative", "calm", "cinematic", "documentary"},
}
SIGNALS = {"explicit_question", "unmet_need", "disagreement", "missing_context", "use_case", "experience_evidence", "language_resonance", "extension"}
ALIGNMENTS = {"fulfilled", "partially_fulfilled", "mismatched", "not_yet_verified"}
STRENGTHS = {"strong", "useful", "ordinary", "weak", "review"}


def now_iso() -> str:
    return datetime.now().astimezone().isoformat(timespec="seconds")


def atomic_json(path: Path, value: Any) -> None:
    fd, temporary = tempfile.mkstemp(prefix=f".{path.name}.", dir=path.parent)
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as handle:
            json.dump(value, handle, ensure_ascii=False, indent=2)
            handle.write("\n")
            handle.flush()
            os.fsync(handle.fileno())
        os.replace(temporary, path)
    finally:
        if os.path.exists(temporary):
            os.unlink(temporary)


def validate_comments(value: dict) -> list[str]:
    errors = []
    if value.get("status") not in ("passed", "review", "partial", "unavailable"):
        errors.append("comments status must be passed, review, partial, or unavailable")
    comments = value.get("comments") or []
    if len(comments) > 10:
        errors.append("comments must contain at most ten retained comments")
    ranks = [item.get("rank") for item in comments]
    if ranks != list(range(1, len(comments) + 1)):
        errors.append("comment ranks must be consecutive from 1")
    for index, item in enumerate(comments, 1):
        if not str(item.get("text") or "").strip():
            errors.append(f"comment {index} has no text")
        if not item.get("locator"):
            errors.append(f"comment {index} has no locator")
        if item.get("like_count") is None or item.get("like_count") < MIN_COMMENT_LIKES:
            errors.append(f"comment {index} must have at least {MIN_COMMENT_LIKES} verified likes")
        if not meaningful_comment(item.get("text")):
            errors.append(f"comment {index} is too fragmentary or generic to retain")
    status = value.get("topic_signal_status")
    signals = value.get("topic_signals") or []
    confirmed_no_source = value.get("status") == "unavailable" and not comments
    if status not in ("analyzed", "no_meaningful_signal") and not (confirmed_no_source and status == "no_evidence_available"):
        errors.append("topic_signal_status must be analyzed or no_meaningful_signal")
    if status == "no_meaningful_signal" and signals:
        errors.append("no_meaningful_signal requires an empty topic_signals array")
    if status == "analyzed" and not signals:
        errors.append("analyzed requires at least one topic signal")
    if not comments and value.get("status") == "passed" and status != "no_meaningful_signal":
        errors.append("passed empty comments require no_meaningful_signal")
    for index, signal in enumerate(signals, 1):
        if signal.get("type") not in SIGNALS:
            errors.append(f"topic signal {index} has invalid type")
        if not signal.get("judgment") or not signal.get("evidence_comment_ranks"):
            errors.append(f"topic signal {index} needs judgment and evidence_comment_ranks")
    return errors


def validate_cover(value: dict) -> list[str]:
    errors = []
    tags = value.get("tags") or {}
    for key, allowed in VOCAB.items():
        candidate = tags.get(key)
        if isinstance(candidate, list):
            invalid = [item for item in candidate if item not in allowed and item != "other"]
            if invalid:
                errors.append(f"cover tag {key} has invalid values: {invalid}")
        elif candidate not in allowed and candidate != "other":
            errors.append(f"cover tag {key} has invalid value: {candidate!r}")
    if value.get("promise_payoff_alignment") not in ALIGNMENTS:
        errors.append("invalid promise_payoff_alignment")
    for key in ("one_second_comprehension", "cover_promise", "image_text_division", "transfer_judgment"):
        if not str(value.get(key) or "").strip():
            errors.append(f"cover analysis needs {key}")
    if value.get("card_admission") not in ("admit", "do_not_admit"):
        errors.append("card_admission must be admit or do_not_admit")
    strengths = value.get("strengths") or {}
    for key in ("click_clarity", "visual_hierarchy", "mobile_legibility", "specificity", "trust", "distinctiveness", "promise_payoff_alignment", "fit_for_user_creation"):
        if strengths.get(key) not in STRENGTHS - {"review"}:
            errors.append(f"cover strength {key} must be finalized")
    if not value.get("evidence_notes"):
        errors.append("cover analysis needs at least one evidence note")
    return errors


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--package", required=True, type=Path)
    parser.add_argument("--commit", action="store_true")
    parser.add_argument("--scope", choices=("all", "comments", "cover"), default="all")
    args = parser.parse_args()
    package = args.package.expanduser().resolve()
    comments_path = package / "evidence" / "comments" / "top-comments.json"
    cover_path = package / "evidence" / "cover" / "analysis.json"
    comments = json.loads(comments_path.read_text(encoding="utf-8")) if args.scope in ("all", "comments") else None
    cover = json.loads(cover_path.read_text(encoding="utf-8")) if args.scope in ("all", "cover") else None
    comment_errors = validate_comments(comments) if comments is not None else []
    cover_errors = validate_cover(cover) if cover is not None else []
    errors = [f"comments: {item}" for item in comment_errors] + [f"cover: {item}" for item in cover_errors]
    if args.commit:
        manifest_path = package / "manifest.json"
        manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
        completed_gates = set()
        if comments is not None and not comment_errors:
            comment_status = comments.get("status")
            comments["status"] = comment_status if comment_status in ("review", "partial", "unavailable") else "passed"
            atomic_json(comments_path, comments)
            confirmed_no_source = manifest.get("source", {}).get("platform") == "xhs" and comments["status"] == "unavailable"
            manifest["gates"]["comments"] = "passed" if comments["status"] == "passed" or confirmed_no_source else "review"
            manifest["comment_capture_status"] = "confirmed_no_public_source" if confirmed_no_source else comments["status"]
            manifest["comment_topic_signal_status"] = comments.get("topic_signal_status")
            completed_gates.add("comments")
        if cover is not None and not cover_errors:
            cover["status"] = "passed"
            atomic_json(cover_path, cover)
            manifest["gates"]["cover"] = "passed"
            completed_gates.add("cover")
        if completed_gates:
            mark_manifest_change(manifest, "agent_analysis", now_iso())
            atomic_json(manifest_path, manifest)
            from gate_ledger import sync_gate_ledger
            sync_gate_ledger(package, manifest, write=True, completed_gates=completed_gates)
    if errors:
        print("\n".join(errors))
        return 2
    print(json.dumps({"package": str(package), "scope": args.scope, "valid": True, "committed": args.commit}, ensure_ascii=False))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

#!/usr/bin/env python3
"""Validate four-type Agent extraction plus protected user-injected asset cards."""

from __future__ import annotations

import argparse
import hashlib
import json
import os
import re
import sys
import tempfile
from pathlib import Path
from typing import Any


ASSET_TYPES = {"quote", "viewpoint", "hook", "structure", "image", "shot"}
EXTRACTED_ASSET_TYPES = {"quote", "viewpoint", "hook", "structure"}
EXTRACTED_ASSET_TYPE_ORDER = ("quote", "viewpoint", "hook", "structure")
INTERACTION_ASSET_TYPES = {"image", "shot"}
STRENGTHS = {"strong", "useful", "ordinary", "weak"}
ADMISSIONS = {"admitted", "review", "rejected"}
CONFIDENCE = {"verified", "review", "uncertain"}
ORIGINALITY = {"source_distinctive", "well_executed_familiar", "generic"}
TYPE_SCAN_STATUSES = {"admitted", "partial", "no_asset", "not_applicable", "review"}
REVIEW_VERDICTS = {"admitted", "review", "rejected"}
ASSET_ORIGINS = {"agent_generated", "user_injected"}
COMMERCIAL_STATUSES = {"confirmed_commercial", "suspected_commercial", "no_commercial_evidence", "uncertain"}
PROMOTION_STYLES = {"hard_ad", "soft_ad", "not_applicable", "unclear"}
COMMERCIAL_DISCLOSURES = {"explicit", "implicit", "none_seen", "unavailable"}
CONTENT_PRIMARY_TYPES = {
    "新闻分析", "观点评论", "知识解释", "教程方法", "案例复盘", "产品评测",
    "体验记录", "人物访谈", "故事叙事", "清单推荐", "作品解读", "音乐作品",
}
EXPRESSION_MODES = {
    "分析", "论证", "叙事", "对谈", "采访", "演示", "实测", "清单",
    "教程", "观察", "抒情", "复盘",
}
AUDIENCE_VALUES = {
    "获得判断", "理解机制", "解决问题", "行动参考", "信息获取", "情绪共鸣",
    "审美体验", "选择参考",
}
LOCATOR_KINDS = {"time_range", "paragraph_range", "image_region", "image_sequence", "shot_range", "source_metadata", "user_interaction"}
GENERIC_TITLE = re.compile(r"^(第?\s*[一二三四五六七八九十百\d]+\s*(?:章|节|部分|部|段)|继续|接下来|然后|其他|总结|正文|内容|未命名|待整理|待复核)(\s|$)")
HOOK_TYPES = {
    "反常识纠偏", "问题悬念", "结果前置", "风险预警", "冲突反差",
    "情绪共鸣", "身份代入", "场景切入", "事件切入", "新用途发现",
}
STRUCTURE_TYPES = {
    "问题解决型", "递进论证型", "对比论证型", "清单展开型", "过程复盘型",
    "转变叙事型", "循环变奏型", "对话探索型", "教程演示型", "并线汇合型",
}
QUOTE_TYPES = {"情绪金句", "概念金句", "判断金句", "行动金句"}
VIEWPOINT_DISCOURSE_PREFIX = re.compile(
    r"^(?:我(?:发现|觉得|认为|观察到|注意到)|在我看来|"
    r"(?:作者|博主|主播|嘉宾|他|她|申强)(?:表示|认为|指出|提到)|"
    r"(?:最近|上个月|前几天|之前|今天|昨天)?我在[^，。！？]{1,30}(?:发了|写了|说了)[^，。！？]{0,20}(?:表示|认为))"
)
PAYLOAD_FIELDS = {
    "quote": ("verbatim", "context", "why_it_lands"),
    "viewpoint": ("claim", "reasoning", "scope", "counterpoint"),
    "hook": ("hook_type", "surface", "mechanism", "promise", "payoff", "transfer_pattern"),
    "structure": ("structure_type", "pattern", "units", "rhythm", "transfer_pattern"),
    "image": ("subject", "composition", "visual_system", "text_image_role", "transfer_pattern"),
    "shot": ("subject_action", "framing", "camera", "duration", "transition", "purpose", "transfer_pattern"),
}
TEXT_ARTIFACT_SUFFIXES = {".md", ".txt", ".json", ".html", ".htm", ".csv"}
GENERIC_VISUAL_COPY = re.compile(
    r"画面提供.*(?:现场|对象|空间).*文字.*(?:补充|观点|阅读顺序)|"
    r"保留主体与关键操作区域|以动作完成或证据类型变化作为切点|需逐镜复核"
)
CARRIER_APPLICABILITY = {
    "video": {"quote", "viewpoint", "hook", "structure"},
    "audio": {"quote", "viewpoint", "hook", "structure"},
    "music": {"quote", "structure"},
    "image_post": {"quote", "viewpoint", "hook", "structure"},
    "article": {"quote", "viewpoint", "hook", "structure"},
    "document": {"quote", "viewpoint", "hook", "structure"},
}
CARRIER_PROFILES = {
    "video": "video_timeline",
    "audio": "spoken_longform",
    "image_post": "carousel_sequence",
    "article": "argument_article",
    "document": "argument_article",
    "music": "music_form",
}
VALIDATION_CACHE_SCHEMA = "creator-source-assets-validation-cache/v1"
VALIDATION_CACHE_RELATIVE = Path("evidence/.source-assets-validation-cache.json")


def load_json(path: Path, errors: list[str], label: str) -> dict[str, Any] | None:
    if not path.is_file() or path.stat().st_size == 0:
        errors.append(f"missing or empty {label}: {path}")
        return None
    try:
        value = json.loads(path.read_text("utf-8"))
    except (OSError, json.JSONDecodeError) as exc:
        errors.append(f"invalid {label}: {exc}")
        return None
    if not isinstance(value, dict):
        errors.append(f"{label} must be a JSON object")
        return None
    return value


def validation_rule_fingerprint() -> str:
    return hashlib.sha256(Path(__file__).read_bytes()).hexdigest()


def validation_input_signature(package: Path) -> str:
    """Cheaply invalidate a prior full validation on any package file change."""
    values = []
    cache_path = package / VALIDATION_CACHE_RELATIVE
    manifest_path = package / "manifest.json"
    try:
        manifest = json.loads(manifest_path.read_text("utf-8"))
    except (OSError, json.JSONDecodeError):
        manifest = {"invalid_manifest": True}
    if not isinstance(manifest, dict):
        manifest = {"invalid_manifest": True}
    document = manifest.get("document") if isinstance(manifest.get("document"), dict) else {}
    document_relative = str(document.get("path") or "").strip()

    def stable_manifest(value: Any) -> Any:
        if isinstance(value, dict):
            return {
                key: stable_manifest(item)
                for key, item in value.items()
                if key not in {"checked_at", "completed_at", "invalidated_at", "validated_at", "updated_at"}
            }
        if isinstance(value, list):
            return [stable_manifest(item) for item in value]
        return value

    # The source-assets validator does not consume archive/gate bookkeeping.
    # Finalization rewrites those receipts after a successful asset validation,
    # so retaining them here would immediately discard a still-valid cache.
    stable_manifest_root = {
        key: stable_manifest(item)
        for key, item in manifest.items()
        if key not in {"gates", "gate_ledger", "gate_ledger_schema"}
    }
    for path in sorted((item for item in package.rglob("*") if item.is_file()), key=lambda item: str(item)):
        if path == cache_path or path.name == ".gate-fingerprint-cache.json":
            continue
        relative = str(path.relative_to(package))
        if relative == "manifest.json":
            values.append((relative, stable_manifest_root))
            continue
        if relative == document_relative:
            try:
                content = path.read_text("utf-8")
            except (OSError, UnicodeDecodeError):
                stat = path.stat()
                values.append((relative, "unreadable_document", stat.st_size, stat.st_mtime_ns, stat.st_ctime_ns))
                continue
            content = re.sub(
                r'(?m)^(capture_status:\s*)(?:"[^"]*"|\S+)(\s*)$',
                r'\1"<archive-status>"\2',
                content,
                count=1,
            )
            content = re.sub(
                r'(?m)^(>\s+[^\n]*?\s+·\s+[^\n]*?\s+·\s+)(?:failed|review|partial|captured|complete)(\s*)$',
                r'\1<archive-status>\2',
                content,
                count=1,
            )
            values.append((relative, "stable_document_sha256", hashlib.sha256(content.encode("utf-8")).hexdigest()))
            continue
        stat = path.stat()
        values.append((
            relative, stat.st_dev, stat.st_ino, stat.st_size,
            stat.st_mtime_ns, stat.st_ctime_ns,
        ))
    payload = json.dumps(values, ensure_ascii=False, separators=(",", ":"))
    return hashlib.sha256(payload.encode("utf-8")).hexdigest()


def atomic_json(path: Path, value: dict) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    fd, temporary = tempfile.mkstemp(prefix=f".{path.name}.", suffix=".tmp", dir=path.parent)
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as handle:
            json.dump(value, handle, ensure_ascii=False, indent=2)
            handle.write("\n")
            handle.flush()
            os.fsync(handle.fileno())
        os.replace(temporary, path)
    finally:
        if os.path.exists(temporary):
            os.unlink(temporary)


def validate_package_cached(package: Path, allow_review: bool, *, use_cache: bool = True) -> tuple[list[str], bool]:
    package = package.expanduser().resolve()
    cache_path = package / VALIDATION_CACHE_RELATIVE
    input_signature = validation_input_signature(package)
    rule_fingerprint = validation_rule_fingerprint()
    if use_cache and cache_path.is_file():
        cached = load_json(cache_path, [], "validation cache") or {}
        if (
            cached.get("schema") == VALIDATION_CACHE_SCHEMA
            and cached.get("rule_fingerprint") == rule_fingerprint
            and cached.get("input_signature") == input_signature
            and cached.get("allow_review") is allow_review
            and cached.get("status") == "passed"
        ):
            return [], True
    errors = validate_package(package, allow_review)
    if not errors and use_cache:
        atomic_json(cache_path, {
            "schema": VALIDATION_CACHE_SCHEMA,
            "status": "passed",
            "allow_review": allow_review,
            "rule_fingerprint": rule_fingerprint,
            "input_signature": input_signature,
        })
    return errors, False


def nonempty(value: Any) -> bool:
    return isinstance(value, str) and bool(value.strip())


def normalized_source_text(value: str) -> str:
    return re.sub(r"[^0-9A-Za-z\u3400-\u9fff]+", "", value or "").lower()


def strip_viewpoint_discourse_prefix(value: str) -> str:
    return VIEWPOINT_DISCOURSE_PREFIX.sub("", value, count=1).lstrip(" ，,：:。；;—-\t")


def viewpoint_copies_near_complete_evidence(claim: str, excerpt: str, raw_excerpt: str = "") -> bool:
    if len(claim) < 14 or not excerpt:
        return False
    if claim == excerpt:
        return True
    if normalized_source_text(strip_viewpoint_discourse_prefix(raw_excerpt)) == claim:
        return False
    return claim in excerpt and len(claim) / len(excerpt) >= 0.72


def hook_title_repeats_surface(title: str, payload: dict[str, Any]) -> bool:
    """Reject a reader title that merely repeats the source-facing hook copy."""
    concrete = str(title or "").split("｜", 1)[-1]
    normalized_title = normalized_source_text(concrete)
    if not normalized_title:
        return False
    return any(
        normalized_title == normalized_source_text(str(payload.get(field) or ""))
        for field in ("surface", "promise")
        if str(payload.get(field) or "").strip()
    )


def validate_technique_naming(
    asset_type: str,
    origin: str,
    title: Any,
    payload: dict[str, Any],
    prefix: str,
    errors: list[str],
) -> None:
    if origin != "agent_generated" or asset_type not in {"hook", "structure"}:
        return
    field, allowed = (
        ("hook_type", HOOK_TYPES)
        if asset_type == "hook"
        else ("structure_type", STRUCTURE_TYPES)
    )
    technique_type = payload.get(field)
    if technique_type not in allowed:
        errors.append(f"{prefix}.payload.{field} is invalid")
        return
    if not str(title or "").startswith(f"{technique_type}｜"):
        errors.append(f"{prefix}.title must use '类型｜具体做法' naming")
    if asset_type == "hook" and hook_title_repeats_surface(str(title or ""), payload):
        errors.append(f"{prefix}.title must describe the hook mechanism instead of repeating surface or promise")


def safe_artifact(package: Path, relative: Any, errors: list[str], prefix: str) -> Path | None:
    if not nonempty(relative):
        errors.append(f"{prefix}.artifact must be a non-empty package-relative path")
        return None
    target = (package / str(relative)).resolve()
    try:
        target.relative_to(package.resolve())
    except ValueError:
        errors.append(f"{prefix}.artifact escapes package: {relative}")
        return None
    if not target.is_file():
        errors.append(f"{prefix}.artifact does not exist: {relative}")
    return target


def validate_locator(locator: Any, package: Path, prefix: str, errors: list[str]) -> Path | None:
    if not isinstance(locator, dict):
        errors.append(f"{prefix} must be an object")
        return None
    kind = locator.get("kind")
    if kind not in LOCATOR_KINDS:
        errors.append(f"{prefix}.kind is invalid: {kind!r}")
        return None
    artifact_path = safe_artifact(package, locator.get("artifact"), errors, prefix)
    verification = locator.get("verification")
    if verification not in CONFIDENCE:
        errors.append(f"{prefix}.verification is invalid: {verification!r}")
    if kind in {"time_range", "shot_range"}:
        start, end = locator.get("start"), locator.get("end")
        if not isinstance(start, (int, float)) or not isinstance(end, (int, float)) or start < 0 or end <= start:
            errors.append(f"{prefix} requires numeric 0 <= start < end")
        if "章节版" in str(locator.get("artifact") or "") and not nonempty(locator.get("chapter_id")):
            errors.append(f"{prefix}.chapter_id is required for chapter evidence")
    if kind == "paragraph_range":
        start, end = locator.get("paragraph_start"), locator.get("paragraph_end")
        if not isinstance(start, int) or not isinstance(end, int) or start < 1 or end < start:
            errors.append(f"{prefix} requires integer 1 <= paragraph_start <= paragraph_end")
        if "章节版" in str(locator.get("artifact") or "") and not nonempty(locator.get("chapter_id")):
            errors.append(f"{prefix}.chapter_id is required for chapter evidence")
    if kind in {"image_region", "image_sequence"}:
        if not isinstance(locator.get("image_index"), int) or locator["image_index"] < 1:
            errors.append(f"{prefix} requires a positive image_index")
        end_index = locator.get("image_end_index")
        if end_index is not None and (not isinstance(end_index, int) or end_index < locator.get("image_index", 1)):
            errors.append(f"{prefix}.image_end_index must be >= image_index")
    if kind == "image_region" and locator.get("region") is not None:
        region = locator.get("region")
        if not isinstance(region, list) or len(region) != 4 or any(not isinstance(x, (int, float)) or x < 0 or x > 1 for x in region):
            errors.append(f"{prefix}.region must be normalized [x,y,width,height]")
        elif region[0] + region[2] > 1 or region[1] + region[3] > 1:
            errors.append(f"{prefix}.region must stay within normalized image bounds")
    if kind == "shot_range" and not nonempty(locator.get("visual_description")):
        errors.append(f"{prefix}.visual_description is required for a shot range")
    if kind == "user_interaction":
        if not nonempty(locator.get("interaction_id")):
            errors.append(f"{prefix}.interaction_id is required for user interaction evidence")
        if not nonempty(locator.get("excerpt")):
            errors.append(f"{prefix}.excerpt is required for user interaction evidence")
    excerpt = locator.get("excerpt")
    if nonempty(excerpt) and artifact_path and artifact_path.is_file() and artifact_path.suffix.lower() in TEXT_ARTIFACT_SUFFIXES:
        try:
            artifact_text = artifact_path.read_text("utf-8", errors="ignore")
        except OSError as exc:
            errors.append(f"{prefix}.artifact could not be read: {exc}")
        else:
            if normalized_source_text(str(excerpt)) not in normalized_source_text(artifact_text):
                errors.append(f"{prefix}.excerpt is not present in the cited artifact")
    return artifact_path


def validate_candidate_evidence(evidence: Any, package: Path, prefix: str, errors: list[str]) -> None:
    """Validate review evidence while grandfathering already stored v1 strings.

    New semantic work writes structured locator objects. Older settled receipts
    used human-readable locator strings; rejecting those while applying one new
    asset type would make an unrelated historical type impossible to preserve.
    """
    if not isinstance(evidence, list) or not evidence:
        errors.append(f"{prefix} must contain at least one source locator")
        return
    for index, locator in enumerate(evidence):
        locator_prefix = f"{prefix}[{index}]"
        if isinstance(locator, dict):
            validate_locator(locator, package, locator_prefix, errors)
        elif not nonempty(locator):
            errors.append(f"{locator_prefix} must be a source locator object or a legacy locator string")


def validate_counterevidence_entry(counter: Any, package: Path, prefix: str, errors: list[str]) -> None:
    if not isinstance(counter, dict) or not nonempty(counter.get("explanation")):
        errors.append(f"{prefix} requires a locator object and explanation")
        return
    locator = counter.get("locator")
    if locator is None and counter.get("kind") in LOCATOR_KINDS:
        # Accept the compact flat form emitted by early structured-locator
        # prompts, but validate it with exactly the same evidence rules.
        locator = {key: value for key, value in counter.items() if key != "explanation"}
    if not isinstance(locator, dict):
        errors.append(f"{prefix} requires a locator object and explanation")
        return
    validate_locator(locator, package, f"{prefix}.locator", errors)


def locator_signature(locator: dict[str, Any]) -> tuple[Any, ...]:
    return (
        locator.get("kind"), locator.get("artifact"), locator.get("chapter_id"),
        locator.get("start"), locator.get("end"), locator.get("paragraph_start"),
        locator.get("paragraph_end"), locator.get("image_index"), locator.get("image_end_index"),
        tuple(locator.get("region") or []),
    )


def locator_position(locator: dict[str, Any]) -> tuple[float, float] | None:
    if locator.get("kind") in {"time_range", "shot_range"}:
        return float(locator.get("start", 0)), float(locator.get("end", 0))
    if locator.get("kind") == "paragraph_range":
        chapter_match = re.search(r"(\d+)$", str(locator.get("chapter_id") or ""))
        chapter_offset = float(chapter_match.group(1)) * 1_000_000 if chapter_match else 0.0
        return (
            chapter_offset + float(locator.get("paragraph_start", 0)),
            chapter_offset + float(locator.get("paragraph_end", 0)),
        )
    if locator.get("kind") in {"image_region", "image_sequence"}:
        return float(locator.get("image_index", 0)), float(locator.get("image_end_index") or locator.get("image_index", 0))
    return None


def validate_source_understanding(package: Path, document: dict[str, Any], strict: bool) -> list[str]:
    """Require a brief, evidence-grounded whole-source reading before asset judgment."""
    errors: list[str] = []
    understanding = document.get("source_understanding")
    if not isinstance(understanding, dict):
        return ["source_understanding must be an object"]
    status = understanding.get("status")
    if status not in {"review", "reviewed"}:
        errors.append("source_understanding.status must be review or reviewed")
    if strict and status != "reviewed":
        errors.append("strict validation requires reviewed source_understanding")
    if status != "reviewed":
        return errors
    for field in ("summary", "core_topic", "core_message", "content_intent", "audience_takeaway"):
        if not nonempty(understanding.get(field)):
            errors.append(f"source_understanding.{field} is required when reviewed")
    if len(str(understanding.get("summary") or "")) > 500:
        errors.append("source_understanding.summary must stay concise (<= 500 characters)")
    classification = understanding.get("classification")
    if not isinstance(classification, dict):
        errors.append("source_understanding.classification must be an object when reviewed")
    else:
        if classification.get("primary_type") not in CONTENT_PRIMARY_TYPES:
            errors.append("source_understanding.classification.primary_type is invalid")
        for field, allowed, minimum, maximum in (
            ("expression_modes", EXPRESSION_MODES, 1, 2),
            ("audience_values", AUDIENCE_VALUES, 1, 2),
        ):
            values = classification.get(field)
            if not isinstance(values, list) or not minimum <= len(values) <= maximum or len(set(map(str, values))) != len(values):
                errors.append(f"source_understanding.classification.{field} must contain {minimum}-{maximum} unique values")
            elif any(value not in allowed for value in values):
                errors.append(f"source_understanding.classification.{field} contains an invalid value")
        topic_tags = classification.get("topic_tags")
        if not isinstance(topic_tags, list) or not 1 <= len(topic_tags) <= 3 or len(set(map(str, topic_tags))) != len(topic_tags):
            errors.append("source_understanding.classification.topic_tags must contain 1-3 unique tags")
        elif any(not nonempty(value) or len(value.strip()) > 16 for value in topic_tags):
            errors.append("source_understanding.classification.topic_tags must be non-empty and <= 16 characters")
    commercial = understanding.get("commercial_assessment")
    if not isinstance(commercial, dict):
        errors.append("source_understanding.commercial_assessment must be an object when reviewed")
    else:
        commercial_status = commercial.get("status")
        promotion_style = commercial.get("promotion_style")
        if commercial_status not in COMMERCIAL_STATUSES:
            errors.append("source_understanding.commercial_assessment.status is invalid")
        if promotion_style not in PROMOTION_STYLES:
            errors.append("source_understanding.commercial_assessment.promotion_style is invalid")
        if commercial.get("disclosure") not in COMMERCIAL_DISCLOSURES:
            errors.append("source_understanding.commercial_assessment.disclosure is invalid")
        if not nonempty(commercial.get("reason")):
            errors.append("source_understanding.commercial_assessment.reason is required")
        # A promoted subject exists only when the source is confirmed or
        # suspected commercial content. Requiring a placeholder such as “无”
        # for no-commercial evidence turns a semantically correct assessment
        # into an apply failure and blocks every downstream asset type.
        if commercial_status in {"confirmed_commercial", "suspected_commercial"} and not nonempty(commercial.get("subject")):
            errors.append("source_understanding.commercial_assessment.subject is required for commercial content")
        if commercial_status in {"confirmed_commercial", "suspected_commercial"} and promotion_style not in {"hard_ad", "soft_ad"}:
            errors.append("commercial or suspected commercial content requires hard_ad or soft_ad")
        if commercial_status == "no_commercial_evidence" and promotion_style != "not_applicable":
            errors.append("no_commercial_evidence requires promotion_style: not_applicable")
        if commercial_status == "uncertain" and promotion_style != "unclear":
            errors.append("uncertain commercial status requires promotion_style: unclear")
        commercial_evidence = commercial.get("evidence")
        if not isinstance(commercial_evidence, list) or not commercial_evidence:
            errors.append("source_understanding.commercial_assessment.evidence must contain at least one locator")
        else:
            for index, locator in enumerate(commercial_evidence):
                validate_locator(locator, package, f"source_understanding.commercial_assessment.evidence[{index}]", errors)
    evidence = understanding.get("evidence")
    if not isinstance(evidence, list) or not evidence:
        errors.append("source_understanding.evidence must contain at least one locator when reviewed")
    else:
        for index, locator in enumerate(evidence):
            validate_locator(locator, package, f"source_understanding.evidence[{index}]", errors)
    return errors


def validate_review_receipt(
    package: Path,
    document: dict[str, Any],
    carrier: str,
    agent_asset_ids: set[str],
    admitted_by_type: dict[str, set[str]],
    strict: bool,
) -> list[str]:
    """Validate four independent type gates plus their aggregate receipt state."""
    errors: list[str] = []
    review = document.get("review")
    if not isinstance(review, dict):
        return ["review must be an object"]
    if review.get("schema_version") != "asset-review/v1":
        errors.append("review.schema_version must be asset-review/v1")
    status = review.get("status")
    if status not in {"review", "partial", "passed"}:
        errors.append("review.status must be review, partial, or passed")
    if strict and status != "passed":
        errors.append("strict validation requires review.status: passed")
    if status in {"partial", "passed"} and review.get("reviewed_source_revision") != document.get("source_revision"):
        errors.append("review.reviewed_source_revision must match source_revision")
    if status in {"partial", "passed"}:
        if (document.get("source_understanding") or {}).get("status") != "reviewed":
            errors.append("settled type gates require reviewed source_understanding")
        if not nonempty(review.get("source_main_line")):
            errors.append("review.source_main_line is required once type review begins")
        audience_path = review.get("audience_path")
        if not isinstance(audience_path, dict):
            errors.append("review.audience_path must be an object")
        else:
            for field in ("attract", "carry", "leave", "transfer"):
                if not nonempty(audience_path.get(field)):
                    errors.append(f"review.audience_path.{field} is required once type review begins")

    scans = review.get("type_scan")
    if not isinstance(scans, list):
        errors.append("review.type_scan must be an array")
        scans = []
    scan_by_type: dict[str, dict[str, Any]] = {}
    for index, scan in enumerate(scans):
        prefix = f"review.type_scan[{index}]"
        if not isinstance(scan, dict):
            errors.append(f"{prefix} must be an object")
            continue
        asset_type = scan.get("type")
        if asset_type not in ASSET_TYPES:
            errors.append(f"{prefix}.type is invalid")
            continue
        if asset_type in scan_by_type:
            errors.append(f"review.type_scan repeats {asset_type}")
        scan_by_type[asset_type] = scan
        scan_status = scan.get("status")
        if scan_status not in TYPE_SCAN_STATUSES:
            errors.append(f"{prefix}.status is invalid")
        if not nonempty(scan.get("reason")):
            errors.append(f"{prefix}.reason is required")
        if scan_status != "review":
            for field in ("reviewed_by", "reviewed_at"):
                if not nonempty(scan.get(field)):
                    errors.append(f"{prefix}.{field} is required when this type gate is settled")
            type_revisions = document.get("type_revisions") if isinstance(document.get("type_revisions"), dict) else {}
            if type_revisions and scan.get("reviewed_evidence_revision") != type_revisions.get(asset_type):
                errors.append(f"{prefix}.reviewed_evidence_revision must match the current type revision")
    if set(scan_by_type) != EXTRACTED_ASSET_TYPES:
        missing = sorted(EXTRACTED_ASSET_TYPES - set(scan_by_type))
        extra = sorted(set(scan_by_type) - EXTRACTED_ASSET_TYPES)
        errors.append(f"review.type_scan must cover the four extracted types; missing={missing}, extra={extra}")
    settled_types = {
        asset_type for asset_type, scan in scan_by_type.items()
        if scan.get("status") in {"admitted", "no_asset", "not_applicable"}
    }
    partial_types = {asset_type for asset_type, scan in scan_by_type.items() if scan.get("status") == "partial"}
    pending_types = {
        asset_type for asset_type, scan in scan_by_type.items()
        if scan.get("status") in {"review", "partial"}
    }
    substantively_settled = {
        asset_type for asset_type, scan in scan_by_type.items()
        if scan.get("status") in {"admitted", "partial", "no_asset"}
    }
    expected_status = "passed" if not pending_types else "partial" if substantively_settled else "review"
    if status != expected_status:
        errors.append(f"review.status must be the aggregate {expected_status} for the four independent type gates")
    applicable = CARRIER_APPLICABILITY.get(carrier, set())
    for asset_type in EXTRACTED_ASSET_TYPE_ORDER:
        scan = scan_by_type.get(asset_type) or {}
        scan_status = scan.get("status")
        admitted_ids = admitted_by_type.get(asset_type, set())
        if asset_type not in applicable and scan_status != "not_applicable":
            errors.append(f"review.type_scan {asset_type} must be not_applicable for {carrier}")
        if asset_type in applicable and scan_status == "not_applicable":
            errors.append(f"review.type_scan {asset_type} is applicable for {carrier}")
        if admitted_ids and scan_status not in {"admitted", "partial"}:
            errors.append(f"review.type_scan {asset_type} must be admitted or partial when admitted assets exist")
        if not admitted_ids and scan_status in {"admitted", "partial"}:
            errors.append(f"review.type_scan {asset_type} claims admission without an admitted asset")

    candidates = review.get("candidates")
    if not isinstance(candidates, list):
        errors.append("review.candidates must be an array")
        candidates = []
    receipt_by_id: dict[str, dict[str, Any]] = {}
    for index, candidate in enumerate(candidates):
        prefix = f"review.candidates[{index}]"
        if not isinstance(candidate, dict):
            errors.append(f"{prefix} must be an object")
            continue
        asset_id = candidate.get("asset_id")
        if not nonempty(asset_id):
            errors.append(f"{prefix}.asset_id is required")
            continue
        if asset_id in receipt_by_id:
            errors.append(f"review.candidates repeats {asset_id}")
        receipt_by_id[str(asset_id)] = candidate
        if candidate.get("type") not in EXTRACTED_ASSET_TYPES:
            errors.append(f"{prefix}.type must be one of the four source-extracted types")
        verdict = candidate.get("verdict")
        if verdict not in REVIEW_VERDICTS:
            errors.append(f"{prefix}.verdict is invalid")
        candidate_type = candidate.get("type")
        type_gate_settled = candidate_type in settled_types
        type_gate_partial = candidate_type in partial_types
        if verdict == "review" and type_gate_settled and not type_gate_partial:
            errors.append(f"{prefix}: a settled type gate cannot retain a review verdict")
        if verdict == "admitted" and not (type_gate_settled or type_gate_partial):
            errors.append(f"{prefix}: an admitted verdict requires its own type gate to be settled")
        migration = candidate.get("migration_receipt")
        if isinstance(migration, dict) and migration.get("schema") == "legacy-admission-migration/v1":
            if verdict != "admitted":
                errors.append(f"{prefix}.migration_receipt is only valid for an admitted legacy result")
            for field in ("reviewed_by", "reviewed_at", "reason"):
                if not nonempty(migration.get(field)):
                    errors.append(f"{prefix}.migration_receipt.{field} is required")
            validate_candidate_evidence(candidate.get("evidence"), package, f"{prefix}.evidence", errors)
            continue
        for field in ("main_line_relation", "audience_effect", "mechanism", "boundary", "anti_copy"):
            if not nonempty(candidate.get(field)):
                errors.append(f"{prefix}.{field} is required")
        validate_candidate_evidence(candidate.get("evidence"), package, f"{prefix}.evidence", errors)
        if verdict != "admitted":
            for field in ("failure_stage", "failure_code", "repair_request", "next_stage"):
                if not nonempty(candidate.get(field)):
                    errors.append(f"{prefix}.{field} is required for non-admitted candidates")

    extracted_asset_ids = {
        asset_id for asset_type, ids in admitted_by_type.items()
        if asset_type in EXTRACTED_ASSET_TYPES for asset_id in ids
    }
    admitted_ids = extracted_asset_ids
    receipt_admitted = {asset_id for asset_id, item in receipt_by_id.items() if item.get("verdict") == "admitted"}
    if receipt_admitted != admitted_ids:
        errors.append("review admitted candidate IDs must exactly match admitted asset IDs")
    missing_receipts = sorted(agent_asset_ids - set(receipt_by_id))
    if missing_receipts:
        errors.append(f"review receipt is missing asset records: {missing_receipts}")
    return errors


def validate_asset_bundle(
    package: Path,
    document: dict[str, Any],
    allow_review: bool,
    preserved_user_asset_ids: set[str] | None = None,
    validate_understanding: bool = True,
) -> list[str]:
    """Validate a bundle, optionally grandfathering unchanged protected user cards.

    ``preserved_user_asset_ids`` is only for the Agent incremental-apply path. It
    lets that transaction validate its own lane without rewriting or rejecting a
    user card that predates the current schema. Identity, type, origin, and ID
    collision checks still run before the legacy card is skipped.

    ``validate_understanding=False`` is reserved for a type-scoped incremental
    commit. Source understanding has its own revision and review transaction;
    an older understanding locator must not reject an independently reviewed
    quote, viewpoint, hook, or structure update.
    """
    errors: list[str] = []
    preserved_user_asset_ids = preserved_user_asset_ids or set()
    if document.get("schema_version") != "creator-assets/v1":
        errors.append("asset schema_version must be creator-assets/v1")
    for field in ("source_id", "source_revision", "generated_at"):
        if not nonempty(document.get(field)):
            errors.append(f"asset bundle missing non-empty {field}")
    type_revisions = document.get("type_revisions")
    if type_revisions is not None:
        if not isinstance(type_revisions, dict) or set(type_revisions) != EXTRACTED_ASSET_TYPES or any(not nonempty(value) for value in type_revisions.values()):
            errors.append("asset bundle type_revisions must cover quote, viewpoint, hook, and structure")
    carrier = document.get("carrier")
    if carrier not in {"video", "audio", "image_post", "article", "document", "music"}:
        errors.append(f"asset bundle carrier is invalid: {carrier!r}")
    elif document.get("analysis_profile") != CARRIER_PROFILES[carrier]:
        errors.append(
            f"asset bundle analysis_profile must be {CARRIER_PROFILES[carrier]!r} for {carrier}"
        )
    if document.get("analysis_status") not in {"review", "passed", "partial"}:
        errors.append("asset bundle analysis_status is invalid")
    if not allow_review and document.get("analysis_status") != "passed":
        errors.append("strict validation requires analysis_status: passed")
    strict = not allow_review or document.get("analysis_status") == "passed"
    manifest = load_json(package / "manifest.json", errors, "source manifest") or {}
    dialect_limited = (manifest.get("transcript") or {}).get("lexical_review_status") == "limited_dialect"
    if validate_understanding:
        errors.extend(validate_source_understanding(package, document, strict))
    assets = document.get("assets")
    if not isinstance(assets, list):
        return errors + ["assets must be an array"]
    ids: set[str] = set()
    agent_asset_ids: set[str] = set()
    admitted_by_type: dict[str, set[str]] = {asset_type: set() for asset_type in ASSET_TYPES}
    admissions_in_bundle: list[str] = []
    for index, asset in enumerate(assets):
        prefix = f"assets[{index}]"
        if not isinstance(asset, dict):
            errors.append(f"{prefix} must be an object")
            continue
        asset_id, asset_type = asset.get("id"), asset.get("type")
        if not nonempty(asset_id):
            errors.append(f"{prefix}.id is required")
        elif asset_id in ids:
            errors.append(f"duplicate asset id: {asset_id}")
        else:
            ids.add(asset_id)
        if asset_type not in ASSET_TYPES:
            errors.append(f"{prefix}.type is invalid: {asset_type!r}")
            continue
        origin = asset.get("origin")
        if origin not in ASSET_ORIGINS:
            errors.append(f"{prefix}.origin must be agent_generated or user_injected")
        if origin == "user_injected" and str(asset_id or "") in preserved_user_asset_ids:
            injection = asset.get("injection") if isinstance(asset.get("injection"), dict) else {}
            if (injection.get("source_extracted_lineage") is True
                    and asset_type in EXTRACTED_ASSET_TYPES
                    and (asset.get("quality") or {}).get("admission") == "admitted"
                    and nonempty(asset_id)):
                # Editing a source-extracted card transfers ownership to the
                # user without erasing its already reviewed source lineage.
                # It still satisfies the original type receipt, while ordinary
                # user-created thoughts never acquire this marker.
                agent_asset_ids.add(str(asset_id))
                admitted_by_type[asset_type].add(str(asset_id))
            continue
        if origin == "agent_generated" and nonempty(asset_id):
            is_stale = asset.get("stale") is True or (
                isinstance(type_revisions, dict) and asset.get("evidence_revision") != type_revisions.get(asset_type)
            )
            if not is_stale:
                agent_asset_ids.add(str(asset_id))
            if strict and asset.get("stale") is True:
                errors.append(f"{prefix}: stale Agent asset cannot be finalized")
            if strict and isinstance(type_revisions, dict) and asset.get("evidence_revision") != type_revisions.get(asset_type):
                errors.append(f"{prefix}.evidence_revision must match the current type revision")
        if asset_type in INTERACTION_ASSET_TYPES and origin != "user_injected":
            errors.append(f"{prefix}: {asset_type} assets can only be user_injected")
        if origin == "user_injected":
            injection = asset.get("injection")
            if not isinstance(injection, dict):
                errors.append(f"{prefix}.injection must be an object for user_injected assets")
            else:
                if injection.get("status") != "standardized":
                    errors.append(f"{prefix}.injection.status must be standardized")
                if injection.get("created_by") != "user":
                    errors.append(f"{prefix}.injection.created_by must be user")
                if injection.get("protected_from_agent") is not True:
                    errors.append(f"{prefix}.injection.protected_from_agent must be true")
                for field in ("interaction_id", "injected_at"):
                    if not nonempty(injection.get(field)):
                        errors.append(f"{prefix}.injection.{field} is required")
        if not nonempty(asset.get("title")):
            errors.append(f"{prefix}.title is required")
        payload = asset.get("payload")
        if not isinstance(payload, dict):
            errors.append(f"{prefix}.payload must be an object")
            payload = {}
        for field in PAYLOAD_FIELDS[asset_type]:
            value = payload.get(field)
            if field == "units":
                if not isinstance(value, list) or len(value) < 2:
                    errors.append(f"{prefix}.payload.units must contain at least two ordered units")
                else:
                    for unit_index, unit in enumerate(value):
                        if not isinstance(unit, dict) or not nonempty(unit.get("label")) or not nonempty(unit.get("function")) or not isinstance(unit.get("locator"), dict):
                            errors.append(f"{prefix}.payload.units[{unit_index}] requires label, function, and locator")
                        else:
                            validate_locator(unit["locator"], package, f"{prefix}.payload.units[{unit_index}].locator", errors)
            elif not nonempty(value):
                errors.append(f"{prefix}.payload.{field} is required")
        if asset_type == "quote" and payload.get("quote_type") is not None and payload.get("quote_type") not in QUOTE_TYPES:
            errors.append(f"{prefix}.payload.quote_type is invalid")
        validate_technique_naming(asset_type, str(origin or ""), asset.get("title"), payload, prefix, errors)
        evidence = asset.get("evidence")
        if not isinstance(evidence, list) or not evidence:
            errors.append(f"{prefix}.evidence must contain at least one locator")
            evidence = []
        for evidence_index, locator in enumerate(evidence):
            validate_locator(locator, package, f"{prefix}.evidence[{evidence_index}]", errors)
        if origin == "agent_generated" and any(isinstance(locator, dict) and locator.get("kind") == "user_interaction" for locator in evidence):
            errors.append(f"{prefix}: agent_generated assets cannot claim user_interaction evidence")
        quality = asset.get("quality")
        if not isinstance(quality, dict):
            errors.append(f"{prefix}.quality must be an object")
            quality = {}
        if quality.get("strength") not in STRENGTHS:
            errors.append(f"{prefix}.quality.strength is invalid")
        if quality.get("admission") not in ADMISSIONS:
            errors.append(f"{prefix}.quality.admission is invalid")
        if quality.get("evidence_confidence") not in CONFIDENCE:
            errors.append(f"{prefix}.quality.evidence_confidence is invalid")
        if quality.get("originality") not in ORIGINALITY:
            errors.append(f"{prefix}.quality.originality is invalid")
        reuse = asset.get("reuse")
        if not isinstance(reuse, dict):
            errors.append(f"{prefix}.reuse must be an object")
            reuse = {}
        for field in ("conditions", "boundary", "anti_copy"):
            if not nonempty(reuse.get(field)):
                errors.append(f"{prefix}.reuse.{field} is required")
        if not isinstance(asset.get("uncertainties"), list):
            errors.append(f"{prefix}.uncertainties must be an array")
        else:
            for uncertainty_index, uncertainty in enumerate(asset.get("uncertainties") or []):
                if not isinstance(uncertainty, dict) or any(not nonempty(uncertainty.get(field)) for field in ("code", "detail", "locator", "impact")):
                    errors.append(f"{prefix}.uncertainties[{uncertainty_index}] requires code, detail, locator, and impact")
        if not isinstance(asset.get("counterevidence"), list):
            errors.append(f"{prefix}.counterevidence must be an array")
        else:
            for counter_index, counter in enumerate(asset.get("counterevidence") or []):
                validate_counterevidence_entry(
                    counter,
                    package,
                    f"{prefix}.counterevidence[{counter_index}]",
                    errors,
                )
        if not nonempty(asset.get("created_at")):
            errors.append(f"{prefix}.created_at is required")
        admission = quality.get("admission")
        admissions_in_bundle.append(str(admission or ""))
        if admission == "admitted":
            if origin == "agent_generated" and nonempty(asset_id) and str(asset_id) in agent_asset_ids:
                admitted_by_type[asset_type].add(str(asset_id))
            if quality.get("strength") not in {"strong", "useful"}:
                errors.append(f"{prefix}: admitted asset must be strong or useful")
            if quality.get("evidence_confidence") != "verified":
                errors.append(f"{prefix}: admitted asset requires verified evidence confidence")
            if quality.get("originality") == "generic":
                errors.append(f"{prefix}: generic material cannot be admitted")
            if any(isinstance(item, dict) and item.get("verification") != "verified" for item in evidence):
                errors.append(f"{prefix}: admitted asset has unverified locator")
            if asset.get("uncertainties"):
                errors.append(f"{prefix}: admitted asset cannot retain material uncertainties")
            if evidence and all(isinstance(item, dict) and item.get("kind") == "source_metadata" for item in evidence):
                errors.append(f"{prefix}: source metadata cannot be the sole evidence for an admitted asset")
            if dialect_limited:
                unsafe_transcript_locators = [
                    item for item in evidence
                    if isinstance(item, dict)
                    and str(item.get("artifact") or "").startswith("evidence/transcript/")
                    and item.get("verification_method") not in {"bounded_audio_review", "synchronized_caption"}
                ]
                if unsafe_transcript_locators:
                    errors.append(
                        f"{prefix}: limited-dialect transcript evidence requires "
                        "verification_method bounded_audio_review or synchronized_caption before admission"
                    )
        if admission == "rejected" and not nonempty(asset.get("rejection_reason")):
            errors.append(f"{prefix}: rejected asset requires rejection_reason")
        if asset_type == "quote" and evidence:
            exact = normalized_source_text(str(payload.get("verbatim") or ""))
            excerpts = normalized_source_text(" ".join(str(item.get("excerpt") or "") for item in evidence if isinstance(item, dict)))
            if exact and exact not in excerpts:
                errors.append(f"{prefix}: quote verbatim is not present in evidence excerpt")
        if asset_type == "viewpoint" and admission == "admitted" and origin == "agent_generated":
            raw_claim = str(payload.get("claim") or "").strip()
            claim = normalized_source_text(raw_claim)
            evidence_excerpts = [
                (normalized_source_text(str(item.get("excerpt") or "")), str(item.get("excerpt") or ""))
                for item in evidence if isinstance(item, dict)
            ]
            if claim and any(viewpoint_copies_near_complete_evidence(claim, excerpt, raw_excerpt) for excerpt, raw_excerpt in evidence_excerpts):
                errors.append(f"{prefix}: admitted Agent viewpoint claim must be a synthesized paraphrase, not copied source evidence")
            if VIEWPOINT_DISCOURSE_PREFIX.match(raw_claim):
                errors.append(f"{prefix}: admitted Agent viewpoint claim must remove non-load-bearing discourse prefixes")
        if asset_type == "hook" and admission != "rejected":
            roles = [item.get("role") for item in evidence if isinstance(item, dict)]
            if roles.count("promise") != 1 or roles.count("payoff") != 1:
                errors.append(f"{prefix}: hook requires exactly one promise and one payoff evidence role")
            by_role = {item.get("role"): item for item in evidence if isinstance(item, dict)}
            promise, payoff = by_role.get("promise"), by_role.get("payoff")
            if not promise or not payoff or not nonempty(promise.get("excerpt")) or not nonempty(payoff.get("excerpt")):
                errors.append(f"{prefix}: hook promise/payoff require source excerpts")
            elif normalized_source_text(str(promise.get("excerpt"))) == normalized_source_text(str(payoff.get("excerpt"))):
                errors.append(f"{prefix}: hook payoff evidence must differ from its promise")
            if promise and payoff and promise.get("artifact") == payoff.get("artifact"):
                left, right = locator_position(promise), locator_position(payoff)
                if left and right and right[0] <= left[0]:
                    errors.append(f"{prefix}: hook payoff must occur after its promise")
        if asset_type == "structure" and admission != "rejected":
            units = payload.get("units") if isinstance(payload.get("units"), list) else []
            locators = [unit.get("locator") for unit in units if isinstance(unit, dict) and isinstance(unit.get("locator"), dict)]
            signatures = [locator_signature(item) for item in locators]
            if len(signatures) != len(set(signatures)):
                errors.append(f"{prefix}: structure units must cite distinct source spans")
            positions = [locator_position(item) for item in locators]
            if positions and all(item is not None for item in positions):
                if any(current[0] <= previous[0] for previous, current in zip(positions, positions[1:])):
                    errors.append(f"{prefix}: structure unit locators must follow source order")
            if any(not nonempty(item.get("excerpt")) for item in locators):
                errors.append(f"{prefix}: every structure unit requires a source excerpt")
        if asset_type == "shot" and not any(isinstance(item, dict) and item.get("kind") == "shot_range" for item in evidence):
            errors.append(f"{prefix}: shot asset requires shot_range evidence")
        if asset_type == "image" and not any(isinstance(item, dict) and item.get("kind") in {"image_region", "image_sequence"} for item in evidence):
            errors.append(f"{prefix}: image asset requires image evidence")
        if carrier in {"audio", "music"} and asset_type == "shot" and admission == "admitted":
            errors.append(f"{prefix}: {carrier} cannot admit a shot without verified motion evidence")
        if admission == "admitted" and asset_type == "image":
            if GENERIC_VISUAL_COPY.search(str(payload.get("text_image_role") or "")):
                errors.append(f"{prefix}: admitted image uses a generic text-image-role template")
            if any(Path(str(item.get("artifact") or "")).suffix.lower() not in {".jpg", ".jpeg", ".png", ".webp", ".gif", ".heic", ".avif"} for item in evidence if isinstance(item, dict)):
                errors.append(f"{prefix}: admitted image evidence must cite archived image files")
        if admission == "admitted" and asset_type == "shot":
            production = " ".join(str(payload.get(field) or "") for field in ("camera", "duration", "transition"))
            if GENERIC_VISUAL_COPY.search(production):
                errors.append(f"{prefix}: admitted shot uses placeholder production language")
            if any(Path(str(item.get("artifact") or "")).suffix.lower() in {".md", ".txt"} for item in evidence if isinstance(item, dict)):
                errors.append(f"{prefix}: admitted shot must cite media, frames, or visual-analysis evidence")
    admitted_quote_texts = {
        normalized_source_text(str((asset.get("payload") or {}).get("verbatim") or "")): str(asset.get("id") or "")
        for asset in assets
        if isinstance(asset, dict)
        and asset.get("origin") == "agent_generated"
        and asset.get("type") == "quote"
        and (asset.get("quality") or {}).get("admission") == "admitted"
    }
    for index, asset in enumerate(assets):
        if not isinstance(asset, dict) or asset.get("origin") != "agent_generated" or asset.get("type") != "viewpoint":
            continue
        if (asset.get("quality") or {}).get("admission") != "admitted":
            continue
        claim = normalized_source_text(str((asset.get("payload") or {}).get("claim") or ""))
        if claim and claim in admitted_quote_texts:
            errors.append(
                f"assets[{index}]: admitted viewpoint duplicates admitted quote {admitted_quote_texts[claim]}; "
                "keep the quote or provide a distinct bounded viewpoint with its own reasoning path"
            )
    if document.get("analysis_status") == "passed" and "review" in admissions_in_bundle:
        errors.append("analysis_status passed cannot retain review assets; resolve, reject, or mark the bundle review")
    generation = document.get("generation_receipt") if isinstance(document.get("generation_receipt"), dict) else {}
    receipts = document.get("review_receipts") if isinstance(document.get("review_receipts"), dict) else {}
    generated_by = str(generation.get("generated_by") or "").strip()
    for asset_type, receipt in receipts.items():
        if asset_type not in EXTRACTED_ASSET_TYPES or not isinstance(receipt, dict):
            errors.append(f"review_receipts.{asset_type} is invalid")
            continue
        reviewed_by = str(receipt.get("reviewed_by") or "").strip()
        if generated_by and reviewed_by and generated_by == reviewed_by:
            errors.append(f"review_receipts.{asset_type} must be independent from candidate generation")
    errors.extend(validate_review_receipt(package, document, str(carrier or ""), agent_asset_ids, admitted_by_type, strict))
    return errors


def validate_chapters(package: Path, path: Path, carrier: str, allow_review: bool) -> list[str]:
    errors: list[str] = []
    doc = load_json(path, errors, f"{carrier} chapter document")
    if doc is None:
        return errors
    expected_schemas = (
        {"chaptered-transcript/v2", "chaptered-transcript/v3"}
        if carrier in {"video", "audio", "music"}
        else {"chaptered-article/v1"}
    )
    if doc.get("schema_version") not in expected_schemas:
        errors.append(
            f"{carrier} chapter schema_version must be one of {', '.join(sorted(expected_schemas))}"
        )
    review_statuses = {"provisional", "partial", "source_sectioned", "unsectioned"}
    if doc.get("status") not in review_statuses | {"reviewed"}:
        errors.append(f"{carrier} chapter status is invalid")
    article_source_structure_settled = carrier == "article" and doc.get("status") in {"source_sectioned", "unsectioned"}
    if not allow_review and doc.get("status") != "reviewed" and not article_source_structure_settled:
        errors.append(f"strict validation requires reviewed {carrier} chapters")
    chapters = doc.get("chapters")
    if not isinstance(chapters, list) or not chapters:
        return errors + [f"{carrier} chapters must be a non-empty array"]
    ids, titles, all_source_ids = set(), set(), []
    previous_end: float | int | None = None
    for index, chapter in enumerate(chapters):
        prefix = f"chapters[{index}]"
        if not isinstance(chapter, dict):
            errors.append(f"{prefix} must be an object")
            continue
        chapter_id, title = chapter.get("id"), str(chapter.get("title") or "").strip()
        if not nonempty(chapter_id) or chapter_id in ids:
            errors.append(f"{prefix}.id is missing or duplicate")
        ids.add(chapter_id)
        allowed_article_navigation = carrier == "article" and chapter.get("kind") in {"preface", "unsectioned"} and title in {"导读", "正文"}
        if not title or title in titles or (GENERIC_TITLE.search(title) and not allowed_article_navigation):
            errors.append(f"{prefix}.title is empty, duplicate, or generic: {title!r}")
        titles.add(title)
        paragraphs = chapter.get("paragraphs")
        if not isinstance(paragraphs, list) or not paragraphs or any(not nonempty(item) for item in paragraphs):
            errors.append(f"{prefix}.paragraphs must contain continuous non-empty source prose")
        title_status = chapter.get("title_status")
        fidelity = chapter.get("text_fidelity")
        if title_status not in {"review", "reviewed", "source_structure"}:
            errors.append(f"{prefix}.title_status is invalid")
        if fidelity not in {"verbatim_raw", "verbatim_corrected", "review"}:
            errors.append(f"{prefix}.text_fidelity is invalid")
        settled_title = title_status == "reviewed" or (
            carrier == "article" and allowed_article_navigation and title_status == "source_structure"
        )
        if not allow_review and (not settled_title or fidelity not in {"verbatim_raw", "verbatim_corrected"}):
            errors.append(f"{prefix} is not title/text reviewed")
        if carrier in {"video", "audio", "music"}:
            start, end = chapter.get("start"), chapter.get("end")
            source_ids = chapter.get("source_segment_ids")
            if not isinstance(start, (int, float)) or not isinstance(end, (int, float)) or start < 0 or end <= start:
                errors.append(f"{prefix} requires 0 <= start < end")
            elif previous_end is not None and start < previous_end:
                errors.append(f"{prefix} overlaps the prior chapter")
            previous_end = end if isinstance(end, (int, float)) else previous_end
            if not isinstance(source_ids, list) or not source_ids:
                errors.append(f"{prefix}.source_segment_ids must be non-empty")
                source_ids = []
            if chapter.get("source_segment_count") != len(source_ids):
                errors.append(f"{prefix}.source_segment_count does not match IDs")
        else:
            start, end = chapter.get("paragraph_start"), chapter.get("paragraph_end")
            source_ids = chapter.get("source_paragraph_ids")
            if not isinstance(start, int) or not isinstance(end, int) or start < 1 or end < start:
                errors.append(f"{prefix} requires 1 <= paragraph_start <= paragraph_end")
            elif previous_end is not None and start != previous_end + 1:
                errors.append(f"{prefix} is not contiguous with the prior chapter")
            previous_end = end if isinstance(end, int) else previous_end
            if not isinstance(source_ids, list) or not source_ids:
                errors.append(f"{prefix}.source_paragraph_ids must be non-empty")
                source_ids = []
            if isinstance(start, int) and isinstance(end, int) and len(source_ids) != end - start + 1:
                errors.append(f"{prefix}.source_paragraph_ids do not cover the declared range")
        if len(set(map(str, source_ids))) != len(source_ids):
            errors.append(f"{prefix} repeats source IDs")
        all_source_ids.extend(map(str, source_ids))
    if len(set(all_source_ids)) != len(all_source_ids):
        errors.append("source IDs appear in more than one chapter")

    if carrier in {"video", "audio", "music"}:
        raw_path = package / "evidence/transcript/transcript-raw.json"
        if raw_path.is_file():
            raw = load_json(raw_path, errors, "raw transcript") or {}
            qa_path = package / "evidence/transcript/transcript-qa.json"
            qa = load_json(qa_path, errors, "transcript QA") if qa_path.is_file() else {}
            excluded_indices = {
                int(item["segment"])
                for field in ("reader_exclusions", "ignored_ranges")
                for item in (qa or {}).get(field) or []
                if isinst

import importlib.util
import sys
import unittest
from pathlib import Path


SCRIPT_DIR = Path(__file__).resolve().parents[1] / "scripts"
sys.path.insert(0, str(SCRIPT_DIR))


def load_module(name: str, path: Path):
    spec = importlib.util.spec_from_file_location(name, path)
    module = importlib.util.module_from_spec(spec)
    assert spec.loader is not None
    spec.loader.exec_module(module)
    return module


transcribe = load_module("transcribe_package", SCRIPT_DIR / "transcribe_package.py")
publisher = load_module("capture_publisher_transcript", SCRIPT_DIR / "capture_publisher_transcript.py")


class BoundedTranscriptReviewTests(unittest.TestCase):
    def test_retry_range_accepts_clock_or_seconds(self):
        self.assertEqual(transcribe.parse_retry_range("00:40-01:00", 120), (40.0, 60.0))
        self.assertEqual(transcribe.parse_retry_range("40-60", 120), (40.0, 60.0))

    def test_retry_range_rejects_unbounded_or_empty_spans(self):
        with self.assertRaises(ValueError):
            transcribe.parse_retry_range("10-10", 120)
        with self.assertRaises(ValueError):
            transcribe.parse_retry_range("0-181", 500)

    def test_publisher_decoder_recovers_timestamp_speaker_and_content(self):
        html = (
            r'children\":\"00:00:42\" children\":\"重轻\" '
            r'content\":\"这世界上不缺播客节目，你没有想说的，你可以不说。\"'
        )
        self.assertEqual(
            publisher.decode_next_transcript(html),
            [{"timestamp": "00:00:42", "speaker": "重轻", "content": "这世界上不缺播客节目，你没有想说的，你可以不说。"}],
        )


if __name__ == "__main__":
    unittest.main()

#!/usr/bin/env python3

import importlib.util
import tempfile
import unittest
from pathlib import Path
from unittest import mock


SCRIPT = Path(__file__).parents[1] / "scripts" / "capture_source.py"
SPEC = importlib.util.spec_from_file_location("capture_source", SCRIPT)
MODULE = importlib.util.module_from_spec(SPEC)
assert SPEC.loader
SPEC.loader.exec_module(MODULE)


class ImagePostOCRStrategyTests(unittest.TestCase):
    def test_xhs_missing_title_uses_only_first_description_line(self):
        description = "脑子是浆糊吗？\n用着顶级AI却不知道做什么？\n1.做浏览器插件"
        self.assertEqual(MODULE.xhs_display_title("", description, "fallback"), "脑子是浆糊吗？")

    def test_xhs_explicit_title_is_preserved_as_one_line(self):
        self.assertEqual(MODULE.xhs_display_title("  正式\n标题  ", "正文", "fallback"), "正式 标题")

    def test_informative_source_text_still_requires_all_page_ocr(self):
        source = {
            "title": "用公交重新认识一座城",
            "body": "这篇图文介绍如何选择一条贯穿城市的公交线路。先观察沿途街区变化，再在感兴趣的站点下车步行，最后记录不同区域的生活节奏与空间关系。",
        }
        decision = MODULE.image_post_ocr_decision(source)
        self.assertEqual(decision["status"], "all_pages_ocr")
        self.assertEqual(decision["policy"], "mandatory_per_image_v1")

    def test_sparse_source_text_requires_all_page_ocr(self):
        decision = MODULE.image_post_ocr_decision({"title": "看图", "body": "路线在图片里"})
        self.assertEqual(decision["status"], "all_pages_ocr")

    def test_named_pages_are_priority_hints_not_exclusive_targets(self):
        decision = MODULE.image_post_ocr_decision(
            {"title": "看图", "body": "路线在图片里"},
            target_pages=[2, 4, 4],
        )
        self.assertEqual(decision["status"], "all_pages_ocr")
        self.assertEqual(decision["priority_pages"], [2, 4])

    def test_image_post_skip_request_cannot_disable_required_ocr(self):
        source = {"title": "标题", "body": "足够长的正文" * 30}
        forced = MODULE.image_post_ocr_decision(source, force_ocr=True)
        skipped = MODULE.image_post_ocr_decision(source, skip_ocr=True)
        self.assertEqual(forced["status"], "all_pages_ocr")
        self.assertTrue(forced["force_requested"])
        self.assertEqual(skipped["status"], "all_pages_ocr")
        self.assertTrue(skipped["skip_request_ignored"])

    def test_ocr_organization_classifies_substantial_sparse_and_empty_pages(self):
        substantial, sparse, empty = MODULE.organize_ocr_records([
            {"text": "这是连续的大段文字，用来说明图片中真正表达的完整观点，并为后续理解提供证据。"},
            {"text": "路线提示"},
            {"text": ""},
        ])
        self.assertEqual(substantial["text_class"], "substantial_text")
        self.assertEqual(sparse["text_class"], "sparse_text")
        self.assertEqual(empty["text_class"], "no_text")


class XiaohongshuCommentStrategyTests(unittest.TestCase):
    def test_missing_public_comments_are_a_terminal_unavailable_result(self):
        result = MODULE.normalize_xhs_comments({"comments": {"list": [], "hasMore": True}})
        self.assertEqual(result["status"], "unavailable")
        self.assertEqual(result["ranking_scope"], "not_in_public_payload")
        self.assertEqual(result["topic_signal_status"], "no_meaningful_signal")
        self.assertEqual(result["comments"], [])
        self.assertIn("no authenticated retry is required", result["reason"])


class StreamingDownloadTests(unittest.TestCase):
    def test_download_streams_to_an_atomic_file_without_reading_response_content(self):
        response = mock.Mock()
        response.headers = {"content-type": "audio/mp4", "content-length": "1024"}
        response.iter_content.return_value = [b"a" * 600, b"b" * 424]
        with tempfile.TemporaryDirectory() as temp, mock.patch.object(MODULE, "request", return_value=response) as request:
            path, returned = MODULE.download(mock.Mock(), "https://media.example/test", Path(temp) / "source", referer="https://example.com")
            self.assertEqual(path.stat().st_size, 1024)
        self.assertEqual(returned, response)
        request.assert_called_once_with(mock.ANY, "https://media.example/test", referer="https://example.com", stream=True)
        response.close.assert_called_once()

    def test_download_removes_partial_file_when_stream_exceeds_limit(self):
        response = mock.Mock()
        response.headers = {"content-type": "video/mp4"}
        response.iter_content.return_value = [b"a" * 600, b"b" * 600]
        with tempfile.TemporaryDirectory() as temp, mock.patch.object(MODULE, "request", return_value=response):
            root = Path(temp)
            with self.assertRaisesRegex(ValueError, "safety limit"):
                MODULE.download(mock.Mock(), "https://media.example/test", root / "source", referer="https://example.com", max_bytes=1000)
            self.assertEqual(list(root.iterdir()), [])


if __name__ == "__main__":
    unittest.main()

#!/usr/bin/env python3

import unittest

from merge_comments import reject_login_comment_input


class CommentLoginGuardTests(unittest.TestCase):
    def test_rejects_account_bearing_methods(self):
        for method in (
            "authorized_browser_dom",
            "authenticated_api",
            "logged-in browser",
            "browser_cookie_web_api",
            "browser_profile_dom",
        ):
            with self.subTest(method=method):
                with self.assertRaises(ValueError):
                    reject_login_comment_input(method)

    def test_accepts_no_login_public_methods(self):
        for method in (
            "bilibili_anonymous_guest_hot_comments_api",
            "xiaoyuzhou_next_data",
            "douyin_public_page_no_login",
            "xhs_initial_state",
        ):
            with self.subTest(method=method):
                reject_login_comment_input(method)


if __name__ == "__main__":
    unittest.main()

#!/usr/bin/env python3

import unittest

from comment_selection import comment_enrichment_decision, deferred_comment_result, meaningful_comment, select_comments


class CommentSelectionTests(unittest.TestCase):
    def test_retains_at_most_ten_by_likes(self):
        items = [
            {"text": f"这是第{i}条包含完整判断和具体信息的评论", "like_count": i * 10}
            for i in range(1, 13)
        ]
        selected, _ = select_comments(items)
        self.assertEqual([item["like_count"] for item in selected], [120, 110, 100, 90, 80, 70, 60, 50, 40, 30])
        self.assertEqual([item["rank"] for item in selected], list(range(1, 11)))

    def test_rejects_low_likes_and_fragmentary_reactions(self):
        selected, discarded = select_comments([
            {"text": "哈哈哈哈", "like_count": 900},
            {"text": "这条评论信息完整，但点赞不足，所以不应保留。", "like_count": 9},
            {"text": "为什么这个方法在长视频里反而更有效？", "like_count": 12},
        ])
        self.assertEqual([item["text"] for item in selected], ["为什么这个方法在长视频里反而更有效？"])
        self.assertEqual(discarded["low_information"], 1)
        self.assertEqual(discarded["below_min_likes"], 1)

    def test_short_but_complete_question_is_meaningful(self):
        self.assertTrue(meaningful_comment("这个字体叫什么？"))
        self.assertFalse(meaningful_comment("好棒！"))

    def test_adaptive_policy_skips_cheap_modes_and_low_volume_sources(self):
        self.assertFalse(comment_enrichment_decision("bilibili", "capture", {"comments": 900})["eligible"])
        low = comment_enrichment_decision("bilibili", "full", {"comments": 19})
        self.assertFalse(low["eligible"])
        self.assertEqual(deferred_comment_result(low)["comment_capture_status"], "skipped_by_policy")

    def test_adaptive_policy_accepts_bounded_high_signal_sources(self):
        self.assertTrue(comment_enrichment_decision("bilibili", "analyze", {"comments": 20})["eligible"])
        self.assertTrue(comment_enrichment_decision(
            "douyin", "full", {"comments": None}, visible_sample_available=True,
        )["eligible"])


if __name__ == "__main__":
    unittest.main()

import importlib.util
import json
import sys
import tempfile
import unittest
from pathlib import Path
from unittest import mock


SCRIPT = Path(__file__).parents[1] / "scripts/gate_ledger.py"
spec = importlib.util.spec_from_file_location("gate_ledger", SCRIPT)
gate_ledger = importlib.util.module_from_spec(spec)
assert spec.loader
sys.modules[spec.name] = gate_ledger
spec.loader.exec_module(gate_ledger)


DOCUMENT_NAME = "视频-小红书-测试-测试作者-20260825.md"


class GateLedgerTests(unittest.TestCase):
    def make_video_package(self, root: Path, *, raw: bool = True) -> tuple[Path, dict]:
        package = root / "S-xhs-example"
        media = package / "evidence/media/source.mp4"
        media.parent.mkdir(parents=True)
        media.write_bytes(b"video")
        if raw:
            transcript = package / "evidence/transcript/transcript-raw.json"
            transcript.parent.mkdir(parents=True)
            transcript.write_text('{"segments": []}', "utf-8")
        (package / "evidence/comments").mkdir(parents=True)
        (package / "evidence/comments/top-comments.json").write_text("{}", "utf-8")
        (package / "evidence/cover").mkdir(parents=True)
        (package / "evidence/cover/analysis.json").write_text("{}", "utf-8")
        (package / "evidence/assets").mkdir(parents=True)
        (package / "evidence/assets/assets.json").write_text('{"assets": []}', "utf-8")
        (package / DOCUMENT_NAME).write_text("## 内容拆解\n\n待审计\n\n## 素材卡片\n\n待审计\n", "utf-8")
        manifest = {
            "mode": "full",
            "content_form": "video",
            "source": {"platform": "xhs", "stable_id": "example", "canonical_url": "https://example.com/note"},
            "document": {"path": DOCUMENT_NAME, "naming_version": 4},
            "media": {"path": "evidence/media/source.mp4"},
            "artifacts": [{"path": "evidence/media/source.mp4", "kind": "video", "sha256": "media-hash"}],
            "gates": {
                "identity": "passed", "acquisition": "passed", "extraction": "review",
                "fidelity": "review", "analysis": "review", "comments": "review",
                "cover": "review", "cards": "review", "archive": "passed",
            },
            "errors": [],
        }
        (package / "manifest.json").write_text(json.dumps(manifest), "utf-8")
        return package, manifest

    def test_missing_required_output_becomes_resumable_stale_gate(self):
        with tempfile.TemporaryDirectory() as directory:
            package, manifest = self.make_video_package(Path(directory), raw=False)
            plan = gate_ledger.execution_plan(package, manifest)
            self.assertIn("extraction", {item["gate"] for item in plan["run"]})

    def test_current_review_with_output_waits_without_repeating_compute(self):
        with tempfile.TemporaryDirectory() as directory:
            package, manifest = self.make_video_package(Path(directory), raw=True)
            gate_ledger.sync_gate_ledger(package, manifest, write=False, completed_gates={"extraction"})
            plan = gate_ledger.execution_plan(package, manifest)
            self.assertNotIn("extraction", {item["gate"] for item in plan["run"]})
            self.assertIn("extraction", {item["gate"] for item in plan["review"]})

    def test_unchanged_stale_acquisition_repairs_dependency_only_cover_cascade(self):
        with tempfile.TemporaryDirectory() as directory:
            package, manifest = self.make_video_package(Path(directory), raw=True)
            manifest["gates"] = {gate: "passed" for gate in gate_ledger.GATE_SPECS}
            ledger, _ = gate_ledger.sync_gate_ledger(
                package, manifest, write=False, completed_gates=set(gate_ledger.GATE_SPECS),
            )
            manifest["gate_ledger"] = ledger
            manifest["gate_ledger"]["acquisition"]["status"] = "stale"
            manifest["gate_ledger"]["cover"].update({
                "status": "stale",
                "invalidation_reason": "dependency_not_settled",
                "invalidated_at": "2026-08-25T08:00:00+08:00",
            })

            repaired, invalidated = gate_ledger.sync_gate_ledger(package, manifest, write=False)

            self.assertEqual(repaired["acquisition"]["status"], "passed")
            self.assertEqual(repaired["cover"]["status"], "passed")
            self.assertNotIn("acquisition", invalidated)
            self.assertNotIn("cover", invalidated)

    def test_partial_gate_remains_reviewable_instead_of_becoming_failed(self):
        with tempfile.TemporaryDirectory() as directory:
            package, manifest = self.make_video_package(Path(directory), raw=True)
            manifest["gates"]["analysis"] = "partial"
            gate_ledger.sync_gate_ledger(package, manifest, write=False, completed_gates={"analysis"})
            plan = gate_ledger.execution_plan(package, manifest)
            self.assertIn("analysis", {item["gate"] for item in plan["review"]})
            self.assertNotIn("analysis", {item["gate"] for item in plan["run"]})

    def test_image_ocr_conventions_count_as_extraction_and_fidelity_outputs(self):
        with tempfile.TemporaryDirectory() as directory:
            package, manifest = self.make_video_package(Path(directory), raw=False)
            manifest["content_form"] = "image_post"
            manifest["gates"]["extraction"] = "passed"
            manifest["gates"]["fidelity"] = "passed"
            ocr = package / "evidence/ocr"
            ocr.mkdir(parents=True)
            (ocr / "ocr-raw.json").write_text('{"images": []}', "utf-8")
            (ocr / "image-001.raw.txt").write_text("原始 OCR", "utf-8")
            (ocr / "image-001.corrected.md").write_text("校正 OCR", "utf-8")
            gate_ledger.sync_gate_ledger(
                package,
                manifest,
                write=False,
                completed_gates={"extraction", "fidelity"},
            )
            plan = gate_ledger.execution_plan(package, manifest)
            rerun = {item["gate"] for item in plan["run"]}
            self.assertNotIn("extraction", rerun)
            self.assertNotIn("fidelity", rerun)

    def test_image_analysis_waits_for_fidelity_and_cards_wait_for_analysis(self):
        with tempfile.TemporaryDirectory() as directory:
            package, manifest = self.make_video_package(Path(directory), raw=False)
            manifest["content_form"] = "image_post"
            manifest["gates"]["extraction"] = "passed"
            manifest["gates"]["fidelity"] = "review"
            manifest["gates"]["analysis"] = "passed"
            manifest["gates"]["cards"] = "passed"
            ocr = package / "evidence/ocr"
            ocr.mkdir(parents=True)
            (ocr / "ocr-raw.json").write_text('[]', "utf-8")
            (ocr / "image-001.raw.txt").write_text("原始 OCR", "utf-8")
            (ocr / "image-001.corrected.md").write_text("校正 OCR", "utf-8")
            ledger, _ = gate_ledger.sync_gate_ledger(
                package,
                manifest,
                write=False,
                completed_gates={"extraction", "analysis", "cards"},
            )
            self.assertEqual(ledger["analysis"]["status"], "stale")
            self.assertEqual(ledger["analysis"]["invalidation_reason"], "dependency_not_settled")
            self.assertEqual(ledger["cards"]["status"], "stale")

    def test_rule_change_invalidates_only_gate_and_transitive_dependants(self):
        with tempfile.TemporaryDirectory() as directory:
            package, manifest = self.make_video_package(Path(directory), raw=True)
            gate_ledger.sync_gate_ledger(
                package,
                manifest,
                write=False,
                completed_gates=set(gate_ledger.GATE_SPECS),
            )
            old = gate_ledger.GATE_SPECS["analysis"]
            changed = gate_ledger.GateSpec("analysis-v3", old.depends_on, old.executor, old.context_policy)
            with mock.patch.dict(gate_ledger.GATE_SPECS, {"analysis": changed}):
                plan = gate_ledger.execution_plan(package, manifest)
            rerun = {item["gate"] for item in plan["run"]}
            self.assertTrue({"analysis", "cards", "archive"} <= rerun)
            self.assertFalse({"identity", "acquisition", "extraction", "comments", "cover"} & rerun)

    def test_local_lexical_correction_does_not_reopen_passed_analysis_or_cards(self):
        with tempfile.TemporaryDirectory() as directory:
            package, manifest = self.make_video_package(Path(directory), raw=True)
            transcript = package / "evidence/transcript"
            (transcript / "transcript-review.json").write_text('{"status":"passed","corrections":[]}', "utf-8")
            manifest["gates"] = {gate: "passed" for gate in gate_ledger.GATE_SPECS}
            gate_ledger.sync_gate_ledger(
                package,
                manifest,
                write=False,
                completed_gates=set(gate_ledger.GATE_SPECS),
            )

            (transcript / "transcript-review.json").write_text(
                '{"status":"passed","corrections":[{"from":"Abi","to":"ami"}]}',
                "utf-8",
            )
            gate_ledger.sync_gate_ledger(package, manifest, write=False, completed_gates={"fidelity"})
            plan = gate_ledger.execution_plan(package, manifest)
            rerun = {item["gate"] for item in plan["run"]}
            self.assertNotIn("analysis", rerun)
            self.assertNotIn("cards", rerun)


if __name__ == "__main__":
    unittest.main()

import importlib.util
import json
import sys
import tempfile
import unittest
from pathlib import Path


SCRIPTS = Path(__file__).parents[1] / "scripts"
sys.path.insert(0, str(SCRIPTS))
SPEC = importlib.util.spec_from_file_location("apply_image_fidelity_review", SCRIPTS / "apply_image_fidelity_review.py")
MODULE = importlib.util.module_from_spec(SPEC)
assert SPEC.loader
SPEC.loader.exec_module(MODULE)


class ImageFidelityReviewTests(unittest.TestCase):
    def test_empty_ocr_receipt_cannot_pass_an_image_inventory(self):
        with tempfile.TemporaryDirectory() as directory:
            package = Path(directory)
            image = package / "evidence/images/image-001.jpg"
            image.parent.mkdir(parents=True)
            image.write_bytes(b"image")
            decision = package / "evidence/ocr/decision.json"
            decision.parent.mkdir(parents=True)
            decision.write_text(json.dumps({"status": "all_pages_ocr", "policy": "mandatory_per_image_v1"}), "utf-8")
            manifest = {
                "content_form": "image_post",
                "image_inventory": [{"index": 1, "relative_path": "evidence/images/image-001.jpg"}],
            }
            receipt = {
                "schema_version": MODULE.SCHEMA,
                "reviewed_by": "tester",
                "reviewed_at": "2026-08-24T00:00:00+08:00",
                "pages": [],
            }
            errors, _ = MODULE.validate_receipt(package, manifest, receipt)
            self.assertIn("source images lack raw OCR: [1]", errors)
            self.assertIn("source images lack corrected OCR: [1]", errors)
            self.assertIn("pages must cover corrected OCR indices exactly once: [1]", errors)


if __name__ == "__main__":
    unittest.main()

import importlib.util
import unittest
from pathlib import Path


SCRIPT = Path(__file__).resolve().parents[1] / "scripts" / "plan_transcript_issues.py"
SPEC = importlib.util.spec_from_file_location("plan_transcript_issues", SCRIPT)
MODULE = importlib.util.module_from_spec(SPEC)
assert SPEC.loader
SPEC.loader.exec_module(MODULE)


class TranscriptIssuePlanningTest(unittest.TestCase):
    def test_ignores_clean_subsecond_timing_density_warnings(self):
        qa = {"low_confidence_ranges": [
            {"start": 1924.5, "end": 1925.06, "text": "我觉得就是开始到", "reasons": ["implausibly_dense_text"]},
            {"start": 2564.26, "end": 2564.54, "text": "我不知道", "reasons": ["implausibly_dense_text"]},
        ]}
        self.assertEqual(MODULE.issue_ranges(qa, {}, 3334.4), [])

    def test_keeps_corrupt_or_multi_reason_short_ranges(self):
        qa = {"low_confidence_ranges": [
            {"start": 100, "end": 100.6, "text": "螺丝刀＝刀＝", "reasons": ["implausibly_dense_text"]},
            {"start": 200, "end": 200.4, "text": "专名", "reasons": ["implausibly_dense_text", "low_confidence"]},
        ]}
        self.assertEqual(len(MODULE.issue_ranges(qa, {}, 300)), 2)

    def test_does_not_schedule_reader_exclusions_for_lexical_review(self):
        qa = {
            "reader_exclusions": [{"segment": 6, "start": 68.44, "end": 89.76}],
            "low_confidence_ranges": [
                {"segment": 6, "start": 68.44, "end": 89.76, "text": "去吃" * 50, "reasons": ["high_compression_ratio"]},
                {"segment": 9, "start": 154.86, "end": 158.2, "text": "新电脑到了�", "reasons": ["replacement_character"]},
            ],
        }
        ranges = MODULE.issue_ranges(qa, {}, 220)
        self.assertEqual(len(ranges), 1)
        self.assertLessEqual(ranges[0]["start"], 154.86)
        self.assertGreaterEqual(ranges[0]["end"], 158.2)


if __name__ == "__main__":
    unittest.main()

#!/usr/bin/env python3

import importlib.util
import json
import tempfile
import unittest
from pathlib import Path


SCRIPT = Path(__file__).parents[1] / "scripts" / "transcribe_package.py"
SPEC = importlib.util.spec_from_file_location("transcribe_package", SCRIPT)
MODULE = importlib.util.module_from_spec(SPEC)
assert SPEC.loader
SPEC.loader.exec_module(MODULE)


class TranscriptQaTests(unittest.TestCase):
    def test_decoder_loop_without_reliable_text_is_no_reliable_speech(self):
        qa = MODULE.score_segments([
            {
                "start": 23.32,
                "end": 24.72,
                "text": "超" + "萎" * 110,
                "compression_ratio": 18.5,
                "avg_logprob": -0.13,
                "no_speech_prob": 0.0,
            },
            {"start": 24.72, "end": 24.72, "text": "", "compression_ratio": 0.27},
        ], 26.033, "en")

        self.assertEqual(qa["status"], "rejected")
        self.assertEqual(qa["failure_reason"], "music_or_no_reliable_speech")
        self.assertFalse(qa["reader_eligible"])
        self.assertEqual(qa["reliable_character_count"], 0)

    def test_decoder_loop_does_not_hide_other_reliable_text(self):
        qa = MODULE.score_segments([
            {"start": 0, "end": 3, "text": "这里有一段清晰的人声", "compression_ratio": 1.0},
            {"start": 3, "end": 4, "text": "坏" * 80, "compression_ratio": 12.0},
        ], 4, "zh")

        self.assertEqual(qa["status"], "failed")
        self.assertEqual(qa["failure_reason"], "non_monotonic_or_hallucinated_asr")
        self.assertGreater(qa["reliable_character_count"], 0)

    def test_decoder_loop_with_tiny_terminal_crumb_is_no_reliable_speech(self):
        qa = MODULE.score_segments([
            {
                "start": 22.04,
                "end": 23.44,
                "text": "请" * 220,
                "compression_ratio": 36.83,
                "avg_logprob": -0.34,
                "no_speech_prob": 0.0,
            },
            {
                "start": 23.44,
                "end": 23.98,
                "text": "you",
                "compression_ratio": 0.27,
                "avg_logprob": -0.95,
                "no_speech_prob": 0.0,
            },
        ], 24.775692, "en")

        self.assertEqual(qa["status"], "rejected")
        self.assertEqual(qa["failure_reason"], "music_or_no_reliable_speech")
        self.assertFalse(qa["reader_eligible"])
        self.assertEqual(qa["reliable_character_count"], 3)
        self.assertLessEqual(qa["reliable_speech_seconds"], 0.75)

    def test_decoder_loop_with_short_thank_you_tail_is_no_reliable_speech(self):
        qa = MODULE.score_segments([
            {
                "start": 18.68,
                "end": 20.08,
                "text": "前世" + "前" * 219,
                "compression_ratio": 28.826,
                "avg_logprob": -0.142,
                "no_speech_prob": 0.0,
            },
            {
                "start": 20.5,
                "end": 21.02,
                "text": "Thank you.",
                "compression_ratio": 0.556,
                "avg_logprob": -0.895,
                "no_speech_prob": 0.0,
            },
        ], 33.133333, "en")

        self.assertEqual(qa["policy_version"], "speech-qa-v10-repeated-generic-settlement")
        self.assertEqual(qa["status"], "rejected")
        self.assertEqual(qa["failure_reason"], "music_or_no_reliable_speech")
        self.assertTrue(qa["sparse_speech"]["decoder_loop_dominates_remainder"])

    def test_spaced_low_confidence_thank_you_hallucinations_do_not_create_speech(self):
        qa = MODULE.score_segments([
            {
                "start": 0.0, "end": 13.94, "text": "in in in in in",
                "compression_ratio": 37.529, "avg_logprob": -0.597,
                "words": [{"word": " in", "probability": 0.01}],
            },
            {
                "start": 16.16, "end": 16.64, "text": "in in in",
                "compression_ratio": 37.529, "avg_logprob": -0.597,
                "words": [{"word": " in", "probability": 0.66}],
            },
            {
                "start": 16.64, "end": 17.20, "text": "in in in",
                "compression_ratio": 37.529, "avg_logprob": -0.597,
                "words": [{"word": " in", "probability": 0.76}],
            },
            {
                "start": 17.20, "end": 17.30, "text": "in in in in in in",
                "compression_ratio": 37.529, "avg_logprob": -0.597,
                "words": [{"word": " in", "probability": 0.83}],
            },
            *[
                {
                    "start": start, "end": start + span, "text": "Thank you.",
                    "compression_ratio": 0.556, "avg_logprob": logprob,
                    "words": [
                        {"word": " Thank", "probability": first_probability},
                        {"word": " you.", "probability": 0.99},
                    ],
                }
                for start, span, logprob, first_probability in [
                    (19.22, 1.4, -0.446, 0.125),
                    (51.08, 2.8, -0.325, 0.206),
                    (78.92, 1.2, -0.398, 0.166),
                    (113.06, 2.8, -0.527, 0.149),
                ]
            ],
        ], 125.0, "en")

        self.assertEqual(qa["status"], "rejected")
        self.assertEqual(qa["failure_reason"], "music_or_no_reliable_speech")
        self.assertEqual(qa["reliable_character_count"], 0)
        self.assertEqual(
            sum("repeated_low_confidence_generic_decoder_phrase" in item["reasons"] for item in qa["reader_exclusions"]),
            4,
        )

    def test_one_real_thank_you_is_not_suppressed(self):
        qa = MODULE.score_segments([{
            "start": 2.0, "end": 3.0, "text": "Thank you.",
            "compression_ratio": 0.556, "avg_logprob": -0.2,
            "words": [
                {"word": " Thank", "probability": 0.91},
                {"word": " you.", "probability": 0.99},
            ],
        }], 5.0, "en")

        self.assertEqual(qa["status"], "passed")
        self.assertTrue(qa["reader_eligible"])

    def test_repeated_weak_thank_you_is_not_suppressed_beside_substantial_speech(self):
        qa = MODULE.score_segments([
            {
                "start": 0.0, "end": 12.0,
                "text": "This is a substantial spoken explanation that should remain ordinary transcript evidence.",
                "compression_ratio": 1.0, "avg_logprob": -0.1,
            },
            *[
                {
                    "start": start, "end": start + 0.8, "text": "Thank you.",
                    "compression_ratio": 0.556, "avg_logprob": -0.46,
                    "words": [
                        {"word": "Thank", "probability": 0.12},
                        {"word": "you", "probability": 0.99},
                    ],
                }
                for start in (36.2, 64.44, 99.14, 120.58)
            ],
        ], 124.92, "en")

        self.assertTrue(qa["reader_eligible"])
        self.assertFalse(any(
            "repeated_low_confidence_generic_decoder_phrase" in item["reasons"]
            for item in qa["reader_exclusions"]
        ))

    def test_one_weak_latin_crumb_plus_repeated_generic_noise_is_no_speech(self):
        qa = MODULE.score_segments([
            {
                "start": 0.58, "end": 2.8, "text": "I love you.",
                "compression_ratio": 0.579, "avg_logprob": -0.661,
                "words": [
                    {"word": "I", "probability": 0.061},
                    {"word": "love", "probability": 0.194},
                    {"word": "you", "probability": 0.958},
                ],
            },
            *[
                {
                    "start": start, "end": start + 0.5, "text": "Thank you.",
                    "compression_ratio": 0.556, "avg_logprob": -0.46,
                    "words": [
                        {"word": "Thank", "probability": 0.12},
                        {"word": "you", "probability": 0.99},
                    ],
                }
                for start in (36.2, 64.44, 99.14, 120.58)
            ],
        ], 124.92, "en")

        self.assertEqual(qa["status"], "rejected")
        self.assertFalse(qa["reader_eligible"])
        self.assertEqual(qa["reliable_character_count"], 0)
        self.assertTrue(any(
            "isolated_low_confidence_latin_remainder" in item["reasons"]
            for item in qa["reader_exclusions"]
        ))

    def test_weak_latin_phrase_is_not_suppressed_without_repeated_decoder_noise(self):
        qa = MODULE.score_segments([{
            "start": 0.58, "end": 2.8, "text": "I love you.",
            "compression_ratio": 0.579, "avg_logprob": -0.661,
            "words": [
                {"word": "I", "probability": 0.061},
                {"word": "love", "probability": 0.194},
                {"word": "you", "probability": 0.958},
            ],
        }], 124.92, "en")

        self.assertEqual(qa["status"], "passed")
        self.assertTrue(qa["reader_eligible"])

    def test_fragmented_local_decoder_loop_keeps_coherent_remainder_eligible(self):
        segments = [
            {"start": 0, "end": 12, "text": "如果 AI 接手了回忆，那遗忘是否也要付费解锁。", "compression_ratio": 1.0},
        ]
        segments.extend(
            {
                "start": 40 + index * 0.2,
                "end": 40 + (index + 1) * 0.2,
                "text": "主任",
                "compression_ratio": 18.0,
                "avg_logprob": -1.2,
            }
            for index in range(40)
        )
        segments.extend([
            {"start": 150, "end": 162, "text": "新电脑到了，现在回看过去，很多旧物承载的生活记忆仍然非常清楚。", "compression_ratio": 1.0},
            {"start": 190, "end": 202, "text": "曾经幻想做个画家，如今准备离开生活多年的小地方，去另一座城市开始新的生活。", "compression_ratio": 1.0},
        ])

        qa = MODULE.score_segments(segments, 240, "zh")

        self.assertEqual(qa["status"], "review")
        self.assertTrue(qa["reader_eligible"])
        self.assertIsNone(qa["failure_reason"])
        self.assertTrue(qa["sparse_speech"]["localized_decoder_loop_omitted"])
        self.assertFalse(qa["catastrophic_impact"]["systemic"])

    def test_no_reliable_speech_review_refresh_is_idempotent(self):
        with tempfile.TemporaryDirectory() as temp:
            evidence = Path(temp)
            review_path = evidence / "transcript-review.json"
            review_path.write_text(json.dumps({
                "status": "not_applicable_no_reliable_speech",
                "failure_reason": "music_or_no_reliable_speech",
                "reviewed_at": "2026-08-24T12:00:00+08:00",
            }), encoding="utf-8")

            MODULE.settle_no_reliable_speech_review(evidence, {
                "failure_reason": "music_or_no_reliable_speech",
            })

            review = json.loads(review_path.read_text(encoding="utf-8"))
            self.assertEqual(review["reviewed_at"], "2026-08-24T12:00:00+08:00")


if __name__ == "__main__":
    unittest.main()

import json
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path


SCRIPT = Path(__file__).resolve().parents[1] / "scripts" / "record_transcript_range_review.py"


class TranscriptRangeReviewTest(unittest.TestCase):
    def make_package(self, temporary):
        package = Path(temporary)
        transcript = package / "evidence/transcript"
        transcript.mkdir(parents=True)
        document_path = "视频-测试-逐字稿复核-测试-20260826.md"
        (package / document_path).write_text("# 测试\n", encoding="utf-8")
        (transcript / "transcript-review.json").write_text(
            json.dumps({"status": "pending", "corrections": []}), encoding="utf-8"
        )
        (package / "manifest.json").write_text(
            json.dumps({
                "document": {"path": document_path},
                "transcript": {"lexical_review_status": "pending"},
            }), encoding="utf-8"
        )
        return package, transcript

    @staticmethod
    def passed_receipt(**overrides):
        return {
            "status": "passed",
            "reviewer": "test-reviewer",
            "evidence_method": "bounded_transcript_context",
            "checks": {
                "names_and_products": "passed",
                "numbers_and_units": "passed",
                "english_terms": "passed",
                "quote_candidates": "passed",
                "automated_low_confidence_ranges": "passed",
            },
            **overrides,
        }

    def test_apply_records_only_the_declared_bounded_span(self):
        with tempfile.TemporaryDirectory() as temporary:
            package, transcript = self.make_package(temporary)
            staged = package / "span.json"
            staged.write_text(json.dumps(self.passed_receipt(
                index=1,
                total=3,
                start=120,
                end=240,
            )), encoding="utf-8")
            completed = subprocess.run([
                sys.executable,
                str(SCRIPT),
                "apply",
                "--package", str(package),
                "--input", str(staged),
                "--index", "1",
                "--total", "3",
                "--start", "120",
                "--end", "240",
            ], text=True, capture_output=True, check=False)
            self.assertEqual(completed.returncode, 0, completed.stderr)
            review = json.loads((transcript / "transcript-review.json").read_text("utf-8"))
            manifest = json.loads((package / "manifest.json").read_text("utf-8"))
            self.assertEqual(review["status"], "in_progress")
            self.assertEqual(review["range_reviews"][0]["index"], 1)
            self.assertEqual(manifest["transcript"]["lexical_range_reviews"][0]["start"], 120)
            self.assertTrue((transcript / "range-reviews/span-002-of-003.json").is_file())

    def test_apply_fills_omitted_schedule_fields_from_cli_contract(self):
        with tempfile.TemporaryDirectory() as temporary:
            package, transcript = self.make_package(temporary)
            staged = package / "span.json"
            staged.write_text(json.dumps(self.passed_receipt()), encoding="utf-8")
            completed = subprocess.run([
                sys.executable, str(SCRIPT), "apply",
                "--package", str(package), "--input", str(staged),
                "--index", "1", "--total", "3", "--start", "120", "--end", "240",
            ], text=True, capture_output=True, check=False)
            self.assertEqual(completed.returncode, 0, completed.stderr)
            receipt = json.loads((transcript / "range-reviews/span-002-of-003.json").read_text("utf-8"))
            self.assertEqual(
                {key: receipt[key] for key in ("index", "total", "start", "end")},
                {"index": 1, "total": 3, "start": 120.0, "end": 240.0},
            )

    def test_apply_rejects_an_explicit_schedule_conflict(self):
        with tempfile.TemporaryDirectory() as temporary:
            package, _ = self.make_package(temporary)
            staged = package / "span.json"
            staged.write_text(json.dumps(self.passed_receipt(index=2)), encoding="utf-8")
            completed = subprocess.run([
                sys.executable, str(SCRIPT), "apply",
                "--package", str(package), "--input", str(staged),
                "--index", "1", "--total", "3", "--start", "120", "--end", "240",
            ], text=True, capture_output=True, check=False)
            self.assertEqual(completed.returncode, 2)
            self.assertIn("receipt index does not match", completed.stderr)

    def test_targeted_finalize_accepts_zero_flagged_ranges_without_global_quote_approval(self):
        with tempfile.TemporaryDirectory() as temporary:
            package = Path(temporary)
            transcript = package / "evidence/transcript"
            transcript.mkdir(parents=True)
            document_path = "视频-bilibili-零问题范围-测试-20260826.md"
            (package / document_path).write_text("# 测试\n", encoding="utf-8")
            (transcript / "transcript-review.json").write_text(
                json.dumps({"status": "pending", "range_reviews": [], "unresolved": []}), encoding="utf-8"
            )
            (package / "manifest.json").write_text(json.dumps({
                "mode": "full",
                "content_form": "video",
                "document": {"path": document_path},
                "source": {"platform": "bilibili", "stable_id": "sample"},
                "gates": {"extraction": "review", "fidelity": "review"},
                "transcript": {
                    "lexical_review_status": "pending",
                    "lexical_review_policy": "targeted_issues_v1",
                    "lexical_issue_ranges": [],
                },
            }), encoding="utf-8")
            completed = subprocess.run([
                sys.executable, str(SCRIPT), "finalize-coverage", "--package", str(package),
            ], text=True, capture_output=True, check=False)
            self.assertEqual(completed.returncode, 0, completed.stderr)
            review = json.loads((transcript / "transcript-review.json").read_text("utf-8"))
            manifest = json.loads((package / "manifest.json").read_text("utf-8"))
            self.assertEqual(review["status"], "targeted_passed")
            self.assertEqual(manifest["transcript"]["lexical_review_status"], "targeted_passed")
            self.assertFalse(manifest["transcript"]["quotation_ready"])

    def test_targeted_finalize_settles_one_inconclusive_range_as_locally_limited(self):
        with tempfile.TemporaryDirectory() as temporary:
            package, transcript = self.make_package(temporary)
            manifest_path = package / "manifest.json"
            manifest = json.loads(manifest_path.read_text("utf-8"))
            manifest.update({
                "mode": "full",
                "content_form": "podcast_episode",
                "source": {"platform": "xiaoyuzhou", "stable_id": "sample"},
                "gates": {"extraction": "review", "fidelity": "review"},
            })
            manifest["transcript"].update({
                "lexical_review_policy": "targeted_issues_v1",
                "lexical_issue_ranges": [{"index": 0, "total": 1, "start": 10, "end": 20}],
            })
            manifest_path.write_text(json.dumps(manifest), encoding="utf-8")
            staged = package / "span.json"
            staged.write_text(json.dumps({
                "status": "limited",
                "reviewer": "test-reviewer",
                "evidence_method": "bounded_audio_retranscription_inconclusive",
                "unresolved": [{"start": 12, "end": 14, "text": "无法确认的专名"}],
            }, ensure_ascii=False), encoding="utf-8")
            applied = subprocess.run([
                sys.executable, str(SCRIPT), "apply",
                "--package", str(package), "--input", str(staged),
                "--index", "0", "--total", "1", "--start", "10", "--end", "20",
            ], text=True, capture_output=True, check=False)
            self.assertEqual(applied.returncode, 0, applied.stderr)
            finalized = subprocess.run([
                sys.executable, str(SCRIPT), "finalize-coverage", "--package", str(package),
            ], text=True, capture_output=True, check=False)
            self.assertEqual(finalized.returncode, 0, finalized.stderr)
            review = json.loads((transcript / "transcript-review.json").read_text("utf-8"))
            manifest = json.loads(manifest_path.read_text("utf-8"))
            self.assertEqual(review["status"], "targeted_limited")
            self.assertEqual(manifest["transcript"]["lexical_review_status"], "targeted_limited")
            self.assertEqual(manifest["gates"]["fidelity"], "passed")
            self.assertEqual(len(manifest["transcript"]["lexical_limited_ranges"]), 1)
            self.assertFalse(manifest["transcript"]["quotation_ready"])


if __name__ == "__main__":
    unittest.main()

#!/usr/bin/env python3

import copy
import importlib.util
import json
import tempfile
import unittest
from pathlib import Path
from unittest import mock


SCRIPT = Path(__file__).parents[1] / "scripts" / "validate_source_assets.py"
SPEC = importlib.util.spec_from_file_location("validate_source_assets", SCRIPT)
MODULE = importlib.util.module_from_spec(SPEC)
assert SPEC.loader
SPEC.loader.exec_module(MODULE)
FIXTURE = Path(__file__).parent / "fixtures" / "valid-article"


class SourceAssetValidationTests(unittest.TestCase):
    def test_hook_and_structure_use_controlled_retrieval_types(self):
        self.assertIn("hook_type", MODULE.PAYLOAD_FIELDS["hook"])
        self.assertIn("structure_type", MODULE.PAYLOAD_FIELDS["structure"])
        self.assertIn("反常识纠偏", MODULE.HOOK_TYPES)
        self.assertIn("递进论证型", MODULE.STRUCTURE_TYPES)

    def test_agent_technique_title_must_begin_with_its_type(self):
        errors = []
        MODULE.validate_technique_naming(
            "hook", "agent_generated", "亲测推翻流行说法",
            {"hook_type": "反常识纠偏"}, "assets[0]", errors,
        )
        self.assertIn("assets[0].title must use '类型｜具体做法' naming", errors)
        errors = []
        MODULE.validate_technique_naming(
            "structure", "agent_generated", "递进论证型｜从实测推到退出成本",
            {"structure_type": "递进论证型"}, "assets[1]", errors,
        )
        self.assertEqual(errors, [])

    def test_music_quote_and_structure_are_applicable(self):
        self.assertEqual(MODULE.CARRIER_APPLICABILITY["music"], {"quote", "structure"})

    def test_legacy_audio_manifest_uses_music_asset_carrier_without_speech_chapters(self):
        with tempfile.TemporaryDirectory() as temp:
            root = Path(temp)
            (root / "evidence/assets").mkdir(parents=True)
            (root / "manifest.json").write_text(json.dumps({"mode": "full", "content_form": "audio"}), "utf-8")
            (root / "evidence/assets/assets.json").write_text(json.dumps({"carrier": "music"}), "utf-8")
            with mock.patch.object(MODULE, "validate_asset_bundle", return_value=[]), mock.patch.object(MODULE, "validate_chapters") as chapters:
                self.assertEqual(MODULE.validate_package(root, allow_review=False), [])
            chapters.assert_not_called()

    def test_explicit_no_reliable_speech_video_does_not_require_chapters(self):
        with tempfile.TemporaryDirectory() as temp:
            root = Path(temp)
            (root / "evidence/assets").mkdir(parents=True)
            (root / "evidence/transcript").mkdir(parents=True)
            (root / "manifest.json").write_text(json.dumps({"mode": "full", "content_form": "video"}), "utf-8")
            (root / "evidence/assets/assets.json").write_text(json.dumps({"carrier": "video"}), "utf-8")
            (root / "evidence/transcript/transcript-qa.json").write_text(json.dumps({
                "status": "rejected",
                "failure_reason": "music_or_no_reliable_speech",
                "reader_eligible": False,
            }), "utf-8")
            with mock.patch.object(MODULE, "validate_asset_bundle", return_value=[]), mock.patch.object(MODULE, "validate_chapters") as chapters:
                self.assertEqual(MODULE.validate_package(root, allow_review=False), [])
            chapters.assert_not_called()

    def test_no_speech_chapter_bypass_requires_all_three_qa_fields(self):
        variants = [
            {"status": "review", "failure_reason": "music_or_no_reliable_speech", "reader_eligible": False},
            {"status": "rejected", "failure_reason": "other", "reader_eligible": False},
            {"status": "rejected", "failure_reason": "music_or_no_reliable_speech", "reader_eligible": True},
        ]
        for qa in variants:
            with self.subTest(qa=qa), tempfile.TemporaryDirectory() as temp:
                root = Path(temp)
                (root / "evidence/assets").mkdir(parents=True)
                (root / "evidence/transcript").mkdir(parents=True)
                (root / "manifest.json").write_text(json.dumps({"mode": "full", "content_form": "video"}), "utf-8")
                (root / "evidence/assets/assets.json").write_text(json.dumps({"carrier": "video"}), "utf-8")
                (root / "evidence/transcript/transcript-qa.json").write_text(json.dumps(qa), "utf-8")
                with mock.patch.object(MODULE, "validate_asset_bundle", return_value=[]), mock.patch.object(MODULE, "validate_chapters", return_value=["chapters required"]) as chapters:
                    self.assertEqual(MODULE.validate_package(root, allow_review=False), ["chapters required"])
                chapters.assert_called_once()

    def test_audio_cannot_use_video_no_speech_bypass(self):
        with tempfile.TemporaryDirectory() as temp:
            root = Path(temp)
            (root / "evidence/assets").mkdir(parents=True)
            (root / "evidence/transcript").mkdir(parents=True)
            (root / "manifest.json").write_text(json.dumps({"mode": "full", "content_form": "audio"}), "utf-8")
            (root / "evidence/assets/assets.json").write_text(json.dumps({"carrier": "audio"}), "utf-8")
            (root / "evidence/transcript/transcript-qa.json").write_text(json.dumps({
                "status": "rejected",
                "failure_reason": "music_or_no_reliable_speech",
                "reader_eligible": False,
            }), "utf-8")
            with mock.patch.object(MODULE, "validate_asset_bundle", return_value=[]), mock.patch.object(MODULE, "validate_chapters", return_value=["chapters required"]) as chapters:
                self.assertEqual(MODULE.validate_package(root, allow_review=False), ["chapters required"])
            chapters.assert_called_once()

    def test_valid_article_package(self):
        self.assertEqual(MODULE.validate_package(FIXTURE, allow_review=False), [])

    def test_whole_package_validation_grandfathers_protected_user_assets(self):
        with tempfile.TemporaryDirectory() as temp:
            package = Path(temp) / "fixture"
            import shutil
            shutil.copytree(FIXTURE, package)
            asset_path = package / "evidence/assets/assets.json"
            document = json.loads(asset_path.read_text("utf-8"))
            document["assets"].append({
                "id": "legacy-user-audio-clip",
                "type": "shot",
                "origin": "user_injected",
                "title": "片段 0:10–0:20",
                "injection": {
                    "status": "standardized",
                    "created_by": "user",
                    "protected_from_agent": True,
                },
            })
            asset_path.write_text(json.dumps(document, ensure_ascii=False), "utf-8")

            self.assertTrue(MODULE.validate_asset_bundle(package, document, False))
            self.assertEqual(MODULE.validate_package(package, allow_review=False), [])

    def test_reviewed_understanding_requires_bounded_classification(self):
        document = json.loads((FIXTURE / "evidence/assets/assets.json").read_text("utf-8"))
        document["source_understanding"]["classification"]["expression_modes"] = ["论证", "分析", "叙事"]
        errors = MODULE.validate_asset_bundle(FIXTURE, document, False)
        self.assertIn(
            "source_understanding.classification.expression_modes must contain 1-2 unique values",
            errors,
        )

    def test_topic_tags_reject_empty_or_unbounded_labels(self):
        document = json.loads((FIXTURE / "evidence/assets/assets.json").read_text("utf-8"))
        document["source_understanding"]["classification"]["topic_tags"] = [""]
        errors = MODULE.validate_asset_bundle(FIXTURE, document, False)
        self.assertIn(
            "source_understanding.classification.topic_tags must be non-empty and <= 16 characters",
            errors,
        )

    def test_carrier_requires_matching_analysis_profile(self):
        document = json.loads((FIXTURE / "evidence/assets/assets.json").read_text("utf-8"))
        document["analysis_profile"] = "video_timeline"
        errors = MODULE.validate_asset_bundle(FIXTURE, document, False)
        self.assertIn("asset bundle analysis_profile must be 'argument_article' for article", errors)

    def test_summary_chapter_is_rejected(self):
        document = json.loads((FIXTURE / "evidence/article/正文-章节版.json").read_text("utf-8"))
        document["chapters"][0]["title"] = "第一部分"
        document["chapters"][0]["source_paragraph_ids"] = ["P0001"]
        with tempfile.TemporaryDirectory() as temp:
            root = Path(temp)
            (root / "evidence/article").mkdir(parents=True)
            (root / "evidence/article/正文-章节版.json").write_text(json.dumps(document, ensure_ascii=False), "utf-8")
            errors = MODULE.validate_chapters(root, root / "evidence/article/正文-章节版.json", "article", False)
        self.assertTrue(any("generic" in item for item in errors))
        self.assertTrue(any("declared range" in item for item in errors))

    def test_quota_filler_asset_is_rejected(self):
        document = json.loads((FIXTURE / "evidence/assets/assets.json").read_text("utf-8"))
        weak = copy.deepcopy(document["assets"][0])
        weak["id"] = "weak"
        weak["quality"]["strength"] = "ordinary"
        document["assets"].append(weak)
        errors = MODULE.validate_asset_bundle(FIXTURE, document, False)
        self.assertTrue(any("admitted asset must be strong or useful" in item for item in errors))

    def test_passed_bundle_requires_whole_source_understanding(self):
        document = json.loads((FIXTURE / "evidence/assets/assets.json").read_text("utf-8"))
        document.pop("source_understanding")
        errors = MODULE.validate_asset_bundle(FIXTURE, document, False)
        self.assertIn("source_understanding must be an object", errors)

    def test_type_scoped_validation_can_skip_unrelated_source_understanding(self):
        document = json.loads((FIXTURE / "evidence/assets/assets.json").read_text("utf-8"))
        document.pop("source_understanding")
        errors = MODULE.validate_asset_bundle(
            FIXTURE,
            document,
            False,
            validate_understanding=False,
        )
        self.assertFalse(any(error.startswith("source_understanding") for error in errors), errors)

    def test_reviewed_understanding_requires_commercial_assessment(self):
        document = json.loads((FIXTURE / "evidence/assets/assets.json").read_text("utf-8"))
        document["source_understanding"].pop("commercial_assessment")
        errors = MODULE.validate_asset_bundle(FIXTURE, document, False)
        self.assertIn("source_understanding.commercial_assessment must be an object when reviewed", errors)

    def test_suspected_commercial_requires_ad_style(self):
        document = json.loads((FIXTURE / "evidence/assets/assets.json").read_text("utf-8"))
        assessment = document["source_understanding"]["commercial_assessment"]
        assessment.update({"status": "suspected_commercial", "promotion_style": "not_applicable"})
        errors = MODULE.validate_asset_bundle(FIXTURE, document, False)
        self.assertIn("commercial or suspected commercial content requires hard_ad or soft_ad", errors)

    def test_no_commercial_evidence_allows_empty_subject(self):
        document = json.loads((FIXTURE / "evidence/assets/assets.json").read_text("utf-8"))
        document["source_understanding"]["commercial_assessment"]["subject"] = ""
        errors = MODULE.validate_asset_bundle(FIXTURE, document, False)
        self.assertFalse(any("commercial_assessment.subject" in item for item in errors))

    def test_suspected_commercial_requires_subject(self):
        document = json.loads((FIXTURE / "evidence/assets/assets.json").read_text("utf-8"))
        assessment = document["source_understanding"]["commercial_assessment"]
        assessment.update({"status": "suspected_commercial", "promotion_style": "soft_ad", "subject": ""})
        errors = MODULE.validate_asset_bundle(FIXTURE, document, False)
        self.assertIn(
            "source_understanding.commercial_assessment.subject is required for commercial content",
            errors,
        )

    def test_passed_bundle_requires_complete_four_type_review(self):
        document = json.loads((FIXTURE / "evidence/assets/assets.json").read_text("utf-8"))
        document["review"]["type_scan"] = document["review"]["type_scan"][:-1]
        errors = MODULE.validate_asset_bundle(FIXTURE, document, False)
        self.assertTrue(any("must cover the four extracted types" in item for item in errors))

    def test_review_receipt_must_match_admitted_assets(self):
        document = json.loads((FIXTURE / "evidence/assets/assets.json").read_text("utf-8"))
        document["review"]["candidates"][0]["verdict"] = "rejected"
        document["review"]["candidates"][0].update({
            "failure_stage": "story_review",
            "failure_code": "main_line_mismatch",
            "repair_request": "重新扫描观点候选。",
            "next_stage": "candidate_discovery",
        })
        errors = MODULE.validate_asset_bundle(FIXTURE, document, False)
        self.assertTrue(any("admitted candidate IDs" in item for item in errors))

    def test_one_pending_type_does_not_invalidate_other_settled_type_gates(self):
        document = json.loads((FIXTURE / "evidence/assets/assets.json").read_text("utf-8"))
        document["analysis_status"] = "partial"
        document["review"]["status"] = "partial"
        for scan in document["review"]["type_scan"]:
            if scan["type"] == "quote":
                scan.update({
                    "status": "review",
                    "reason": "金句证据还要复核。",
                    "reviewed_by": "",
                    "reviewed_at": "",
                })
        self.assertEqual(MODULE.validate_asset_bundle(FIXTURE, document, True), [])

    def test_aggregate_review_status_must_match_independent_type_gates(self):
        document = json.loads((FIXTURE / "evidence/assets/assets.json").read_text("utf-8"))
        document["analysis_status"] = "partial"
        document["review"]["status"] = "passed"
        document["review"]["type_scan"][0].update({
            "status": "review",
            "reason": "仍在复核。",
            "reviewed_by": "",
            "reviewed_at": "",
        })
        errors = MODULE.validate_asset_bundle(FIXTURE, document, True)
        self.assertTrue(any("aggregate partial" in item for item in errors))

    def test_one_type_can_render_admitted_quote_and_retain_pending_quote_candidate(self):
        document = json.loads((FIXTURE / "evidence/assets/assets.json").read_text("utf-8"))
        document["analysis_status"] = "partial"
        document["review"]["status"] = "partial"
        quote = copy.deepcopy(document["assets"][0])
        quote.update({"id": "valid-article:quote:01", "type": "quote", "title": "组织验证比接入更难"})
        quote["payload"] = {
            "verbatim": "真正的困难是重写责任边界和验证流程。",
            "context": "作者在区分模型接入和真正的组织转型。",
            "why_it_lands": "用一句话把技术动作与组织难题拉开。",
        }
        document["assets"].append(quote)
        for scan in document["review"]["type_scan"]:
            if scan["type"] == "quote":
                scan.update({
                    "status": "partial",
                    "reason": "已有一条精确正文金句准入，另一条口播候选待局部复核。",
                    "reviewed_by": "test-reviewer",
                    "reviewed_at": "2026-08-20T20:00:00+08:00",
                })
        shared = {
            "type": "quote",
            "main_line_relation": "压缩来源的组织转型主线。",
            "audience_effect": "让读者重新判断项目难点。",
            "mechanism": "用技术动作与组织难题形成反差。",
            "evidence": ["chapter-01 paragraphs 1-2"],
            "boundary": "只适用于组织型 AI 落地讨论。",
            "anti_copy": "不脱离组织语境套用。",
        }
        document["review"]["candidates"].extend([
            {
                **shared,
                "asset_id": "valid-article:quote:01",
                "verdict": "admitted",
                "failure_stage": None,
                "failure_code": None,
                "repair_request": None,
                "next_stage": None,
            },
            {
                **shared,
                "asset_id": "valid-article:quote:02",
                "verdict": "review",
                "failure_stage": "verbatim_review",
                "failure_code": "speech_span_unverified",
                "repair_request": "仅复核候选对应音频片段。",
                "next_stage": "quote_span_review",
            },
        ])
        self.assertEqual(MODULE.validate_asset_bundle(FIXTURE, document, True), [])

    def test_zero_assets_pass_after_explicit_completed_scan(self):
        document = json.loads((FIXTURE / "evidence/assets/assets.json").read_text("utf-8"))
        document["assets"] = []
        document["review"]["candidates"] = []
        for scan in document["review"]["type_scan"]:
            if scan["status"] == "admitted":
                scan["status"] = "no_asset"
                scan["reason"] = "完成该类型扫描后，没有发现达到准入门槛的候选。"
        self.assertEqual(MODULE.validate_asset_bundle(FIXTURE, document, False), [])

    def test_user_injected_viewpoint_does_not_enter_agent_review_receipt(self):
        document = json.loads((FIXTURE / "evidence/assets/assets.json").read_text("utf-8"))
        injected = copy.deepcopy(document["assets"][0])
        injected["id"] = "user:viewpoint:01"
        injected["origin"] = "user_injected"
        injected["injection"] = {
            "status": "standardized",
            "created_by": "user",
            "protected_from_agent": True,
            "interaction_id": "interaction-test-01",
            "injected_at": "2026-08-20T16:00:00+08:00",
        }
        document["assets"].append(injected)
        self.assertEqual(MODULE.validate_asset_bundle(FIXTURE, document, False), [])

    def test_agent_cannot_generate_image_or_shot_assets(self):
        document = json.loads((FIXTURE / "evidence/assets/assets.json").read_text("utf-8"))
        document["assets"][0]["type"] = "image"
        errors = MODULE.validate_asset_bundle(FIXTURE, document, False)
        self.assertTrue(any("can only be user_injected" in item for item in errors))

    def test_agent_viewpoint_cannot_duplicate_an_admitted_quote_verbatim(self):
        document = json.loads((FIXTURE / "evidence/assets/assets.json").read_text("utf-8"))
        viewpoint = document["assets"][0]
        exact = viewpoint["evidence"][0]["excerpt"]
        viewpoint["payload"]["claim"] = exact
        quote = copy.deepcopy(viewpoint)
        quote["id"] = "valid-article:quote:01"
        quote["type"] = "quote"
        quote["title"] = "组织验证才是转型难点"
        quote["payload"] = {"quote_type": "判断金句", "verbatim": exact, "context": "文章讨论企业 AI 转型。", "why_it_lands": "用对比压缩核心判断。"}
        document["assets"].append(quote)
        errors = MODULE.validate_asset_bundle(FIXTURE, document, False)
        self.assertTrue(any("admitted viewpoint duplicates admitted quote" in item for item in errors), errors)

    def test_agent_viewpoint_claim_must_not_copy_its_evidence_excerpt(self):
        document = json.loads((FIXTURE / "evidence/assets/assets.json").read_text("utf-8"))
        document["assets"][0]["payload"]["claim"] = document["assets"][0]["evidence"][0]["excerpt"]
        errors = MODULE.validate_asset_bundle(FIXTURE, document, False)
        self.assertTrue(any("viewpoint claim must be a synthesized paraphrase" in item for item in errors), errors)

    def test_agent_viewpoint_may_isolate_a_short_proposition_from_longer_evidence(self):
        document = json.loads((FIXTURE / "evidence/assets/assets.json").read_text("utf-8"))
        document["assets"][0]["payload"]["claim"] = "用 AI 制定旅行计划可能是伪需求"
        document["assets"][0]["evidence"][0]["excerpt"] = "上个月我在黄即发了条表示用 AI 制定旅行计划可能是伪需求的观点，因为制定计划本身也是旅行乐趣的一部分。"
        errors = MODULE.validate_asset_bundle(FIXTURE, document, False)
        self.assertFalse(any("viewpoint claim must be a synthesized paraphrase" in item for item in errors), errors)

    def test_agent_viewpoint_claim_removes_discourse_prefix(self):
        document = json.loads((FIXTURE / "evidence/assets/assets.json").read_text("utf-8"))
        document["assets"][0]["payload"]["claim"] = "我发现，现在旅行的矛盾是移动越来越快，深度连接却越来越慢"
        errors = MODULE.validate_asset_bundle(FIXTURE, document, False)
        self.assertTrue(any("remove non-load-bearing discourse prefixes" in item for item in errors), errors)

    def test_agent_viewpoint_may_equal_evidence_after_removing_only_discourse_prefix(self):
        document = json.loads((FIXTURE / "evidence/assets/assets.json").read_text("utf-8"))
        document["assets"][0]["payload"]["claim"] = "现在旅行的矛盾是移动越来越快，深度连接却越来越慢"
        document["assets"][0]["evidence"][0]["excerpt"] = "我发现，现在旅行的矛盾是移动越来越快，深度连接却越来越慢"
        errors = MODULE.validate_asset_bundle(FIXTURE, document, False)
        self.assertFalse(any("viewpoint claim must be a synthesized paraphrase" in item for item in errors), errors)

    def test_video_chapter_accepts_current_and_legacy_transcript_schemas(self):
        document = {
            "schema_version": "chaptered-transcript/v3",
            "status": "reviewed",
            "chapters": [{
                "id": "chapter-01",
                "title": "按偏好筛选地点",
                "paragraphs": ["先记录用户的偏好，再筛选合适的地点。"],
                "source_segment_ids": ["asr-0001"],
                "source_segment_range": [0, 0],
                "source_segment_locators": [{
                    "segment_id": "asr-0001",
                    "segment_index": 0,
                    "start": 0,
                    "end": 3,
                    "excerpt": "先记录用户的偏好，再筛选合适的地点。",
                }],
                "start": 0,
                "end": 3,
                "title_status": "reviewed",
                "text_fidelity": "verbatim_corrected",
            }],
        }
        with tempfile.TemporaryDirectory() as temp:
            root = Path(temp)
            path = root / "evidence/transcript/逐字稿-章节版.json"
            path.parent.mkdir(parents=True)
            for schema_version in ("chaptered-transcript/v3", "chaptered-transcript/v2"):
                document["schema_version"] = schema_version
                path.write_text(json.dumps(document, ensure_ascii=False), "utf-8")
                errors = MODULE.validate_chapters(root, path, "video", False)
                self.assertFalse(
                    any("chapter schema_version" in item for item in errors),
                    errors,
                )


if __name__ == "__main__":
    unittest.main()