# Cohort Profile Template

Fill in the sections below to describe the cohort database. This profile drives
the intersection matrix and feasibility checks.

---

## Basic Information

- **Cohort name:**
- **Institution/Organization:**
- **Country:**
- **Population type:** (general population / health checkup / disease registry / claims data / hospital EMR)
- **Enrollment period:** (e.g., 2002-2019)
- **Total N at baseline:**
- **N with follow-up data:**
- **Mean/median follow-up duration:**
- **Measurement intervals:** (e.g., annual, biennial, at-event)

## Variable Categories

Check all that apply and list key variables in each category:

- [ ] **Demographics**: (age, sex, BMI, smoking, alcohol, exercise, income, education)
- [ ] **Laboratory**: (CBC, metabolic panel, lipid panel, liver function, kidney function, tumor markers, HbA1c, ...)
- [ ] **Imaging**: (chest X-ray, CT, ultrasound, DEXA, mammography, ...)
- [ ] **Questionnaires**: (PHQ-9, IPAQ, diet, sleep, quality of life, ...)
- [ ] **Vital signs**: (BP, heart rate, ...)
- [ ] **Anthropometry**: (height, weight, waist circumference, body composition, ...)
- [ ] **Medications**: (prescription records, drug categories, ...)
- [ ] **Procedures**: (surgery codes, intervention records, ...)
- [ ] **Diagnoses**: (ICD codes, physician diagnosis, ...)

## Endpoints Available

Check all that apply:

- [ ] **All-cause mortality** (linkage to: ___)
- [ ] **Cause-specific mortality** (categories: ___)
- [ ] **Cancer incidence** (linkage to: ___)
- [ ] **Cardiovascular events** (definition: ___)
- [ ] **Hospitalization** (source: ___)
- [ ] **Disease incidence** (ICD-based / physician-confirmed / registry)
- [ ] **Other**: ___

## Special Strengths

What makes this cohort unique? (check all that apply)

- [ ] **Serial measurements** (same variables measured repeatedly over time)
- [ ] **Large scale** (>100K participants)
- [ ] **Long follow-up** (>10 years)
- [ ] **National registry linkage** (mortality, cancer, insurance claims)
- [ ] **Screening-based** (no referral bias — general population health checkups)
- [ ] **Unique population** (ethnicity, occupation, geography not well-studied)
- [ ] **Rich phenotyping** (imaging + labs + questionnaires)
- [ ] **Biobank/genetic data available**
- [ ] **Other**: ___

## Known Limitations

- [ ] **Healthy volunteer bias** (participants may be healthier than general population)
- [ ] **Attrition** (estimated dropout rate: ___%)
- [ ] **Missing data** (key variables with >20% missing: ___)
- [ ] **Limited demographics** (e.g., single sex, narrow age range, single institution)
- [ ] **Claims-only diagnoses** (no clinical validation of ICD codes)
- [ ] **No imaging data**
- [ ] **No medication data**
- [ ] **Other**: ___

## Existing Publications

List known papers already published from this cohort (to avoid topic duplication):

1. (Author, Year, Topic, Journal)
2. ...

## Data Access

- **IRB status:** (approved / needs application)
- **Access method:** (on-site analysis center / remote access / direct download)
- **Estimated turnaround:** (application to data receipt)
- **Cost:** (if applicable)

---

## Variable Cluster Map (Auto-generated)

If a codebook / data dictionary / CSV export is available, do not fill this in by hand
and do not summarise the file yourself — run the input adapter:

```bash
python3 "${CLAUDE_SKILL_DIR}/scripts/build_cohort_profile.py" --codebook <file> --out-dir .
```

It writes `cohort_profile.md` + `cohort_profile.json` with:

| Section | Content |
|---------|---------|
| Variable cluster map | every variable copied **verbatim**, with its source (`file:row`) and the keyword that placed it in its cluster |
| Serial / repeated measures | measurement groups that genuinely repeat (evidence for **P1**) |
| Endpoint candidates | mortality / cancer / CVD / hospitalisation variables (evidence for **P2**) |
| `[UNKNOWN]` list | sample size, follow-up, IRB, prior publications — **ask the user; never guess** |

Variables that match no cluster keyword are reported as `unclassified` rather than forced
into a bucket. Review them: the lexicon is not exhaustive, and a mis-clustered exposure
variable will distort the intersection matrix.

# Research Topic Proposal — One-Pager

## [Working Title]

**Rank:** #X of Y | **6-Pattern Score:** X/6 | **Saturation Grade:** Green Field
**Target Journal:** [Journal Name] | **Estimated Timeline:** X months

---

### Background

[3-4 sentences establishing the clinical problem, current evidence gaps, and why this
topic matters now. End with the "Zero Papers" claim if applicable: "To our knowledge,
no study has examined [specific gap] using serial measurements in a [population type]."]

### Comparison Table

| Feature | Author1 (Year) | Author2 (Year) | Author3 (Year) | **THIS STUDY** |
|---------|----------------|----------------|----------------|----------------|
| Design | | | | |
| N | | | | |
| Serial data | | | | |
| Hard endpoint | | | | |
| Population | | | | |
| Ethnicity | | | | |
| Follow-up | | | | |
| Key gap | | | | |

**Unique differentiators: X** (minimum 3 required)

### Objective

**Primary:** [PICO format research question]

**Secondary (optional):** [1-2 secondary questions]

### Methods Summary

- **Study design:** Retrospective cohort study
- **Population:** [Inclusion/exclusion criteria]
- **Exposure:** [Key variable(s) and operational definition]
- **Outcome:** [Primary endpoint and ascertainment method]
- **Statistical approach:** [Key methods — Cox regression, trajectory analysis, etc.]
- **Sample size justification:** [N eligible, expected events, EPV ratio]

### 6-Pattern Score Card

| Pattern | Status | Evidence |
|---------|--------|----------|
| P1 Longitudinal Advantage | [+/-] | [one-line justification] |
| P2 Endpoint Upgrade | [+/-] | [one-line justification] |
| P3 Cohort Uniqueness | [+/-] | [one-line justification] |
| P4 PI-Topic Alignment | [+/-] | [one-line justification] |
| P5 Comparison Table 3+ | [+/-] | [one-line justification] |
| P6 Complementary Design | [+/-] | [one-line justification] |

### PI Role

**Corresponding Author:** [Name, Title, Affiliation]
**Relevance:** [Why this PI is the right CA — society role, expertise, journal connections]

### Feasibility

- **Go / Conditional Go / No-Go**
- **Sample size:** N = X, expected events = Y, EPV = Z
- **Key variables:** Available / needs derivation / missing
- **IRB:** Covered / needs new application
- **Data access:** Ready / X weeks to obtain

### Timeline

| Phase | Duration | Milestone |
|-------|----------|-----------|
| Data preparation | X weeks | Clean dataset, operational definitions |
| Analysis | X weeks | Primary + sensitivity analyses |
| Drafting | X weeks | Full manuscript |
| Internal review | X weeks | Co-author feedback |
| Submission | Target date | [Journal] |

### Clinical Implications ("So What")

1. [Implication for clinical practice]
2. [Implication for screening/prevention policy]
3. [Implication for future research directions]

# 6-Pattern Scoring Rubric

## Overview

Score each pattern 0 (absent) or 1 (present). Total: 0-6 points.
Interpretation: 5-6 = top-tier, 3-4 = specialty journal, 1-2 = restructure or kill.

---

## P1: Longitudinal Advantage

**Question:** Does the cohort's serial/repeated measurement structure create a clear
edge over existing cross-sectional studies?

**Score 1 if ALL of:**
- The cohort has 3+ measurement timepoints for the key exposure variable
- No prior study on this topic used serial/trajectory data
- The research question benefits from temporal modeling (change over time, trajectory
  clusters, time-to-event with time-varying exposure)

**Score 0 if ANY of:**
- The exposure is a one-time measurement (e.g., genetic variant, birth weight)
- Prior longitudinal studies already exist for this topic
- Serial data adds no interpretive value (e.g., stable demographic variable)

**Examples:**
- Score 1: Serial body composition → sarcopenia trajectory → mortality (no prior serial study)
- Score 0: Blood type → cancer risk (blood type doesn't change over time)

**Theoretical basis:** Repeated measures increase statistical efficiency by reducing
within-subject variance and enabling trajectory-based phenotyping that cross-sectional
designs cannot achieve (Lee et al., 2014, PMID 25464127).

---

## P2: Endpoint Upgrade

**Question:** Can we escalate to a harder endpoint than existing studies?

**Score 1 if BOTH of:**
- The cohort links to mortality, cancer, or major cardiovascular event registries
- Existing studies on this topic used only surrogate endpoints (biomarkers, imaging
  findings, composite scores) without hard clinical outcomes

**Score 0 if ANY of:**
- The cohort lacks hard endpoint linkage
- Prior studies already reported hard endpoints for this topic
- The research question is inherently about a surrogate (e.g., mechanism study)

**Examples:**
- Score 1: Existing studies link fatty liver to liver enzymes only; our cohort links to
  liver-related mortality and HCC incidence
- Score 0: Existing studies already report all-cause mortality for this exposure

**Endpoint hierarchy** (strongest to weakest):
1. All-cause mortality
2. Cause-specific mortality
3. Major adverse events (MACE, cancer diagnosis)
4. Hospitalization
5. Disease incidence (physician diagnosis)
6. Surrogate markers (lab values, imaging scores)

---

## P3: Cohort Uniqueness

**Question:** Is the cohort's population, scale, or setting distinctive?

**Score 1 if ANY of:**
- Largest cohort for this topic (>5x larger than existing studies)
- First study in this ethnic/geographic population
- Screening-based population (no referral bias) when prior studies used hospital cohorts
- Unique data linkage not available elsewhere (e.g., national registry + health checkup)
- Community-dwelling general population when prior studies used disease-specific cohorts

**Score 0 if:**
- Similar-sized cohorts with the same population type have published on this topic

**Examples:**
- Score 1: 486K health checkup participants vs existing studies of 3-9K referral patients
- Score 0: Another 500K cohort from the same country already published on this topic

---

## P4: PI-Topic Alignment

**Question:** Does the PI's expertise and reputation strengthen this topic?

**Score 1 if ANY of:**
- PI holds a society leadership role directly relevant to the topic
- PI has 5+ first/corresponding author papers in this specific domain
- PI is an editorial board member of a target journal in this field

**Score 0 if:**
- PI's expertise is only tangentially related
- No specific PI identified (skip this pattern; score out of 5 instead)

**Why this matters:** A PI with society standing in the topic area signals that the
study has expert oversight. Editors recognize this. The PI's name also guides target
journal selection (e.g., hepatology society president -> J Hepatol).

**When no PI is specified:** Remove this pattern from scoring. Interpret:
4-5/5 = top-tier, 2-3/5 = specialty, 0-1/5 = restructure.

---

## P5: Comparison Table Gaps (3+)

**Question:** Does the THIS STUDY column show 3+ unique features vs all existing papers?

**Score 1 if:**
- The comparison table has at least 3 rows where THIS STUDY has a checkmark/advantage
  that NO prior paper has

**Score 0 if:**
- Fewer than 3 unique differentiators

**Common differentiator categories:**
1. Study design (longitudinal vs cross-sectional)
2. Sample size (order of magnitude larger)
3. Serial measurements (multiple timepoints vs single)
4. Hard endpoints (mortality vs surrogate)
5. Population type (screening vs referral)
6. Ethnicity/geography (first in this population)
7. Subgroup analyses (age/sex/comorbidity stratification)
8. Adjustment for key confounders (missing in prior studies)
9. Exposure definition (validated operational definition vs ICD-only)
10. Follow-up duration (significantly longer)

**Construction method:**
1. Identify 3-5 most relevant existing papers from saturation scan
2. Create table with Feature rows and Paper columns + THIS STUDY column
3. For each feature, check whether each paper and THIS STUDY address it
4. Count features unique to THIS STUDY

---

## P6: Complementary Design

**Question:** Can this topic pair with another study from the same cohort?

**Score 1 if ANY of:**
- A complementary analysis using the same DB but different population subset is
  feasible (e.g., diabetic vs non-diabetic; viral vs non-viral liver disease)
- The same exposure can be studied against a different outcome in a companion paper
- The topic creates a "series" with a previously published paper from the same cohort

**Score 0 if:**
- The topic is standalone with no natural complement
- The complementary analysis would be trivially similar (not publishable separately)

**Why this matters:** Paired papers from the same cohort strengthen both: the second
paper can reference the first as "in this cohort, we previously showed..." and reviewers
see a programmatic research line, not a one-off analysis.

---

## Quick Reference Card

```
Pattern         | Key Signal
----------------|------------------------------------------
P1 Longitudinal | "No prior study used serial data for this"
P2 Endpoint     | "We add mortality/cancer to surrogate-only literature"
P3 Uniqueness   | "Largest / first in this population / no referral bias"
P4 PI Alignment | "PI is society president in this exact field"
P5 Comparison   | "3+ checkmarks unique to THIS STUDY"
P6 Complement   | "Natural pair study exists in same DB"
```

# Literature Saturation Query Templates

## Purpose

These templates help construct PubMed queries for the Phase 3 saturation scan.
Adapt the bracketed terms to the specific topic.

---

## Basic Saturation Query

```
([exposure MeSH] OR [exposure free text]) AND ([outcome MeSH] OR [outcome free text])
AND (cohort OR longitudinal OR prospective OR "follow-up")
```

**Filters:** English, Humans, last 20 years (to capture the full landscape)

## Longitudinal-Specific Query

To check if anyone has used serial/repeated measurements for this topic:

```
([exposure] OR [exposure synonym]) AND ([outcome] OR [outcome synonym])
AND ("repeated measure*" OR "serial" OR "trajectory" OR "longitudinal change"
OR "time-varying" OR "growth curve" OR "latent class trajectory")
```

## Meta-Analysis Check Query

To verify if a meta-analysis already exists:

```
([exposure] OR [exposure synonym]) AND ([outcome] OR [outcome synonym])
AND ("meta-analysis"[Publication Type] OR "systematic review"[Publication Type])
```

If a meta-analysis exists → Red grade (avoid unless doing NMA).

## Population-Specific Queries

### Korean/Asian population filter
```
AND (Korea* OR Korean OR "Republic of Korea" OR Asia* OR Japan* OR China OR Chinese
OR Taiwan*)
```

### Health checkup / screening population filter
```
AND ("health checkup" OR "health screening" OR "health examination" OR "medical checkup"
OR "periodic health exam*" OR "annual exam*")
```

### Large cohort filter (to find comparator studies)
```
AND ("national health insurance" OR "claims data" OR "administrative data"
OR "population-based" OR "nationwide" OR "registry")
```

---

## Saturation Grading Protocol

After running the basic saturation query:

### Step 1: Count total results
- 0-2: Blue Ocean
- 3-10: Possible Green Field (proceed to Step 2)
- 10-30: Possible Yellow (proceed to Step 2)
- 30+: Likely Red (check for MA in Step 3)

### Step 2: Check longitudinal gap
Run the longitudinal-specific query.
- 0 results with serial/trajectory data → upgrade one grade
- 1-2 results → maintain current grade
- 3+ results → no upgrade

### Step 3: Check meta-analysis existence
Run the MA check query.
- MA exists and is comprehensive → Red (firm)
- MA exists but outdated (>5 years) or limited scope → Yellow (update MA possible)
- No MA → maintain current grade

### Step 4: Final grade assignment

| Base Count | Longitudinal Papers | MA Exists? | Final Grade |
|------------|-------------------|------------|-------------|
| 0-2 | 0 | No | Blue Ocean |
| 3-10 | 0 | No | **Green Field** |
| 3-10 | 1-2 | No | Yellow |
| 10-30 | 0 | No | Green Field (upgraded) |
| 10-30 | 1-2 | No | Yellow |
| 10-30 | 3+ | No | Yellow |
| 30+ | Any | No | Yellow (borderline Red) |
| Any | Any | Yes (recent) | Red |
| Any | Any | Yes (outdated) | Yellow |

---

## Example: Fatty Liver and Cardiovascular Mortality

### Basic query
```
("fatty liver" OR "hepatic steatosis" OR NAFLD OR MASLD) AND
("cardiovascular mortality" OR "cardiac death" OR "MACE")
AND (cohort OR longitudinal OR prospective)
```
Result: ~45 papers → base grade Red

### Longitudinal check
```
("fatty liver" OR "hepatic steatosis") AND ("cardiovascular mortality")
AND ("trajectory" OR "serial" OR "repeated measure*" OR "longitudinal change")
```
Result: 2 papers → no upgrade

### MA check
```
("fatty liver" OR NAFLD) AND ("cardiovascular mortality")
AND ("meta-analysis"[PT] OR "systematic review"[PT])
```
Result: 3 MAs → confirmed Red

**Conclusion:** Avoid this topic unless using truly unique data angle.

---

## Tips for Effective Saturation Scanning

1. **Start broad, then narrow.** If the broad query returns >30, add population or
   design filters to find the exact niche.

2. **Check the "last 3 years" subset.** A topic with 20 total papers but 15 in the
   last 3 years is trending (good for timeliness, bad for novelty).

3. **Read the most recent review article.** It maps the field faster than scanning
   individual papers. Look for "future research directions" sections.

4. **Check for registered protocols.** Search PROSPERO or ClinicalTrials.gov for
   ongoing studies that haven't published yet — these are invisible competitors.

5. **Use Semantic Scholar** for citation network analysis. A paper with 200+ citations
   on this exact topic means the field is well-established.

#!/usr/bin/env python3
"""Build a cohort profile from a local codebook (and optional domain context).

`/find-cohort-gap` used to start from a *named* database — NHIS, UK Biobank, and the
handful of registries the skill knows about. Most researchers do not have one of those.
They have an institutional registry, a single-centre EMR export, or a specialty cohort,
described by a data dictionary nobody else has ever seen (issue #69).

This is the input layer that lets them in. It reads their codebook and emits the same
cohort profile the skill already consumes, so everything downstream — the intersection
matrix, saturation scan, 6-pattern scoring — is unchanged.

WHY A SCRIPT AND NOT JUST READING THE FILE. A language model asked to "summarise this
codebook" will paraphrase a variable name, merge two variables that look alike, or
quietly invent one that the cohort does not have — and every downstream claim inherits
it. So the variable inventory is *enumerated*, never generated: each variable is copied
verbatim from the file and carries its provenance (`file:row`), which is exactly the
dictionary-first discipline that a reviewer expects of a DB-backed study.

WHAT IT REFUSES TO DO. A codebook lists variables; it does not state the sample size,
the enrollment window, the follow-up duration, or the IRB status. Those are emitted as
`[UNKNOWN - ask the user]`, never guessed — a fabricated N is worse than a missing one,
because it survives all the way to a feasibility gate that then passes for the wrong
reason.

WHAT IT INFERS (and shows its work). Two structural facts *are* derivable from variable
names alone, and both feed patterns the skill already scores:

  * serial / repeated measures (`bp_v1`, `bp_v2`, `visit2_hba1c`, ...) -> P1 Longitudinal
    Advantage. Reported as the actual variable groups, so the claim is auditable.
  * endpoint-like variables (`death_date`, `cvd_event`, `cancer_incidence`) -> P2
    Endpoint Upgrade.

Every cluster assignment records the keyword that triggered it, and anything that matches
nothing is left `unclassified` rather than forced into a bucket.

Usage:
    build_cohort_profile.py --codebook dict.csv [--codebook more.xlsx ...] \\
        [--context review.pdf --context https://example.org/guideline] \\
        [--cohort-name "Institutional CT registry"] [--out-dir .]

Formats: .csv / .tsv / .json / .md / .txt (stdlib), .xlsx (needs openpyxl),
.pdf (needs `pdftotext` from poppler). Context URLs are fetched with stdlib urllib;
a paywalled or JavaScript-rendered page will not extract, and that is reported rather
than papered over.
"""

from __future__ import annotations

import argparse
import csv
import html.parser
import json
import re
import shutil
import subprocess
import sys
import urllib.error
import urllib.request
from pathlib import Path

UNKNOWN = "[UNKNOWN - ask the user]"

# Variable-name lexicons. Each cluster maps to the template's category names, and the
# matched keyword is recorded so a wrong assignment is visible instead of silent.
#
# Short keys are matched as WHOLE TOKENS, not substrings (see `classify`). Substring
# matching on a two-letter abbreviation is a false-positive machine: `us` (ultrasound)
# fires on `statin_use`, and `age` fires on `storage_temp`. Long keys keep substring
# matching so that `smok` still catches `smoking_status`.
CLUSTERS: dict[str, tuple[str, ...]] = {
    "demographics": (
        "age", "sex", "gender", "birth", "race", "ethnic", "income", "educat",
        "marital", "occupation", "smok", "alcohol", "drink", "exercise", "activity",
    ),
    "anthropometry": ("height", "weight", "bmi", "waist", "hip", "circumf", "body_fat", "muscle"),
    "vital_signs": ("sbp", "dbp", "bp", "blood_pressure", "pulse", "heart_rate", "hr", "resp_rate", "temp"),
    "laboratory": (
        "glucose", "glu", "hba1c", "chol", "lipid", "ldl", "hdl", "triglyc", "tg", "ast", "alt",
        "ggt", "bilirubin", "albumin", "creatinin", "egfr", "bun", "uric", "crp", "esr",
        "hb", "hgb", "hct", "wbc", "rbc", "platelet", "psa", "cea", "afp", "ca19", "tsh",
        "insulin", "lab",
    ),
    "imaging": (
        "ct", "mri", "xray", "x_ray", "cxr", "ultrasound", "us", "sono", "dexa",
        "dxa", "mammo", "echo", "angio", "pet", "cac", "calcium_score", "imaging", "radiol",
    ),
    "questionnaire": (
        "phq", "gad", "psqi", "ipaq", "sf36", "sf_36", "eq5d", "eq_5d", "qol", "questionn",
        "survey", "scale", "sleep", "diet", "food_freq", "ffq", "stress", "depress",
    ),
    "medication": ("med", "drug", "rx", "prescri", "statin", "antihyp", "therapy", "treat"),
    "procedure": ("surg", "operat", "procedur", "biopsy", "resect", "ablat", "stent", "pci", "cabg", "intervention"),
    "diagnosis": ("icd", "diag", "dx", "disease", "hypertension", "diabetes", "history_of", "hx", "comorbid"),
    "identifier_admin": ("id", "key", "index", "seq", "code", "visit_date", "site", "center"),
}

# Below this length a keyword is matched as a whole token, never as a substring.
TOKEN_ONLY = 4

# Endpoint-like variables. A cohort's value proposition is usually its hard endpoints, and
# the skill scores that (P2), so they get both a cluster of their own (`outcome_endpoint`,
# assigned only when no other cluster claims the variable) and an explicit candidate list.
ENDPOINT_HINTS = (
    "death", "mortal", "expire", "died", "survival", "cancer", "malign", "incid",
    "cvd", "chd", "mace", "stroke", "myocard", "mi", "infarct", "event", "outcome",
    "hospital", "admission", "readmit", "recurrence", "progression", "relapse",
)

# Serial / repeated measures: <stem><separator><index>, or an explicit visit/wave marker.
SERIAL_SUFFIX = re.compile(r"^(?P<stem>.+?)[ _\-.]?(?:v|t|w|wave|visit|yr|y|round|r|time)?(?P<idx>\d{1,2})$", re.I)
SERIAL_PREFIX = re.compile(r"^(?:v|t|w|wave|visit|yr|y|round|time)(?P<idx>\d{1,2})[ _\-.](?P<stem>.+)$", re.I)

# A column in a codebook that holds the variable NAME (as opposed to its description).
NAME_COL = re.compile(r"^\s*(variable|var|var_?name|name|field|field_?name|column|col|item|code)\s*$", re.I)
DESC_COL = re.compile(r"^\s*(desc|description|label|definition|meaning|explanation|comment|note|한글|설명)", re.I)


class _Strip(html.parser.HTMLParser):
    """Minimal HTML -> text. Not a browser: a JS-rendered page yields nothing, and the
    caller is told so rather than being handed an empty 'context'."""

    def __init__(self) -> None:
        super().__init__()
        self.chunks: list[str] = []
        self._skip = 0

    def handle_starttag(self, tag, attrs):
        if tag in ("script", "style", "nav", "footer"):
            self._skip += 1

    def handle_endtag(self, tag):
        if tag in ("script", "style", "nav", "footer") and self._skip:
            self._skip -= 1

    def handle_data(self, data):
        if not self._skip and data.strip():
            self.chunks.append(data.strip())


# --------------------------------------------------------------------------------------
# Variable extraction — one function per format, all returning (name, description, where)
# --------------------------------------------------------------------------------------

Var = tuple[str, str, str]


def _from_delimited(path: Path, delim: str) -> list[Var]:
    """A .csv/.tsv is either a CODEBOOK (rows are variables) or a DATA export (columns are
    variables). Decide by whether a header column names a variable column."""
    with path.open(newline="", encoding="utf-8-sig", errors="replace") as fh:
        rows = list(csv.reader(fh, delimiter=delim))
    if not rows:
        return []
    header = rows[0]
    name_idx = next((i for i, h in enumerate(header) if NAME_COL.match(h or "")), None)

    if name_idx is None:  # data export: the header row IS the variable list
        return [
            (h.strip(), "", f"{path.name}:1 (column {i + 1})")
            for i, h in enumerate(header)
            if h and h.strip()
        ]

    desc_idx = next((i for i, h in enumerate(header) if DESC_COL.match(h or "")), None)
    out: list[Var] = []
    for r, row in enumerate(rows[1:], start=2):
        if name_idx >= len(row) or not (row[name_idx] or "").strip():
            continue
        desc = row[desc_idx].strip() if desc_idx is not None and desc_idx < len(row) else ""
        out.append((row[name_idx].strip(), desc, f"{path.name}:{r}"))
    return out


def _from_json(path: Path) -> list[Var]:
    data = json.loads(path.read_text(encoding="utf-8"))
    out: list[Var] = []
    if isinstance(data, dict):
        for i, (k, v) in enumerate(data.items(), start=1):
            desc = v if isinstance(v, str) else (v.get("description", "") if isinstance(v, dict) else "")
            out.append((str(k), str(desc), f"{path.name}:key {i}"))
    elif isinstance(data, list):
        for i, item in enumerate(data, start=1):
            if isinstance(item, str):
                out.append((item, "", f"{path.name}:item {i}"))
            elif isinstance(item, dict):
                name = next((item[k] for k in ("variable", "var", "name", "field", "column") if k in item), None)
                if name:
                    desc = next((item[k] for k in ("description", "desc", "label", "definition") if k in item), "")
                    out.append((str(name), str(desc), f"{path.name}:item {i}"))
    return out


def _from_markdown(text: str, origin: str) -> list[Var]:
    """A markdown/plain codebook: a pipe table, or `var` — description lines."""
    out: list[Var] = []
    for i, line in enumerate(text.splitlines(), start=1):
        s = line.strip()
        if not s or set(s) <= set("|-: "):  # separator row
            continue
        if s.startswith("|"):
            cells = [c.strip().strip("`*") for c in s.strip("|").split("|")]
            if len(cells) >= 1 and cells[0] and not NAME_COL.match(cells[0]):
                out.append((cells[0], cells[1] if len(cells) > 1 else "", f"{origin}:{i}"))
            continue
        m = re.match(r"^[-*+]?\s*`([^`]+)`\s*[-–—:]?\s*(.*)$", s)
        if m:
            out.append((m.group(1).strip(), m.group(2).strip(), f"{origin}:{i}"))
    return out


def _from_xlsx(path: Path) -> list[Var]:
    try:
        from openpyxl import load_workbook  # type: ignore
    except ImportError:
        raise SystemExit(
            f"reading {path.name} needs openpyxl (`pip install openpyxl`). "
            "Or export the sheet to .csv and pass that — the CSV path is stdlib-only."
        )
    ws = load_workbook(path, read_only=True, data_only=True).active
    rows = [[("" if c is None else str(c)) for c in row] for row in ws.iter_rows(values_only=True)]
    if not rows:
        return []
    header = rows[0]
    name_idx = next((i for i, h in enumerate(header) if NAME_COL.match(h or "")), None)
    if name_idx is None:
        return [(h.strip(), "", f"{path.name}:1 (column {i + 1})") for i, h in enumerate(header) if h.strip()]
    desc_idx = next((i for i, h in enumerate(header) if DESC_COL.match(h or "")), None)
    out: list[Var] = []
    for r, row in enumerate(rows[1:], start=2):
        if name_idx >= len(row) or not row[name_idx].strip():
            continue
        desc = row[desc_idx].strip() if desc_idx is not None and desc_idx < len(row) else ""
        out.append((row[name_idx].strip(), desc, f"{path.name}:{r}"))
    return out


def _pdf_text(path: Path) -> str:
    if not shutil.which("pdftotext"):
        raise SystemExit(
            f"reading {path.name} needs `pdftotext` (poppler: `brew install poppler` / "
            "`apt install poppler-utils`). Or convert the PDF to .md / .txt and pass that."
        )
    p = subprocess.run(["pdftotext", "-layout", str(path), "-"], capture_output=True, text=True)
    if p.returncode != 0:
        raise SystemExit(f"pdftotext failed on {path.name}: {p.stderr.strip()}")
    return p.stdout


def read_codebook(path: Path) -> list[Var]:
    if not path.is_file():
        raise SystemExit(f"not found: {path}")
    suf = path.suffix.lower()
    if suf == ".csv":
        return _from_delimited(path, ",")
    if suf in (".tsv", ".tab"):
        return _from_delimited(path, "\t")
    if suf == ".json":
        return _from_json(path)
    if suf == ".xlsx":
        return _from_xlsx(path)
    if suf == ".pdf":
        return _from_markdown(_pdf_text(path), path.name)
    if suf in (".md", ".markdown", ".txt"):
        return _from_markdown(path.read_text(encoding="utf-8", errors="replace"), path.name)
    raise SystemExit(f"unsupported codebook format: {path.suffix} ({path.name})")


# --------------------------------------------------------------------------------------
# Structure inference — clusters, serial groups, endpoints
# --------------------------------------------------------------------------------------


def _tokens(s: str) -> set[str]:
    return set(re.split(r"[^a-z0-9]+", s.lower())) - {""}


def _hits(key: str, hay: str, toks: set[str]) -> bool:
    """A short key must BE a token; a long key may appear anywhere.

    This is the whole defence against the abbreviation false positive: `us` matching
    `statin_use`, `age` matching `storage_temp`, `id` matching `lipid`.
    """
    if len(key) >= TOKEN_ONLY or "_" in key:
        return key in hay
    return key in toks


def classify(name: str, desc: str) -> tuple[str, str]:
    """Return (cluster, matched_keyword). Unmatched stays 'unclassified' — a variable is
    never forced into a bucket to make the map look complete.

    Endpoints are resolved LAST, and only against variables no other cluster claimed, so a
    cohort's hard endpoint lands in `outcome_endpoint` instead of `unclassified`. Otherwise
    the profile contradicts itself: `death_date` would be listed as "matched nothing —
    review it" in the cluster map and as the P2 evidence two sections below.
    """
    hay = f"{name} {desc}".lower()
    toks = _tokens(hay)
    for cluster, keys in CLUSTERS.items():
        for k in keys:
            if _hits(k, hay, toks):
                return cluster, k
    kw = is_endpoint(name, desc)
    if kw:
        return "outcome_endpoint", kw
    return "unclassified", ""


def is_endpoint(name: str, desc: str) -> str:
    hay = f"{name} {desc}".lower()
    toks = _tokens(hay)
    return next((k for k in ENDPOINT_HINTS if _hits(k, hay, toks)), "")


def serial_groups(names: list[str]) -> dict[str, list[str]]:
    """Group variables that look like the same measurement repeated over time.

    Only a stem seen with >= 2 distinct indices counts: a lone `visit1_bp` is not evidence
    of serial data, and claiming otherwise would hand P1 a free point it has not earned.
    """
    stems: dict[str, dict[str, str]] = {}
    for n in names:
        for rx in (SERIAL_PREFIX, SERIAL_SUFFIX):
            m = rx.match(n)
            if m:
                stem = m.group("stem").strip("_- .").lower()
                if stem:
                    stems.setdefault(stem, {})[m.group("idx")] = n
                break
    return {
        stem: [v for _, v in sorted(idx.items(), key=lambda kv: int(kv[0]))]
        for stem, idx in stems.items()
        if len(idx) >= 2
    }


# --------------------------------------------------------------------------------------
# Domain context (a review, a guideline page) — extracted, never summarised here
# --------------------------------------------------------------------------------------


def read_context(src: str) -> tuple[str, str]:
    """Return (text, provenance). The LLM reads this; the script only fetches it."""
    if src.startswith(("http://", "https://")):
        try:
            req = urllib.request.Request(src, headers={"User-Agent": "medsci-skills/find-cohort-gap"})
            with urllib.request.urlopen(req, timeout=30) as r:  # noqa: S310 - user-supplied context URL
                raw = r.read().decode("utf-8", errors="replace")
        except (urllib.error.URLError, TimeoutError, ValueError) as exc:
            raise SystemExit(
                f"could not fetch {src}: {exc}\n"
                "If the page is paywalled or JavaScript-rendered (UpToDate and many guideline "
                "portals are), save it as PDF/markdown and pass the file instead."
            )
        p = _Strip()
        p.feed(raw)
        text = "\n".join(p.chunks)
        if len(text.split()) < 50:
            raise SystemExit(
                f"{src} yielded almost no text ({len(text.split())} words) — it is probably "
                "JavaScript-rendered or paywalled. Save it as PDF/markdown and pass the file."
            )
        return text, src
    path = Path(src)
    if not path.is_file():
        raise SystemExit(f"not found: {src}")
    if path.suffix.lower() == ".pdf":
        return _pdf_text(path), path.name
    return path.read_text(encoding="utf-8", errors="replace"), path.name


# --------------------------------------------------------------------------------------


def build(codebooks: list[Path], contexts: list[str], cohort_name: str | None) -> tuple[dict, str]:
    variables: list[dict] = []
    seen: set[str] = set()
    for cb in codebooks:
        for name, desc, where in read_codebook(cb):
            if name.lower() in seen:
                continue
            seen.add(name.lower())
            cluster, kw = classify(name, desc)
            variables.append(
                {
                    "name": name,
                    "description": desc,
                    "source": where,
                    "cluster": cluster,
                    "matched_keyword": kw,
                    "endpoint_hint": is_endpoint(name, desc),
                }
            )

    names = [v["name"] for v in variables]
    serial = serial_groups(names)
    endpoints = [v for v in variables if v["endpoint_hint"]]

    by_cluster: dict[str, list[str]] = {}
    for v in variables:
        by_cluster.setdefault(v["cluster"], []).append(v["name"])

    ctx: list[dict] = []
    for c in contexts:
        text, prov = read_context(c)
        ctx.append({"source": prov, "words": len(text.split()), "text": text})

    profile = {
        "cohort_name": cohort_name or UNKNOWN,
        "codebooks": [str(c) for c in codebooks],
        "n_variables": len(variables),
        "variables": variables,
        "clusters": {k: sorted(v) for k, v in sorted(by_cluster.items())},
        "serial_groups": serial,
        "endpoint_candidates": [v["name"] for v in endpoints],
        "context_documents": [{"source": c["source"], "words": c["words"]} for c in ctx],
        # Not derivable from a codebook. Guessing any of these corrupts the feasibility gate.
        "must_ask_user": {
            "n_baseline": UNKNOWN,
            "n_with_followup": UNKNOWN,
            "enrollment_period": UNKNOWN,
            "followup_duration": UNKNOWN,
            "measurement_intervals": UNKNOWN,
            "irb_status": UNKNOWN,
            "existing_publications": UNKNOWN,
            "known_limitations": UNKNOWN,
        },
    }
    return profile, "\n\n".join(f"# Context: {c['source']}\n\n{c['text']}" for c in ctx)


def render_markdown(p: dict) -> str:
    L = [
        "# Cohort Profile (auto-generated)",
        "",
        f"**Cohort name:** {p['cohort_name']}",
        f"**Codebook(s):** {', '.join(p['codebooks']) or UNKNOWN}",
        f"**Variables enumerated:** {p['n_variables']}",
        "",
        "> Every variable below is copied verbatim from the codebook and carries its source",
        "> location. Nothing here is inferred except the cluster assignment and the serial /",
        "> endpoint flags, each of which shows the keyword that triggered it.",
        "",
        "## Variable Cluster Map",
        "",
        "| Cluster | N | Variables |",
        "|---------|--:|-----------|",
    ]
    for cluster, vs in p["clusters"].items():
        shown = ", ".join(f"`{v}`" for v in vs[:12])
        if len(vs) > 12:
            shown += f", … (+{len(vs) - 12})"
        L.append(f"| {cluster} | {len(vs)} | {shown} |")

    L += ["", "## Serial / repeated measures (evidence for P1 Longitudinal Advantage)", ""]
    if p["serial_groups"]:
        L += ["| Measurement | Timepoints | Variables |", "|-------------|-----------:|-----------|"]
        for stem, vs in sorted(p["serial_groups"].items()):
            L.append(f"| {stem} | {len(vs)} | {', '.join(f'`{v}`' for v in vs)} |")
    else:
        L.append("None detected from variable names. If the cohort *does* have repeated measures,")
        L.append("say so — the naming convention may simply not encode the timepoint.")

    L += ["", "## Endpoint candidates (evidence for P2 Endpoint Upgrade)", ""]
    if p["endpoint_candidates"]:
        L += [", ".join(f"`{v}`" for v in p["endpoint_candidates"])]
    else:
        L.append("None detected. Without a hard endpoint, P2 cannot score and the cohort is")
        L.append("limited to cross-sectional questions — confirm before proceeding.")

    if p["context_documents"]:
        L += ["", "## Domain context supplied", ""]
        for c in p["context_documents"]:
            L.append(f"- {c['source']} ({c['words']:,} words) — see `context_extract.md`")

    L += [
        "",
        "## Not derivable from a codebook — ASK THE USER before Phase 2",
        "",
        "A data dictionary lists variables. It does not state any of the following, and a",
        "guessed value here would silently pass the Phase 5 feasibility gate for the wrong reason.",
        "",
    ]
    L += [f"- **{k.replace('_', ' ')}:** {v}" for k, v in p["must_ask_user"].items()]
    return "\n".join(L) + "\n"


def main() -> int:
    ap = argparse.ArgumentParser(
        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
    )
    ap.add_argument("--codebook", action="append", type=Path, required=True,
                    help="data dictionary / codebook / CSV header (repeatable)")
    ap.add_argument("--context", action="append", default=[],
                    help="domain context: a review/guideline file (.md/.txt/.pdf) or a URL (repeatable)")
    ap.add_argument("--cohort-name", help="what the cohort is called")
    ap.add_argument("--out-dir", type=Path, default=Path("."))
    a = ap.parse_args()

    profile, context_text = build(a.codebook, a.context, a.cohort_name)
    if not profile["n_variables"]:
        raise SystemExit(
            "no variables found. If this is a data export, the first row must be the header; "
            "if it is a codebook, one column must be named variable / var / name / field / column."
        )

    a.out_dir.mkdir(parents=True, exist_ok=True)
    (a.out_dir / "cohort_profile.json").write_text(json.dumps(profile, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")
    (a.out_dir / "cohort_profile.md").write_text(render_markdown(profile), encoding="utf-8")
    if context_text:
        (a.out_dir / "context_extract.md").write_text(context_text + "\n", encoding="utf-8")

    print(f"{profile['n_variables']} variables enumerated from {len(a.codebook)} codebook(s)")
    for cluster, vs in profile["clusters"].items():
        print(f"  {cluster:<18} {len(vs)}")
    print(f"  serial groups      {len(profile['serial_groups'])}")
    print(f"  endpoint candidates {len(profile['endpoint_candidates'])}")
    unclassified = len(profile["clusters"].get("unclassified", []))
    if unclassified:
        print(f"\n{unclassified} variable(s) matched no cluster — review them; the lexicon is not exhaustive.")
    print(f"\nwrote {a.out_dir / 'cohort_profile.md'} + cohort_profile.json")
    print("ASK THE USER for: " + ", ".join(k.replace("_", " ") for k in profile["must_ask_user"]))
    return 0


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env bash
# Regression test for skills/find-cohort-gap/scripts/build_cohort_profile.py — the
# local-codebook / document input layer (issue #69).
#
# The contract under test is as much about what the adapter REFUSES to do as what it
# extracts: variables are enumerated verbatim with provenance (never paraphrased or
# invented), a sample size that the codebook does not state stays [UNKNOWN], and a serial
# structure is only claimed when a measurement really does repeat.
set -u

REPO_ROOT="$(cd "$(dirname "$0")/../../.." && pwd)"
B="$REPO_ROOT/skills/find-cohort-gap/scripts/build_cohort_profile.py"
TMP="$(mktemp -d)"
trap 'rm -rf "$TMP"' EXIT

pass=0
fail=0
ck() {
  local label="$1" expected="$2" actual="$3"
  if [ "$expected" = "$actual" ]; then
    printf '  PASS  %-54s exit=%s\n' "$label" "$actual"
    pass=$((pass + 1))
  else
    printf '  FAIL  %-54s expected=%s actual=%s\n' "$label" "$expected" "$actual"
    fail=$((fail + 1))
  fi
}

# --- a CODEBOOK (rows are variables, one column names them) ---
cat > "$TMP/codebook.csv" <<'CSV'
variable,description,type
subject_id,Study identifier,char
age,Age at baseline in years,num
sex,Biological sex,char
sbp_v1,Systolic blood pressure at visit 1,num
sbp_v2,Systolic blood pressure at visit 2,num
sbp_v3,Systolic blood pressure at visit 3,num
hba1c,Glycated haemoglobin,num
ct_lung_nodule,Lung nodule seen on chest CT,char
phq9_total,PHQ-9 depression score,num
statin_use,Statin prescription at baseline,char
death_date,Date of death from national registry,date
cvd_event,Incident cardiovascular event,char
weird_unmatchable_token,,char
CSV

# --- a DATA EXPORT (columns are the variables; no codebook column) ---
printf 'age,sex,ldl_chol,mri_brain,death_date\n45,M,130,normal,\n' > "$TMP/export.csv"

# --- a markdown codebook (pipe table AND bullet/backtick lines) ---
cat > "$TMP/codebook.md" <<'MD'
| variable | description |
|----------|-------------|
| waist_circumference | Waist circumference (cm) |

Additional variables:

- `alt` — Alanine aminotransferase
MD

# --- a JSON codebook ---
cat > "$TMP/codebook.json" <<'JSON'
{"egfr": "Estimated glomerular filtration rate", "cancer_incidence": "Incident cancer from registry"}
JSON

# 1) a codebook is read and every variable is enumerated
python3 "$B" --codebook "$TMP/codebook.csv" --cohort-name "Test registry" --out-dir "$TMP/o1" > /dev/null 2>&1
ck "codebook (rows = variables) is read" 0 "$?"

python3 - "$TMP/o1/cohort_profile.json" <<'PY'
import json, sys
p = json.load(open(sys.argv[1]))
names = [v["name"] for v in p["variables"]]
assert len(names) == 13, f"expected 13 variables, got {len(names)}: {names}"
# verbatim, not paraphrased
for must in ("sbp_v1", "ct_lung_nodule", "phq9_total", "weird_unmatchable_token"):
    assert must in names, f"{must} missing — variables must be copied verbatim"
# provenance points at the real row
src = {v["name"]: v["source"] for v in p["variables"]}
assert src["age"].endswith(":3"), f"bad provenance for age: {src['age']}"
PY
ck "variables verbatim + provenance (file:row)" 0 "$?"

# 2) clusters are assigned, and an unmatchable variable is NOT forced into a bucket
python3 - "$TMP/o1/cohort_profile.json" <<'PY'
import json, sys
p = json.load(open(sys.argv[1]))
c = {v["name"]: v["cluster"] for v in p["variables"]}
assert c["age"] == "demographics", c["age"]
assert c["hba1c"] == "laboratory", c["hba1c"]
assert c["ct_lung_nodule"] == "imaging", c["ct_lung_nodule"]
assert c["phq9_total"] == "questionnaire", c["phq9_total"]
assert c["statin_use"] == "medication", c["statin_use"]
assert c["weird_unmatchable_token"] == "unclassified", c["weird_unmatchable_token"]
# a hard endpoint gets its own cluster — it must never read as "matched nothing, review it"
# in the cluster map while being cited as P2 evidence two sections below.
assert c["death_date"] == "outcome_endpoint", c["death_date"]
assert c["cvd_event"] == "outcome_endpoint", c["cvd_event"]
assert "death_date" not in p["clusters"].get("unclassified", []), "endpoint landed in unclassified"
# every assignment shows the keyword that caused it
assert all(v["matched_keyword"] for v in p["variables"] if v["cluster"] != "unclassified")
PY
ck "clusters assigned; endpoints get outcome_endpoint" 0 "$?"

# 2b) the abbreviation false positive: a short keyword must match a whole TOKEN, never a
# substring. `us` (ultrasound) inside `statin_use` once made a statin an imaging variable.
# -B: importing the module must not leave a __pycache__ artifact in the skill directory.
python3 -B - "$B" <<'PY'
import importlib.util, sys
spec = importlib.util.spec_from_file_location("bcp", sys.argv[1])
m = importlib.util.module_from_spec(spec); spec.loader.exec_module(m)
cases = [
    ("statin_use", "Statin prescription", "medication"),   # not imaging via "us"
    ("lipid_panel", "Lipid panel", "laboratory"),          # not identifier_admin via "id"
    ("ct_lung_nodule", "Chest CT nodule", "imaging"),      # token "ct" still works
    ("sbp_v1", "Systolic BP visit 1", "vital_signs"),
]
for name, desc, want in cases:
    got, kw = m.classify(name, desc)
    assert got == want, f"{name!r} -> {got} (via {kw!r}), expected {want}"
PY
ck "short keywords match whole tokens, not substrings" 0 "$?"

# 3) serial structure detected — the P1 evidence — and only when it really repeats
python3 - "$TMP/o1/cohort_profile.json" <<'PY'
import json, sys
p = json.load(open(sys.argv[1]))
sg = p["serial_groups"]
assert "sbp" in sg, f"sbp_v1/v2/v3 not detected as serial: {sg}"
assert sg["sbp"] == ["sbp_v1", "sbp_v2", "sbp_v3"], sg["sbp"]
# phq9_total ends in a digit but never repeats -> must NOT be claimed as serial
assert "phq" not in sg and "phq9_total" not in sg, f"false serial group: {sg}"
PY
ck "serial group found; a lone digit-suffixed var is not one" 0 "$?"

# 4) endpoints surfaced (P2 evidence)
python3 - "$TMP/o1/cohort_profile.json" <<'PY'
import json, sys
p = json.load(open(sys.argv[1]))
e = set(p["endpoint_candidates"])
assert {"death_date", "cvd_event"} <= e, e
assert "age" not in e
PY
ck "endpoint candidates surfaced (death, cvd event)" 0 "$?"

# 5) THE ANTI-HALLUCINATION CONTRACT: what a codebook cannot state stays UNKNOWN
python3 - "$TMP/o1/cohort_profile.json" "$TMP/o1/cohort_profile.md" <<'PY'
import json, re, sys
p = json.load(open(sys.argv[1]))
must = p["must_ask_user"]
for k in ("n_baseline", "enrollment_period", "followup_duration", "irb_status"):
    assert must[k] == "[UNKNOWN - ask the user]", f"{k} was invented: {must[k]!r}"
md = open(sys.argv[2]).read()
assert "ASK THE USER" in md.upper()
# no fabricated sample size anywhere in the rendered profile
assert not re.search(r"\bN\s*=\s*[\d,]+", md), "a sample size appeared from nowhere"
PY
ck "un-stated facts stay [UNKNOWN], never guessed" 0 "$?"

# 6) a DATA EXPORT (header row = variables) is routed correctly
python3 "$B" --codebook "$TMP/export.csv" --out-dir "$TMP/o2" > /dev/null 2>&1
python3 - "$TMP/o2/cohort_profile.json" <<'PY'
import json, sys
p = json.load(open(sys.argv[1]))
names = [v["name"] for v in p["variables"]]
assert names == ["age", "sex", "ldl_chol", "mri_brain", "death_date"], names
assert not any(v["name"] == "45" for v in p["variables"]), "read a data row as a variable"
PY
ck "data export (header = variables) routed correctly" 0 "$?"

# 7) markdown + json codebooks, and multiple codebooks merge without duplicates
python3 "$B" --codebook "$TMP/codebook.md" --codebook "$TMP/codebook.json" \
  --codebook "$TMP/codebook.csv" --out-dir "$TMP/o3" > /dev/null 2>&1
python3 - "$TMP/o3/cohort_profile.json" <<'PY'
import json, sys
p = json.load(open(sys.argv[1]))
names = [v["name"] for v in p["variables"]]
for must in ("waist_circumference", "alt", "egfr", "cancer_incidence", "sbp_v1"):
    assert must in names, f"{must} missing from merged profile"
assert len(names) == len(set(n.lower() for n in names)), "duplicate variables across codebooks"
PY
ck "markdown + json + csv codebooks merge, de-duplicated" 0 "$?"

# 8) a local document is attached as domain context (no URL needed)
cat > "$TMP/review.md" <<'MD'
# Narrative review: coronary calcium and mortality
Coronary artery calcium scoring predicts cardiovascular events across populations.
Repeated scanning and its incremental value remain debated.
MD
python3 "$B" --codebook "$TMP/codebook.csv" --context "$TMP/review.md" --out-dir "$TMP/o4" > /dev/null 2>&1
ck "local document accepted as domain context" 0 "$?"
grep -q "coronary calcium" "$TMP/o4/context_extract.md"
ck "context text extracted to context_extract.md" 0 "$?"

# 9) a missing file fails loudly rather than proceeding with an empty profile
python3 "$B" --codebook "$TMP/does_not_exist.csv" --out-dir "$TMP/o5" > /dev/null 2>&1
ck "missing codebook fails loudly" 1 "$?"

# 10) an unparseable codebook fails rather than emitting an empty profile
printf 'just some prose with no table and no variables at all\n' > "$TMP/prose.txt"
python3 "$B" --codebook "$TMP/prose.txt" --out-dir "$TMP/o6" > /dev/null 2>&1
ck "codebook with no variables fails (no empty profile)" 1 "$?"

echo "----"
echo "test_cohort_profile: $pass passed, $fail failed"
[ "$fail" -eq 0 ]