# Anti-patterns — skill smells and why they hurt

> Load this when a draft "feels off" or to review an edit before keeping it. Each
> item is a documented Agent Skills anti-pattern with the reason, so you fix the
> cause, not the symptom.

## Description / triggering
- **First-person or mixed point of view** ("I can help with…"). → Inconsistent POV
  hurts skill discovery. Use third person ("Processes…", "Exports…").
- **Vague description missing user keywords.** → The skill never triggers because
  the words the user actually types aren't in the only text the model sees
  pre-trigger. Add the literal keywords.
- **ALL-CAPS imperatives in the description** (`CRITICAL`, `ALWAYS`, `MUST`). → A
  yellow flag that *over*-triggers current models. Fix over-triggering with
  specificity, not volume.
- **Key use case buried late.** → The listing truncates at ~1,536 chars; trigger
  words past the cut are invisible. Front-load.

## Body
- **Narrating why at length instead of instructing.** → Every body line is a
  recurring per-session token cost. State what to do; give a rule's reason briefly.
- **A wall of ALL-CAPS MUST/NEVER rules.** → Today's models follow *reasoning*
  better than rigid rules; reframe as "do X because Y". Reserve emphasis for the
  one thing that genuinely breaks if ignored.
- **Body over budget (>500 lines / ~5k tokens).** → Move detail into `references/`;
  the body is paid for on every trigger, references only when read.
- **Overfitting to the eval tasks.** → Fiddly task-specific rules hurt a skill used
  many times. Prefer general patterns/metaphors.

## References / structure
- **Nested references** (a reference file pointing to another). → The agent may
  only partially read a file (e.g. `head -100`) and miss the pointer, yielding
  incomplete information. Keep every reference **one level deep**, linked directly
  from SKILL.md.
- **A reference with no pointer.** → SKILL.md must say what each file contains and
  *when* to load it, or the model won't know to open it.
- **Long reference with no table of contents** (>300 lines). → Hard to navigate /
  partially read; add an early TOC.
- **Broken links.** → A `(references/…)` / `(scripts/…)` link to a missing file
  wastes a load attempt.

## Scripts
- **Re-implementing the same helper in prose across runs.** → Bundle it as a script
  the skill executes; code is repeatable where prose is only likely.
- **Not stating execute-vs-read intent.** → The agent doesn't know whether to run
  the script (output-only token cost) or read it as reference. Say which.
- **A stub script** (docstring / `...` / `pass` only) or one with no `--self-check`.
  → Nothing proves it runs, so the "deterministic" step silently isn't. Write real
  code with a self-check `validate()` can execute.

# Concepts — optimizing a skill package

> The authoring model below is first-party (Anthropic Agent Skills docs +
> skill-creator + the engineering blog; see Sources). When the optimizer edits a
> skill package, these are the rules that make the edit a *better skill*, not just
> different text. Load it for grounding; `SKILL.md` links the sibling references.

## What a skill package is
```
skill-name/
├── SKILL.md          (required: YAML frontmatter + Markdown body)
├── references/*.md   (docs loaded on demand)
├── scripts/          (code the agent EXECUTES; source never enters context)
└── assets/           (templates/icons/fonts used in the skill's OUTPUT)
```

## Progressive disclosure (the core idea)
Skills use a **three-level loading model**; optimize for the cheapest that works:

1. **Metadata** (`name` + `description`) — **always** loaded at startup into the
   system prompt, ~100 tokens per skill. The `description` is the **primary
   triggering mechanism**: it must say WHAT the skill does AND WHEN to use it.
2. **SKILL.md body** — loaded **when the skill triggers**, then it **stays in
   context for the rest of the session** (a recurring cost). Keep it **≤500 lines**
   (skill-creator's guidance, enforced by `validate`). When it grows, move detail
   into `references/` and add an explicit pointer.
3. **Bundled resources** — references, scripts and assets, loaded/executed **only as
   needed**, effectively unlimited. A reference costs **zero** context until read; a
   script runs via bash **without its code entering context at all** — only its
   *output* costs tokens.

## Frontmatter rules (hard invariants)
- **`name`**: ≤64 chars, lowercase `[a-z0-9-]` only, **no XML tags**, no reserved
  words (`anthropic`, `claude`).
- **`description`**: non-empty, ≤1024 chars, **no XML tags**. Should contain a
  "use when" clause (the triggering signal). A block scalar (`description: >`) is
  fine — the validator reads it.

**Listing truncation (host-specific).** The Claude Code skill listing truncates the
combined `description + when_to_use` text at 1,536 chars by default (configurable via
`maxSkillDescriptionChars`); another host may differ. That is tighter in practice than
the 1024-char validation limit, so **front-load the key use case**.

## Deterministic code beats prose
A body rule is *likely* to be followed; a script that runs is repeatable. The strong
signals to convert a step into `scripts/`: traces where the agent **skips the step**,
**re-implements the same helper**, or hand-executes a deterministic transform. Write
working code (not a stub), give it a `--self-check` so an edit that breaks it is
caught before rollouts are paid for, and **state execute-vs-read intent** in the body
so the agent runs it instead of reading it. Reserve prose for judgment.

## Organize by variant when the skill spans domains
A selection body plus one reference per variant (`references/aws.md`, `gcp.md`, …),
each linked **directly** from SKILL.md with a what/when pointer, so only the relevant
one is ever read. Keep references **one level deep**: a reference that points at
another reference can be missed when the agent reads only part of the first.

## Measurement is core-owned — do not re-implement it
cap-evolve already owns evaluation and acceptance, so this capability deliberately does
**not** carry skill-creator's own eval harness (`evals.json`, assertions, graders): a
second private eval loop would duplicate — and could contradict — the framework's own.
The one skill-specific measurement it adds is trigger rate
(`scripts/trigger_eval.py`), because a skill that never fires is invisible to task
reward.

## What `validate` decides
- **Fails** (hard problem): no `SKILL.md`; bad/missing `name`; empty/oversize/XML
  `description`; body >500 lines; a broken `references|scripts|assets/…` link; a
  bundled script that does not compile or whose declared `--self-check` fails.
- **Warns**: first-person POV, ALL-CAPS `CRITICAL/ALWAYS/MUST/NEVER`, a description
  near the host listing cap, a body over ~5k tokens (this repo's own heuristic, not a
  skill-creator rule), a nested or orphan reference, a long reference with no real
  table of contents, a stub script, a script with no `--self-check`, and
  network/subprocess/`eval` use in bundled code.

## Sources
- Anthropic Agent Skills docs — overview & best-practices:
  https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview ·
  https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices
- skill-creator skill (anthropics/skills, main):
  https://raw.githubusercontent.com/anthropics/skills/main/skills/skill-creator/SKILL.md
- Claude Code skills docs: https://code.claude.com/docs/en/skills
- Engineering blog, "Equipping agents for the real world with Agent Skills":
  https://www.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skills

# Description optimization — the trigger-tuning playbook

> The `description` is the **only** thing loaded before a skill triggers, so it is
> the single highest-leverage edit. This file is the depth behind lever 1 in
> `SKILL.md`. Load it when fixing under/over-triggering.

## What a good description does
A description is read by the model to decide, among 100+ skills, whether THIS one
applies to the current task. It must:

1. **Be third person.** "Processes Excel files and generates reports", not "I can
   help you with Excel". Inconsistent point-of-view causes discovery problems.
2. **State WHAT it does AND WHEN to use it.** The when-to-use information lives in
   the description, not the body — the body is not loaded until after the decision.
3. **Use the keywords a user would naturally say.** If users say "export", "CSV",
   "download a table", those words belong in the description. Missing keywords are
   the most common cause of a skill that never triggers.
4. **Front-load the key use case.** The listing truncates `description +
   when_to_use` at ~1,536 chars; the most important trigger words must come first.

## Diagnosing the failure direction
- **Under-trigger** (didn't fire when it should have) → the description is too
  vague or missing keywords. Enumerate the phrasings and contexts that should fire
  it, including when the user doesn't name the skill. Claude tends to under-trigger,
  so it is fine to be slightly **pushy**: "Use when the user mentions X, Y, or Z,
  even if they don't say 'skill'."
- **Over-trigger** (fired when it shouldn't have) → make the description **more
  specific** and name the **near-miss cases it does NOT cover**. Do **not** reach
  for ALL-CAPS — `CRITICAL`/`ALWAYS`/`MUST` *increase* over-triggering on current
  models.

## Selecting a description honestly — run the loop, don't eyeball it
A trigger decision is stochastic, so one sample per query is noise and hand-judging
drifts between candidates. `scripts/trigger_eval.py` makes it deterministic:

```bash
python scripts/trigger_eval.py --eval-set trigger_eval.json --skill <skill_dir> \
    --judge-cmd '<a shell command that answers YES/NO on stdout>' \
    --description "<candidate description>" --trials 3
# -> {"train_score": .., "heldout_score": .., "per_query": [..], "select_on": "heldout_score"}
```

1. Write ~20 realistic queries — 8-10 **should-trigger** (varied phrasing, including
   cases where the user never names the skill) and 8-10 **should-NOT-trigger**, whose
   value is in the **near-misses**: same keywords, different actual need. An obviously
   irrelevant negative tests nothing. Save as
   `[{"query": "...", "should_trigger": true}, ...]`.
2. The script splits 60/40 by seed and runs each query `--trials 3` times, so the
   score is a rate rather than a coin flip.
3. Propose candidate descriptions and re-score each one with `--description`.
4. **Keep the candidate with the best `heldout_score`** — never the train score. Same
   discipline as cap-evolve's val/test seal, for the same reason.

Queries must be substantive: a trivial one-step request ("read file X") won't trigger
any skill regardless of description quality, so it measures nothing.

Caveat: **trivial single-step tasks may not trigger any skill** regardless of
wording — don't chase those as triggering failures.

## Quick checklist
- [ ] Third person, no "I"/"you can help".
- [ ] Says what AND when.
- [ ] Contains the literal keywords users say.
- [ ] Key use case in the first ~1 sentence (survives the 1,536 truncation).
- [ ] No ALL-CAPS imperatives unless under-triggering is the measured problem.
- [ ] ≤1024 chars, no XML tags.

"""skill-package capability — optimize a WHOLE Agent Skill package.

A skill package is a directory: a required ``SKILL.md`` (YAML frontmatter
``name``/``description`` + a Markdown body) plus optional ``references/`` (docs
loaded on demand), ``scripts/`` (code the agent EXECUTES — its source never
enters context), and ``assets/`` (files used in output). Every one of those is an
editable component here: ``materialize`` exposes them, ``apply`` can create or
rewrite them (including a NEW bundled script), and ``validate`` checks them.

``validate`` encodes the skill-creator / Agent-Skills authoring rules so the
optimizer cannot drift into an invalid package (rules sourced to first-party
Anthropic docs — see references/concepts.md):
  - frontmatter has ``name`` (<=64 chars, [a-z0-9-], no "anthropic"/"claude",
    no XML tags) and a non-empty ``description`` (<=1024 chars, no XML tags) that
    says WHAT + WHEN ("use when").
  - the SKILL.md body stays within the Level-2 budget (<=500 lines): it is a
    recurring per-session token cost.
  - references are one level deep (no nested pointers), each is linked from
    SKILL.md, and a long one (>300 lines) opens with a table of contents.
  - files the body links to exist.
  - bundled scripts COMPILE (``ast.parse``), are not stub-only, and — when they
    declare a ``--self-check`` entry point — that self-check actually passes.
    Deterministic code the agent runs is only deterministic if it runs.

Soft authoring lints (warnings, not failures): first-person description (POV
drift hurts discovery), all-caps CRITICAL/ALWAYS/MUST/NEVER in the description
(over-triggers current models), a description long enough to risk the host's
listing truncation, a script with no declared self-check, and a script reaching
for network/subprocess/``eval`` (a bundled script is executable context — the
human must see that in the diff).

Edit ops (mirrored by the mock optimizer):
``{"file", "op": set|append|ensure_contains|remove, "text", "kind"?}``. ``kind``
defaults to the file's location (frontmatter/body/reference/script/asset) and is
checked against the action policy (``inputs/policy.json`` overrides
``DEFAULT_POLICY``), so a run can allow prose edits while forbidding new code.
"""

from __future__ import annotations

import ast
import json
import os
import re
import subprocess
import sys
from pathlib import Path

NAME_RE = re.compile(r"^[a-z0-9-]{1,64}$")
FRONTMATTER_RE = re.compile(r"^---\s*\n(.*?)\n---", re.S)
XML_TAG_RE = re.compile(r"</?[a-zA-Z][^>]*>")  # an actual tag, not a stray "<"
REL_LINK_RE = re.compile(r"\]\(([^)\s]+)\)")  # any Markdown link target
# A description states WHEN either conditionally ("Use when the user asks") or
# POSITIONALLY ("Use after intake", "Use as the last evaluation step") — both are real
# triggering signals, so demanding the literal "use when" bigram flags good prose.
SAYS_WHEN_RE = re.compile(
    r"\bwhen\b|\buse (after|before|as|at|to|between|right|during|only|this|the moment)\b",
    re.I)
MAX_BODY_LINES = 500              # skill-creator's Level-2 budget (hard here)
MAX_BODY_TOKENS = 5000            # cap-evolve heuristic (~chars/4), advisory only
CHARS_PER_TOKEN = 4
LONG_REF_LINES = 300
MIN_TOC_LINKS = 3                 # anchor links a long reference's TOC must show...
TOC_SCAN_CHARS = 1500             # ...within this leading window, so put the TOC FIRST
LISTING_CAP_CHARS = 1536          # Claude Code host default (maxSkillDescriptionChars)
LONG_DESC_CHARS = 1024            # hard cap; also the front-load advisory threshold
MAX_COMPONENT_CHARS = 40000       # per-component cap so one big file can't blow the prompt
SELF_CHECK_TIMEOUT = 30           # seconds per bundled-script self-check
TEXT_SUFFIXES = {".py", ".sh", ".md", ".txt", ".json", ".yaml", ".yml", ".toml",
                 ".js", ".ts", ".csv", ".cfg", ".ini", ""}
RISKY_IMPORTS = {"subprocess", "socket", "urllib", "http", "requests", "httpx", "ftplib",
                 "telnetlib", "smtplib", "ctypes"}
RISKY_BUILTINS = {"eval", "exec", "compile", "__import__"}

# The whole package is editable, so every kind is allowed by default. A run that
# must not gain new executable code drops "script" (and "add") in policy.json —
# the same knob shape the tool capabilities use (cap_evolve.tool_surface).
DEFAULT_POLICY = {"allow": ["frontmatter", "body", "reference", "script", "asset",
                            "add", "remove"]}


def load_policy(capability_dir: Path) -> dict:
    """``policy.json`` in the capability dir if present (it overrides), else the default."""
    f = Path(capability_dir) / "policy.json"
    return json.loads(f.read_text(encoding="utf-8")) if f.exists() else dict(DEFAULT_POLICY)


def _parse_frontmatter(text: str) -> tuple[dict, str]:
    """Parse the YAML frontmatter enough for the authoring lints.

    Handles what a description realistically uses: a one-line value, a quoted
    value, a ``>``/``|`` block scalar, and an indented continuation. A block
    scalar is ordinary YAML for a long description — the highest-leverage field
    this capability edits — so a parser that returned the literal ``'>'`` would
    silently bypass every description lint.
    """
    m = FRONTMATTER_RE.match(text)
    if not m:
        return {}, text
    fm: dict[str, str] = {}
    key: str | None = None
    fold = ""                     # ">" (fold to spaces), "|" (keep newlines), or ""
    for line in m.group(1).splitlines():
        stripped = line.strip()
        indented = line[:1] in (" ", "\t")
        # A block scalar continues while lines are INDENTED; a blank line inside it
        # is not a terminator. An unindented `next-key:` ends it — treating that as
        # continuation swallowed every key after a `>`/`|` description (6 of them on
        # algorithms/evograph) and inflated the measured description past its cap.
        if key and (indented or (fold and not stripped)):
            if not stripped:
                continue
            sep = "\n" if fold == "|" else " "
            fm[key] = (fm[key] + sep + stripped).strip() if fm[key] else stripped
            continue
        if ":" in line and not indented:
            key, _, v = line.partition(":")
            key = key.strip()
            v = v.strip()
            if v in (">", "|", ">-", "|-", ">+", "|+"):
                fold, v = v[0], ""
            else:
                fold = ""
                v = v.strip('"').strip("'")
            fm[key] = v
        else:
            key, fold = None, ""
    return fm, text[m.end():]


def _relative_links(text: str) -> list[str]:
    """Local relative link targets in Markdown, anchors stripped.

    A Markdown link may carry an anchor (``references/x.md#a-heading``); the anchor is
    not part of the path, so existence-checking the raw target reads every deep link as
    broken. Absolute URLs, bare in-document anchors and mailto:/rooted paths are skipped
    — only targets that must resolve to a file next to the linking document come back,
    de-duplicated (the same file linked five times is one problem, not five findings).
    """
    out = []
    for raw in REL_LINK_RE.findall(text):
        if raw.startswith(("#", "/", "mailto:")) or "://" in raw:
            continue
        target = raw.split("#", 1)[0].strip()
        if target:
            out.append(target)
    return list(dict.fromkeys(out))


def _subpackages(capability_dir: Path) -> list[Path]:
    """Immediate sub-directories that are themselves skill packages (contain SKILL.md).

    A capability_path may hold ONE skill (SKILL.md at the top) or SEVERAL shared
    skills as immediate sub-packages (e.g. seed_capability/{docx,pptx,xlsx,pdf}/).
    Returns the sub-package dirs (sorted) when this is a MULTI-skill root, else []."""
    # A missing/typoed/non-dir path is NOT a multi-skill root — return [] so the caller
    # falls through to the single-skill path and reports a clean validation error
    # ("no SKILL.md") instead of raising FileNotFoundError on iterdir().
    if not capability_dir.is_dir() or (capability_dir / "SKILL.md").exists():
        return []
    return sorted(
        sub for sub in capability_dir.iterdir()
        if sub.is_dir() and (sub / "SKILL.md").exists()
    )


def _read_component(f: Path) -> str:
    """Component text for one bundled file; binaries become an inventory stub.

    An asset (icon/font/template) is part of the package the optimizer must SEE,
    but its bytes are worthless in a text prompt — so list it, don't inline it.
    """
    if f.suffix.lower() not in TEXT_SUFFIXES:
        return f"<binary asset, {f.stat().st_size} bytes>"
    try:
        text = f.read_text(encoding="utf-8")
    except UnicodeDecodeError:
        return f"<binary asset, {f.stat().st_size} bytes>"
    if len(text) > MAX_COMPONENT_CHARS:
        return text[:MAX_COMPONENT_CHARS] + f"\n<truncated at {MAX_COMPONENT_CHARS} chars>"
    return text


def _materialize_one(skill_dir: Path, prefix: str = "") -> dict:
    """Flatten ONE skill package — SKILL.md, references, scripts, assets — into components."""
    parts = {}
    skill_md = skill_dir / "SKILL.md"
    if skill_md.exists():
        parts[f"{prefix}SKILL.md"] = skill_md.read_text(encoding="utf-8")
    for sub in ("references", "scripts", "assets"):
        d = skill_dir / sub
        if not d.is_dir():
            continue
        for f in sorted(d.rglob("*")):
            if f.is_file() and "__pycache__" not in f.parts:
                rel = f.relative_to(skill_dir).as_posix()
                parts[f"{prefix}{rel}"] = _read_component(f)
    return parts


def materialize(capability_dir: Path) -> dict:
    """Flatten the whole package into named components (SKILL.md + refs + scripts + assets).

    ``scripts/<name>`` is code the downstream agent EXECUTES: its source costs the
    agent no context, only its output — which is why converting a skipped prose
    step into a script is the determinism lever.

    Supports BOTH a single-skill capability_path (SKILL.md at the top) and a
    MULTI-skill root holding several immediate sub-packages: components from each
    sub-package are namespaced by ``<skill>/`` (e.g. ``docx/SKILL.md``,
    ``pdf/references/forms.md``)."""
    capability_dir = Path(capability_dir)
    subs = _subpackages(capability_dir)
    if subs:
        parts: dict = {}
        for sub in subs:
            parts.update(_materialize_one(sub, prefix=f"{sub.name}/"))
        return parts
    return _materialize_one(capability_dir)


def _kind_of(rel: str) -> str:
    """The action kind implied by a component's location in the package."""
    head = rel.split("/", 1)[0]
    if head == "references":
        return "reference"
    if head == "scripts":
        return "script"
    if head == "assets":
        return "asset"
    return "body"          # SKILL.md — "frontmatter" is the same file, callers may say so


def apply(capability_dir: Path, edits: list[dict] | None = None) -> dict:
    """Apply edits to any part of the package. Returns {changed, refused}.

    Every write is contained to the capability dir (a ``../`` target is refused,
    not raised — the same {changed, refused} contract the tool capabilities use)
    and checked against the action policy, so a run can permit prose edits while
    forbidding new executable code.
    """
    capability_dir = Path(capability_dir)
    root = capability_dir.resolve()
    allow = set(load_policy(capability_dir).get("allow", []))
    report: dict = {"changed": [], "refused": []}

    def refuse(edit, reason):
        report["refused"].append({"edit": edit, "reason": reason})

    for e in edits or []:
        rel = str(e.get("file", ""))
        op = e.get("op", "set")
        text = e.get("text", "")
        target = capability_dir / rel
        try:
            resolved = target.resolve()
        except OSError as exc:                       # pragma: no cover — exotic paths
            refuse(e, f"unresolvable path: {exc}")
            continue
        if resolved != root and root not in resolved.parents:
            refuse(e, f"path '{rel}' escapes the capability dir")
            continue
        kind = e.get("kind") or _kind_of(rel)
        if kind == "frontmatter":
            kind = "frontmatter" if "frontmatter" in allow else "body"
        if kind not in allow:
            refuse(e, f"action '{kind}' not allowed by policy")
            continue
        exists = target.exists()
        if op == "remove":
            if "remove" not in allow:
                refuse(e, "action 'remove' not allowed by policy")
            elif exists:
                target.unlink()
                report["changed"].append(rel)
            continue
        if not exists and "add" not in allow:
            refuse(e, f"creating '{rel}' needs the 'add' action")
            continue
        cur = target.read_text(encoding="utf-8") if exists else ""
        if op == "set":
            new = text
        elif op == "append":
            new = cur + text
        elif op == "ensure_contains":
            new = cur if (text.strip() and text.strip() in cur) else cur + text
        else:
            refuse(e, f"unknown op {op!r}")
            continue
        if new != cur:
            target.parent.mkdir(parents=True, exist_ok=True)
            target.write_text(new, encoding="utf-8")
            report["changed"].append(rel)
    return report


def is_empty(capability_dir: Path) -> bool:
    """Return True when the capability directory has no meaningful skill content yet.

    An empty directory (no SKILL.md, no sub-packages) is an accepted starting state
    so the optimizer can create the initial skill from failing trajectories."""
    capability_dir = Path(capability_dir)
    subs = _subpackages(capability_dir)
    if subs:
        return False
    return not (capability_dir / "SKILL.md").exists()


def validate(capability_dir: Path) -> dict:
    """Enforce the Agent-Skills authoring rules. Returns {ok, problems, warnings, scripts}.

    An empty capability (no SKILL.md, no sub-packages) is accepted as a valid
    starting state so the optimizer can create the initial skill from failing
    trajectories.

    For a MULTI-skill root (several immediate sub-packages), validate EACH
    sub-package and aggregate: problems/warnings are namespaced by ``<skill>:`` and
    ``ok`` is True only if every sub-package is valid."""
    capability_dir = Path(capability_dir)
    if is_empty(capability_dir):
        return {"ok": True, "empty": True, "name": "", "problems": [], "warnings": [],
                "scripts": []}
    subs = _subpackages(capability_dir)
    if subs:
        problems: list[str] = []
        warnings: list[str] = []
        names: list[str] = []
        scripts: list[dict] = []
        for sub in subs:
            v = _validate_one(sub)
            names.append(v.get("name", sub.name))
            problems += [f"{sub.name}: {p}" for p in v["problems"]]
            warnings += [f"{sub.name}: {w}" for w in v["warnings"]]
            scripts += [{**s, "skill": sub.name} for s in v.get("scripts", [])]
        return {"ok": not problems, "name": ",".join(names),
                "problems": problems, "warnings": warnings, "scripts": scripts}
    return _validate_one(capability_dir)


def _validate_frontmatter(fm: dict, problems: list, warnings: list) -> str:
    name = fm.get("name", "")
    if not name:
        problems.append("frontmatter missing 'name'")
    elif not NAME_RE.match(name):
        problems.append(f"name {name!r} must be <=64 chars, lowercase [a-z0-9-]")
    if "anthropic" in name.lower() or "claude" in name.lower():
        problems.append("name must not contain 'anthropic' or 'claude'")
    if XML_TAG_RE.search(name):
        problems.append("name must not contain XML tags")

    desc = fm.get("description", "")
    if not desc.strip():
        problems.append("frontmatter missing a non-empty 'description'")
        return name
    if len(desc) > LONG_DESC_CHARS:
        problems.append(f"description is {len(desc)} chars (>{LONG_DESC_CHARS})")
    if XML_TAG_RE.search(desc):
        problems.append("description must not contain XML tags")
    if not SAYS_WHEN_RE.search(desc):
        warnings.append("description should say WHEN to use the skill "
                        "('Use when …') — it is the primary triggering signal")
    # point-of-view drift: descriptions must be third person.
    if re.search(r"(?<![A-Za-z])I(?![A-Za-z])|I'?m\b|I can\b|you can help", desc):
        warnings.append("description should be third person (e.g. 'Processes X "
                        "…'), not first person ('I can …') — POV drift hurts discovery")
    # all-caps imperatives over-trigger current models.
    if re.search(r"\b(CRITICAL|ALWAYS|MUST|NEVER)\b", desc):
        warnings.append("avoid all-caps CRITICAL/ALWAYS/MUST/NEVER in the "
                        "description — it over-triggers; say plainly 'Use when …'")
    # host listing truncation (Claude Code default; configurable per host).
    if len(desc) > LISTING_CAP_CHARS - 256:
        warnings.append(f"description is {len(desc)} chars; the Claude Code host "
                        f"truncates description + when_to_use at {LISTING_CAP_CHARS} "
                        "by default (configurable) — front-load the key use case")
    return name


def _validate_references(pkg: Path, body: str, problems: list, warnings: list) -> None:
    # Relative links the body points at must resolve. Anchors are stripped first
    # (_relative_links), or every deep link like "references/x.md#heading" reads as broken.
    for rel in _relative_links(body):
        if rel.split("/", 1)[0] in ("references", "scripts", "assets") \
                and not (pkg / rel).exists():
            problems.append(f"SKILL.md links '{rel}' which does not exist")

    refs = pkg / "references"
    if not refs.is_dir():
        return
    # sorted(): filesystem iteration order differs between machines, and callers (the
    # repo's own blocking authoring lint, and the optimizer's warning list) need the
    # findings to come out in the same order everywhere.
    for sub in sorted(refs.iterdir()):
        if sub.is_dir():
            warnings.append(f"references/{sub.name}/ is nested >1 level deep; "
                            "keep references one level deep")
    for f in sorted(refs.glob("*.md")):
        text = f.read_text(encoding="utf-8")
        head = text[:TOC_SCAN_CHARS]
        n = text.count("\n") + 1
        # A long reference needs a real table of contents, and the check is POSITIONAL:
        # the agent may read only the head, so a perfect TOC sitting behind a preamble
        # is a TOC the reader never sees. Anchor links preferred, section headings
        # accepted as the weaker form.
        if n > LONG_REF_LINES and not (head.count("](#") >= MIN_TOC_LINKS
                                       or head.count("\n## ") >= MIN_TOC_LINKS):
            warnings.append(f"references/{f.name} is {n} lines and shows no table of "
                            f"contents in its first {TOC_SCAN_CHARS} chars: put "
                            f"{MIN_TOC_LINKS}+ anchor links ('- [Section](#section)') "
                            "at the very TOP, above any orientation prose")
        # One level deep is about POINTERS, not directories: a reference that points at
        # another reference can be missed when the agent reads only part of it. Judged
        # on the RESOLVED target — a substring/`refs / link` test reads a legitimate
        # "../SKILL.md" back-link as a sibling reference (it resolves through refs/).
        for target in _relative_links(text):
            # A reference written as if it were SKILL.md ("references/b.md") still MEANS
            # the sibling, so resolve that form against references/ — otherwise the
            # commonest ref->ref shape reads as a mere broken link.
            rel = re.sub(r"^(\./)?references/", "", target)
            hop = ((refs if rel != target else f.parent) / rel).resolve()
            if hop.suffix == ".md" and hop.parent == refs.resolve() and hop != f.resolve():
                warnings.append(f"references/{f.name} points at another reference "
                                f"('{target}') — keep references one level deep, linked "
                                "directly from SKILL.md")
            elif not hop.exists():
                warnings.append(f"references/{f.name} links '{target}' "
                                "which does not exist")
        if f.name not in body and f"references/{f.name}" not in body:
            warnings.append(f"references/{f.name} is an orphan — SKILL.md never points "
                            "at it, so the agent will not know to load it")


def _script_self_check(f: Path, pkg: Path) -> dict:
    """Run a bundled script's declared ``--self-check`` and report the outcome.

    Only a script that DECLARES a self-check is executed — an entry point that
    needs real arguments would otherwise "fail" for the wrong reason. Runs with a
    timeout, in the package dir, with no inherited proxy/API env, so validation
    cannot quietly reach the network on the optimizer's behalf.
    """
    src = f.read_text(encoding="utf-8", errors="replace")
    rel = f.relative_to(pkg).as_posix()
    if "--self-check" not in src:
        return {"file": rel, "self_check": "absent"}
    if os.environ.get("CAPEVOLVE_NO_SCRIPT_EXEC"):
        return {"file": rel, "self_check": "skipped (CAPEVOLVE_NO_SCRIPT_EXEC)"}
    # PATH/HOME/PYTHONPATH pass through (a bundled script may import its own package);
    # nothing else does — no API keys — and the proxy vars are blanked so a self-check
    # cannot quietly reach the network on the optimizer's behalf.
    env = {"PATH": os.environ.get("PATH", ""), "HOME": os.environ.get("HOME", ""),
           "PYTHONPATH": os.environ.get("PYTHONPATH", ""),
           "PYTHONDONTWRITEBYTECODE": "1", "NO_NETWORK": "1",
           "http_proxy": "", "https_proxy": "", "HTTP_PROXY": "", "HTTPS_PROXY": ""}
    try:
        p = subprocess.run([sys.executable, str(f.resolve()), "--self-check"],
                           cwd=str(pkg.resolve()),
                           capture_output=True, text=True, timeout=SELF_CHECK_TIMEOUT,
                           env=env)
    except subprocess.TimeoutExpired:
        return {"file": rel, "self_check": "timeout", "ok": False,
                "exit_code": None, "stderr_tail": f"timed out after {SELF_CHECK_TIMEOUT}s"}
    except OSError as exc:                                # pragma: no cover
        return {"file": rel, "self_check": "error", "ok": False,
                "exit_code": None, "stderr_tail": str(exc)[-400:]}
    # A failing self-check often reports on stdout (an assertion printer, a FAIL
    # line), so carry both tails — the optimizer can only fix what it is shown.
    return {"file": rel, "self_check": "ran", "ok": p.returncode == 0,
            "exit_code": p.returncode, "stderr_tail": (p.stderr or "")[-400:],
            "stdout_tail": (p.stdout or "")[-400:]}


def _is_stub(tree: ast.Module) -> bool:
    """True when a module has no real body — only a docstring / pass / ``...``."""
    for node in tree.body:
        if isinstance(node, ast.Expr) and isinstance(node.value, ast.Constant):
            continue                                      # docstring or bare ``...``
        if isinstance(node, ast.Pass):
            continue
        return False
    return True


def _risky(tree: ast.Module) -> list[str]:
    """Network / subprocess / dynamic-exec surface a human should see in the diff.

    Judged from the AST (imports and calls), not substrings, so a script that
    merely mentions the word is not flagged.
    """
    found: set[str] = set()
    for node in ast.walk(tree):
        if isinstance(node, ast.Import):
            found |= {a.name.split(".")[0] for a in node.names} & RISKY_IMPORTS
        elif isinstance(node, ast.ImportFrom) and node.module:
            found |= {node.module.split(".")[0]} & RISKY_IMPORTS
        elif isinstance(node, ast.Call) and isinstance(node.func, ast.Name) \
                and node.func.id in RISKY_BUILTINS:
            found.add(node.func.id + "()")
    return sorted(found)


def _validate_scripts(pkg: Path, problems: list, warnings: list) -> list[dict]:
    """Check bundled code the way the downstream agent will meet it: as something run.

    Deterministic code is the point of ``scripts/`` — a script that does not
    compile, or whose declared self-check fails, is not determinism, it is a file.
    """
    d = pkg / "scripts"
    if not d.is_dir():
        return []
    reports: list[dict] = []
    for f in sorted(d.rglob("*.py")):
        if "__pycache__" in f.parts:
            continue
        rel = f.relative_to(pkg).as_posix()
        src = f.read_text(encoding="utf-8", errors="replace")
        try:
            tree = ast.parse(src, filename=str(f))
        except SyntaxError as exc:
            problems.append(f"{rel} does not compile: {exc.msg} (line {exc.lineno})")
            reports.append({"file": rel, "compiles": False, "error": exc.msg})
            continue
        rep: dict = {"file": rel, "compiles": True}
        if _is_stub(tree):
            warnings.append(f"{rel} has no real body (docstring/pass/... only) — a "
                            "bundled script must be working code, not a placeholder")
            rep["stub"] = True
        risky = _risky(tree)
        if risky:
            warnings.append(f"{rel} uses {', '.join(risky)} — a bundled script is "
                            "executable context; confirm this is intended in the diff")
            rep["risky"] = risky
        rep.update(_script_self_check(f, pkg))
        if rep.get("self_check") == "absent":
            warnings.append(f"{rel} has no declared '--self-check' entry point, so "
                            "nothing verifies it still runs after an edit — add one")
        elif rep.get("ok") is False:
            detail = (rep.get("stderr_tail") or "").strip() or (rep.get("stdout_tail") or "").strip()
            problems.append(f"{rel} --self-check failed (exit {rep.get('exit_code')}): "
                            f"{detail[-300:]}")
        reports.append(rep)
    return reports


def _validate_one(capability_dir: Path) -> dict:
    """Enforce the Agent-Skills authoring rules on ONE skill package."""
    capability_dir = Path(capability_dir)
    problems: list[str] = []
    warnings: list[str] = []

    skill_md = capability_dir / "SKILL.md"
    if not skill_md.exists():
        return {"ok": False, "name": "", "problems": ["no SKILL.md in the package"],
                "warnings": [], "scripts": []}

    text = skill_md.read_text(encoding="utf-8")
    fm, body = _parse_frontmatter(text)
    name = _validate_frontmatter(fm, problems, warnings)

    n_body = body.count("\n") + 1
    if n_body > MAX_BODY_LINES:
        problems.append(f"SKILL.md body is {n_body} lines (>{MAX_BODY_LINES}); the body "
                        "is a recurring per-session cost — split detail into references/")
    body_tokens = len(body) // CHARS_PER_TOKEN
    if body_tokens > MAX_BODY_TOKENS:
        warnings.append(f"SKILL.md body is ~{body_tokens} tokens (>{MAX_BODY_TOKENS}, this "
                        "repo's heuristic) — move detail into references/")

    _validate_references(capability_dir, body, problems, warnings)
    scripts = _validate_scripts(capability_dir, problems, warnings)

    return {"ok": not problems, "name": name, "problems": problems,
            "warnings": warnings, "scripts": scripts}

"""Self-test: round-trip the WHOLE package through materialize -> apply -> validate.

One case per rule, each with a deliberately broken fixture, so every authoring
rule is a checked property instead of a claim. The end-to-end case is the point
of the capability: apply() CREATES a new bundled script, materialize() shows it as
a component, and validate() runs its self-check.
"""

from __future__ import annotations

import json
import sys
import tempfile
from pathlib import Path

import _bootstrap  # noqa: F401

import abstract
import token_report
import trigger_eval

GOOD = ("---\nname: demo-skill\n"
        "description: Do a thing. Use when the user wants the thing done.\n---\n"
        "# Demo\nBody.\n")


def _pkg(d: Path, skill_md: str = GOOD) -> Path:
    d.mkdir(parents=True, exist_ok=True)
    (d / "SKILL.md").write_text(skill_md, encoding="utf-8")
    return d


def main() -> int:
    report = {"skill": "skill-package", "ok": False, "problems": [], "notes": []}
    fail = report["problems"].append
    note = report["notes"].append

    with tempfile.TemporaryDirectory() as tmp:
        root = Path(tmp)

        # --- the whole package is materialized (SKILL.md + refs + scripts + assets)
        cap = _pkg(root / "full")
        (cap / "references").mkdir()
        (cap / "references" / "deep.md").write_text("# Deep\n", encoding="utf-8")
        (cap / "scripts").mkdir()
        (cap / "scripts" / "helper.py").write_text("print(1)\n", encoding="utf-8")
        (cap / "assets").mkdir()
        (cap / "assets" / "tpl.html").write_text("<p>x</p>\n", encoding="utf-8")
        (cap / "assets" / "logo.png").write_bytes(b"\x89PNG\r\n\x1a\n" + b"\x00" * 40)
        parts = abstract.materialize(cap)
        missing = [k for k in ("SKILL.md", "references/deep.md", "scripts/helper.py",
                               "assets/tpl.html", "assets/logo.png") if k not in parts]
        if missing:
            fail(f"materialize missed package components: {missing}")
        if "binary asset" not in parts.get("assets/logo.png", ""):
            fail("a binary asset must be inventoried as a stub, not inlined")
        note("materialize exposes SKILL.md + references + scripts + assets")

        # --- a valid package passes
        v = abstract.validate(_pkg(root / "ok"))
        if not v["ok"]:
            fail(f"valid skill rejected: {v['problems']}")

        # --- frontmatter: bad name / XML tag in name
        if abstract.validate(_pkg(root / "n1", "---\nname: Bad_Name\ndescription: x\n---\n#x\n"))["ok"]:
            fail("invalid name not rejected")
        if abstract.validate(_pkg(root / "n2", "---\nname: bad-<tag>\n"
                                  "description: Do it. Use when needed.\n---\n#x\n"))["ok"]:
            fail("XML tag in name not rejected")

        # --- soft lints fire on a ONE-LINE description
        w = " ".join(abstract.validate(_pkg(
            root / "l1", "---\nname: lint-demo\n"
            "description: I can help. ALWAYS use when the user wants it.\n---\n#x\n"))["warnings"])
        if "third person" not in w or "over-triggers" not in w:
            fail(f"POV/all-caps lints did not fire: {w!r}")

        # --- ... and on a FOLDED (block scalar) description, which used to bypass them
        folded = abstract.validate(_pkg(
            root / "l2", "---\nname: lint-folded\ndescription: >\n"
            "  I can help with things.\n  ALWAYS use when the user wants it.\n---\n#x\n"))
        wf = " ".join(folded["warnings"])
        if "third person" not in wf or "over-triggers" not in wf:
            fail(f"block-scalar description bypassed the lints: {folded}")
        note("block-scalar (`description: >`) frontmatter is parsed and linted")

        # --- body budget is enforced, not merely advised
        big = abstract.validate(_pkg(root / "big", GOOD + "line\n" * 600))
        if big["ok"] or not any("lines" in p for p in big["problems"]):
            fail(f"a 600-line body must be a hard problem: {big}")

        # --- references: nested pointer, orphan, fake TOC
        cap = _pkg(root / "refs", GOOD + "See [a](references/a.md).\n")
        (cap / "references").mkdir()
        (cap / "references" / "a.md").write_text("# A\nSee [b](references/b.md)\n", encoding="utf-8")
        (cap / "references" / "b.md").write_text("# B\n", encoding="utf-8")
        (cap / "references" / "long.md").write_text(
            "# L\n" + "orientation prose that pushes the TOC out of the window\n" * 30 +
            "## Contents\n- [A](#a)\n- [B](#b)\n- [C](#c)\n" + "x\n" * 400,
            encoding="utf-8")   # a real TOC, but behind a preamble -> still warns
        w = " ".join(abstract.validate(cap)["warnings"])
        for want, label in (("one level deep", "nested reference pointer"),
                            ("orphan", "orphan reference"),
                            ("table of contents", "missing TOC in a long reference")):
            if want not in w:
                fail(f"{label} not warned: {w!r}")
        note("reference structure: nested pointers, orphans, missing TOC all warn")

        # --- broken link the body points at
        if abstract.validate(_pkg(root / "bl", GOOD + "See [x](references/gone.md).\n"))["ok"]:
            fail("a broken reference link must be a hard problem")

        # --- scripts: a syntax error is a hard problem
        cap = _pkg(root / "badpy")
        (cap / "scripts").mkdir()
        (cap / "scripts" / "x.py").write_text("def broken(:\n", encoding="utf-8")
        v = abstract.validate(cap)
        if v["ok"] or not any("does not compile" in p for p in v["problems"]):
            fail(f"a bundled script that does not compile must fail validation: {v}")

        # --- scripts: stub body + missing self-check warn
        cap = _pkg(root / "stub")
        (cap / "scripts").mkdir()
        (cap / "scripts" / "s.py").write_text('"""todo."""\n...\n', encoding="utf-8")
        w = " ".join(abstract.validate(cap)["warnings"])
        if "no real body" not in w or "--self-check" not in w:
            fail(f"stub/self-check warnings did not fire: {w!r}")

        # --- scripts: a FAILING declared self-check is a hard problem
        cap = _pkg(root / "failing")
        (cap / "scripts").mkdir()
        (cap / "scripts" / "f.py").write_text(
            "import sys\nif '--self-check' in sys.argv:\n"
            "    print('boom', file=sys.stderr); sys.exit(1)\n", encoding="utf-8")
        v = abstract.validate(cap)
        if v["ok"] or not any("--self-check failed" in p for p in v["problems"]):
            fail(f"a failing script self-check must fail validation: {v}")

        # --- apply(): path traversal is refused, not written
        cap = _pkg(root / "esc")
        r = abstract.apply(cap, [{"file": "../escaped.txt", "op": "set", "text": "pwned"}])
        if r["changed"] or not r["refused"] or (root / "escaped.txt").exists():
            fail(f"apply must refuse an edit escaping the capability dir: {r}")

        # --- apply(): the action policy gates a script edit
        cap = _pkg(root / "policy")
        (cap / "policy.json").write_text(json.dumps({"allow": ["body", "reference", "add"]}),
                                        encoding="utf-8")
        r = abstract.apply(cap, [{"file": "scripts/no.py", "op": "set", "text": "x=1\n"}])
        if r["changed"] or "not allowed by policy" not in json.dumps(r["refused"]):
            fail(f"a script edit must be refusable by policy: {r}")
        note("apply() contains writes to the package and honors the action policy")

        # --- END TO END: apply() CREATES a bundled script, it materializes as a
        #     component, and validate() runs its self-check.
        cap = _pkg(root / "e2e", GOOD + "Run [the helper](scripts/helper.py).\n")
        script = ("import sys\n\n"
                  "def normalize(s):\n    return ' '.join(str(s).split()).lower()\n\n"
                  'if __name__ == "__main__":\n'
                  "    if '--self-check' in sys.argv:\n"
                  "        assert normalize('  A  B ') == 'a b'\n"
                  "        print('ok')\n")
        r = abstract.apply(cap, [{"file": "scripts/helper.py", "op": "set", "text": script}])
        if "scripts/helper.py" not in r["changed"]:
            fail(f"apply did not create the new bundled script: {r}")
        if "scripts/helper.py" not in abstract.materialize(cap):
            fail("a newly created script must appear as a component")
        v = abstract.validate(cap)
        checked = [s for s in v["scripts"] if s["file"] == "scripts/helper.py"]
        if not v["ok"]:
            fail(f"the created script package must validate: {v['problems']}")
        if not checked or checked[0].get("self_check") != "ran" or not checked[0].get("ok"):
            fail(f"the created script's self-check did not run and pass: {v['scripts']}")
        note("end-to-end: optimizer-created script -> component -> self-check ran and passed")

        # --- reporters
        tr = token_report.report(cap)
        if "body_tokens" not in tr or "over_budget" not in tr:
            fail(f"token_report missing budget fields: {tr}")
        if not tr.get("scripts") or tr.get("scripts_context_cost") != 0:
            fail(f"token_report must inventory scripts with context_cost 0: {tr}")
        if trigger_eval.main(["--self-check"]) != 0:
            fail("trigger_eval --self-check failed")
        note("token_report inventories scripts; trigger_eval self-check passes")

    report["ok"] = not report["problems"]
    print(json.dumps(report, indent=2))
    return 0 if report["ok"] else 1


if __name__ == "__main__":
    sys.exit(main())

"""Expose a skill package as a Candidate and report authoring-rule validity."""

from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path

import _bootstrap  # noqa: F401

from cap_evolve import Candidate

import abstract


def main(argv=None) -> int:
    p = argparse.ArgumentParser(prog="skill-package")
    p.add_argument("--path", required=True, help="the skill package dir (contains SKILL.md)")
    args = p.parse_args(argv)
    parts = abstract.materialize(Path(args.path))
    v = abstract.validate(Path(args.path))
    cand = Candidate(id="seed", component="skill-package", text_parts=parts, dir=str(args.path))
    print(json.dumps({"candidate": cand.to_dict(), "valid": v}, indent=2))
    return 0 if v["ok"] else 1


if __name__ == "__main__":
    sys.exit(main())

"""Report the progressive-disclosure token budget of a skill package.

Level 2 (the SKILL.md body) is loaded on every trigger and stays in context for
the whole session — a *recurring* token cost — so it has a budget (<=500 lines;
~5k tokens is this repo's own heuristic). Level 3 costs **zero** context until
used: a reference until it is read, a script until it is run (and a script's
source is never loaded at all — only its output). So the report states
``context_cost: 0`` for both, and inventories ``scripts/`` — the deterministic
surface — instead of only sizing the cheap thing.

Deterministic, dependency-free (no cap-evolve bootstrap) — run it directly:

    python scripts/token_report.py --path <skill_dir>
    python scripts/token_report.py --self-check

Exit code is 0 always (advisory); the JSON `over_budget` flag carries the signal.
"""

from __future__ import annotations

import argparse
import json
import re
import sys
from pathlib import Path

CHARS_PER_TOKEN = 4
MAX_BODY_TOKENS = 5000
FRONTMATTER_RE = re.compile(r"^---\s*\n.*?\n---", re.S)


def _tokens(text: str) -> int:
    return len(text) // CHARS_PER_TOKEN


def report(skill_dir: Path) -> dict:
    skill_dir = Path(skill_dir)
    out: dict = {"skill_dir": str(skill_dir), "references": {}}

    skill_md = skill_dir / "SKILL.md"
    if not skill_md.exists():
        return {"error": "no SKILL.md", **out}
    text = skill_md.read_text(encoding="utf-8")
    body = FRONTMATTER_RE.sub("", text, count=1)
    out["body_tokens"] = _tokens(body)
    out["body_lines"] = body.count("\n") + 1
    out["over_budget"] = out["body_tokens"] > MAX_BODY_TOKENS
    out["budget_tokens"] = MAX_BODY_TOKENS

    refs = skill_dir / "references"
    if refs.is_dir():
        for f in sorted(refs.glob("*.md")):
            out["references"][f.name] = _tokens(f.read_text(encoding="utf-8"))
    out["reference_tokens_total"] = sum(out["references"].values())
    # Level 3 is free until used — say so, so a big reference is not mistaken for
    # an expensive one and the optimizer reads the budget correctly.
    out["references_context_cost"] = 0
    out["scripts"] = _scripts(skill_dir)
    out["scripts_context_cost"] = 0        # source never loads; only the output costs
    return out


def _scripts(skill_dir: Path) -> list[dict]:
    """Inventory the deterministic surface: size, entry point, declared self-check."""
    d = skill_dir / "scripts"
    if not d.is_dir():
        return []
    rows = []
    for f in sorted(d.rglob("*")):
        if not f.is_file() or "__pycache__" in f.parts:
            continue
        try:
            src = f.read_text(encoding="utf-8")
        except (UnicodeDecodeError, OSError):
            src = ""
        rows.append({"file": f.relative_to(skill_dir).as_posix(),
                     "bytes": f.stat().st_size,
                     "entry_point": '__main__' in src or f.suffix == ".sh",
                     "self_check": "--self-check" in src})
    return rows


def _self_check() -> int:
    import tempfile
    with tempfile.TemporaryDirectory() as d:
        pkg = Path(d)
        (pkg / "SKILL.md").write_text("---\nname: t\ndescription: d\n---\n# t\nbody\n",
                                      encoding="utf-8")
        (pkg / "scripts").mkdir()
        (pkg / "scripts" / "h.py").write_text(
            'if __name__ == "__main__":\n    pass  # --self-check\n', encoding="utf-8")
        r = report(pkg)
    assert r["scripts"] and r["scripts"][0]["file"] == "scripts/h.py", r
    assert r["scripts"][0]["entry_point"] and r["scripts"][0]["self_check"], r
    assert r["scripts_context_cost"] == 0 and r["references_context_cost"] == 0, r
    assert r["body_lines"] > 0 and r["over_budget"] is False, r
    print(json.dumps({"self_check": "ok"}))
    return 0


def main(argv=None) -> int:
    p = argparse.ArgumentParser(prog="token-report")
    p.add_argument("--path", help="skill package dir (contains SKILL.md)")
    p.add_argument("--self-check", action="store_true")
    args = p.parse_args(argv)
    if args.self_check:
        return _self_check()
    if not args.path:
        p.error("--path is required (or use --self-check)")
    print(json.dumps(report(Path(args.path)), indent=2))
    return 0


if __name__ == "__main__":
    sys.exit(main())

"""Measure a candidate `description`'s trigger rate on a held-out split.

The `description` is the only text loaded before a skill fires, so triggering is
its own measurable objective. Doing this by hand costs a decision per query and
drifts run to run; this script makes it deterministic: it splits the eval set by
seed, asks a judge the same question N times per query (a trigger decision is
stochastic — one sample is noise), scores both halves, and prints JSON. Select
the description by the HELD-OUT score, never the train score.

Eval set (JSON list, the shape skill-creator uses):

    [{"query": "the user prompt", "should_trigger": true}, ...]

The judge is whatever the host has, so this stays model-agnostic: `--judge-cmd`
is shelled once per (query, trial) with the prompt on stdin and must print a
verdict containing `yes`/`trigger` or `no`. Example:

    python trigger_eval.py --eval-set eval.json --skill ../my-skill \
        --judge-cmd 'llm -m gpt-4o-mini' --trials 3

    {"train_score": 0.83, "heldout_score": 0.75, "per_query": [...]}

`--self-check` runs the whole pipeline against a built-in keyword judge (no
model, no network) and asserts the scoring is right, so the plumbing is verified
without spending anything.
"""

from __future__ import annotations

import argparse
import json
import random
import re
import subprocess
import sys
import tempfile
from pathlib import Path

PROMPT = """You decide whether one skill should be consulted for a user request.

Skill name: {name}
Skill description: {description}

User request: {query}

Answer with one word: YES if the skill should be consulted, NO if it should not."""


def _description(skill_dir: Path) -> tuple[str, str]:
    """(name, description) from a skill package's frontmatter."""
    text = (Path(skill_dir) / "SKILL.md").read_text(encoding="utf-8")
    m = re.match(r"^---\s*\n(.*?)\n---", text, re.S)
    fm = {}
    if m:
        key = None
        for line in m.group(1).splitlines():
            if ":" in line and not line[:1].isspace():
                key, _, v = line.partition(":")
                key = key.strip()
                fm[key] = v.strip().strip('"').strip("'")
            elif key and line.strip():
                fm[key] = (fm[key] + " " + line.strip()).strip()
    return fm.get("name", ""), fm.get("description", "")


def _keyword_judge(prompt: str) -> str:
    """Offline stand-in used by --self-check: does the request share the description's words?"""
    desc = re.search(r"Skill description: (.*)", prompt).group(1).lower()
    query = re.search(r"User request: (.*)", prompt, re.S).group(1).lower()
    words = {w for w in re.findall(r"[a-z]{4,}", desc)}
    hits = sum(1 for w in re.findall(r"[a-z]{4,}", query) if w in words)
    return "YES" if hits >= 2 else "NO"


def _ask(judge_cmd: str | None, prompt: str) -> bool:
    if judge_cmd is None:
        verdict = _keyword_judge(prompt)
    else:
        p = subprocess.run(judge_cmd, shell=True, input=prompt, text=True,
                           capture_output=True, timeout=120)
        verdict = (p.stdout or "").strip()
    low = verdict.lower()
    return ("yes" in low or "trigger" in low) and "no" != low[:2]


def evaluate(items: list[dict], name: str, description: str, *, trials: int = 3,
             judge_cmd: str | None = None) -> list[dict]:
    """Per-query trigger rate over `trials` samples, plus whether it matches expectation."""
    out = []
    for it in items:
        prompt = PROMPT.format(name=name, description=description, query=it["query"])
        fired = [_ask(judge_cmd, prompt) for _ in range(trials)]
        rate = sum(fired) / len(fired)
        want = bool(it["should_trigger"])
        out.append({"query": it["query"], "should_trigger": want,
                    "trigger_rate": rate,
                    "score": rate if want else 1.0 - rate})
        out[-1]["correct"] = out[-1]["score"] > 0.5
    return out


def _mean(rows: list[dict]) -> float:
    return round(sum(r["score"] for r in rows) / len(rows), 4) if rows else 0.0


def split(items: list[dict], *, seed: int = 0, train_frac: float = 0.6) -> tuple[list, list]:
    """Deterministic train/held-out split (skill-creator uses 60/40)."""
    idx = list(range(len(items)))
    random.Random(seed).shuffle(idx)
    cut = max(1, int(round(len(items) * train_frac)))
    return [items[i] for i in idx[:cut]], [items[i] for i in idx[cut:]]


def run(eval_set: list[dict], skill_dir: Path, *, trials: int = 3, seed: int = 0,
        train_frac: float = 0.6, judge_cmd: str | None = None,
        description: str | None = None) -> dict:
    name, current = _description(skill_dir)
    desc = description if description is not None else current
    train, heldout = split(eval_set, seed=seed, train_frac=train_frac)
    tr = evaluate(train, name, desc, trials=trials, judge_cmd=judge_cmd)
    ho = evaluate(heldout, name, desc, trials=trials, judge_cmd=judge_cmd)
    return {"skill": name, "trials": trials, "seed": seed,
            "n_train": len(tr), "n_heldout": len(ho),
            "train_score": _mean(tr), "heldout_score": _mean(ho),
            "per_query": tr + ho,
            "select_on": "heldout_score"}


def _self_check() -> int:
    """Prove the split + scoring + judge plumbing with no model and no network."""
    items = [{"query": f"please export the sales table to csv {i}", "should_trigger": True}
             for i in range(5)]
    items += [{"query": f"write a haiku about rain {i}", "should_trigger": False}
              for i in range(5)]
    with tempfile.TemporaryDirectory() as d:
        pkg = Path(d)
        (pkg / "SKILL.md").write_text(
            "---\nname: csv-export\ndescription: Exports records to csv table files. "
            "Use when the user asks to export or download a sales table.\n---\n# x\n",
            encoding="utf-8")
        out = run(items, pkg, trials=3, seed=0)
    a, b = split(items, seed=0), split(items, seed=0)
    assert [i["query"] for i in a[0]] == [i["query"] for i in b[0]], "split is not deterministic"
    assert out["n_train"] == 6 and out["n_heldout"] == 4, out
    assert out["heldout_score"] > 0.5, f"keyword judge should mostly agree: {out}"
    assert all(0.0 <= r["trigger_rate"] <= 1.0 for r in out["per_query"])
    print(json.dumps({"self_check": "ok", "train_score": out["train_score"],
                      "heldout_score": out["heldout_score"]}))
    return 0


def main(argv=None) -> int:
    p = argparse.ArgumentParser(prog="trigger-eval")
    p.add_argument("--self-check", action="store_true", help="offline pipeline check")
    p.add_argument("--eval-set", help="JSON [{query, should_trigger}]")
    p.add_argument("--skill", help="skill package dir (contains SKILL.md)")
    p.add_argument("--description", default=None, help="candidate description to test "
                                                       "instead of the one in SKILL.md")
    p.add_argument("--judge-cmd", default=None, help="shell command; prompt on stdin, "
                                                     "YES/NO on stdout")
    p.add_argument("--trials", type=int, default=3)
    p.add_argument("--seed", type=int, default=0)
    p.add_argument("--train-frac", type=float, default=0.6)
    args = p.parse_args(argv)
    if args.self_check:
        return _self_check()
    if not (args.eval_set and args.skill):
        p.error("--eval-set and --skill are required (or use --self-check)")
    items = json.loads(Path(args.eval_set).read_text(encoding="utf-8"))
    print(json.dumps(run(items, Path(args.skill), trials=args.trials, seed=args.seed,
                         train_frac=args.train_frac, judge_cmd=args.judge_cmd,
                         description=args.description), indent=2))
    return 0


if __name__ == "__main__":
    sys.exit(main())

"""Thin shim: locate cap_evolve, then defer to cap_evolve._bootstrap.

Skill scripts ``import _bootstrap`` first. The real path-resolution logic lives
ONCE in ``cap_evolve._bootstrap`` (so it can't drift across skills); this shim
only has to find that package, which means a minimal upward walk for ``core/`` —
the single bit of bootstrapping that genuinely must run before cap_evolve is
importable. Everything else delegates.
"""

from __future__ import annotations

import os
import sys
from pathlib import Path


def _seed_path() -> None:
    """Minimal: put a dir containing the cap_evolve package on sys.path.

    ``CAPEVOLVE_CORE`` is honoured BEFORE any ambient import. An editable install of a
    *different* cap-evolve checkout registers a ``sys.meta_path`` finder, which outranks
    both ``sys.path`` and ``PYTHONPATH`` — so "cap_evolve imports fine" is not evidence that
    it imports the checkout you are standing in. Deferring to the ambient package here made
    an explicit override unreachable, and the symptom was a stale core silently answering
    for this one (``ModuleNotFoundError: cap_evolve.constraints`` from a checkout that
    predates that module). An explicit env var wins.
    """
    env = os.environ.get("CAPEVOLVE_CORE")
    want = Path(env).resolve() if env else None
    if want and (want / "cap_evolve" / "__init__.py").exists():
        loaded = sys.modules.get("cap_evolve")
        already = getattr(loaded, "__file__", None)
        if already and Path(already).resolve().parent.parent == want:
            return                      # right checkout already imported: touch nothing
        p = str(want)
        if p in sys.path:
            sys.path.remove(p)
        sys.path.insert(0, p)
        if loaded is not None:
            # Evicting a module makes a re-import yield a DIFFERENT object, so anything
            # already holding a reference fails an `is` check. Only ever do it when the
            # loaded package really is the wrong checkout — otherwise this "fix" becomes
            # the bug (it broke two identity assertions in core/tests exactly once).
            for name in [m for m in sys.modules
                         if m == "cap_evolve" or m.startswith("cap_evolve.")]:
                sys.modules.pop(name, None)
        for finder in list(sys.meta_path):
            if "cap_evolve" in getattr(finder, "MAPPING", {}):
                sys.meta_path.remove(finder)
        return
    # A checkout's own core outranks an ambient install. Without this, a skill script run
    # from checkout X silently executed against checkout Y's cap_evolve (an editable install
    # registers a sys.meta_path finder, which outranks sys.path), and the only symptom was
    # missing modules — or, worse, a green result measured against the wrong tree.
    here = Path(__file__).resolve()
    own = next((p / "core" for p in here.parents
                if (p / "core" / "cap_evolve" / "__init__.py").exists()), None)
    if own is not None:
        os.environ.setdefault("CAPEVOLVE_CORE", str(own))
        return _seed_path()
    try:
        import cap_evolve  # noqa: F401
        return
    except Exception:
        pass
    cands = []
    for parent in here.parents:
        cands.append(parent / "core")
        cands.append(parent)
    for c in cands:
        if (c / "cap_evolve" / "__init__.py").exists():
            p = str(c)
            if p not in sys.path:
                sys.path.insert(0, p)
            return


_seed_path()
from cap_evolve._bootstrap import ensure_core  # noqa: E402

# Anchor the upward walk at THIS skill script's location (not the core module's).
ensure_core(Path(__file__).resolve())