{
  "skill": "alterlab-link-health",
  "evals": [
    {
      "id": "full-repo-audit",
      "prompt": "Our docs repo's lychee link-check GitHub Action is failing with over a thousand broken-link errors. Audit and repair the link health of AlterLab-IEU/our-docs and get the checker back to green.",
      "expected_output": "Invokes alterlab-link-health: dispatches the full four-tier pipeline (config hardening, intra-repo file-ref fixes, external URL substitutions, residual exclusions) per playbooks/full-audit.md, classifies each failing link by tier, and applies fixes while enforcing the Tier 3 substitution guardrail so no previously-passing link regresses.",
      "assertions": [
        { "type": "should_trigger", "value": true },
        { "type": "output_contains", "value": "tier" },
        { "type": "behavior", "value": "Runs the four-tier audit-and-repair pipeline for a lychee-based checker and reports per-tier link classifications." }
      ]
    },
    {
      "id": "residual-followup",
      "prompt": "The lychee link checker is down from 1208 to 67 errors after the first pass. Close out the remaining residual broken links.",
      "expected_output": "Invokes alterlab-link-health and dispatches the targeted residual followup pass (playbooks/followup-pass.md), resolving the remaining links via Tier 2/3 substitutions or Tier 4 exclusions without regressing any link that already passes.",
      "assertions": [
        { "type": "should_trigger", "value": true },
        { "type": "output_contains", "value": "residual" },
        { "type": "behavior", "value": "Handles the residual-pass scenario rather than re-running a full audit, preserving the no-regression guardrail." }
      ]
    },
    {
      "id": "dead-links-generic-checker",
      "prompt": "Can you find and fix all the dead links and 404s in our Markdown docs? We use markdown-link-check in CI.",
      "expected_output": "Invokes alterlab-link-health: applies the same tiered methodology generalized from lychee to markdown-link-check, hardening checker config, repairing intra-repo file references, substituting dead external URLs, and excluding irreducible residuals, with the Tier 3 regression guardrail in force.",
      "assertions": [
        { "type": "should_trigger", "value": true },
        { "type": "output_contains", "value": "tier" },
        { "type": "behavior", "value": "Recognizes the generic markdown-link-check checker and applies the same four-tier link-repair methodology, keeping the no-regression guardrail in force." }
      ]
    },
    {
      "id": "near-miss-citation-verifier",
      "prompt": "Check whether the references in my paper's bibliography actually exist — I'm worried some DOIs are fabricated or hallucinated.",
      "expected_output": "Does NOT invoke alterlab-link-health; defers to alterlab-citation-verifier. The user wants scholarly-citation existence verification against Crossref/OpenAlex/etc., not Markdown link-health auditing of a docs repo. Link-health repairs broken hyperlinks in docs; it does not validate that a cited paper exists.",
      "assertions": [
        { "type": "should_not_trigger", "value": true },
        { "type": "output_contains", "value": "citation-verifier" },
        { "type": "behavior", "value": "Distinguishes bibliographic existence-verification (citation-verifier) from Markdown link-health repair and defers." }
      ]
    }
  ]
}

# Example — PR #1 Retrospective

The source audit that generated this skill. Real numbers, real commits, one real self-correction.

- **Repo**: `AlterLab-IEU/AlterLab-Academic-Skills`
- **PR**: [#1 fix(ci): harden link-check workflow and repair broken refs (link-health audit)](https://github.com/AlterLab-IEU/AlterLab-Academic-Skills/pull/1)
- **Audit + merge**: 2026-04-21, squash commit `93a72fe`
- **Branch**: `fix/link-health-audit` (deleted on merge)

## Before / after

Against the weekly `Check Links` workflow run 24662239329 (scheduled, 2026-04-20) for baseline, and `workflow_dispatch` run 24710712721 on the branch for final:

| Metric | Baseline (main) | After (fix/link-health-audit) | Δ |
|--------|----------------:|-------------------------------:|---:|
| Total links | 1966 | 1912 | -54 |
| ✅ Successful | 684 | **1497** | **+813** |
| 🔀 Redirected | 26 | 265 | +239 |
| 👻 Excluded | 42 | 148 | +106 |
| 🚫 **Errors** | **1208** | **0** | **-1208** |
| ⏳ Timeouts | 4 | 0 | -4 |
| ⛔ Unsupported | 2 | 2 | 0 |

Why the totals shifted by 54:
- Broken intra-repo `file://` refs that resolved in Tier 2 collapsed several near-duplicate entries (the same missing target was linked from multiple markdown files).
- The wider accept set + broader `.lycheeignore` reclassified many entries as Redirected or Excluded instead of probed-and-rejected.

## Commits on the branch (8 + 1 post-merge)

| # | SHA | Tier | Message |
|---|-----|------|---------|
| 1 | `96b5c27` | Tier 1 config | ci(links): add .lychee.toml, .lycheeignore, harden check-links workflow |
| 2 | `91079e9` | Tier 2 file refs | docs(refs): fix broken intra-repo Markdown links in skills tree |
| 3 | `0c8c272` | Tier 3 substitutions | docs(refs): replace moved / dead external URLs with current locations |
| 4 | `f025748` | Tier 4 exclusions | ci(links): extend .lycheeignore for unfixable / placeholder URLs |
| 5 | `9cbd801` | Fix-up | fix(links): repair post-dispatch regressions and broaden exclude list |
| 6 | `31fa0f5` | Fix-up | fix(links): close last 5 residual errors |
| 7 | `204e591` | Fix-up | fix(links): bump retries + exclude two chronically-flaky hosts |
| 8 | `426363c` | Post-merge cleanup | chore(links): finalize benchling substitution, document link debt |

Merged as `93a72fe` on `main`.

## Error trajectory

| Stage | Errors | Delta |
|-------|-------:|------:|
| Baseline (main) | 1208 | — |
| After Tier 1 (commit 1) + Tier 2 (commit 2) + Tier 3 (commit 3) + Tier 4 (commit 4) — first dispatch | 67 | **-1141** |
| After fix-up (commit 5) | 5 | -62 |
| After fix-up (commit 6) | 1 | -4 |
| After fix-up (commit 7) | 0 | -1 |

## The self-correction (why the Tier 3 guardrail exists)

Commit 3 (`0c8c272`) applied 57 URL substitutions across 46 files via a single batched `sed` pass. When the first dispatch landed at 67 errors (not the predicted 10-40), the triage surfaced three classes of regressions the Tier 3 commit had introduced:

1. **Double-append from prefix rules.** `s|/v3/|/v3/docs|g` also matched URLs already ending in `/v3/docs`, producing `/v3/docsdocs`. Affected: `api-v3.monarchinitiative.org`.
2. **Concatenation onto specific paths.** `s|pylabrobot.org/api/|pylabrobot.org/user_guide/index.html|g` merged onto trailing segments, producing `user_guide/index.htmlpylabrobot.liquid_handling.html`. Affected: 5 pylabrobot reference files.
3. **Chain-substitution overreach.** `s|clinvar/docs/|clinvar/docs/help/|g` rewrote specific sub-paths like `clinvar/docs/xsd_public/` into non-existent `clinvar/docs/help/xsd_public/`. Affected: 4 ClinVar reference files.

Plus one Tier 2 crossover bug: `s|reference/|references/|g` also matched inside an external URL `plotly.com/python-api-reference/`, turning it into `python-api-references/` (404).

Commit 5 (`9cbd801`) repaired all 11 regressions with narrower sed patterns. The Tier 3 guardrail rule (`references/tier3-substitution.md`) was derived from this experience: **rerun the link checker after each substitution pass and diff the baseline's 200-set against the post's 200-set**. Deletions from that diff are guaranteed regressions.

## Post-merge cleanup (commit 8 / PR `chore`)

The `benchling/benchling-sdk` GitHub URL was left in `.lycheeignore` during Tier 4 because the obvious substitute `benchling/benchling-api-client` returned 404 on the GitHub API (verified with `gh api repos/benchling/benchling-api-client` before choosing exclude over replace — an application of the Verification-First Rule). Later, `curl -sSI https://pypi.org/project/benchling-sdk/` returned 200, so the URL was substituted to PyPI in the post-merge cleanup, and the exclusion block was removed.

Filed as follow-up issues alongside commit 8:
- [#2 Pylabrobot docs refresh after MyST migration settles](https://github.com/AlterLab-IEU/AlterLab-Academic-Skills/issues/2) — tracks the `docs.pylabrobot.org/*` blanket exclusion.
- [#3 SKILL.md schema drift audit](https://github.com/AlterLab-IEU/AlterLab-Academic-Skills/issues/3) — schema observations surfaced during the audit, deliberately kept out of the link-health PR.

Created `.github/KNOWN_LINK_DEBT.md` documenting every current `.lycheeignore` entry with reason, date, and revisit trigger or tracking issue.

## Lessons that shaped this skill

1. **Config before content.** Tier 1 alone cut 900+ errors. Fixing intra-repo refs and replacing dead URLs on top of a broken `--accept` flag would have wasted cycles on work the config fix obviated.
2. **Verify every substitute.** One unverified `benchling-api-client` guess that actually 404s would have ridden to main and confused maintainers for months. `gh api repos/<owner>/<name>` is cheap; trust is expensive.
3. **Broad sed is a trap.** Applying 50+ prefix substitutions in one pass produced three classes of regressions not one of which was caught by reading the commit diff. The guardrail diff caught all of them on the first dispatch.
4. **Scope discipline compounds.** Every observation from the audit that *wasn't* link-health (SKILL.md schema, license-field normalization, category renames) got filed as a separate issue. The PR stayed one-topic and reviewable.
5. **The rationale is the PR body, not the commit.** Root-cause explanations survive longer in PR descriptions than in squashed commit messages. A future maintainer reads the PR first.

## Reproducing this audit on another repo

The 10-agent playbook in `playbooks/full-audit.md` is the dispatch prompt. Swap `AlterLab-IEU/AlterLab-Academic-Skills` for the target repo, point the scratch dir at a fresh location, and follow the tier-by-tier APPLY sequence. Expect 30-60 minutes from cold start to PR green on a repo of similar size (~2000 links).

# Playbook — Follow-up pass (3-agent targeted residual)

Use when: the initial APPLY phase reduced errors significantly (e.g. 1208 → 67) but some residuals remain. Typical residuals cluster into three independent classes, each suited to its own subagent.

## Shape

```
fetch latest CI run  →  classify residuals  →  3 parallel subagents  →  single fix-up commit  →  redispatch  →  verify
```

## Fetch and classify

Pull the latest failing run's log:

```bash
gh run view <id> --log-failed > audit/after1.log
grep -E '\[ERROR\]|\[40[0-9]\]|\[5[0-9][0-9]\]|\[TIMEOUT\]' audit/after1.log \
  | sort -u > audit/residuals.txt
```

Classify each residual into one of three buckets:

| Class | Examples | Typical cause |
|-------|----------|----------------|
| **Substitution regressions** | `/v3/docsdocs`, `/api/swagger-ui/index.htmlswagger-ui/...`, over-greedy prefix rewrites | Tier 3 sed bugs — concatenation, double-append, chain-substitution overreach. |
| **Genuinely dead / flaky infra** | Upstream docs sites that 404'd a page, hosts with consistent CI-side failures, SSL/OCSP issues | Need either a better substitute (verify!) or an exclusion with rationale. |
| **Config-fixable** | Flaky-but-live hosts that need more retries or a longer timeout; one endpoint that answers GET with 400 (e.g. a POST-only GraphQL URL) | Tune `max_retries` / `timeout`, or exclude that exact URL in `.lycheeignore` with a comment. Widen `accept` only for statuses that are benign on every host (403/429) — never 400 or 5xx. |

## Parallel agent dispatch (3 subagents)

| Agent | Name | Responsibility |
|-------|------|----------------|
| **agent-R** | `regression-repair` | For each substitution-regression URL: grep the repo to find the broken pattern, write a targeted corrective sed (narrow scope — don't re-use the broad Tier 3 rules). Verify each repair with curl. Produce the fix-up commit content for these files. |
| **agent-D** | `dead-infra-triage` | For each dead/flaky URL: probe from this box (curl with reasonable timeout) to confirm the host state. Choose exclude-with-rationale vs. find-a-real-substitute. Verify substitutes before proposing them. Output `.lycheeignore` additions + any URL rewrites. |
| **agent-C** | `config-tuning` | Identify config-fixable residuals. Propose `accept = [...]` additions or `max_retries` bumps. Keep additions narrow — never blanket-accept 5xx. |

All three write their output to `audit/followup-<name>.md` and do not commit.

## Synthesize into one commit

Single commit with message describing all three classes:

```
fix(links): repair Tier 3 regressions + broaden exclude list

<paragraph describing the regression bugs>
<paragraph describing the dead-infra exclusions>
<paragraph describing the config tuning>
```

Why one commit: reviewers can see the full cleanup logic in one place, and the guardrail diff (baseline-vs-post-this-commit) shows one net effect.

## Re-dispatch + verify

After pushing the fix-up:

1. `gh workflow run check-links.yml --ref <branch>`.
2. Wait for completion via a polling monitor.
3. Run the Tier 3 guardrail diff once more. Any residual that's a regression goes back through agent-R. Any new dead-infra gets agent-D.

## Stopping criteria

Stop when one of:

- Errors = 0 (ideal).
- Errors = N (small, single-digit) and every residual has a rationale comment in `.lycheeignore` / `KNOWN_LINK_DEBT.md`.
- Consecutive dispatches show different residuals each time — that's CI flake, and the config should be tuned (bump retries) rather than pursuing individual URLs.

# Playbook — Full audit (10-agent dispatch)

Use when: a repo's `Check Links` workflow is failing with a large error count (hundreds+), no prior link-health work has been done, and you want a complete audit + fix cycle.

## Shape

```
pre-flight  →  10 parallel subagents  →  master report  →  4-tier APPLY  →  CI dispatch  →  post-dispatch cleanup if residuals
```

## Pre-flight (sequential, fail fast)

1. Verify tools: `gh auth status`, `git --version`, `node --version`, `lychee --version` (install if needed — falls back to `markdown-link-check` via `npx` if lychee not installable).
2. Verify network egress to `github.com` and `raw.githubusercontent.com`.
3. Create scratch dir: `~/work/<repo>-audit-$(date +%Y%m%d-%H%M)`.
4. Clone: `gh repo clone <owner/repo>` (full clone, not shallow — need the full markdown tree).
5. Log current default branch, latest commit SHA, last CI run status via `gh run list --limit 5`.
6. Grab the latest failing `Check Links` run log: `gh run view <id> --log-failed > audit/baseline.log`. Parse the summary block for total / successful / errors counts.

## Parallel agent dispatch (10 subagents, single message)

Each agent gets a self-contained prompt with the repo path, the baseline log path, and the specific deliverable filename under `audit/`.

| Agent | Name | Responsibility |
|-------|------|----------------|
| agent-1 | `lychee-config` | Draft corrected `.lychee.toml` + rationale. Confirm no existing config. |
| agent-2 | `broken-file-refs` | Classify every `[ERROR] file://` entry as PATH_TYPO / WRONG_PATH / MISSING_FILE. Per-file fix table. |
| agent-3 | `<worst-skill>-deep-dive` | Deep dive on the worst-affected skill (usually one concentrates the bulk of intra-repo breakage). Directory restructure plan. |
| agent-4 | `genuine-404s` | For every unique 404 URL, WebFetch to categorize DEAD_PERMANENT / MOVED / TRANSIENT / PLACEHOLDER. Replacement URL table. |
| agent-5 | `network-ssl-issues` | For every `[ERROR] network` / `[TIMEOUT]`, probe from this box with curl. Categorize DEAD_INFRA / SSL_EXPIRED / TRANSIENT_CI / RATE_LIMITED. |
| agent-6 | `doi-redirects` | Confirm publisher 403 redirects (BMJ, Oxford, Sage, OECD, Ensembl) are fixable via config alone. No URL changes. |
| agent-7 | `skill-md-structural` | Spot-check 20 random `SKILL.md` files against the canonical frontmatter schema. Drive-by integrity pass. |
| agent-8 | `ci-pipeline` | Inspect `.github/workflows/`. Draft hardened `check-links.yml`. Web-check current `actions/checkout` and `lychee-action` majors. |
| agent-9 | `badges-and-shields` | Confirm badge / shield URLs are false positives. Draft `.lycheeignore` badge block. |
| agent-10 | `prioritization-and-pr-plan` | **Sequential; depends on 1-9.** Synthesize 4-tier plan + PR title/body + consolidated human-decision list. |

**Dispatch 1-9 in parallel** (single message with 9 Agent tool calls). Then dispatch agent-10 after 1-9 complete.

Each agent's prompt must include:
- Absolute repo path.
- Absolute baseline log path.
- Output file path (`audit/agent-N.md`).
- "No commits, no pushes. Write only to `audit/`."
- Concrete budgets for web-probing agents (e.g. "up to 60 URLs max").

## Synthesize

After all 10 reports land:
- Write `audit/MASTER_REPORT.md` — executive summary pointing to agent-10.
- Write `audit/diff-preview.txt` — concrete file changes per tier.
- **Pause for human review** before APPLY. Present the consolidated human-decision list.

## APPLY (4 tiered commits on `fix/link-health-audit`)

Per `references/tier1-config.md` through `references/tier4-exclusions.md`. Each tier = one commit. See `examples/pr-1-retrospective.md` for real commit boundaries.

After all four commits: push, `gh pr create`, trigger `workflow_dispatch` on the branch, link the run in a PR comment.

## Post-dispatch

Run the guardrail diff (`references/tier3-substitution.md`). If residuals:
- **Regressions** (previously-200 URLs now failing) — revert the matching substitutions via a fix-up commit.
- **New residuals** (URLs that were also broken in baseline, still broken now) — cleanup-pass via `playbooks/followup-pass.md`, or additional `.lycheeignore` entries.

When errors hit 0 (or the agreed residual target), the PR is ready to merge.

## Expected timeline

- Pre-flight: ~2 min
- 9 parallel agents: 3-6 min total (slowest one gates)
- Agent-10 synthesis: ~2-3 min
- User review: human-paced
- APPLY 4 commits: 5-10 min
- First dispatch + analysis: ~3 min
- Post-dispatch cleanup: 1-3 additional fix-up commits, 5-10 min per dispatch cycle

Total assistant time: 30-60 minutes from cold start to PR green.

# Playbook — Post-merge cleanup (4-agent)

Use when: the link-health PR is CI-green and mergeable, but some human-decision items from the audit deserve finalization before merge. Converts deferred decisions into verified substitutions, follow-up issues, and durable documentation.

## Shape

```
pre-flight  →  4 parallel subagents  →  patch any cross-agent references  →  single cleanup commit  →  push  →  wait push-CI  →  squash-merge --delete-branch
```

## Pre-flight

1. `cd` to the cloned repo. Confirm current branch.
2. `gh pr view <N>` — confirm state is OPEN, mergeable CLEAN.
3. `git pull --ff-only` — local must be current with origin.
4. `gh auth status` — token must have `repo` scope.

## Parallel agent dispatch (4 subagents)

| Agent | Name | Responsibility |
|-------|------|----------------|
| **agent-A** | `<dead-sub>-substitution` | Take one or more specific dead URLs that were left excluded in Tier 4 and verify alternative targets. Example: replace `github.com/<dead-repo>` with the verified-alive PyPI/npm landing page. **Must `curl -sSI`** or `gh api` before committing. Remove the corresponding exclusion block from `.lycheeignore`. |
| **agent-B** | `followup-issues` | Open GitHub issues for out-of-scope observations from the audit. Each issue needs a title, labels (create missing labels first via `gh label create`), and a body referencing the audit PR. Typical issues: upstream migration tracking, schema drift, deferred substitutions. |
| **agent-C** | `link-debt-doc` | Create `.github/KNOWN_LINK_DEBT.md` populated from the actual `.lycheeignore` plus audit artefacts. Five categories (see `references/known-debt-template.md`). Link any temporary exclusions to agent-B's issue numbers. |
| **agent-D** | `guardrail-in-skill` | Locate the installed link-health skill on the user's system (`~/.claude/skills/`, plugin caches, or an AlterLab skill repo). If present, append the Tier 3 guardrail rule to a `## Guardrails` section. If absent, report and skip. |

All four run in parallel. Agents B, C, D don't modify the skills repo (agent-B opens issues, agent-C writes `.github/KNOWN_LINK_DEBT.md`, agent-D touches an unrelated skill file or reports absent).

## Patch cross-agent references

Agent-C's output references agent-B's issue numbers (tracking-issue cells for the "Upstream migrations" category). After both land:

- Replace the placeholder `<ISSUE_NUMBER_*>` strings in `KNOWN_LINK_DEBT.md` with the real issue numbers from agent-B's output.

## Single cleanup commit

One commit covering everything agents A + C produced (agent-B's work is on the GitHub issue tracker, agent-D's work is in another repo/skill dir):

```
chore(links): finalize <dead-sub> substitution, document link debt
```

Body paragraphs should cover:
- The verified substitution (what was excluded, what replaced it, how you verified).
- The new `KNOWN_LINK_DEBT.md` and what it tracks.
- References to the follow-up issues agent-B opened.

## Push and merge

```bash
git push origin <branch>
# Wait for push-triggered CI (e.g. Validate Skills) to complete.
# Check Links does not auto-trigger on push — only schedule + workflow_dispatch.
gh pr merge <N> --squash --delete-branch
```

Squash-merge gives a clean single commit on `main` with the full audit narrative in the title and the expanded 7+1-commit detail in the squash-merge description (which GitHub pre-populates from the PR body).

## Verify

- `git fetch && git log origin/main --oneline -2` — confirm the merge commit is on main.
- `gh api repos/<owner>/<repo>/branches/<branch>` should return 404 (branch deleted).
- Check `KNOWN_LINK_DEBT.md` exists on `main` and its issue-number placeholders resolved.

## Sanity checks that tend to fail in this playbook

- **Labels don't exist.** `gh issue create --label <x>` fails if the label isn't in the repo. Agent-B should always `gh label list` first and create missing labels.
- **Substitute URL turns out to be archived.** The substitute was 200 when you probed, but if the repo is archived, the URL is still "alive" yet no longer maintained. Prefer `gh api repos/<owner>/<name>` and check `"archived": false`.
- **`.lycheeignore` removal without `KNOWN_LINK_DEBT.md` removal.** If agent-A removes an exclusion, agent-C must not list it under "Dead substitutions." Run a consistency grep before committing.

# `KNOWN_LINK_DEBT.md` — Template

Every repo that runs this skill should ship a `.github/KNOWN_LINK_DEBT.md` documenting every `.lycheeignore` entry with rationale. The file is the durable memory of **why** each exclusion exists.

## Structure

Five category sections, in this order. Empty sections keep their header with "No current entries." — the presence of the category keeps maintainers from re-categorizing when future entries land.

```markdown
# Known Link Debt

URLs excluded from link checking that warrant future review. Each entry explains *why* the exclusion exists so maintainers do not need to reconstruct context. Keep this file in sync with `.lycheeignore` — if you add or remove an exclusion pattern, update the corresponding row here (or add a new one with rationale).

## Categories

### Infrastructure hostility (permanent)

URLs where the host systematically blocks or fails for CI runners, regardless of any fix on our side. Rows are permanent unless the host changes policy.

| URL pattern | Reason | Date excluded | Revisit trigger |
| --- | --- | --- | --- |
| `<host>/*` | <why CI always fails> | YYYY-MM-DD | <what would change to revisit> |

### Upstream migrations (temporary)

Exclusions expected to be removed once upstream projects finish doc restructures. Each row tracks an issue to reassess.

| URL pattern | Reason | Date excluded | Tracking issue |
| --- | --- | --- | --- |
| `<host>/*` | <upstream state> | YYYY-MM-DD | #<issue> |

### Pedagogical placeholders (deliberate)

URLs that are intentional filler in template or example files. They demonstrate a citation FORMAT, not real resources. These should stay excluded indefinitely.

| URL pattern | Reason |
| --- | --- |
| `<regex>` | <pedagogical use case> |

### <Language/domain> journal / resource DOIs (needs native reader review)

<Optional: only if the repo has DOIs that a native-language or domain-expert reader should verify against source citations. Omit section if N/A.>

- `<DOI>` ← <note on suspected issue>

### Dead substitutions (review for replacement)

URLs where the original target is dead and no verified replacement has been committed. Empty state is the goal — populating this section means somebody guessed and left a breadcrumb for future verification.

No current entries.

## Maintenance

- When adding to `.lycheeignore`, add a row here with reason + date.
- When removing an entry from `.lycheeignore` (because upstream fixed the issue), delete the corresponding row here.
- Entries in "Upstream migrations" should link a tracking issue; close the entry when the issue resolves.
```

## Sync discipline

A `.lycheeignore` entry without a corresponding `KNOWN_LINK_DEBT.md` row is a bug. Add a CI check (or a CODEOWNERS comment on `.lycheeignore` changes) that requires both files to change together in any PR that modifies either. The two files must stay in sync or the debt doc becomes lying-by-omission.

## Writing good reasons

Good reason text answers three questions in one or two sentences:

1. What's wrong with the URL from CI's perspective?
2. Why can't we fix it with a substitute or accept-list change?
3. What would have to change upstream (or in tooling) for us to re-include this URL?

Example:

> **`*.stlouisfed.org/*`** — Consistently fails HTTP/2 handshakes from GitHub Actions runners (live from desktop). `max_retries = 5` doesn't help; handshake fails deterministically per-host. **Revisit when** lychee's HTTP/2 handling improves or GitHub Actions runners update their libcurl.

Bad reason text:

> **`*.stlouisfed.org/*`** — Fails in CI.

The bad version loses the "HTTP/2 handshake" detail and the revisit trigger — six months later someone reading this has to rediscover both.

# Tier 1 — Config

Goal: eliminate CI-level false positives with a single reviewable commit that introduces three artefacts at the repo root.

## Files to create

1. `.lychee.toml` — lychee's config file. lychee only auto-loads `lychee.toml` (no leading dot) from the working directory, so pass this one with `--config`.
2. `.lycheeignore` — regex-per-line exclusion list.
3. `.github/workflows/check-links.yml` — hardened workflow.

## `.lychee.toml` template

```toml
# .lychee.toml — pass it explicitly with `--config .lychee.toml` (see workflow below).

# HTTP status codes treated as success. Both this key and the CLI `--accept`
# flag REPLACE lychee's default set (100..=103, 200..=299) rather than extending
# it — a common footgun — so the 2xx range is listed explicitly.
accept = ["200..=299", "301", "302", "304", "308", "403", "429"]

# Cache results between scheduled runs. lychee writes `.lycheecache`.
cache = true
# Drop cached results after a day so a transient outage is not pinned as OK/ERROR.
max_cache_age = "1d"

# Per-request timeout (lychee default: 20). 30 gives slow academic mirrors a fair chance.
timeout = 30

# Back-off before retrying a failed request (default: 1).
retry_wait_time = 5

# Retry transient failures (5xx, network errors, 429) up to this many times
# (default: 3). 5 absorbs most DOI / publisher flakes without bloating CI time.
max_retries = 5

# Follow redirect chains common to DOIs and archive.org links (default: 10).
max_redirects = 10

# Paths to skip entirely. Entries are REGULAR EXPRESSIONS matched against the
# path, not literal paths: a bare ".git" would also match "digital-humanities"
# (any character + "git"). Anchor and escape them.
exclude_path = ["(^|/)node_modules/", "(^|/)\\.git/", "(^|/)\\.github/ISSUE_TEMPLATE/"]

# Placeholder / example DOIs deliberately included in templates.
exclude = [
  "^https?://doi\\.org/x+$",
  "^https?://doi\\.org/10\\.x+",
  "^https?://doi\\.org/xx\\.",
  "^https?://doi\\.org/10\\.xxx/yyy",
  "^https?://doi\\.org/10\\.xxxx/yyyy",
  "^https?://doi\\.org/xx\\.xxx/yyyy",
  "^https?://your-tenant\\.benchling\\.com/.*",
]
```

### Why each `accept` code

- `200..=299` — normal success.
- `301, 302, 304, 308` — redirects; combined with `max_redirects = 10` covers DOI → publisher chains.
- `403` — academic / publisher sites (BMJ, Oxford Academic, Sage, OECD, Ensembl) bot-block automated requests from CI runners even when the content is public.
- `429` — rate-limited responses on retry are transient, not broken links.

**Do not** accept `400`. A 400 means the host actively rejected the request, and accepting it globally hides genuinely broken or moved API endpoints and malformed URLs — exactly the rot the check exists to surface. If one specific endpoint (for example a GraphQL URL that only answers POST) returns 400 to a GET probe, exclude that exact URL in `.lycheeignore` with a comment. This repo's source audit originally accepted 400 for such endpoints and later removed it after re-probing showed nothing relied on it.

**Do not** blanket-accept `500..=504`. Those indicate upstream infrastructure problems. Handle per-host in `.lycheeignore` with a comment.

lychee sends `GET` by default. Hosts that reject bot traffic sometimes accept a different method; current lychee (v0.24.x) accepts `--method head,get` to try each in turn, but prefer a per-host exclusion over widening methods globally.

## `.lycheeignore` starter

```
# Badge / shield services — always 200 but noisy.
^https?://img\.shields\.io/.*
^https?://shields\.io/.*
^https?://awesome\.re/.*
^https?://capsule-render\.vercel\.app/.*

# Anti-bot / auth-walled hosts.
^https?://([a-z0-9-]+\.)?linkedin\.com/.*
^https?://(twitter\.com|x\.com|t\.co)/.*

# gitter.im 301s to matrix.to and trips max-redirects.
^https?://gitter\.im/.*
```

Add host-specific entries during Tier 4 (see `tier4-exclusions.md`) with a `#` comment above each group explaining why the host is excluded.

## Workflow (`.github/workflows/check-links.yml`)

```yaml
name: Check Links

on:
  schedule:
    - cron: "0 9 * * 1"
  workflow_dispatch:

permissions:
  contents: read

jobs:
  check-links:
    name: Check for Dead Links
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v7

      - name: Restore lychee cache
        uses: actions/cache@v6
        with:
          path: .lycheecache
          key: cache-lychee-${{ github.sha }}
          restore-keys: cache-lychee-

      # lychee writes its cache to .lycheecache itself (cache = true in the config).
      # `output` is the Markdown REPORT path — never point it at the cache file,
      # or each run overwrites the cache with the report.
      - name: Check links in Markdown files
        uses: lycheeverse/lychee-action@v2
        with:
          args: --config .lychee.toml --no-progress "**/*.md"
          output: ./lychee/out.md
          jobSummary: true
          fail: true
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
```

### Hardening checklist

- Current majors as of 2026-09: `actions/checkout@v7`, `actions/cache@v6`, `lycheeverse/lychee-action@v2` (v2.9.x, which installs lychee v0.24.2 by default). Pin to a commit SHA if your org requires it, and re-check majors when you touch the workflow.
- `actions/cache` keyed on SHA — consecutive scheduled runs reuse already-verified results (bounded by `max_cache_age`).
- `output` is the report path (default `lychee/out.md`), distinct from `.lycheecache`; `jobSummary: true` puts the report in the run summary.
- Args reference `.lychee.toml` via `--config` — keep inline args to the glob only.
- `fail: true` — CI fails on residual errors so regressions are visible.
- `permissions: contents: read` — least privilege; the workflow doesn't need write.
- `GITHUB_TOKEN` exposed — lychee authenticates github.com URLs (higher rate limit).

## The gotcha that motivated this tier

The original workflow passed `args: --verbose --no-progress --accept 403 "**/*.md"`. The `--accept` flag **replaces** the default accept set instead of extending it — so every `200 OK` was reclassified as a failure. That single line accounted for the bulk of the 1208 baseline errors. Moving the config to `.lychee.toml` makes the accepted-status set explicit and reviewable, and survives future CLI-flag drift.

**Write the root cause into the PR body**, not just the commit message: "the `--accept 403` flag on the lychee CLI replaces the default accept set rather than extending it — every 200 OK was rejected as a result." That one sentence saves the next person from rediscovering the footgun in six months.

# Tier 2 — Intra-repo file references

Goal: eliminate every `[ERROR] file://` entry in the lychee log. These are broken relative-path references between Markdown files in the same repo — always real breakage, never a CI artefact.

## Extract the target list

After Tier 1 lands and the link checker reruns, grep the log for local-file errors:

```bash
grep '\[ERROR\] file:///' baseline.log \
  | sed 's|^.*file:///home/runner/work/<repo>/<repo>/||' \
  | sort -u
```

Strip the CI runner's checkout prefix so what remains is repo-relative. Dedupe — the same broken target often appears in multiple files.

## Classification

For each unique missing path, decide one of three categories:

| Category | Signal | Fix |
|----------|--------|-----|
| **PATH_TYPO** | Target would exist under a near-identical name (`reference/` vs `references/`, `mermaid_diagrams/` vs `diagrams/`, `X.md` vs `x.md`). | Mechanical `replace_all` on the referring file(s). |
| **WRONG_PATH** | Target file exists elsewhere in the repo; link is missing `../` or has an extra prefix. | Rewrite the link with the correct relative path. |
| **MISSING_FILE** | Target genuinely doesn't exist, and the link is inside a **template file** with illustrative prose. | Decide per Mermaid-placeholder policy (see below). |

## Common bug patterns

### Singular/plural directory name drift

When a skill is ported from an upstream project, the link URLs often preserve the upstream folder name (`reference/`) while the ported folder uses a different convention (`references/`). Grep + replace per-file:

```bash
grep -c '](reference/' skills/<skill>/SKILL.md
# N occurrences

# Safe if `reference/` is never a substring of `references/` (it isn't):
sed -i 's|](reference/|](references/|g' skills/<skill>/SKILL.md
```

**Check for collision**: before a bare `s/OLD/NEW/g`, grep for the candidate `NEW` text already in the file — if the substitution would corrupt existing occurrences, scope the match with a prefix (e.g. `](reference/` instead of `reference/`).

### Missing path prefix

Templates referencing style guides at `../X.md` when the guides actually live at `../references/X.md`. Same sed pattern, scoped to link syntax:

```bash
sed -i 's|](../markdown_style_guide\.md|](../references/markdown_style_guide.md|g' \
       skills/<skill>/templates/*.md
```

### YAML frontmatter indentation

If the repo's `validate-skills.yml` workflow parses YAML frontmatter, subtle indentation bugs under `metadata:` (e.g. a field indented 4 spaces under a `metadata:` block at column 0 with sibling fields at 2 spaces) will break the parser. These surface as Tier-2-adjacent issues worth catching while you're in the tree — but file anything beyond trivial indentation fixes as a separate schema PR.

## Mermaid-placeholder policy (pedagogical links)

Some template files deliberately include links to paths that **don't exist in the skill repo** — they illustrate the filename convention the skill's downstream consumer should adopt (`../adr/ADR-001-<slug>.md`, `../../docs/project/issues/issue-00000001-<slug>.md`, etc.).

Three options per placeholder:

| Option | Result | When |
|--------|--------|------|
| **A. Backtick-wrap** | ``[Label](../adr/foo.md)`` → ``Label — `../adr/foo.md` `` — inline code span. Lychee skips it; pedagogical value preserved. | **Default**. Preserves the filename example as visible code. |
| **B. Strip the link** | ``[Related ADR](../adr/foo.md)`` → `Related ADR` | When the path adds no teaching value. |
| **C. Create stub file** | Author a minimal `adr/ADR-001-<slug>.md` | When the referenced file is part of the skill's own promised content. |

**Default to A.** Option C should only be picked when the surrounding prose treats the file as skill-internal authoritative content, not an example for downstream consumers.

A precise sed for option A (note: POSIX ERE, `~` delimiter to avoid URL-slash collision):

```bash
sed -i -E 's~\[([^]]+)\]\((\.\./adr/[^)]*|\.\./workflow_guide\.md|\.\./operational_readiness\.md|\.\./\.\./docs/project/[^)]+)\)~\1 — `\2`~g' \
     skills/<skill>/templates/*.md
```

## Verify before committing

After the Tier 2 pass:

```bash
git diff --stat
# Line edits should be symmetric (N insertions, N deletions) — pure renames.

# Any asymmetric file-count line suggests unintended changes; inspect.
```

The PR's commit message should list the pattern types you applied, not every individual file — reviewers want to see the *rules*, not 300 lines of renames.

# Tier 3 — External URL substitution

Goal: replace MOVED or DEAD external URLs with verified replacements. This tier is where most self-inflicted regressions happen. Two rules are non-negotiable.

---

## THE GUARDRAIL

> **After any URL substitution pass, re-run the link checker and diff against the baseline success set. Any URL that returned 200 OK in baseline and is non-200 after substitution is a regression and MUST be reverted before commit.**
>
> **Self-check:**
>
> ```bash
> diff <(grep "^\[200\]" baseline.log | sort -u) \
>      <(grep "^\[200\]" post.log | sort -u)
> ```
>
> **Deletions are regressions.** The diff should show only additions (URLs that were failing in baseline and now pass). Any `< ` line is a URL you broke.

### Why this rule exists

During the source audit of this skill, a broad `sed` substitution pipeline applied prefix rewrites across the skill tree. Three of those rewrites had unintended collateral:

- `/v3/` → `/v3/docs` also matched URLs already ending in `/v3/docs`, producing `/v3/docsdocs`.
- `docs.pylabrobot.org/api/` → `docs.pylabrobot.org/user_guide/index.html` concatenated onto more specific paths, producing `user_guide/index.htmlpylabrobot.*.html`.
- `clinvar/docs/` → `clinvar/docs/help/` rewrote specific sub-paths like `clinvar/docs/xsd_public/` into non-existent `clinvar/docs/help/xsd_public/`.

None of these was caught by the commit's own review — they surfaced only when the link checker re-ran on the branch. **Assume your Tier 3 pass will have at least one such bug. Plan for the diff check.**

---

## THE VERIFICATION RULE

> **DEFAULT to probe-verification before any URL substitution. Unverified substitutions are how phantom URLs land in public skills.**

For every proposed `[old] → [new]`:

| Target type | Probe |
|-------------|-------|
| GitHub repo or user | `gh api repos/owner/name` — status must be 200 and `"archived": false` |
| HTTP URL | `curl -sSI -L --max-time 15 '<new>'` — final status 200 after redirects |
| PyPI / npm / crates package | Hit the registry landing page or API directly |
| Google Cloud / marketplace | Often fails HTTP/2 handshakes from CI even when alive — probe with `--http1.1` flag before trusting |

**If verification fails**, the correct action is to **exclude the dead target** via `.lycheeignore` with a commented rationale — **not** to guess a plausible-looking replacement.

Real example from the source audit:

```
Proposed: github.com/benchling/benchling-sdk → github.com/benchling/benchling-api-client
Probe:    gh api repos/benchling/benchling-api-client  →  404 Not Found
Action:   Exclude github.com/benchling/benchling-sdk.* in .lycheeignore.
          (Later, verified pypi.org/project/benchling-sdk/ returns 200 — substituted to PyPI in the post-merge cleanup.)
```

A wrong substitution that looks plausible is worse than an acknowledged exclusion: it misleads readers and is harder to spot in review.

---

## Sed-safety patterns

Bulk URL rewriting via `sed -f <scriptfile>` is faster than a hand-written `Edit` per file, but has sharp edges.

### Use a delimiter URLs don't contain

URLs contain `/` constantly and sometimes `&`, `|`, `%`. Pick `|` only if no URL has a pipe; otherwise `~` is a safe default:

```bash
sed -E 's~OLD~NEW~g'
```

### Anchor prefix substitutions

A rule like `s~cbioportal.org/api/~cbioportal.org/api/swagger-ui/index.html~g` will catch more specific paths you didn't intend. Anchor by the end of the URL when possible:

```bash
# Wrong — catches cbioportal.org/api/foo → cbioportal.org/api/swagger-ui/index.htmlfoo
s~cbioportal\.org/api/~cbioportal.org/api/swagger-ui/index.html~g

# Right — only matches the bare endpoint
s~cbioportal\.org/api/"~cbioportal.org/api/swagger-ui/index.html"~g
```

Or: make the old string more specific (include surrounding context) so accidentally overlapping targets aren't matched.

### Order matters

When one rule's `NEW` is a prefix of another rule's `OLD`, chain-substitutions can cascade. Either:

- Apply specific rules first, general rules last (so general rules don't see the specific rule's output), **or**
- Run each rule against the **original** file content (use `-e` with `sed` and read-modify-write explicitly, or apply rules one at a time with `git diff` inspection between).

### Scripted batches

For many substitutions, put them in a `.sedfile` and apply once per file:

```bash
find skills -type f -name '*.md' -print0 \
  | xargs -0 sed -i -f tier3.sedfile
```

This is ~100× faster than `xargs sed -i -e '<expr>' -e '<expr>' ...` for every rule.

---

## Probe-before-commit workflow

1. Compile the `[old] → [new]` proposal list (from agent reports or your own triage).
2. Batch-probe: loop over each `[new]` with `curl -sSI -L --max-time 15` or `gh api`. Emit a TSV of `old \t new \t status`.
3. Drop every proposal where `[new]` is not 200/2xx from the TSV. Those move to Tier 4 exclusions.
4. Apply the filtered sed script.
5. **Rerun the link checker.**
6. Run the guardrail diff. If any previously-200 URL is now non-200, that substitution is a regression — revert the specific sed rule or manually restore the URL.
7. Only then commit.

---

## Dealing with unavoidable regressions

Sometimes a correct substitution still surfaces a new error downstream: e.g. your new GCP marketplace URL for AlphaFold is correct but returns HTTP/2 errors from the CI runner. That's not a regression of a previously-working URL — the baseline didn't include this URL at all — so it's a **new residual**, not a violation of the guardrail.

New residuals get cleaned up in Tier 4 (`tier4-exclusions.md`), not reverted.

The guardrail is strict about one thing only: **don't make a previously-passing URL fail**.

# Tier 4 — Exclusions

Goal: get the link checker to zero errors (or to a stable documented residual set) by excluding URLs that genuinely cannot be fixed by content changes. Each entry in `.lycheeignore` is a policy decision — not a "make CI green" shortcut.

## Decision: exclude or substitute?

Walk the decision tree before adding an entry:

1. **Is the URL dead?** (`curl -sSI` times out / resolves to 404 / 5xx with no redirect)
   - **Is there a verified-alive replacement?** → Tier 3 substitute.
   - **No verified replacement?** → Exclude here with a `# Reason: <why>` comment. Document in `KNOWN_LINK_DEBT.md`.

2. **Is the URL alive in a browser but failing from CI?**
   - SSL / OCSP revocation failures → Exclude. Cannot be fixed repo-side.
   - Bot-block (anti-scraping 403/451/999) → Exclude. Likewise.
   - HTTP/2 handshake errors from GitHub Actions runners → Exclude. lychee limitation.
   - Transient 5xx or timeout on the flakiest few percent of runs → **First**, bump `max_retries` in `.lychee.toml`. Only exclude if retries don't help.

3. **Is the URL a deliberate placeholder in a template or example?**
   - `doi.org/xxxxx`-style pedagogical filler → Exclude via regex in `.lychee.toml`'s `exclude` array (not `.lycheeignore`) — these belong with config, not policy.
   - `your-tenant.<domain>` tenant placeholders → Same.
   - `localhost:port/*` dev-server references → `.lycheeignore`.

4. **Is the URL in a skill whose upstream is mid-migration?**
   - Example: `docs.pylabrobot.org/*` during the MyST rewrite.
   - **Blanket-exclude with a tracking issue**. Document in `KNOWN_LINK_DEBT.md` § "Upstream migrations" with the issue number. When upstream stabilizes, the issue's closure is the signal to re-audit that skill and remove the exclusion.

## Category rubric

Every `.lycheeignore` block should fit one of these five categories:

| Category | Permanence | Example patterns |
|----------|-----------|------------------|
| **Infrastructure hostility** | Permanent unless host policy changes | `linkedin.com`, `twitter.com`, `*.stlouisfed.org`, `dicom.nema.org`, FDA deep links that bot-block |
| **Pedagogical placeholders** | Permanent (deliberate) | `doi.org/x+`, `your-tenant.benchling.com`, `localhost:*` |
| **Upstream migrations** | Temporary; track in issue | `docs.pylabrobot.org/*` during MyST rewrite |
| **Dead infrastructure, no substitute** | Permanent until someone finds replacement content | `iqtree.org/workshop/molevol2022*` (SSL expired), `fged.org/projects/miame/?` (org defunct) |
| **Fabricated / example DOIs** | Permanent (deliberate) | Illustrative citations in writing-tool templates |

Each block of lines in `.lycheeignore` should have a `# <category>: <specific reason>` comment above it. Maintainers reading the file six months from now need to know the **why** for each pattern.

## `.lycheeignore` syntax

One regex per line. Lychee treats lines as case-sensitive POSIX-ish regex anchored with `^`. Standard escapes apply.

```
# Category + specific reason on the line above each block.

# Infrastructure hostility — bot-blocks on CI runners.
^https?://([a-z0-9-]+\.)?linkedin\.com/.*
^https?://(twitter\.com|x\.com|t\.co)/.*

# Pedagogical placeholders — template-example DOIs.
^https?://doi\.org/10\.1038/nrd\.2023\.001$
```

**Prefer host-level exclusions over full-URL exclusions** when the category is permanent (e.g. `linkedin.com/*`), because individual page changes on those hosts would otherwise require ignoring-list maintenance each time a skill touches the host.

**Use specific-URL exclusions** for dead-infrastructure cases, because excluding the whole host would hide legitimate future-working pages on the same domain.

## Retries before exclusions

If a URL is live-in-browser but flaky from CI, exhaust the config knobs first:

- `max_retries = 5` in `.lychee.toml` (up from the default 3).
- `retry_wait_time = 5` or higher — gives transient services time to recover.
- `timeout = 30` or higher for slow academic mirrors.

Only exclude a flaky URL after retries fail across multiple dispatches. A single CI flake is not evidence of a dead link.

## Documenting debt

Every exclusion gets a row in `.github/KNOWN_LINK_DEBT.md` (see `references/known-debt-template.md`). The file is the durable record of **why** each exclusion exists — `.lycheeignore` comments get lost in diffs over time, but a well-structured debt doc stays reviewable.

Minimum row content: URL pattern, category, reason, date excluded, revisit trigger (or tracking issue).