Claude Skill

fetch-url-as-markdown

Fetch a web page (URL) and return clean Markdown via local trafilatura, with Exa MCP as a fallback for JS-rendered or anti-bot pages. Use when the user asks to read, fetch, scrape, summarize, or quote a URL — prefer this over the built-in WebFetch tool. Don't use for binary files

LLM Mart · 0 points · 1 views 0 listing impressions 0 install-command copies
Virus-scanned Reviewed automatically before listing.

Full trust report

Download codealive-ai-ai-driven-development-skills_fetch-url-as-markdown-68a302a.zip · 8 KB
codealive-ai/ai-driven-development 143 11 forks MIT Updated 8d ago
Part of codealive-ai/ai-driven-development — 21 skills

Install

skills CLI npx skills add https://github.com/CodeAlive-AI/ai-driven-development/tree/main/skills/fetch-url-as-markdown
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install codealive-ai-ai-driven-development@llmmart
Git git clone https://github.com/CodeAlive-AI/ai-driven-development.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole codealive-ai/ai-driven-development collection as a plugin from our marketplace. Git is the plain clone.

README

fetch-url-as-markdown

Fetch any web URL and get clean, readable Markdown — main content only, no navigation, ads, or footer. Runs locally on trafilatura with a real-browser User-Agent and structured exit codes that tell the host agent when to fall back to a remote crawler (Exa MCP).

Install

npx skills add CodeAlive-AI/ai-driven-development@fetch-url-as-markdown -g -y

Prerequisites

  • Python 3.10+

  • trafilatura ≥ 2.0:

    python3 -m pip install --break-system-packages trafilatura

    (The script exits with code 3 and prints this hint if the import fails.)

  • Optional — an Exa MCP server in your agent host (e.g. mcp__exa__web_search_advanced_exa). Used only as a fallback when local extraction can't recover the page.

Quick start

After installing, ask your agent things like:

> Read https://github.com/adbar/trafilatura and summarize the README
> Fetch https://docs.python.org/3/library/json.html and quote the section on encoders
> Pull this blog post as Markdown so I can paste it into my notes

The agent will run the bundled script directly:

python3 ~/.claude/skills/fetch-url-as-markdown/scripts/fetch_url.py "https://example.com"
python3 ~/.claude/skills/fetch-url-as-markdown/scripts/fetch_url.py "https://example.com" --no-metadata --min-body 0

What it does

One entry point — a single CLI script with one job: URL → clean Markdown to stdout.

Stage Behaviour
Download trafilatura.fetch_response() with a real Chrome User-Agent and 30 s timeout (config in scripts/settings.cfg)
Content-Type guard Anything outside text/html \| application/xhtml+xml \| text/plain \| application/xml \| text/xml is rejected up-front (exit 4) so PDFs/images/archives don't get mis-parsed as HTML
Anti-stub guard Sniffs the raw HTML for Cloudflare / "Please enable JavaScript" / Imperva / DataDome wall markers and bails with exit 2 instead of returning a useless 30-character "Just a moment…" page
Extract trafilatura.extract(output_format="markdown", include_formatting=True, include_links=True, include_tables=True, favor_recall=True, deduplicate=True, with_metadata=True) — keeps headings/lists/code where the source HTML uses real <h1..h6>, with a YAML frontmatter (title, author, date, url, hostname) on top
Min-body guard Bodies under 50 chars (configurable via --min-body N, 0 to disable) are treated as stubs → exit 2

Exit codes (the contract for the host agent)

Code Meaning Recommended action
0 Markdown printed to stdout done
1 DownloadError — network/HTTP/timeout/anti-bot block at fetch fall back to Exa MCP
2 ExtractionError — empty extract, JS/Cloudflare wall, or stub body fall back to Exa MCP
3 trafilatura not installed install (see Prerequisites), then retry
4 UnsupportedContentTypeError — URL is binary don't fall back to Exa; route to a content-specific skill (e.g. pdf for PDFs)

SKILL.md instructs the agent on this fallback flow, so for the common case the user just says "fetch this URL" and gets Markdown — local first, Exa second, no manual orchestration.

Key features

  • Local-first, free, no API key needed for the happy path — extraction runs entirely on trafilatura ≥ 2.0
  • Real browser User-Agent baked into settings.cfg — fixes the silent failure where github.com and other anti-bot sites return empty bodies for trafilatura's default UA
  • Structured exit codes 0/1/2/3/4 — the script tells the host agent why it failed, so the fallback decision is mechanical, not interpretive
  • Content-Type and anti-stub guards — prevent the classic "trafilatura returned 30 chars from a Cloudflare interstitial, so we silently passed garbage downstream" failure mode
  • Defaults tuned for LLM-friendly output — include_formatting=True, favor_recall=True, deduplicate=True, YAML metadata header on by default
  • Drop-in replacement for the built-in WebFetch — the description in SKILL.md instructs the agent to prefer this skill whenever the user asks to "read / fetch / scrape / summarize / quote a URL"

Sources and methodology

  • trafilatura by Adrien Barbaresi — GitHub, docs. Configuration patterns (use_config, settings.cfg, USER_AGENTS) follow the official Settings and Downloads docs.
  • Extract flag selection — informed by Barbaresi 2021 (ACL anthology) and the Bevendorff et al. 2023 extraction benchmark, which rank trafilatura first among open-source extractors on ROUGE-LSum.
  • Real-world reference implementation — vakovalskii/searcharvester (simple_tavily_adapter/main.py) uses trafilatura.extract(output_format="markdown", include_formatting=True, include_links=True, include_tables=True, favor_recall=True) for its /extract and /search endpoints — the same flag set we ship as default.
  • Anti-stub markers — collected from Cloudflare interstitial copy ("Just a moment…", "Verifying you are human"), Imperva (Incapsula Incident ID), DataDome (captcha-delivery.com) and standard <noscript> patterns. Matched on a case-insensitive snippet of the first 8 KB of the response body.

File structure

skills/fetch-url-as-markdown/
├── SKILL.md                     # agent-facing contract (workflow, exit-code routing)
├── README.md                    # this file
└── scripts/
    ├── fetch_url.py             # CLI entry point
    └── settings.cfg             # trafilatura config: real-browser UA, 30s timeout, retries

License

MIT

Skill manifest

URL to Markdown

Fetch any web URL and get clean, readable Markdown — main content only, no navigation/footer/ads. Local + free by default; smart fallback to Exa MCP when the page can't be extracted locally.

Workflow (the only thing the agent needs to remember)

  1. Try trafilatura first:

    python3 ~/.claude/skills/fetch-url-as-markdown/scripts/fetch_url.py "<URL>"

  2. If exit code is 1 or 2 → fall back to Exa MCP with the same URL:

    mcp__exa__web_search_advanced_exa(
    query="<URL>",
    includeDomains=["<host of URL>"],
    numResults=1,
    textMaxCharacters=50000,
    type="auto"
)

    (mcp__exa__crawling works too if the server exposes it; the web_search_advanced_exa call above is the always-available variant — pin the host with includeDomains and use the URL itself as the query.)

  3. Exit code 3 means trafilatura is not installed — install once:

    python3 -m pip install --break-system-packages trafilatura

Exit codes (what they mean for the fallback decision)

Code Meaning Action
0 Markdown printed to stdout done
1 DownloadError — network/HTTP/timeout/anti-bot block at fetch fall back to Exa
2 ExtractionError — empty extract, JS/Cloudflare wall, or stub body (<200 chars) fall back to Exa
3 trafilatura missing install (see above), then retry
4 UnsupportedContentTypeError — URL is binary (PDF, image, archive) don't fall back to Exa; use the right specialized skill (e.g. pdf for PDFs)

Defaults baked into the script

  • output_format="markdown", include_formatting=True — keeps headings/lists/code structure where the source HTML uses real <h1..h6> etc.
  • include_links=True, include_tables=True
  • with_metadata=True → emits a YAML frontmatter (title, author, date, url, hostname)
  • favor_recall=True, deduplicate=True — readable but trims duplicates
  • Real-browser User-Agent + 30s timeout configured in scripts/settings.cfg
  • Anti-stub guards (built into the script):
    • rejects Content-Type other than text/html|application/xhtml+xml|text/plain|application/xml|text/xml → exit 4
    • sniffs raw HTML for Cloudflare / "Please enable JavaScript" / Imperva / DataDome wall markers → exit 2
    • rejects extracted bodies under 50 chars (configurable via --min-body N, 0 to disable) → exit 2

Useful flags

... fetch_url.py "<URL>" --no-links     # strip hyperlinks
... fetch_url.py "<URL>" --no-tables    # strip tables
... fetch_url.py "<URL>" --no-metadata  # omit YAML header
... fetch_url.py "<URL>" --comments     # include user comments (off by default — usually noise)
... fetch_url.py "<URL>" --images       # include image refs (experimental)
... fetch_url.py "<URL>" --precision    # terser output, drops borderline content

When to choose what

Situation Tool
Article, blog post, docs, README, wiki trafilatura (default) — local, free
JS-heavy SPA, login-walled, Cloudflare Exa fallback (the script will signal exit 2)
Bulk / many URLs trafilatura — no quota, no API key
Already failed twice on a domain Exa directly
Files (ai-driven-development)
  • scripts
    • fetch_url.py 8.3 KB 
      #!/usr/bin/env python3
"""Convert a URL to Markdown using trafilatura (https://github.com/adbar/trafilatura).

Default extractor for fetch-url-as-markdown skill. Runs entirely locally,
no API key required. Sends a real-browser User-Agent (configured in
settings.cfg next to this script) so anti-bot sites like github.com return
real HTML instead of a stub.

Exit codes:
  0  success — Markdown printed to stdout
  1  download failed (network, HTTP 4xx/5xx, timeout, anti-bot 403)
  2  extraction failed — page downloaded but yielded no usable main
     content (SPA shell, JS-only render, captcha/Cloudflare wall, login wall)
  4  unsupported content type (binary: PDF, image, archive, video, octet-stream)
  3  trafilatura is not installed
The skill's fallback (Exa MCP crawl) should be tried on exit 1 or 2 — not on 4.
"""

import argparse
import re
import sys
from pathlib import Path

try:
    import trafilatura
    from trafilatura.settings import use_config
except ImportError:
    sys.stderr.write(
        "trafilatura not installed. Install with:\n"
        "  python3 -m pip install --break-system-packages trafilatura\n"
    )
    sys.exit(3)

SETTINGS_PATH = Path(__file__).with_name("settings.cfg")
_CONFIG = use_config(str(SETTINGS_PATH)) if SETTINGS_PATH.exists() else use_config()

# Content types we accept. Anything outside this set triggers UnsupportedContentTypeError.
_HTML_CT_RE = re.compile(
    r"^(text/html|application/xhtml\+xml|text/plain|application/xml|text/xml)\b",
    re.IGNORECASE,
)

# Substrings that strongly indicate the page is a JS/anti-bot wall, not real content.
# Matched on a case-insensitive normalized snippet of the *raw HTML*.
_WALL_MARKERS = (
    "please enable javascript",
    "javascript is required",
    "javascript is disabled",
    "enable javascript to run",
    "you need to enable javascript",
    "checking your browser before accessing",  # Cloudflare interstitial
    "just a moment...",                          # Cloudflare interstitial
    "verifying you are human",                   # Cloudflare Turnstile
    "verify you are a human",
    "attention required! | cloudflare",
    "ddos protection by cloudflare",
    "access denied | cloudflare",
    "ray id:",                                    # cloudflare error page
    "captcha-delivery.com",                       # DataDome captcha
    "incapsula incident id",                      # Imperva
    "<title>access denied</title>",
    "<title>403 forbidden</title>",
    "<title>error 1020</title>",                  # Cloudflare access denied
)

# Below this length the extracted markdown is almost certainly a stub
# (e.g. "Loading..." or "Please enable JS"). Tuned low enough to accept
# tiny-but-valid pages like example.com (~110 body chars) while still
# catching SPA shells.
_MIN_MARKDOWN_BODY_CHARS = 50


class DownloadError(RuntimeError):
    """Network/HTTP/anti-bot failure — could not retrieve the page bytes."""


class UnsupportedContentTypeError(RuntimeError):
    """The URL points to a binary/non-HTML resource (PDF, image, archive, …)."""


class ExtractionError(RuntimeError):
    """HTML retrieved, but contains no extractable main content (or a JS/anti-bot wall)."""


def _looks_like_wall(html: str) -> str | None:
    """Return the marker that matched, or None if no wall detected."""
    snippet = html[:8000].lower()
    for marker in _WALL_MARKERS:
        if marker in snippet:
            return marker
    return None


def _markdown_body_chars(markdown: str, with_metadata: bool) -> int:
    """Count characters in the body of the Markdown, excluding the YAML frontmatter."""
    if with_metadata and markdown.startswith("---"):
        # Strip the leading YAML block: from the first '---' to the next one
        end = markdown.find("\n---", 3)
        if end != -1:
            return len(markdown[end + 4 :].strip())
    return len(markdown.strip())


def fetch_url_as_markdown(
    url: str,
    include_links: bool = True,
    include_tables: bool = True,
    include_images: bool = False,
    include_comments: bool = False,
    with_metadata: bool = True,
    favor_precision: bool = False,
    favor_recall: bool = True,
    min_body_chars: int = _MIN_MARKDOWN_BODY_CHARS,
) -> str:
    """Fetch a URL and return its main content as Markdown.

    Defaults aim at "clean readable article body": metadata header on,
    structural formatting (headings, lists) on, links on, recall favored.

    Raises:
        DownloadError: page could not be retrieved.
        UnsupportedContentTypeError: response is binary (PDF, image, …).
        ExtractionError: HTML retrieved but no usable main content
            (empty extract, JS/anti-bot wall, or below min_body_chars).
    """
    response = trafilatura.fetch_response(
        url, with_headers=True, decode=True, config=_CONFIG
    )
    if response is None or response.html is None:
        raise DownloadError(f"Failed to download: {url}")

    content_type = (response.headers or {}).get("content-type", "")
    if content_type and not _HTML_CT_RE.match(content_type):
        raise UnsupportedContentTypeError(
            f"Unsupported content-type {content_type!r} at {url} — "
            "this skill only handles HTML/XML."
        )

    html = response.html
    wall_marker = _looks_like_wall(html)
    if wall_marker is not None:
        raise ExtractionError(
            f"Anti-bot or JS wall detected at {url} (marker: {wall_marker!r})"
        )

    markdown = trafilatura.extract(
        html,
        url=url,
        output_format="markdown",
        include_formatting=True,
        include_links=include_links,
        include_tables=include_tables,
        include_images=include_images,
        include_comments=include_comments,
        with_metadata=with_metadata,
        favor_precision=favor_precision,
        favor_recall=favor_recall and not favor_precision,
        deduplicate=True,
        config=_CONFIG,
    )
    if not markdown:
        raise ExtractionError(f"No extractable main content at: {url}")

    body_chars = _markdown_body_chars(markdown, with_metadata)
    if body_chars < min_body_chars:
        raise ExtractionError(
            f"Extracted body is too short ({body_chars} chars < {min_body_chars}) "
            f"at {url} — likely a JS-only render or stub page."
        )
    return markdown


def main() -> None:
    parser = argparse.ArgumentParser(
        description="Fetch a URL and print clean Markdown to stdout (trafilatura).",
    )
    parser.add_argument("url", help="URL to fetch")
    parser.add_argument(
        "--no-links", action="store_true", help="Strip hyperlinks from output"
    )
    parser.add_argument(
        "--no-tables", action="store_true", help="Strip tables from output"
    )
    parser.add_argument(
        "--images", action="store_true", help="Include images (experimental)"
    )
    parser.add_argument(
        "--comments", action="store_true", help="Include user comments"
    )
    parser.add_argument(
        "--no-metadata", action="store_true", help="Skip YAML metadata header"
    )
    parser.add_argument(
        "--precision",
        action="store_true",
        help="Favor precision over recall (terser, drops borderline content)",
    )
    parser.add_argument(
        "--min-body",
        type=int,
        default=_MIN_MARKDOWN_BODY_CHARS,
        help=f"Minimum body chars for a successful extraction (default: {_MIN_MARKDOWN_BODY_CHARS}). "
        "Set to 0 to disable the stub-page check.",
    )
    args = parser.parse_args()

    try:
        markdown = fetch_url_as_markdown(
            args.url,
            include_links=not args.no_links,
            include_tables=not args.no_tables,
            include_images=args.images,
            include_comments=args.comments,
            with_metadata=not args.no_metadata,
            favor_precision=args.precision,
            min_body_chars=args.min_body,
        )
    except DownloadError as e:
        sys.stderr.write(f"DownloadError: {e}\n")
        sys.exit(1)
    except ExtractionError as e:
        sys.stderr.write(f"ExtractionError: {e}\n")
        sys.exit(2)
    except UnsupportedContentTypeError as e:
        sys.stderr.write(f"UnsupportedContentTypeError: {e}\n")
        sys.exit(4)

    sys.stdout.write(markdown)
    if not markdown.endswith("\n"):
        sys.stdout.write("\n")


if __name__ == "__main__":
    main()
    • settings.cfg 893 B 
      # Settings for trafilatura — all keys live under [DEFAULT].
# See https://trafilatura.readthedocs.io/en/latest/settings.html

[DEFAULT]

# Download
DOWNLOAD_TIMEOUT = 30
MAX_FILE_SIZE = 20000000
MIN_FILE_SIZE = 10

# sleep between requests (used in batch mode; single-URL CLI usage rarely hits it)
SLEEP_TIME = 2.0

# one line per user-agent — real-browser UA so anti-bot sites (github.com,
# Cloudflare, etc.) don't return empty bodies for the default UA.
USER_AGENTS =
    Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36

COOKIE =

MAX_REDIRECTS = 5


# Extraction
MIN_EXTRACTED_SIZE = 200
MIN_EXTRACTED_COMM_SIZE = 1
MIN_OUTPUT_SIZE = 1
MIN_OUTPUT_COMM_SIZE = 1

MAX_TREE_SIZE =

EXTRACTION_TIMEOUT = 30


# Deduplication
MIN_DUPLCHECK_SIZE = 100
MAX_REPETITIONS = 2


EXTENSIVE_DATE_SEARCH = on
EXTERNAL_URLS = off
  • README.md 6 KB 
    # fetch-url-as-markdown

Fetch any web URL and get clean, readable Markdown — main content only, no navigation, ads, or footer. Runs locally on [trafilatura](https://github.com/adbar/trafilatura) with a real-browser User-Agent and structured exit codes that tell the host agent when to fall back to a remote crawler (Exa MCP).

## Install

```bash
npx skills add CodeAlive-AI/ai-driven-development@fetch-url-as-markdown -g -y
```

## Prerequisites

- Python 3.10+
- [`trafilatura`](https://pypi.org/project/trafilatura/) ≥ 2.0:

  ```bash
  python3 -m pip install --break-system-packages trafilatura
  ```

  (The script exits with code `3` and prints this hint if the import fails.)

- *Optional* — an Exa MCP server in your agent host (e.g. `mcp__exa__web_search_advanced_exa`). Used only as a fallback when local extraction can't recover the page.

## Quick start

After installing, ask your agent things like:

```
> Read https://github.com/adbar/trafilatura and summarize the README
> Fetch https://docs.python.org/3/library/json.html and quote the section on encoders
> Pull this blog post as Markdown so I can paste it into my notes
```

The agent will run the bundled script directly:

```bash
python3 ~/.claude/skills/fetch-url-as-markdown/scripts/fetch_url.py "https://example.com"
python3 ~/.claude/skills/fetch-url-as-markdown/scripts/fetch_url.py "https://example.com" --no-metadata --min-body 0
```

## What it does

One entry point — a single CLI script with one job: *URL → clean Markdown to stdout.*

| Stage | Behaviour |
|---|---|
| **Download** | `trafilatura.fetch_response()` with a real Chrome User-Agent and 30 s timeout (config in `scripts/settings.cfg`) |
| **Content-Type guard** | Anything outside `text/html \| application/xhtml+xml \| text/plain \| application/xml \| text/xml` is rejected up-front (exit `4`) so PDFs/images/archives don't get mis-parsed as HTML |
| **Anti-stub guard** | Sniffs the raw HTML for Cloudflare / "Please enable JavaScript" / Imperva / DataDome wall markers and bails with exit `2` instead of returning a useless 30-character "Just a moment…" page |
| **Extract** | `trafilatura.extract(output_format="markdown", include_formatting=True, include_links=True, include_tables=True, favor_recall=True, deduplicate=True, with_metadata=True)` — keeps headings/lists/code where the source HTML uses real `<h1..h6>`, with a YAML frontmatter (title, author, date, url, hostname) on top |
| **Min-body guard** | Bodies under 50 chars (configurable via `--min-body N`, `0` to disable) are treated as stubs → exit `2` |

### Exit codes (the contract for the host agent)

| Code | Meaning | Recommended action |
|---:|---|---|
| `0` | Markdown printed to stdout | done |
| `1` | `DownloadError` — network/HTTP/timeout/anti-bot block at fetch | fall back to Exa MCP |
| `2` | `ExtractionError` — empty extract, JS/Cloudflare wall, or stub body | fall back to Exa MCP |
| `3` | trafilatura not installed | install (see Prerequisites), then retry |
| `4` | `UnsupportedContentTypeError` — URL is binary | **don't** fall back to Exa; route to a content-specific skill (e.g. `pdf` for PDFs) |

`SKILL.md` instructs the agent on this fallback flow, so for the common case the user just says "fetch this URL" and gets Markdown — local first, Exa second, no manual orchestration.

## Key features

- **Local-first, free, no API key needed for the happy path** — extraction runs entirely on `trafilatura` ≥ 2.0
- **Real browser User-Agent baked into `settings.cfg`** — fixes the silent failure where `github.com` and other anti-bot sites return empty bodies for trafilatura's default UA
- **Structured exit codes 0/1/2/3/4** — the script tells the host agent *why* it failed, so the fallback decision is mechanical, not interpretive
- **Content-Type and anti-stub guards** — prevent the classic "trafilatura returned 30 chars from a Cloudflare interstitial, so we silently passed garbage downstream" failure mode
- **Defaults tuned for LLM-friendly output** — `include_formatting=True`, `favor_recall=True`, `deduplicate=True`, YAML metadata header on by default
- **Drop-in replacement for the built-in `WebFetch`** — the description in `SKILL.md` instructs the agent to prefer this skill whenever the user asks to "read / fetch / scrape / summarize / quote a URL"

## Sources and methodology

- **trafilatura** by Adrien Barbaresi — [GitHub](https://github.com/adbar/trafilatura), [docs](https://trafilatura.readthedocs.io). Configuration patterns (`use_config`, `settings.cfg`, `USER_AGENTS`) follow the official [Settings](https://trafilatura.readthedocs.io/en/latest/settings.html) and [Downloads](https://trafilatura.readthedocs.io/en/latest/downloads.html) docs.
- **Extract flag selection** — informed by Barbaresi 2021 ([ACL anthology](https://aclanthology.org/2021.acl-demo.15/)) and the [Bevendorff et al. 2023 extraction benchmark](https://webis.de/downloads/publications/papers/bevendorff_2023b.pdf), which rank trafilatura first among open-source extractors on ROUGE-LSum.
- **Real-world reference implementation** — [`vakovalskii/searcharvester`](https://github.com/vakovalskii/searcharvester) (`simple_tavily_adapter/main.py`) uses `trafilatura.extract(output_format="markdown", include_formatting=True, include_links=True, include_tables=True, favor_recall=True)` for its `/extract` and `/search` endpoints — the same flag set we ship as default.
- **Anti-stub markers** — collected from Cloudflare interstitial copy ("Just a moment…", "Verifying you are human"), Imperva (`Incapsula Incident ID`), DataDome (`captcha-delivery.com`) and standard `<noscript>` patterns. Matched on a case-insensitive snippet of the first 8 KB of the response body.

## File structure

```
skills/fetch-url-as-markdown/
├── SKILL.md                     # agent-facing contract (workflow, exit-code routing)
├── README.md                    # this file
└── scripts/
    ├── fetch_url.py             # CLI entry point
    └── settings.cfg             # trafilatura config: real-browser UA, 30s timeout, retries
```

## License

MIT
  • SKILL.md 3.6 KB 
    ---
name: fetch-url-as-markdown
description: Fetch a web page (URL) and return clean Markdown via local trafilatura, with Exa MCP as a fallback for JS-rendered or anti-bot pages. Use when the user asks to read, fetch, scrape, summarize, or quote a URL — prefer this over the built-in WebFetch tool. Don't use for binary files (PDFs, images, archives) or for fetching API/JSON endpoints.
---

# URL to Markdown

Fetch any web URL and get clean, readable Markdown — main content only, no
navigation/footer/ads. Local + free by default; smart fallback to Exa MCP
when the page can't be extracted locally.

## Workflow (the only thing the agent needs to remember)

1. **Try trafilatura first**:

   ```bash
   python3 ~/.claude/skills/fetch-url-as-markdown/scripts/fetch_url.py "<URL>"
   ```

2. **If exit code is 1 or 2 → fall back to Exa MCP** with the same URL:

   ```
   mcp__exa__web_search_advanced_exa(
       query="<URL>",
       includeDomains=["<host of URL>"],
       numResults=1,
       textMaxCharacters=50000,
       type="auto"
   )
   ```

   (`mcp__exa__crawling` works too if the server exposes it; the `web_search_advanced_exa`
   call above is the always-available variant — pin the host with `includeDomains` and
   use the URL itself as the query.)

3. Exit code `3` means trafilatura is not installed — install once:

   ```bash
   python3 -m pip install --break-system-packages trafilatura
   ```

## Exit codes (what they mean for the fallback decision)

| Code | Meaning | Action |
|---|---|---|
| 0 | Markdown printed to stdout | done |
| 1 | DownloadError — network/HTTP/timeout/anti-bot block at fetch | fall back to Exa |
| 2 | ExtractionError — empty extract, JS/Cloudflare wall, or stub body (<200 chars) | fall back to Exa |
| 3 | trafilatura missing | install (see above), then retry |
| 4 | UnsupportedContentTypeError — URL is binary (PDF, image, archive) | **don't** fall back to Exa; use the right specialized skill (e.g. `pdf` for PDFs) |

## Defaults baked into the script

- `output_format="markdown"`, `include_formatting=True` — keeps headings/lists/code structure where the source HTML uses real `<h1..h6>` etc.
- `include_links=True`, `include_tables=True`
- `with_metadata=True` → emits a YAML frontmatter (`title`, `author`, `date`, `url`, `hostname`)
- `favor_recall=True`, `deduplicate=True` — readable but trims duplicates
- Real-browser User-Agent + 30s timeout configured in `scripts/settings.cfg`
- Anti-stub guards (built into the script):
  - rejects `Content-Type` other than `text/html|application/xhtml+xml|text/plain|application/xml|text/xml` → exit `4`
  - sniffs raw HTML for Cloudflare / "Please enable JavaScript" / Imperva / DataDome wall markers → exit `2`
  - rejects extracted bodies under 50 chars (configurable via `--min-body N`, `0` to disable) → exit `2`

## Useful flags

```bash
... fetch_url.py "<URL>" --no-links     # strip hyperlinks
... fetch_url.py "<URL>" --no-tables    # strip tables
... fetch_url.py "<URL>" --no-metadata  # omit YAML header
... fetch_url.py "<URL>" --comments     # include user comments (off by default — usually noise)
... fetch_url.py "<URL>" --images       # include image refs (experimental)
... fetch_url.py "<URL>" --precision    # terser output, drops borderline content
```

## When to choose what

| Situation | Tool |
|---|---|
| Article, blog post, docs, README, wiki | trafilatura (default) — local, free |
| JS-heavy SPA, login-walled, Cloudflare | Exa fallback (the script will signal exit 2) |
| Bulk / many URLs | trafilatura — no quota, no API key |
| Already failed twice on a domain | Exa directly |

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.

Related

Claude Prompt
Tighten-my-prose editing prompt
Claude Skill
Adversarial code-review skill for Claude
Claude Workflow
Deep-research workflow (fan-out + verify)
Claude Skill
Conventional-commit writer for Claude