webextrator
Render and extract web page content via AceDataCloud's WebExtrator API. Use when scraping a page's final rendered HTML, or extracting typed structured data (Article, Product, Recipe, Video, Discussion, Job) plus clean markdown/text from any URL. Real headless Chromium with schema
Install
npx skills add https://github.com/AceDataCloud/Skills/tree/main/skills/webextrator
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install acedatacloud-skills@llmmart
git clone https://github.com/AceDataCloud/Skills.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole acedatacloud/skills collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
WebExtrator Web Render & Extract
Render and extract web content through AceDataCloud's WebExtrator API — real headless Chromium plus a three-tier extraction pipeline (schema.org JSON-LD mapper → LLM typed extractor → Readability/markdown fallback).
Setup: See authentication for token setup.
Quick Start
curl -X POST https://api.acedata.cloud/webextrator/extract \
-H "Authorization: Bearer $ACEDATACLOUD_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "expected_type": "general"}'
Returns synchronously in seconds — no task polling needed.
Endpoints
| Path | Purpose |
|---|---|
POST /webextrator/render |
Headless Chromium render → raw HTML + clean text + title |
POST /webextrator/extract |
Render + structured extraction (schema.org + LLM types) + markdown |
POST /webextrator/tasks |
Look up historical render/extract task envelopes (7-day retention, free) |
Workflows
1. Extract typed content
POST /webextrator/extract
{
"url": "https://example.com",
"expected_type": "general"
}
Real response (trimmed):
{
"success": true,
"task_id": "604b1cfb-6c5a-42c9-b900-a281e1b9c3c5",
"trace_id": "f2a7c0b0-c17c-4bc9-b6e7-9c59746dd366",
"elapsed": 0.003,
"data": {
"kind": "extract",
"url": "https://example.com",
"finalUrl": "https://example.com/",
"contentType": "general",
"title": "Example Domain",
"description": "This domain is for use in documentation examples without needing permission. Avoid use in operations.",
"language": "en",
"images": [],
"links": [],
"markdown": "...",
"text": "...",
"structured": { "schemaOrg": {}, "openGraph": {}, "jsonLd": [] }
}
}
2. Render raw HTML
POST /webextrator/render
{
"url": "https://example.com",
"wait_until": "load",
"block_resources": ["image", "media", "font"]
}
Returns data.html, data.text, data.title, data.status, data.finalUrl.
3. Look up a task
POST /webextrator/tasks
{
"action": "retrieve",
"id": "604b1cfb-6c5a-42c9-b900-a281e1b9c3c5"
}
Parameters
Render & Extract (shared)
| Parameter | Required | Description |
|---|---|---|
url |
Yes | Page URL to render (http(s)://) |
wait_until |
No | load / domcontentloaded / networkidle / commit (default load) |
timeout |
No | Navigation timeout in positive integer seconds (minimum 1, default 30), not a total processing deadline |
delay |
No | Extra wait in nonnegative integer seconds after wait_until (for SPAs; default 0) |
wait_for_selector |
No | CSS selector to wait for before ready |
block_resources |
No | Drop image/font/media/stylesheet/xhr/fetch |
headers |
No | Extra request headers for the target site |
user_agent |
No | Override the browser User-Agent |
cookies |
No | Cookie array injected before navigation; each item requires name, value, domain |
bypass_cache |
No | Boolean, default false; skips cache reads, but the result may still be written to cache |
cache_ttl_seconds |
No | Nonnegative integer seconds, default 3600; 0 disables caching this response |
async |
No | true submits without blocking; poll /webextrator/tasks for the result |
callback_url |
No | Posted the final envelope when running asynchronously |
Cookies also accept optional path, Unix-seconds expires, httpOnly, secure, and sameSite (Strict, Lax, None). These nested cookie keys keep their exact casing; the top-level request uses snake_case. Send cookies or private headers only for sites the user authorizes, and never print their values or commit them. For SPA content, prefer wait_for_selector plus an integer delay over assuming all pages reach network idle.
Extract-only
| Parameter | Required | Description |
|---|---|---|
expected_type |
No | product / article / general — skips the heuristic |
enable_llm |
No | Allow LLM extractor when schema.org found nothing (default false) |
Tasks
| Parameter | Required | Description |
|---|---|---|
action |
Yes | retrieve (single) or retrieve_batch (many) |
id / trace_id |
one of | For retrieve |
ids / trace_ids |
one of | For retrieve_batch |
Gotchas
- Parameters use snake_case (
wait_until,block_resources), not camelCase - Cache hits are still billed; identical URLs return in ~0.003s
expected_typeonly allowsproduct/article/general— typed kinds (recipe/video/job) are detected automatically from schema.orgenable_llmhas no effect when the page ships schema.org JSON-LD — the deterministic mapper wins for free- Tasks API is free and retains records for 7 days only
MCP:
pip install mcp-webextrator| Hosted:https://webextrator.mcp.acedata.cloud/mcp| See all MCP servers
Files (skills)
-
SKILL.md 5.5 KB
--- name: webextrator description: Render and extract web page content via AceDataCloud's WebExtrator API. Use when scraping a page's final rendered HTML, or extracting typed structured data (Article, Product, Recipe, Video, Discussion, Job) plus clean markdown/text from any URL. Real headless Chromium with schema.org + LLM extraction. license: Apache-2.0 metadata: author: acedatacloud version: "1.0" compatibility: Requires ACEDATACLOUD_API_TOKEN in .env file (see _shared/authentication.md). Optionally pair with mcp-webextrator for tool-use. --- # WebExtrator Web Render & Extract Render and extract web content through AceDataCloud's WebExtrator API — real headless Chromium plus a three-tier extraction pipeline (schema.org JSON-LD mapper → LLM typed extractor → Readability/markdown fallback). > **Setup:** See [authentication](../_shared/authentication.md) for token setup. ## Quick Start ```bash curl -X POST https://api.acedata.cloud/webextrator/extract \ -H "Authorization: Bearer $ACEDATACLOUD_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com", "expected_type": "general"}' ``` Returns synchronously in seconds — no task polling needed. ## Endpoints | Path | Purpose | |------|---------| | `POST /webextrator/render` | Headless Chromium render → raw HTML + clean text + title | | `POST /webextrator/extract` | Render + structured extraction (schema.org + LLM types) + markdown | | `POST /webextrator/tasks` | Look up historical render/extract task envelopes (7-day retention, free) | ## Workflows ### 1. Extract typed content ```json POST /webextrator/extract { "url": "https://example.com", "expected_type": "general" } ``` Real response (trimmed): ```json { "success": true, "task_id": "604b1cfb-6c5a-42c9-b900-a281e1b9c3c5", "trace_id": "f2a7c0b0-c17c-4bc9-b6e7-9c59746dd366", "elapsed": 0.003, "data": { "kind": "extract", "url": "https://example.com", "finalUrl": "https://example.com/", "contentType": "general", "title": "Example Domain", "description": "This domain is for use in documentation examples without needing permission. Avoid use in operations.", "language": "en", "images": [], "links": [], "markdown": "...", "text": "...", "structured": { "schemaOrg": {}, "openGraph": {}, "jsonLd": [] } } } ``` ### 2. Render raw HTML ```json POST /webextrator/render { "url": "https://example.com", "wait_until": "load", "block_resources": ["image", "media", "font"] } ``` Returns `data.html`, `data.text`, `data.title`, `data.status`, `data.finalUrl`. ### 3. Look up a task ```json POST /webextrator/tasks { "action": "retrieve", "id": "604b1cfb-6c5a-42c9-b900-a281e1b9c3c5" } ``` ## Parameters ### Render & Extract (shared) | Parameter | Required | Description | |-----------|----------|-------------| | `url` | Yes | Page URL to render (`http(s)://`) | | `wait_until` | No | `load` / `domcontentloaded` / `networkidle` / `commit` (default `load`) | | `timeout` | No | Navigation timeout in positive integer seconds (minimum 1, default 30), not a total processing deadline | | `delay` | No | Extra wait in nonnegative integer seconds after `wait_until` (for SPAs; default 0) | | `wait_for_selector` | No | CSS selector to wait for before ready | | `block_resources` | No | Drop `image`/`font`/`media`/`stylesheet`/`xhr`/`fetch` | | `headers` | No | Extra request headers for the target site | | `user_agent` | No | Override the browser User-Agent | | `cookies` | No | Cookie array injected before navigation; each item requires `name`, `value`, `domain` | | `bypass_cache` | No | Boolean, default false; skips cache reads, but the result may still be written to cache | | `cache_ttl_seconds` | No | Nonnegative integer seconds, default 3600; `0` disables caching this response | | `async` | No | `true` submits without blocking; poll `/webextrator/tasks` for the result | | `callback_url` | No | Posted the final envelope when running asynchronously | Cookies also accept optional `path`, Unix-seconds `expires`, `httpOnly`, `secure`, and `sameSite` (`Strict`, `Lax`, `None`). These nested cookie keys keep their exact casing; the top-level request uses snake_case. Send cookies or private headers only for sites the user authorizes, and never print their values or commit them. For SPA content, prefer `wait_for_selector` plus an integer `delay` over assuming all pages reach network idle. ### Extract-only | Parameter | Required | Description | |-----------|----------|-------------| | `expected_type` | No | `product` / `article` / `general` — skips the heuristic | | `enable_llm` | No | Allow LLM extractor when schema.org found nothing (default false) | ### Tasks | Parameter | Required | Description | |-----------|----------|-------------| | `action` | Yes | `retrieve` (single) or `retrieve_batch` (many) | | `id` / `trace_id` | one of | For `retrieve` | | `ids` / `trace_ids` | one of | For `retrieve_batch` | ## Gotchas - Parameters use **snake_case** (`wait_until`, `block_resources`), not camelCase - Cache hits are still billed; identical URLs return in ~0.003s - `expected_type` only allows `product`/`article`/`general` — typed kinds (recipe/video/job) are detected automatically from schema.org - `enable_llm` has no effect when the page ships schema.org JSON-LD — the deterministic mapper wins for free - Tasks API is free and retains records for 7 days only > **MCP:** `pip install mcp-webextrator` | Hosted: `https://webextrator.mcp.acedata.cloud/mcp` | See [all MCP servers](../_shared/mcp-servers.md)
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.