Claude Skill

web-scraper-api

Production-grade web scraping with automatic anti-bot bypass, structured JSON parsing for 40+ targets, and geo-targeting. Use when the user needs to scrape web pages, extract product data, get search results, or collect structured data from supported e-commerce and search platfor

LLM Mart · 0 points · 0 views 0 listing impressions 0 install-command copies
Virus-scanned Reviewed automatically before listing.

Full trust report

Download oxylabs-agent-skills-skills_web-scraper-api-35eb792.zip · 5 KB
oxylabs/agent-skills 875 2 forks MIT Updated 1d ago
Part of oxylabs/agent-skills — 5 skills

Install

skills CLI npx skills add https://github.com/oxylabs/agent-skills/tree/main/skills/web-scraper-api
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install oxylabs-agent-skills@llmmart
Git git clone https://github.com/oxylabs/agent-skills.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole oxylabs/agent-skills collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

Oxylabs Web Scraper API

Authentication

Requires HTTP Basic Auth with credentials from environment variables:

curl -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" ...

Endpoint

POST https://realtime.oxylabs.io/v1/queries   # immediate response
POST https://data.oxylabs.io/v1/queries       # Push-Pull jobs, callbacks, storage
Content-Type: application/json

Core Parameters

Parameter Required Description
source Yes Target scraper (e.g., universal, amazon_product, google_search)
url Conditional URL to scrape (for universal and *_url sources)
query Conditional Search query or product ID (for *_search and *_product sources)
parse No Enable structured data parsing (recommended for supported sources)
render No JavaScript rendering: html or png
geo_location No Geographic targeting: country/state/city, ZIP/postcode, coordinates, or Criteria ID where supported
session_id No Reuse the same proxy IP across multiple jobs
content_encoding No Set to base64 when downloading image files via Realtime or Push-Pull
user_agent_type No Device/browser preset, e.g., desktop_chrome, mobile_ios, tablet_android
locale No Interface language / Accept-Language, e.g., de-DE
callback_url No Push-Pull callback endpoint
storage_type, storage_url No Push-Pull cloud upload target (gcs, s3, tos, s3_compatible)
markdown, xhr No Enable markdown or captured XHR result types
browser_instructions No Rendered browser actions; requires render: "html"
parsing_instructions, parser_preset No Custom parser rules or saved preset; pair with parse: true
client_notes No Client-side job tag saved with the job metadata
domain, subdomain, start_page, pages, limit, store_id, delivery_zip, fulfillment_type Source-specific Marketplace/search/store localization and pagination fields

user_agent_type values: desktop, desktop_chrome, desktop_edge, desktop_firefox, desktop_opera, desktop_safari, mobile, mobile_android, mobile_ios, tablet, tablet_android, tablet_ios.

Context Parameters

Add these as { "key": "...", "value": ... } objects in context:

Key Use
force_headers, headers Merge custom headers with managed headers
force_cookies, cookies Merge custom cookies with managed cookies
http_method, content Use post with Base64-encoded body content
follow_redirects Follow 3xx redirect chains
successful_status_codes Treat specific non-standard HTTP codes as successful

For multi-format output, enable types in the payload (parse, markdown, xhr, render: "png") and request them with ?type=raw,parsed,png,markdown,xhr.

For batch Push-Pull jobs, use POST /v1/queries/batch with arrays only for query or url; keep all other parameters singular. Maximum batch size is 5,000 values.

Quick Start

Scrape any URL:

curl -X POST 'https://realtime.oxylabs.io/v1/queries' \
  -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{"source": "universal", "url": "https://example.com"}'

Google search with parsing:

curl -X POST 'https://realtime.oxylabs.io/v1/queries' \
  -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{"source": "google_search", "query": "best laptops", "parse": true}'

Amazon product by ASIN:

curl -X POST 'https://realtime.oxylabs.io/v1/queries' \
  -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{"source": "amazon_product", "query": "B07FZ8S74R", "parse": true}'

Choosing the Right Source

  1. Use specific sources when available (amazon_product, google_search) - better parsing and reliability
  2. Use universal for unsupported sites - works with any URL
  3. Enable parse: true for structured JSON output on supported sources

Response Structure

{
  "results": [{
    "content": "...",
    "status_code": 200,
    "url": "https://..."
  }]
}

With parse: true, content contains structured data (title, price, reviews, etc.) instead of raw HTML.

Available Sources

For the complete list of 40+ supported sources organized by category, see sources.md.

More Examples

For detailed request/response examples including geo-location, JavaScript rendering, and custom headers, see examples.md.

Error Handling

Code Meaning
200 Success
400 Invalid parameters
401 Authentication failed
403 Access denied
429 Rate limit exceeded

Key Guidelines

  • Always set parse: true for supported sources to get structured data
  • Use ZIP codes for US e-commerce geo-location (e.g., "90210")
  • Use country/state format for search engines (e.g., "California,United States")
  • Add render: "html" for JavaScript-heavy pages
  • Use render: "" only to disable automatic forced rendering for force-rendered pages; set client timeouts near 180 seconds for rendered Realtime or Proxy Endpoint requests
  • Add content_encoding: "base64" when scraping image URLs, then decode results[0].content before saving the file
Files (agent-skills)
  • examples.md 5.9 KB 
    # Request/Response Examples

## Universal URL Scraping

**Request:**
```bash
curl -X POST 'https://realtime.oxylabs.io/v1/queries' \
  -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{
    "source": "universal",
    "url": "https://example.com"
  }'
```

**Response:**
```json
{
  "results": [{
    "content": "<!DOCTYPE html><html>...</html>",
    "created_at": "2024-01-15T10:30:00.000Z",
    "updated_at": "2024-01-15T10:30:01.000Z",
    "page": 1,
    "url": "https://example.com",
    "job_id": "7654321",
    "status_code": 200
  }]
}
```

## Google Search with Parsing

**Request:**
```bash
curl -X POST 'https://realtime.oxylabs.io/v1/queries' \
  -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{
    "source": "google_search",
    "query": "best wireless headphones",
    "geo_location": "United States",
    "parse": true
  }'
```

**Response (parsed):**
```json
{
  "results": [{
    "content": {
      "results": {
        "organic": [
          {
            "pos": 1,
            "url": "https://example.com/headphones",
            "title": "Best Wireless Headphones 2024",
            "desc": "Our top picks for wireless headphones..."
          }
        ],
        "paid": [...],
        "knowledge": {...}
      }
    },
    "status_code": 200
  }]
}
```

## Amazon Product

**Request:**
```bash
curl -X POST 'https://realtime.oxylabs.io/v1/queries' \
  -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{
    "source": "amazon_product",
    "query": "B07FZ8S74R",
    "geo_location": "90210",
    "parse": true
  }'
```

**Response (parsed):**
```json
{
  "results": [{
    "content": {
      "title": "Echo Dot (3rd Gen)",
      "price": 29.99,
      "currency": "USD",
      "rating": 4.7,
      "reviews_count": 845234,
      "availability": "In Stock",
      "seller": "Amazon.com",
      "categories": ["Electronics", "Smart Home"],
      "images": ["https://..."],
      "description": "..."
    },
    "status_code": 200
  }]
}
```

## Amazon Search

**Request:**
```bash
curl -X POST 'https://realtime.oxylabs.io/v1/queries' \
  -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{
    "source": "amazon_search",
    "query": "wireless mouse",
    "geo_location": "10001",
    "parse": true
  }'
```

**Response (parsed):**
```json
{
  "results": [{
    "content": {
      "results": {
        "organic": [
          {
            "pos": 1,
            "asin": "B07CGKQLWG",
            "title": "Logitech M510 Wireless Mouse",
            "price": 24.99,
            "rating": 4.6,
            "reviews_count": 52341
          }
        ]
      }
    },
    "status_code": 200
  }]
}
```

## JavaScript Rendering

For pages that require JavaScript execution:

**Request:**
```bash
curl -X POST 'https://realtime.oxylabs.io/v1/queries' \
  -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{
    "source": "universal",
    "url": "https://example.com/spa-page",
    "render": "html"
  }'
```

## Geo-Location Examples

**For search engines (country/state/city):**
```json
{
  "source": "google_search",
  "query": "coffee shops",
  "geo_location": "New York,New York,United States"
}
```

**For US e-commerce (ZIP code):**
```json
{
  "source": "amazon_product",
  "query": "B07FZ8S74R",
  "geo_location": "90210"
}
```

**For international e-commerce (country):**
```json
{
  "source": "amazon_product",
  "query": "B07FZ8S74R",
  "geo_location": "Germany"
}
```

## Custom Headers

**Request:**
```bash
curl -X POST 'https://realtime.oxylabs.io/v1/queries' \
  -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{
    "source": "universal",
    "url": "https://example.com",
    "context": [
      {
        "key": "headers",
        "value": {
          "Accept-Language": "de-DE",
          "Custom-Header": "value"
        }
      }
    ]
  }'
```

## Browser Instructions

For complex interactions (clicking, scrolling, waiting):

**Request:**
```bash
curl -X POST 'https://realtime.oxylabs.io/v1/queries' \
  -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{
    "source": "universal",
    "url": "https://example.com",
    "render": "html",
    "browser_instructions": [
      {"type": "click", "selector": "button.load-more"},
      {"type": "wait", "wait_time_s": 2},
      {"type": "scroll", "direction": "down", "pixels": 500}
    ]
  }'
```


## Walmart Product

**Request:**
```bash
curl -X POST 'https://realtime.oxylabs.io/v1/queries' \
  -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{
    "source": "walmart_product",
    "query": "123456789",
    "parse": true
  }'
```

## Error Response

**Request with invalid source:**
```json
{
  "source": "invalid_source",
  "url": "https://example.com"
}
```

**Response:**
```json
{
  "error": {
    "code": "INVALID_SOURCE",
    "message": "Source 'invalid_source' is not supported"
  }
}
```

## Python Example

```python
import requests
import os

response = requests.post(
    "https://realtime.oxylabs.io/v1/queries",
    auth=(os.environ["OXY_WSA_USERNAME"], os.environ["OXY_WSA_PASSWORD"]),
    json={
        "source": "amazon_product",
        "query": "B07FZ8S74R",
        "parse": True
    }
)

data = response.json()
product = data["results"][0]["content"]
print(f"Title: {product['title']}")
print(f"Price: ${product['price']}")
```

## JavaScript/Node.js Example

```javascript
const response = await fetch("https://realtime.oxylabs.io/v1/queries", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "Authorization": "Basic " + btoa(`${process.env.OXY_WSA_USERNAME}:${process.env.OXY_WSA_PASSWORD}`)
  },
  body: JSON.stringify({
    source: "google_search",
    query: "web scraping",
    parse: true
  })
});

const data = await response.json();
console.log(data.results[0].content);
```
  • SKILL.md 5.7 KB 
    ---
name: web-scraper-api
description: Production-grade web scraping with automatic anti-bot bypass, structured JSON parsing for 40+ targets, and geo-targeting. Use when the user needs to scrape web pages, extract product data, get search results, or collect structured data from supported e-commerce and search platforms without worrying about getting blocked and when geo targeting is required.
---

# Oxylabs Web Scraper API

## Authentication

Requires HTTP Basic Auth with credentials from environment variables:

```bash
curl -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" ...
```

## Endpoint

```
POST https://realtime.oxylabs.io/v1/queries   # immediate response
POST https://data.oxylabs.io/v1/queries       # Push-Pull jobs, callbacks, storage
Content-Type: application/json
```

## Core Parameters

| Parameter | Required | Description |
|-----------|----------|-------------|
| `source` | Yes | Target scraper (e.g., `universal`, `amazon_product`, `google_search`) |
| `url` | Conditional | URL to scrape (for `universal` and `*_url` sources) |
| `query` | Conditional | Search query or product ID (for `*_search` and `*_product` sources) |
| `parse` | No | Enable structured data parsing (recommended for supported sources) |
| `render` | No | JavaScript rendering: `html` or `png` |
| `geo_location` | No | Geographic targeting: country/state/city, ZIP/postcode, coordinates, or Criteria ID where supported |
| `session_id` | No | Reuse the same proxy IP across multiple jobs |
| `content_encoding` | No | Set to `base64` when downloading image files via Realtime or Push-Pull |
| `user_agent_type` | No | Device/browser preset, e.g., `desktop_chrome`, `mobile_ios`, `tablet_android` |
| `locale` | No | Interface language / `Accept-Language`, e.g., `de-DE` |
| `callback_url` | No | Push-Pull callback endpoint |
| `storage_type`, `storage_url` | No | Push-Pull cloud upload target (`gcs`, `s3`, `tos`, `s3_compatible`) |
| `markdown`, `xhr` | No | Enable markdown or captured XHR result types |
| `browser_instructions` | No | Rendered browser actions; requires `render: "html"` |
| `parsing_instructions`, `parser_preset` | No | Custom parser rules or saved preset; pair with `parse: true` |
| `client_notes` | No | Client-side job tag saved with the job metadata |
| `domain`, `subdomain`, `start_page`, `pages`, `limit`, `store_id`, `delivery_zip`, `fulfillment_type` | Source-specific | Marketplace/search/store localization and pagination fields |

`user_agent_type` values: `desktop`, `desktop_chrome`, `desktop_edge`, `desktop_firefox`, `desktop_opera`, `desktop_safari`, `mobile`, `mobile_android`, `mobile_ios`, `tablet`, `tablet_android`, `tablet_ios`.

## Context Parameters

Add these as `{ "key": "...", "value": ... }` objects in `context`:

| Key | Use |
|-----|-----|
| `force_headers`, `headers` | Merge custom headers with managed headers |
| `force_cookies`, `cookies` | Merge custom cookies with managed cookies |
| `http_method`, `content` | Use `post` with Base64-encoded body content |
| `follow_redirects` | Follow 3xx redirect chains |
| `successful_status_codes` | Treat specific non-standard HTTP codes as successful |

For multi-format output, enable types in the payload (`parse`, `markdown`, `xhr`, `render: "png"`) and request them with `?type=raw,parsed,png,markdown,xhr`.

For batch Push-Pull jobs, use `POST /v1/queries/batch` with arrays only for `query` or `url`; keep all other parameters singular. Maximum batch size is 5,000 values.

## Quick Start

**Scrape any URL:**
```bash
curl -X POST 'https://realtime.oxylabs.io/v1/queries' \
  -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{"source": "universal", "url": "https://example.com"}'
```

**Google search with parsing:**
```bash
curl -X POST 'https://realtime.oxylabs.io/v1/queries' \
  -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{"source": "google_search", "query": "best laptops", "parse": true}'
```

**Amazon product by ASIN:**
```bash
curl -X POST 'https://realtime.oxylabs.io/v1/queries' \
  -u "$OXY_WSA_USERNAME:$OXY_WSA_PASSWORD" \
  -H 'Content-Type: application/json' \
  -d '{"source": "amazon_product", "query": "B07FZ8S74R", "parse": true}'
```

## Choosing the Right Source

1. **Use specific sources when available** (`amazon_product`, `google_search`) - better parsing and reliability
2. **Use `universal` for unsupported sites** - works with any URL
3. **Enable `parse: true`** for structured JSON output on supported sources

## Response Structure

```json
{
  "results": [{
    "content": "...",
    "status_code": 200,
    "url": "https://..."
  }]
}
```

With `parse: true`, `content` contains structured data (title, price, reviews, etc.) instead of raw HTML.

## Available Sources

For the complete list of 40+ supported sources organized by category, see [sources.md](sources.md).

## More Examples

For detailed request/response examples including geo-location, JavaScript rendering, and custom headers, see [examples.md](examples.md).

## Error Handling

| Code | Meaning |
|------|---------|
| 200 | Success |
| 400 | Invalid parameters |
| 401 | Authentication failed |
| 403 | Access denied |
| 429 | Rate limit exceeded |

## Key Guidelines

- Always set `parse: true` for supported sources to get structured data
- Use ZIP codes for US e-commerce geo-location (e.g., `"90210"`)
- Use country/state format for search engines (e.g., `"California,United States"`)
- Add `render: "html"` for JavaScript-heavy pages
- Use `render: ""` only to disable automatic forced rendering for force-rendered pages; set client timeouts near 180 seconds for rendered Realtime or Proxy Endpoint requests
- Add `content_encoding: "base64"` when scraping image URLs, then decode `results[0].content` before saving the file
  • sources.md 5.4 KB 
    # Available Sources

## Universal

| Source | Use Case |
|--------|----------|
| `universal` | Any website by URL |

## Google

| Source | Use Case |
|--------|----------|
| `google_search` | Search results with ads, snippets, knowledge panels |
| `google_url` | Any Google page by URL |
| `google_image_search` | Image search results |
| `google_news_search` | News results |
| `google_local_search` | Local business results |
| `google_trends_explore` | Trends data by keyword |
| `google_travel_hotels` | Hotel search results |
| `google_lens` | Visual search from image URL |
| `google_ai_mode` | AI Mode responses |
| `google_ai_overviews` | AI Overviews data |
| `google_ads` | Paid advertisement data |
| `google_reverse_image_search` | Reverse image lookup |

## Bing

| Source | Use Case |
|--------|----------|
| `bing_search` | Search results |
| `bing_url` | Any Bing page by URL |

## Amazon

| Source | Use Case |
|--------|----------|
| `amazon_search` | Search results |
| `amazon_product` | Product details by ASIN |
| `amazon_pricing` | Price and offer data |
| `amazon_sellers` | Seller information |
| `amazon_best_sellers` | Best seller rankings |
| `amazon_url` | Any Amazon page by URL |

## Walmart

| Source | Use Case |
|--------|----------|
| `walmart_search` | Search results |
| `walmart_product` | Product details |
| `walmart_url` | Any Walmart page by URL |

## eBay

| Source | Use Case |
|--------|----------|
| `ebay_search` | Search results |
| `ebay_product` | Product details |
| `ebay_url` | Any eBay page by URL |

## Target

| Source | Use Case |
|--------|----------|
| `target_search` | Search results |
| `target_product` | Product details |
| `target_category` | Category pages |

## Best Buy

| Source | Use Case |
|--------|----------|
| `bestbuy_search` | Search results |
| `bestbuy_product` | Product details |

## Etsy

| Source | Use Case |
|--------|----------|
| `etsy_search` | Search results |
| `etsy_product` | Product details |
| `etsy_url` | Any Etsy page by URL |

## Costco

| Source | Use Case |
|--------|----------|
| `costco_search` | Search results |
| `costco_product` | Product details |
| `costco_url` | Any Costco page by URL |

## Alibaba

| Source | Use Case |
|--------|----------|
| `alibaba_search` | Search results |
| `alibaba_product` | Product details |
| `alibaba_url` | Any Alibaba page by URL |

## AliExpress

| Source | Use Case |
|--------|----------|
| `aliexpress_search` | Search results |
| `aliexpress_product` | Product details |
| `aliexpress_url` | Any AliExpress page by URL |

## Flipkart

| Source | Use Case |
|--------|----------|
| `flipkart_search` | Search results |
| `flipkart_product` | Product details |
| `flipkart_url` | Any Flipkart page by URL |

## Lazada

| Source | Use Case |
|--------|----------|
| `lazada_search` | Search results |
| `lazada_product` | Product details |
| `lazada_url` | Any Lazada page by URL |

## Mercado Libre

| Source | Use Case |
|--------|----------|
| `mercadolibre_search` | Search results |
| `mercadolibre_product` | Product details |
| `mercadolibre_url` | Any Mercado Libre page by URL |

## YouTube

| Source | Use Case |
|--------|----------|
| `youtube_search` | Video search results |
| `youtube_metadata` | Video metadata |
| `youtube_subtitles` | Closed captions |
| `youtube_channel` | Channel data |

## Real Estate

| Source | Use Case |
|--------|----------|
| `airbnb_homes` | Property listings |
| `airbnb_url` | Any Airbnb page by URL |
| `zillow_url` | Zillow property pages |

## AI Platforms

| Source | Use Case |
|--------|----------|
| `chatgpt` | ChatGPT responses |
| `perplexity` | Perplexity AI responses |

## TikTok Shop

| Source | Use Case |
|--------|----------|
| `tiktok_shop_search` | Search results |
| `tiktok_shop_product` | Product details |
| `tiktok_shop_url` | Any TikTok Shop page by URL |

## Other North American E-Commerce

| Source | Use Case |
|--------|----------|
| `lowes_search`, `lowes_product`, `lowes_url` | Lowe's |
| `kroger_search`, `kroger_product`, `kroger_url` | Kroger |
| `menards_search`, `menards_product`, `menards_url` | Menards |
| `grainger_search`, `grainger_product`, `grainger_url` | Grainger |
| `publix_search`, `publix_product`, `publix_url` | Publix |
| `instacart_search`, `instacart_product`, `instacart_url` | Instacart |
| `petco_search`, `petco_url` | Petco |
| `staples` | Staples |
| `bedbathandbeyond_search`, `bedbathandbeyond_product`, `bedbathandbeyond_url` | Bed Bath & Beyond |
| `bodega_aurrera_search`, `bodega_aurrera_product`, `bodega_aurrera_url` | Bodega Aurrerá |

## Other European E-Commerce

| Source | Use Case |
|--------|----------|
| `allegro_search`, `allegro_product` | Allegro |
| `mediamarkt_search`, `mediamarkt_product`, `mediamarkt_url` | MediaMarkt |
| `cdiscount_search`, `cdiscount_product`, `cdiscount_url` | Cdiscount |
| `idealo` | Idealo price comparison |

## Other Asian E-Commerce

| Source | Use Case |
|--------|----------|
| `indiamart_search`, `indiamart_product`, `indiamart_url` | IndiaMART |
| `rakuten_search`, `rakuten_url` | Rakuten |
| `tokopedia_search`, `tokopedia_url` | Tokopedia |

## Other Latin American E-Commerce

| Source | Use Case |
|--------|----------|
| `mercadolivre_search`, `mercadolivre_product` | Mercado Livre (Brazil) |
| `magazineluiza_search`, `magazineluiza_product`, `magazineluiza_url` | Magazine Luiza |
| `falabella_search`, `falabella_product`, `falabella_url` | Falabella |
| `dcard` | Dcard |

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.

Related

Claude Prompt
Tighten-my-prose editing prompt
Claude Skill
Adversarial code-review skill for Claude
Claude Workflow
Deep-research workflow (fan-out + verify)
Claude Skill
Conventional-commit writer for Claude