# Code recipes — wiring HasData into your code

## Ground rules

- **Base URL:** `https://api.hasdata.com`. Header `x-api-key` on every request.
- **Methods:** Scraper APIs are `GET`; Web Scraping is `POST`; Scraper Jobs use `POST` (submit) + `GET` (status/results) + `DELETE` (stop).
- **Key handling:** read from env (`HASDATA_API_KEY`). Never hardcode, never log.
- **Timeouts:** **client timeout ≥ 300 s.** HasData's deadline is 300 s; shorter clients get phantom failures while still being billed.
- **Retries:** `429` and `5xx` only with exponential backoff + jitter. Never retry `4xx`.
- **Concurrency:** cap at plan limit. Free tier = 1.
- **Success signal:** sync APIs require `body.requestMetadata.status === "ok"`. HTTP 200 alone isn't enough.

## Status codes

| Code | Meaning | Action |
|---|---|---|
| 200 + `status:"ok"` | OK | Use body |
| 401 | Bad/missing key | Fix — don't retry |
| 403 | Quota exhausted | Don't retry |
| 429 | Concurrency cap | Backoff + retry |
| 500 | Server error | Retry |

## Python — minimal client

```python
import os, requests

class HasData:
    BASE = "https://api.hasdata.com"

    def __init__(self, api_key=None, timeout=300):
        self.s = requests.Session()
        self.s.headers["x-api-key"] = api_key or os.environ["HASDATA_API_KEY"]
        self.timeout = timeout

    def get(self, path, **params):
        r = self.s.get(f"{self.BASE}{path}", params=params, timeout=self.timeout)
        r.raise_for_status()
        body = r.json()
        if body.get("requestMetadata", {}).get("status") != "ok":
            raise RuntimeError(f"hasdata not-ok: {body.get('requestMetadata')}")
        return body

    def post(self, path, body):
        r = self.s.post(f"{self.BASE}{path}", json=body, timeout=self.timeout)
        r.raise_for_status()
        return r.json()

hd = HasData()
serp = hd.get("/scrape/google/serp", q="coffee", num=20)["organicResults"]
md   = hd.post("/scrape/web", {"url": "https://example.com", "outputFormat": ["markdown"]})["markdown"]
```

## Python — retry + bounded concurrency

```python
import time, random
from concurrent.futures import ThreadPoolExecutor, as_completed
from requests import HTTPError

def with_retry(fn, attempts=5, base=1.0, cap=60.0):
    for i in range(attempts):
        try:
            return fn()
        except HTTPError as e:
            code = e.response.status_code
            if code == 429 or 500 <= code < 600:
                time.sleep(min(cap, base * 2 ** i) + random.random())
                continue
            raise
    raise RuntimeError("retry exhausted")

def scrape_many(urls, workers=5):
    out = {}
    with ThreadPoolExecutor(max_workers=workers) as ex:
        futs = {ex.submit(lambda u=u: hd.post("/scrape/web", {"url": u, "outputFormat": ["markdown"]})): u
                for u in urls}
        for f in as_completed(futs):
            try:
                out[futs[f]] = f.result().get("markdown")
            except Exception as e:
                out[futs[f]] = e
    return out
```

Cap `workers` at your plan's concurrency — anything higher just generates `429`s.

## TypeScript — minimal client

```typescript
const BASE = "https://api.hasdata.com";
const KEY  = process.env.HASDATA_API_KEY!;

async function get<T = any>(path: string, params: Record<string, string | number> = {}): Promise<T> {
  const qs = new URLSearchParams(Object.entries(params).map(([k, v]) => [k, String(v)]));
  const r = await fetch(`${BASE}${path}?${qs}`, {
    headers: { "x-api-key": KEY },
    signal:  AbortSignal.timeout(300_000),
  });
  if (!r.ok) throw new Error(`HasData ${r.status} ${await r.text()}`);
  const body = await r.json() as any;
  if (body?.requestMetadata?.status && body.requestMetadata.status !== "ok") {
    throw new Error(`HasData not-ok: ${JSON.stringify(body.requestMetadata)}`);
  }
  return body as T;
}

async function post<T = any>(path: string, body: unknown): Promise<T> {
  const r = await fetch(`${BASE}${path}`, {
    method:  "POST",
    headers: { "x-api-key": KEY, "Content-Type": "application/json" },
    body:    JSON.stringify(body),
    signal:  AbortSignal.timeout(300_000),
  });
  if (!r.ok) throw new Error(`HasData ${r.status} ${await r.text()}`);
  return r.json() as Promise<T>;
}

// Bounded concurrency, no deps
async function pool<T, R>(items: T[], n: number, fn: (x: T) => Promise<R>) {
  const out: R[] = []; let i = 0;
  await Promise.all(Array.from({ length: n }, async () => {
    while (i < items.length) { const k = i++; out[k] = await fn(items[k]); }
  }));
  return out;
}
```

## Pagination cheat sheet

| Endpoint family | Pagination |
|---|---|
| Google SERP / Light SERP / Bing | `start` + `num` (max 100) |
| Google Maps Search | `start` (steps of 20) |
| Yelp Search | `start` (steps of 10) |
| Google Maps Reviews / Glassdoor / Airbnb | `nextPageToken` |
| Indeed / YellowPages / Amazon Search | `start` or `page` |
| Shopify Products | `page` (with `limit` ≤ 250) |
| Scraper-Job results | `page` + `limit` (max 100) until `meta.currentPage >= meta.lastPage` |

## Pre-ship checklist

- [ ] Key from env, never logged.
- [ ] All HTTP timeouts ≥ 300 s.
- [ ] `requestMetadata.status === "ok"` checked on every sync response.
- [ ] Backoff on 429 + 5xx; never on 4xx.
- [ ] Concurrency capped at plan limit.
- [ ] Job `id` (from submit response) persisted to durable storage immediately.
- [ ] Webhooks paired with polling fallback.
- [ ] Result files downloaded immediately on `scraper.job.finished`.

# E-commerce APIs — Amazon & Shopify

| Endpoint | Returns |
|---|---|
| `/scrape/amazon/product` | Single product (price, ratings, variants, other sellers, A+) |
| `/scrape/amazon/search` | Search results (sponsored + organic) |
| `/scrape/amazon/seller` | Seller profile |
| `/scrape/amazon/seller-products` | Seller catalog |
| `/scrape/shopify/products` | Products from any Shopify store |
| `/scrape/shopify/collections` | Collections from any Shopify store |

All synchronous `GET`.

## Amazon Product

```python
import requests

resp = requests.get(
    "https://api.hasdata.com/scrape/amazon/product",
    headers={"x-api-key": API_KEY},
    params={"asin": "B0DHJ7SBDR", "domain": "www.amazon.com", "otherSellers": "true"},
    timeout=300,
)
```

| Param | Notes |
|---|---|
| `asin` | **Required.**. |
| `domain` | `www.amazon.com` (default), `.co.uk`, `.de`, `.co.jp`, … |
| `language` | Locale per domain. |
| `deliveryZip` | Affects shipping/availability fields. |
| `shippingLocation` | 2-letter country code. |
| `otherSellers` | `true` (default) to include other-seller block. |

Response: top-level `requestMetadata` + `product`. The `product` object's keys (verified live): `asin`, `url`, `title`, `brand`, `isAvailable`, `primaryFeatures`, `features`, `featureBullets`, `description`, `badges`, `breadcrumbs`, `whatIsInTheBox`, `variants`, `totalImages`, `primaryImage`, `images`, `descriptionImages`, `totalVideos`, `primaryVideo`, `videos`, `specification`, `reviewsInfo` (rating + count + sample reviews live here, not at the root). Pricing fields are surfaced via `variants` and `specification`.

## Amazon Search

```python
params = {"q": "mechanical keyboard", "domain": "www.amazon.com", "page": 1}
```

Params: `q` (required), `domain`, `language`, `page`, `deliveryZip`, `shippingLocation`, `sortBy`.

## Amazon Seller / Seller Products

```python
profile = requests.get(
    "https://api.hasdata.com/scrape/amazon/seller",
    headers={"x-api-key": API_KEY},
    params={"sellerId": "A1MNOPQR", "domain": "www.amazon.com"},
    timeout=300,
).json()

catalog = requests.get(
    "https://api.hasdata.com/scrape/amazon/seller-products",
    headers={"x-api-key": API_KEY},
    params={"sellerId": "A1MNOPQR", "page": 1},
    timeout=300,
).json()
```

Use cases: counterfeit detection, MAP enforcement, competitor catalog mirroring.

## Shopify Products

Works on **any** Shopify storefront with no authentication.

```python
def shopify_all(store_url):
    page, out = 1, []
    while True:
        batch = requests.get(
            "https://api.hasdata.com/scrape/shopify/products",
            headers={"x-api-key": API_KEY},
            params={"url": store_url, "page": page, "limit": 250},
            timeout=300,
        ).json().get("products", [])
        if not batch:
            return out
        out.extend(batch)
        page += 1
```

| Param | Notes |
|---|---|
| `url` | **Required.** Storefront URL. |
| `limit` | 1–250, default `1`. **Bump to 250** for catalog work. |
| `page` | 1-indexed. |
| `collection` | Collection handle filter. |

`/scrape/shopify/collections` has the same shape and returns the collection list.

## Patterns

### Cross-merchant price comparison

```python
a = requests.get("https://api.hasdata.com/scrape/amazon/search",
                 headers={"x-api-key": API_KEY},
                 params={"q": query}, timeout=300).json()
g = requests.get("https://api.hasdata.com/scrape/google/shopping",
                 headers={"x-api-key": API_KEY},
                 params={"q": query, "gl": "us"}, timeout=300).json()
```

### Reviews & bestsellers go through Scraper Jobs

The Product API only includes a sample of reviews. For all reviews use the `amazon-product-reviews` Scraper Job. For bestseller ranks use `amazon-bestsellers` — there's no synchronous API. See `scraper-jobs.md`.

## Gotchas

- **Same ASIN ≠ same product across `domain`s.** `.com` vs `.co.uk` can differ.
- **`deliveryZip` changes availability.** Pass it when stock matters; omit for spec-only scrapes.
- **Shopify `limit` defaults to 1** — always set 250 for catalog crawls.

# Jobs APIs — Indeed & Glassdoor

| Endpoint | Returns |
|---|---|
| `/scrape/indeed/listing` | Indeed search results |
| `/scrape/indeed/job` | Single Indeed job detail |
| `/scrape/glassdoor/listing` | Glassdoor search results |
| `/scrape/glassdoor/job` | Single Glassdoor job (incl. salary band, company snippet) |

All synchronous `GET`.

## Indeed Listing

```python
import requests

resp = requests.get(
    "https://api.hasdata.com/scrape/indeed/listing",
    headers={"x-api-key": API_KEY},
    params={
        "keyword":  "software engineer",
        "location": "New York, NY",
        "sort":     "date",
        "domain":   "www.indeed.com",
        "start":    0,
    },
    timeout=300,
)
```

| Param | Notes |
|---|---|
| `keyword` | **Required.** |
| `location` | **Required.** |
| `sort` | `date`, `relevance` (default). |
| `domain` | Country site — `www.indeed.com`, `uk.indeed.com`, `de.indeed.com`. |
| `start` | Offset, **steps of 10**. |

Response: `jobs` array with `title`, `company`, `location`, `salary`, `description`, `postedAt`, `link`, `jobKey`. Salary is free-form string — parse with regex.

## Indeed Job

Pass `jobKey` from listing → returns full description, requirements, benefits, company URL.

## Glassdoor Listing & Job

```python
params = {"keyword": "software engineer", "location": "New York, NY", "sort": "recent"}
# pagination: pass back nextPageToken
```

| Param | Notes |
|---|---|
| `keyword`, `location` | **Required.** |
| `sort` | `recent` (default), `relevant`. |
| `domain` | Country site. |
| `nextPageToken` | Cursor pagination. |

## Patterns

### Salary band

```python
import re, statistics

def salary_band(role, location):
    page = requests.get(
        "https://api.hasdata.com/scrape/indeed/listing",
        headers={"x-api-key": API_KEY},
        params={"keyword": role, "location": location}, timeout=300,
    ).json()
    nums = [int(m.replace(",", ""))
            for j in page.get("jobs", [])
            for m in re.findall(r"\$([\d,]+)", j.get("salary") or "")]
    if not nums: return None
    return {"n": len(nums), "median": statistics.median(nums)}
```

### Hiring velocity by company

```python
from collections import Counter

page = indeed_listing(role, loc, sort="date")
Counter(j.get("company") for j in page.get("jobs", []))
```

Run weekly; sustained increases often precede earnings/PR signals.

### Pagination differs

```python
# Indeed: numeric start
for p in range(10):
    page = indeed_listing(kw, loc, start=p * 10)

# Glassdoor: cursor token
out, token = [], None
while True:
    page = glassdoor_listing(kw, loc, next_token=token)
    out.extend(page.get("jobs", []))
    token = page.get("nextPageToken")
    if not token: break
```

## Gotchas

- **Salary is free-form string.** Always regex-parse.
- **Indeed = numeric start (10), Glassdoor = token.** Don't mix.
- **`domain` matters for non-US.** `uk.indeed.com`, `ca.indeed.com`, etc.
- **Prefer the API + pagination for bulk.** Reach for the matching Scraper Job only when you want webhook-driven fan-out across many keyword × location pairs without managing the polling loop yourself.

# Local Business APIs — Google Maps, Yelp, YellowPages

| Endpoint | Returns |
|---|---|
| `/scrape/google-maps/search` | Search results in a viewport |
| `/scrape/google-maps/place` | Single place details |
| `/scrape/google-maps/reviews` | Reviews for a place, paginated |
| `/scrape/google-maps/photos` | Photo gallery |
| `/scrape/google-maps/posts` | Owner-published posts (offers, events, announcements) |
| `/scrape/google-maps/contributor-reviews` | All reviews by a Google reviewer |
| `/scrape/yelp/search` | Yelp search |
| `/scrape/yelp/place` | Yelp business detail |
| `/scrape/yellowpages/search` | YellowPages search |
| `/scrape/yellowpages/place` | YellowPages business detail |

All synchronous `GET`.

## Google Maps Search

```python
import requests

resp = requests.get(
    "https://api.hasdata.com/scrape/google-maps/search",
    headers={"x-api-key": API_KEY},
    params={"q": "Pizza", "ll": "@40.7455,-74.0083,14z"},
    timeout=300,
)
```

| Param | Notes |
|---|---|
| `q` | **Required.** Free-form query. |
| `ll` | `@LAT,LNG,ZOOMz` viewport — **lat/lng + zoom, not a city name**. Required for tight pagination. |
| `domain`, `gl`, `hl` | Standard. |
| `start` | Pagination offset, **steps of 20**. |

Response: `localResults` — each entry has `position`, `title`, `placeId`, `dataId`, `kgmid`, `thumbnail`, `phone`, `address`, `website`, `description`, `workingHours` (object with `timezone` + `days[]`), `openState`, `rating`, `reviews`, `type` + `types[]` (categories), `price`, `priceDescription`, `gpsCoordinates`, `serviceOptions[]`, `extensions` (offerings, accessibility, payments, …), `menu`. Feed `placeId`/`dataId` into `/place` and `/reviews`.

## Google Maps Place

```python
params = {"placeId": "ChIJFU2bda4SM4cRKSCRyb6pOB8"}
```

Returns full place detail — coordinates, hours by day, phone, website, popular times, attributes (delivery, dine-in), photo summary.

## Google Maps Reviews

```python
def reviews(place_id=None, data_id=None, sort_by="newestFirst", token=None):
    params = {}
    if place_id: params["placeId"] = place_id
    if data_id:  params["dataId"]  = data_id
    if sort_by:  params["sortBy"]  = sort_by
    if token:    params["nextPageToken"] = token
    return requests.get(
        "https://api.hasdata.com/scrape/google-maps/reviews",
        headers={"x-api-key": API_KEY},
        params=params, timeout=300,
    ).json()
```

| Param | Notes |
|---|---|
| `placeId` / `dataId` | Pass one. `dataId` is the hex pair from Maps results. |
| `sortBy` | `newestFirst`, `highestRating`, `lowestRating`, `mostRelevant`. |
| `topicId` | Filter by review topic. |
| `nextPageToken` | Cursor pagination. |

## Google Maps Posts

```python
resp = requests.get(
    "https://api.hasdata.com/scrape/google-maps/posts",
    headers={"x-api-key": API_KEY},
    params={"placeId": "ChIJ..."},      # or dataId="0x...:0x..."
    timeout=300,
)
for p in resp.json().get("posts", []):
    print(p["postedAt"], p["description"][:120], p.get("cta", {}).get("url"))
```

Either `placeId` **or** `dataId` is required. Optional: `hl` (UI language), `nextPageToken` (cursor pagination). 10 credits/call.

Per-post fields (verified live): `postId`, `locationId`, `title`, `description`, `image`, `cta` (`label` + `url`), `createdAt` (ISO), `postedAt` (human-readable), `shareUrl`, `postUrl`. Response top-level: `posts`, `pagination`, `source`, `requestMetadata`.

Posts surface current offers, holiday hours, events, and product launches the business is actively promoting. Cheaper signal than the homepage scrape, and `cta.url` is the canonical landing page.

## Yelp & YellowPages

```python
# Yelp
params = {"keyword": "McDonald's", "location": "New York, NY", "start": 0}  # steps of 10
# YellowPages
params = {"keyword": "Plumbers", "location": "New York, NY", "page": 1}
```

YellowPages is US-only — EU/APAC searches return nothing useful.

## Patterns

### Lead-gen with emails (Maps + Web Scraping)

Maps results have website + phone but **not email**. Combine with the Web Scraping API's `extractEmails` only for public business contact pages, legitimate outreach, and workflows that honor opt-out, privacy-law, rate, and terms-of-service constraints:

```python
leads = []
for biz in maps_results.get("localResults", []):
    site = biz.get("website")
    if not site: continue
    page = requests.post(
        "https://api.hasdata.com/scrape/web",
        headers={"x-api-key": API_KEY},
        json={"url": site, "extractEmails": True},
        timeout=300,
    ).json()
    leads.append({
        "name":    biz["title"],
        "phone":   biz.get("phone"),
        "website": site,
        "emails":  page.get("extractedEmails") or [],
    })
```

For higher volume, switch to the `contacts` Scraper Job (see `scraper-jobs.md`) only when you have a legitimate purpose, a compliant outreach process, and rate/opt-out controls.

### New-business discovery

Filter Maps by review count `< 5` — usually means recently opened.

```python
new = [b for b in localResults if (b.get("reviews") or 0) < 5]
```

### Multi-location chain mapping

Search the brand name; every `localResults` entry is a branch.

## Gotchas

- **`ll` is a viewport, not a city.** `@lat,lng,zoom`. Pasting "Brooklyn" fails.
- **Pagination steps differ.** Maps `start` = +20, Yelp `start` = +10, Maps Reviews uses `nextPageToken`.
- **`placeId` vs `dataId`** — Place prefers `placeId`; Reviews accepts either.
- **YellowPages is US-only.**

# Real Estate APIs — Zillow, Redfin

| Endpoint | Returns |
|---|---|
| `/scrape/zillow/listing` | Search results by area + filters |
| `/scrape/zillow/property` | Single home (history, agent, schools, taxes) |
| `/scrape/redfin/listing` | Redfin search results |
| `/scrape/redfin/property` | Single Redfin home |

All synchronous `GET`. 5 credits each.

For short-term rentals (Airbnb), hotels (Booking), and flights, see `travel.md`.

## Zillow Listing

Filter params use **bracketed** keys (`price[min]`, `beds[max]`).

```python
import requests

def zillow_search(keyword, listing_type="forSale", **filters):
    r = requests.get(
        "https://api.hasdata.com/scrape/zillow/listing",
        headers={"x-api-key": API_KEY},
        params={"keyword": keyword, "type": listing_type, **filters},
        timeout=300,
    )
    return r.json()

zillow_search("Brooklyn, NY", price={"min": 800000, "max": 2000000})
zillow_search("33321", "sold", daysOnZillow="6m")  # recent comps
```

`requests` + `axios` serialize nested dicts as `price[min]=…&price[max]=…` automatically. With raw `URLSearchParams`, build the bracketed keys yourself.

| Param | Notes |
|---|---|
| `keyword` | **Required.** Area string ("New York, NY", zip, neighborhood). |
| `type` | **Required.** `forSale`, `forRent`, `sold`. |
| `price[min/max]`, `beds[min/max]`, `baths[min/max]`, `sqft[min/max]` | Range filters. |
| `daysOnZillow` | `24h`, `7d`, `14d`, `30d`, `90d`, `6m`, `12m`. |
| `page` | Pagination. |

Response: `requestMetadata`, `searchInformation`, **`properties`** (the listings array — not `listings`), `pagination`.

## Zillow Property

```python
requests.get(
    "https://api.hasdata.com/scrape/zillow/property",
    headers={"x-api-key": API_KEY},
    params={"url": url, "extractAgentEmails": "true"},
    timeout=300,
)
```

Takes a full Zillow URL (not zpid). Returns address, lot/sqft/beds/baths, price + tax history, schools, agent block, photos. Agent emails are best-effort.

## Redfin

```python
# Listing
params = {"keyword": "33321", "type": "forSale", "page": 1}
# Property
params = {"url": "https://www.redfin.com/FL/Tamarac/9...html"}
```

Same bracketed `price[min]`, `beds[min]`, etc. as Zillow. Zip codes work best for `keyword`.

## Patterns

### Sold comps for ROI

```python
sold = zillow_search(zip_code, "sold", daysOnZillow="6m").get("properties", [])
ppsf = [(l["price"] / l["livingArea"]) for l in sold if l.get("livingArea")]
```

## Gotchas

- **Bracketed query keys** — work with `requests`/`axios`, not raw `URLSearchParams`.
- **`type=sold` + `daysOnZillow` = comps recipe.** Without `daysOnZillow`, history is unbounded.
- **Property endpoints take URLs**, not IDs.
- **Agent emails are best-effort.**

# Scraper Jobs — async, bulk

Use only when there's no Scraper-API equivalent (`crawler`, `contacts`, `sec-edgar`, `amazon-bestsellers`, `amazon-product-reviews`) or when you want webhook-driven fan-out without managing your own polling loop. Otherwise the matching Scraper API + paginated client loop is simpler.

| Slug | Notes |
|---|---|
| `crawler` | Recursive site crawl. Accepts every Web Scraping API parameter. |
| `contacts` | URL list → emails / phones / social profiles. |
| `sec-edgar` | Bulk SEC filings by CIK / ticker / company name. |
| `google-serp`, `google-maps`, `google-maps-reviews`, `google-trends` | Bulk Google. |
| `amazon-search`, `amazon-product`, `amazon-product-reviews`, `amazon-seller-products`, `amazon-bestsellers` | Bulk Amazon. |
| `shopify` | Multi-store crawl. |
| `zillow`, `redfin`, `airbnb` | Bulk real estate. |
| `yelp`, `yellow-pages` | Bulk local. |
| `indeed`, `glassdoor` | Bulk jobs. |

## Lifecycle

1. `POST /scrapers/<slug>/jobs` → returns the full job record. **The handle is `body.id` (numeric integer), not `jobId`** despite older doc snippets — store this. Status starts as `pending`.
2. `GET /scrapers/jobs/<id>` — poll status.
3. `GET /scrapers/jobs/<id>/results?page=…&limit=100` — once `status === "finished"`.
4. `DELETE /scrapers/jobs/<id>` — stop early (rows produced before stop are kept).

Status values: `pending` → `in_progress` → `finished` (or `stopped` if cancelled).

**Shortcut for finished jobs:** the status response on a `finished` job carries a `data` object with direct download URLs:

```json
"data": {
  "csv":  "https://f005.backblazeb2.com/file/.../{uuid}.csv",
  "json": "https://f005.backblazeb2.com/file/.../{uuid}.json",
  "xlsx": "https://f005.backblazeb2.com/file/.../{uuid}.xlsx"
}
```

For one-shot ingestion, fetch `data.json` directly instead of paging `/results`. **These URLs are short-lived** — download immediately on `finished`.

## End-to-end (Python)

```python
import os, time, requests

API_KEY = os.environ["HASDATA_API_KEY"]
H = {"x-api-key": API_KEY, "Content-Type": "application/json"}
BASE = "https://api.hasdata.com"

def submit(slug, body):
    r = requests.post(f"{BASE}/scrapers/{slug}/jobs", headers=H, json=body, timeout=60)
    r.raise_for_status()
    return r.json()["id"]                            # numeric job id — not "jobId"

def wait(job_id, poll=10, cap=60, timeout=3600):
    deadline = time.time() + timeout
    while time.time() < deadline:
        s = requests.get(f"{BASE}/scrapers/jobs/{job_id}", headers=H, timeout=60).json()
        if s["status"] in ("finished", "stopped"):
            return s
        time.sleep(poll)
        poll = min(poll * 1.5, cap)
    raise TimeoutError(job_id)

def results(job_id):
    page = 1
    while True:
        body = requests.get(
            f"{BASE}/scrapers/jobs/{job_id}/results",
            headers=H, params={"page": page, "limit": 100}, timeout=120,
        ).json()
        for row in body["data"]:
            yield row["data"]                       # double-wrapped — see below
        if body["meta"]["currentPage"] >= body["meta"]["lastPage"]:
            return
        page += 1
```

### Response shapes

Submit (live):
```json
{
  "id": 416349,                        // ← the job handle, integer
  "scraperId": 26,
  "status": "pending",
  "creditsSpent": 0,
  "dataRowsCount": 0,
  "input": { ... },
  "createdAt": "...", "updatedAt": "...",
  "scraper": { "slug": "contacts", ... },
  "columns": [ ... ]
}
```

Status (live; numeric fields arrive as **strings** when populated):
```json
{
  "id": 416349,
  "status": "finished",
  "creditsSpent": "5",                 // string!
  "dataRowsCount": "1",                // string!
  "input": { ... },
  "data": {
    "csv":  "https://f005.backblazeb2.com/.../{uuid}.csv",
    "json": "https://f005.backblazeb2.com/.../{uuid}.json",
    "xlsx": "https://f005.backblazeb2.com/.../{uuid}.xlsx"
  }
}
```

Results page:
```json
{
  "meta": {
    "total": 1, "perPage": 100,
    "currentPage": 1, "lastPage": 1,
    "firstPage": 1, "firstPageUrl": "/?page=1",
    "lastPageUrl": "/?page=1",
    "nextPageUrl": null, "previousPageUrl": null
  },
  "data": [
    {
      "id": "...", "jobId": 416349, "dataId": "...",
      "data": { /* the actual scraped row */ },
      "createdAt": "...", "updatedAt": "..."
    }
  ]
}
```

**Double `data`** — the row is `body["data"][i]["data"]`; the outer wraps with `id`, `jobId`, `dataId`, `createdAt`, `updatedAt`.

## Common body fields

- `limit` (int) — max rows. `0` = no cap.
- `webhook.url` (string, https), `webhook.events` (any subset of `scraper.job.started`, `scraper.data.scraped`, `scraper.job.finished`), `webhook.headers` (sent on every callback — pin a shared secret here).

## Webhooks

```python
# Submit with webhook
submit("indeed", {
    "keywords":  ["software engineer", "data scientist"],
    "locations": ["New York, NY", "Remote"],
    "limit":     500,
    "webhook":   {
        "url":     "https://your.app/hasdata-hook",
        "events":  ["scraper.data.scraped", "scraper.job.finished"],
        "headers": {"x-shared-secret": SHARED_SECRET},
    },
})
```

```python
from flask import Flask, request, abort
app = Flask(__name__)

@app.post("/hasdata-hook")
def hook():
    if request.headers.get("x-shared-secret") != SHARED_SECRET:
        abort(401)
    e = request.json
    if e["event"] == "scraper.data.scraped":
        save_row(e["jobId"], e["data"])
    elif e["event"] == "scraper.job.finished":
        finalize(e["jobId"])
    return "", 200                  # 2xx prevents retry
```

- Async with **3 retries** on non-2xx. **Order not guaranteed** — payload is the source of truth.
- **No documented HMAC.** Pin a shared secret via `webhook.headers`, or just fetch results via the API on `scraper.job.finished` and ignore per-row deliveries.
- **Always pair webhooks with polling.** A long quiet period probably means missed callbacks.

## Per-scraper bodies

### `crawler` — recursive site crawl

Accepts every Web Scraping API parameter applied to **every page**.

| Field | Notes |
|---|---|
| `urls` | **Required.** Seed URLs. |
| `maxDepth` | Hops from seed. |
| `includePaths` / `excludePaths` | Regex. **Case-sensitive.** |
| `limit` | Cap on pages. `0` = unlimited. |

```python
job = submit("crawler", {
    "urls":         ["https://docs.example.com"],
    "maxDepth":     5,
    "includePaths": "/docs/.+",
    "outputFormat": ["markdown"],
    "excludeTags":  ["script", "style", "nav", "footer"],
    "limit":        2000,
})
```

### `contacts` — URLs → contact info

```python
submit("contacts", {"urls": ["https://example.com/about", "https://example.com/team"]})
```

Verified row schema (one row per input URL):

```json
{
  "url": "https://example.com/about",
  "emails":       ["..."],
  "phoneNumbers": ["..."],
  "linkedin":     ["..."],
  "xcom":         ["..."],          // X / Twitter — note key is "xcom"
  "facebook":     ["..."],
  "instagram":    ["..."],
  "dribbble":     ["..."],
  "clutch":       ["..."]
}
```

Empty arrays for missing categories — never null. If you only have a domain, discover URLs first via SERP `site:example.com`.

### `sec-edgar` — bulk SEC filings

```python
submit("sec-edgar", {
    "limit":       100,
    "ciks":        ["AAPL", "789019", "Alphabet Inc."],
    "filingTypes": "10-K, 10-Q, 8-K",
    "startDate":   "2024-01-01",
    "endDate":     "2025-12-31",
})
```

`ciks` accepts CIKs, tickers, or company names mixed.

### Bulk-API equivalents

`google-serp`, `google-maps`, `amazon-search`, `indeed`, `glassdoor`, etc. Jobs accept arrays of inputs (`keywords[]`, `locations[]`, etc.). Use them when you want webhook fan-out; otherwise the synchronous Scraper API + paginated client loop is simpler.

### Crawler vs Contacts vs Web Scraping batch

- **crawler** — unknown URL set, recursive discovery.
- **contacts** — known URL list, want extracted contact fields.
- **`/scrape/batch/web`** — known URL list, want full HTML/markdown/AI extraction at >1k scale.

## Gotchas

- **Persist the job `id` immediately** (the integer from the submit response — *not* `jobId`). Only handle to status, results, stop.
- **Result file retention is short.** Download right after `finished`.
- **Webhooks are best-effort.** Always poll as a backup.
- **`includePaths` regex is case-sensitive.**
- **Status `stopped` is terminal.** Rows already produced remain available.
- **Don't poll faster than every 10 s** — wastes concurrency cap.
- **Double-wrapped results** — `body["data"][i]["data"]`, not `body["data"][i]`.

# Search & SERP APIs

Pre-parsed JSON for Google, AI Mode, Bing, and the specialized Google panels. Synchronous `GET` under `https://api.hasdata.com`.

| Endpoint | Returns |
|---|---|
| `/scrape/google/serp` | Full SERP — organic + every rich-snippet block |
| `/scrape/google-light/serp` | Organic only |
| `/scrape/google/ai-mode` | Gemini answer + references |
| `/scrape/google/ai-overview` | AI Overview block |
| `/scrape/google/news` | News articles |
| `/scrape/google/shopping` | Shopping carousel |
| `/scrape/google/images` | Image search |
| `/scrape/google/events` | Local events |
| `/scrape/google/short-videos` | Short-video panel |
| `/scrape/google/immersive-product` | Expanded product pop-up |
| `/scrape/google-trends/search` | Trends + related queries |
| `/scrape/bing/serp` | Bing SERP |

For `/scrape/google/flights`, see `travel.md`.

## Google SERP

```python
import requests

resp = requests.get(
    "https://api.hasdata.com/scrape/google/serp",
    headers={"x-api-key": API_KEY},
    params={"q": "coffee beans", "gl": "us", "hl": "en", "num": 100},
    timeout=300,
)
for hit in resp.json().get("organicResults", []):
    print(hit["position"], hit["title"], hit["link"])
```

### Query parameters

| Param | Default | Notes |
|---|---|---|
| `q` | — | **Required.** |
| `location` | — | Canonical, e.g. `"Austin,Texas,United States"`. Hyper-local. |
| `uule` | — | Pre-encoded location (mutually exclusive with `location`). |
| `domain` | `google.com` | `google.co.uk`, `google.de`, … |
| `gl` | — | 2-letter country (`us`, `de`, `jp`). |
| `hl` | — | 2-letter UI language. |
| `lr` | — | Content-language filter (`lang_en`). |
| `tbs` | — | Filters — `qdr:d|w|m|y` for time, `li:1` verbatim, sort, image type. |
| `safe` | — | `active` / `off`. |
| `start` | `0` | Pagination offset. |
| `num` | `10` | Results/page. **Max 100** |
| `tbm` | — | `isch` images, `vid`, `nws`, `shop`, `lcl`. |
| `deviceType` | — | `desktop`, `mobile`, `tablet`. |

### Response keys

```
requestMetadata, searchInformation, organicResults, knowledgeGraph, answerBox,
aiOverview, topStories, newsResults, localResults, inlineShoppingResults,
inlineVideos, inlineImages, recipesResults, perspectives, discussionsAndForums,
relatedQuestions, relatedSearches, adResults, pagination
```

Rich-snippet keys appear **only when the SERP shows that block** — always `data.get(key, default)`.

### Tips

- `gl`/`hl` change ranking, not just localization. Run the same `q` with different `gl` to study geo-bias.
- `location="Austin,Texas,United States"` produces hyperlocal results that differ from `gl=us` alone.

## Google Light SERP

Same params as full SERP, but the response is trimmed to a few keys — typically `requestMetadata`, `searchInformation`, `organicResults`, `relatedSearches`, and `pagination` when present. Use for crawler seeding and link discovery when you don't need the heavier rich-snippet blocks.

## Google AI Mode

```python
resp = requests.get(
    "https://api.hasdata.com/scrape/google/ai-mode",
    headers={"x-api-key": API_KEY},
    params={"q": "is coffee good for health?", "location": "Austin,Texas,United States"},
    timeout=300,
)
```

Params: `q` (required), `location`, `uule`, `gl`. Response:

```json
{
  "requestMetadata": {...},
  "textBlocks": [
    {"type":"heading","snippet":"..."},
    {"type":"paragraph","snippet":"...","snippetHighlightedWords":["..."]},
    {"type":"list","list":[{"snippet":"..."}]},
    {"type":"table","table":{...}},
    {"type":"code","code":"..."}
  ],
  "references": [{"index":1,"link":"...","title":"...","snippet":"...","source":"..."}]
}
```

Block types observed in practice: `heading`, `paragraph`, `list`, `table`, `code`. Always switch on `type` rather than assuming a fixed set.

Pattern: AI Mode for the answer → `/scrape/web` (markdown) on each `references[].link` → cited RAG context.

## Google News / Shopping / Bing

Same shape: `q` + `gl`/`hl`/`location`. News supports `tbs=qdr:d|w|m|y` for time windows. Bing returns the same key set as Google SERP — useful for cross-engine consensus (disagreement = contested topic).

## Patterns

### Pagination

```python
def all_organic(q, target=300):
    out, start = [], 0
    while len(out) < target:
        page = requests.get(
            "https://api.hasdata.com/scrape/google-light/serp",
            headers={"x-api-key": API_KEY},
            params={"q": q, "num": 100, "start": start},
            timeout=300,
        ).json().get("organicResults", [])
        if not page:
            break
        out.extend(page)
        start += 100
    return out[:target]
```

### Reverse lookup (email / phone / domain → identity)

```python
requests.get(
    "https://api.hasdata.com/scrape/google/serp",
    headers={"x-api-key": API_KEY},
    params={"q": f'"{literal}"', "num": 20},
    timeout=300,
).json().get("organicResults", [])
```

Quoted literals (emails, phones, error strings) usually surface the canonical mention.

### Indexation check

```python
def is_indexed(url):
    r = requests.get(
        "https://api.hasdata.com/scrape/google-light/serp",
        headers={"x-api-key": API_KEY},
        params={"q": f"site:{url}", "num": 1}, timeout=300,
    )
    return bool(r.json().get("organicResults"))
```

# Travel APIs — Airbnb, Booking, Google Flights

| Endpoint | Returns |
|---|---|
| `/scrape/airbnb/listing` | Airbnb search results |
| `/scrape/airbnb/property` | Single Airbnb listing |
| `/scrape/booking/search` | Booking.com search results (hotels, apartments) |
| `/scrape/booking/place` | Single Booking.com property with room/rate list |
| `/scrape/google/flights` | Google Flights prices and itineraries |

All synchronous `GET`. Airbnb is 5 credits; Booking is 10; Google Flights is 15.

For activities at the destination see `/scrape/google/events` (in `search.md`); for ground transport, scrape the operator's site with `POST /scrape/web`.

## Airbnb

```python
import requests

def airbnb_search(location, check_in, check_out, **kwargs):
    return requests.get(
        "https://api.hasdata.com/scrape/airbnb/listing",
        headers={"x-api-key": API_KEY},
        params={"location": location, "checkIn": check_in, "checkOut": check_out, **kwargs},
        timeout=300,
    ).json()
```

| Param | Notes |
|---|---|
| `location` | **Required.** Free-form. |
| `checkIn` | **Required.** `YYYY-MM-DD`. |
| `checkOut`, `adults`, `children`, `infants`, `pets` | Optional. |
| `nextPageToken` | Pagination cursor. |

### Token pagination

```python
def airbnb_all(location, check_in, check_out):
    out, token = [], None
    while True:
        page = airbnb_search(location, check_in, check_out,
                             **({"nextPageToken": token} if token else {}))
        out.extend(page.get("listings", []))
        token = page.get("nextPageToken")
        if not token:
            return out
```

### Airbnb Property

```python
requests.get(
    "https://api.hasdata.com/scrape/airbnb/property",
    headers={"x-api-key": API_KEY},
    params={"url": "https://www.airbnb.com/rooms/12345678"},
    timeout=300,
)
```

## Booking Search

```python
import json, requests

def booking_search(keyword, check_in, check_out, *, adults=2, children=0,
                   children_ages=None, rooms=1, **filters):
    params = {
        "keyword":      keyword,
        "checkInDate":  check_in,
        "checkOutDate": check_out,
        "adults":       adults,
        "children":     children,
        "rooms":        rooms,
        **filters,
    }
    if children and children_ages:
        params["childrenAgesJson"] = json.dumps(children_ages)
    return requests.get(
        "https://api.hasdata.com/scrape/booking/search",
        headers={"x-api-key": API_KEY},
        params=params, timeout=300,
    ).json()
```

| Param | Notes |
|---|---|
| `keyword` | **Required.** City, neighborhood, or property name. |
| `checkInDate` / `checkOutDate` | **Required.** `YYYY-MM-DD`. |
| `adults`, `children`, `rooms` | **Required.** Pass `children=0` explicitly when none. |
| `childrenAgesJson` | Required iff `children > 0` — JSON array of ages (0–17), one per child. |
| `price[min]` / `price[max]` | `>= 10` / `>= 20`. Bracketed — `requests`/`axios` serialize nested dicts as `price[min]=…`. |
| `rating[]`, `reviewScore[]`, `propertyType[]`, `facilities[]`, `meals[]`, `bedPreference[]`, `roomFacilities[]`, `propertyAccessibility[]`, `roomAccessibility[]`, `distanceFromCenter[]`, `travelGroup[]`, `onlinePayment[]`, `reservationPolicy[]` | Multi-value filters (OR). |
| `bedrooms`, `bathrooms` | Minimum count. |
| `sort` | `ourTopPicks`, `homesAndApartmentsFirst`, `priceLowestFirst`, `priceHighestFirst`, `bestReviewedAndLowestPrice`, `ratingHighToLow`, `ratingLowToHigh`, `ratingAndPrice`, `distanceFromDowntown`, `topReviewed`. |
| `page` | 1-indexed, 25 results per page. |
| `currency` | ISO code or `hotelCurrency` to keep native. |
| `language` | UI locale. |

Top-level response (verified live): `requestMetadata`, `searchInformation`, `pagination`, `results`. Per-result keys: `hotelId`, `roomId`, `title`, `url`, `location`, `rating`, `reviews`, `price` (object with `total` / `nightly` / `currency`), `room`, `beds`, `bedTypes`, `policies`, `photo`.

## Booking Place

```python
resp = requests.get(
    "https://api.hasdata.com/scrape/booking/place",
    headers={"x-api-key": API_KEY},
    params={
        "url":           "https://www.booking.com/hotel/fr/le-bristol-paris.html",
        "checkInDate":   "2026-07-10",
        "checkOutDate":  "2026-07-13",
        "adults":         2,
        "children":       0,
        "rooms":          1,
    },
    timeout=300,
).json()
```

`url` must be `booking.com` / `www.booking.com`. The remaining stay/guest parameters share the same rules as `booking-search` (including `childrenAgesJson` when `children > 0`).

Response top-level keys: `requestMetadata`, `overview`, `bookingDetails`, `rooms`, `facilities`, `houseRules`, `ratings`, `reviews`, `restaurants`, `breadcrumbs`, `questionsAndAnswers`.

- `overview` → `id`, `title`, `address`, `description`, `propertyType`, `photos`, `highlights`, `mostPopularFacilities`.
- `rooms[i]` → `roomId`, `name`, `bedTypes`, `beds`, `facilities`, `otherFacilities`, `variants[]` (per-rate price/availability). Variants are the actual buyable units; `rooms[i]` is the floor-plan.

## Google Flights

```python
resp = requests.get(
    "https://api.hasdata.com/scrape/google/flights",
    headers={"x-api-key": API_KEY},
    params={
        "departureId":  "JFK",
        "arrivalId":    "LAX",
        "outboundDate": "2026-06-15",
        "returnDate":   "2026-06-22",     # omit for one-way
        "currency":     "USD",
    },
    timeout=300,
).json()
```

| Param | Notes |
|---|---|
| `departureId` / `arrivalId` | **Required.** IATA airport codes (`JFK`, `LAX`). |
| `outboundDate` | **Required.** `YYYY-MM-DD`. |
| `returnDate` | Optional — omit for one-way. |
| `currency` | ISO code. |
| `gl`, `hl` | Country / language. |
| `travelClass` | `1` economy, `2` premium economy, `3` business, `4` first. |
| `stops` | `0` any, `1` non-stop, `2` ≤1 stop, `3` ≤2 stops. |
| `adults`, `children`, `infantsInSeat`, `infantsOnLap` | Passenger counts. |

## Patterns

### STR yield estimate

```python
rentals = airbnb_search(area, ci, co).get("listings", [])           # Airbnb → "listings"
# pair with /scrape/zillow/listing (see real-estate.md) for purchase price
night   = sum(r.get("price", 0) for r in rentals) / max(len(rentals), 1)
```

### Hotel-vs-rental price diff

```python
b = booking_search(city, ci, co, adults=2, children=0, rooms=1, sort="priceLowestFirst")
a = airbnb_search(city, ci, co, adults=2)
def median(xs): xs = sorted(xs); return xs[len(xs)//2] if xs else None
median_hotel = median([r["price"]["nightly"] for r in b.get("results", []) if r.get("price")])
median_str   = median([r["price"]            for r in a.get("listings", []) if r.get("price")])
```

### Full trip cost

```python
flight = requests.get(
    "https://api.hasdata.com/scrape/google/flights",
    headers={"x-api-key": API_KEY},
    params={"departureId": origin, "arrivalId": dest_iata,
            "outboundDate": dep, "returnDate": ret, "currency": "USD"},
    timeout=300,
).json()
cheapest_flight = min((f["price"] for f in flight.get("best_flights", [])), default=None)

stay = booking_search(city, dep, ret, adults=2, children=0, rooms=1, sort="priceLowestFirst")
cheapest_stay = stay.get("results", [{}])[0].get("price", {}).get("total")

total = (cheapest_flight or 0) + (cheapest_stay or 0)
```

## Gotchas

- **Airbnb requires `checkIn`** and uses **token** pagination — store `nextPageToken`, not page numbers.
- **Airbnb property endpoints take URLs**, not IDs.
- **Booking requires `children` even when zero.** Pass `children=0`. When `children > 0`, also pass `childrenAgesJson` with exactly that many ages.
- **Booking `price[min]` / `price[max]`** are bracketed — use a nested dict with `requests`/`axios`.
- **Booking `rooms[i].variants[]` is where prices live** — the parent `rooms[i]` describes the floor-plan, variants are the buyable rates with `priceBreakdown` / `cancellationPolicy` / `mealPlan`.
- **`bookingDetails` carries the resolved stay context** the response was priced for — echo it back when persisting results so future comparisons use the same dates / occupancy.
- **Google Flights uses IATA codes**, not city names. `JFK` not `New York`.
- **Round-trip vs one-way** is determined by `returnDate` presence — pass it for round-trip, omit for one-way.

# Web Scraping API — `POST /scrape/web`

One endpoint to fetch any URL, optionally with JS rendering, proxies, AI extraction, and screenshots. Synchronous.

> Reach for this only when the user gave you a specific URL, or when no Scraper API covers the field. Otherwise the platform-specific APIs return pre-extracted JSON without direct page access. Use only for public pages or content the user is authorized to access.

## Minimal request

```python
import requests

# Multiple outputs (or include "json") → response is a JSON object
resp = requests.post(
    "https://api.hasdata.com/scrape/web",
    headers={"x-api-key": API_KEY},
    json={"url": "https://example.com", "outputFormat": ["markdown", "json"]},
    timeout=300,
)
data = resp.json()
assert data["requestMetadata"]["status"] == "ok"
print(data["markdown"])

# Single non-JSON output → response IS the raw content (markdown/html/text bytes)
resp = requests.post(
    "https://api.hasdata.com/scrape/web",
    headers={"x-api-key": API_KEY},
    json={"url": "https://example.com", "outputFormat": ["markdown"]},
    timeout=300,
)
print(resp.text)            # raw markdown — no JSON parsing
```

## Body parameters

| Parameter | Type | Notes |
|---|---|---|
| `url` | string | **Required.** Absolute URL. |
| `outputFormat` | string[] | `html`, `text`, `markdown`, `json`. **Single non-JSON format → raw content as the body** (not JSON-wrapped); multiple formats → JSON object with one key per format. Always include `"json"` (or another format) when you also need `requestMetadata`. |
| `proxyType` | enum | `datacenter` (default) or `residential` — use residential only for authorized geo/availability testing where terms and access controls permit it. |
| `proxyCountry` | string | ISO 3166-1 alpha-2 — `US`, `UK`, `DE`, `FR`, `IT`, `SE`, `BR`, `CA`, `JP`, `SG`, `IN`, `ID`, `IE`. |
| `jsRendering` | bool | Headless browser — required for SPAs and dynamically-injected content. |
| `wait` / `waitFor` | int (ms) / CSS string | Fixed delay vs. wait-until-selector. Prefer `waitFor`. |
| `jsScenario` | array | Sequence of click/fill/wait/scroll/evaluate. Requires `jsRendering`. |
| `headers` | object | Custom headers. **Cookies go here too — no separate `cookies` parameter.** |
| `screenshot` | bool | Returns a CDN URL in the response. |
| `extractRules` | object | CSS selectors → field text. `@attr` for attributes. **First match only**, missing → `null`. |
| `aiExtractRules` | object | Typed LLM extraction. Types: `string`, `number`, `boolean`, `list`, `item`. |
| `extractEmails` / `extractLinks` | bool | Quick helpers. |
| `blockResources` / `blockAds` | bool | Skip images/CSS/ads — speeds text-only scrapes. |
| `blockUrls` | string[] | Glob patterns to block subresources. |
| `removeBase64Images` | bool | Strip inline base64 from response. |
| `includeOnlyTags` / `excludeTags` | string[] | Trim DOM before serialization. |

## CSS extraction (`extractRules`)

```python
"extractRules": {
    "title": "h1",
    "links": "a @href",   # @attr extracts attribute
    "price": ".price-now",
}
```

First match per selector. For lists of records, use `aiExtractRules` with `type: "list"`.

## AI extraction (`aiExtractRules`)

```python
"aiExtractRules": {
    "title":    {"type": "string"},
    "price":    {"type": "number"},
    "in_stock": {"type": "boolean"},
    "tags":     {"type": "list", "description": "category tags"},
    "author":   {"type": "item", "output": {
        "name":     {"type": "string"},
        "verified": {"type": "boolean"},
    }},
    "reviews":  {"type": "list", "output": {
        "rating": {"type": "number"},
        "text":   {"type": "string"},
    }},
}
```

Use when layout varies across pages; otherwise prefer `extractRules` for determinism and predictability.

## JS scenarios

```python
"jsScenario": [
    {"fill": ["#email", "user@example.com"]},
    {"fill": ["#password", PASSWORD]},
    {"click": "#login"},
    {"waitFor": ".dashboard"},
    {"scrollY": 2000},
    {"waitForAndClick": ".load-more"},
    {"evaluate": "window.__APP_STATE__"},
]
```

Actions: `click`, `fill: [sel, val]`, `wait: ms`, `waitFor: sel`, `waitForAndClick: sel`, `scrollX/scrollY: px`, `evaluate: "JS"`. Sequential. Missing element on `click`/`fill` fails the request — wrap with `waitFor` first.

## Auth via cookies

```python
"headers": {
    "User-Agent": "Mozilla/5.0 ...",
    "Cookie": "session=abc; csrf=xyz",
    "Accept-Language": "en-US,en;q=0.9",
}
```

Capture cookies once in a real browser (devtools → Storage → Cookies), forward via the `Cookie` header. Only with explicit user permission and authority to access that account/content; never use cookies to bypass someone else's access controls.

## Slim response & speed

```python
{
    "blockResources": True,                       # skip images/CSS/fonts
    "blockAds": True,                             # skip ad/tracking
    "blockUrls": ["**.googletagmanager.com/**", "**.doubleclick.net/**"],
    "removeBase64Images": True,
    "excludeTags": ["script", "style", "nav", "footer"],
}
```

Reduces response size 60–90% on noisy pages.

## Response shape

The wrapper is JSON **only when** the response is JSON-wrapped — i.e. multiple `outputFormat` values, or a single value that includes `"json"`. With a single non-JSON format the response body is the raw content (`text/markdown`, `text/html`, `text/plain`).

```json
{
  "requestMetadata": { "id": "uuid", "status": "ok", "url": "..." },
  "headers": { "content-type": "text/html" },
  "screenshot": "https://...jpeg",
  "content": "<!DOCTYPE html>...",     // outputFormat: html
  "markdown": "# Title\n...",           // outputFormat: markdown
  "text":     "Title\n...",
  "extractRules":    { ... },           // present iff sent
  "aiExtractRules":  { ... },           // present iff sent
  "extractedEmails": [ ... ],           // iff extractEmails: true
  "extractedLinks":  [ ... ]            // iff extractLinks: true
}
```

## Batch (`POST /scrape/batch/web`)

Async wrapper for >1k URLs running the same extraction. Returns `jobId`; poll status, page `/results`. Per-batch cap **10,000 URLs**. For small workloads loop the sync endpoint at concurrency = plan limit.

## Gotchas

- **Disable `jsRendering` first**, enable only when the page needs it — most static pages parse fine without a headless browser.
- **`waitFor` > `wait`.** Selector-based waits adapt to network speed.
- **Cookies via `headers["Cookie"]` only.**
- **`extractRules` returns first match** — for arrays use `aiExtractRules` `type: "list"`.
- **Set client timeout ≥ 300 s** to match the server deadline.
- **`requestMetadata.status === "ok"` is the only success signal.**

# YouTube APIs

| Endpoint | Returns |
|---|---|
| `/scrape/youtube/search` | Search results — videos, shorts, channels, playlists |
| `/scrape/youtube/video` | Single video metadata (stats, captions, related) |
| `/scrape/youtube/channel` | Channel home / videos / shorts / playlists / community |
| `/scrape/youtube/transcript` | Full transcript with millisecond offsets |

All synchronous `GET`. 10 credits each.

## YouTube Search

```python
import requests

resp = requests.get(
    "https://api.hasdata.com/scrape/youtube/search",
    headers={"x-api-key": API_KEY},
    params={"q": "anthropic claude", "sortBy": "views", "date": "month"},
    timeout=300,
)
for v in resp.json().get("videoResults", []):
    print(v["title"], v.get("extractedViews"), v["link"])
```

### Query parameters

| Param | Notes |
|---|---|
| `q` | **Required.** Free-text query. |
| `sortBy` | `relevance` (default), `date`, `views`, `rating`, `popularity`. |
| `date` | Upload window: `hour`, `today`, `week`, `month`, `year`. |
| `length` | Duration bucket: `under4`, `between420`, `plus20`. |
| `videoType` | `video`, `shorts`, `channel`, `playlist`, `movie`. |
| `filters[]` | Feature flags ANDed: `hd`, `k4`, `hdr`, `subtitles`, `cc`, `d3`, `d360`, `vr180`, `live`, `bought`, `location`. |
| `gl` / `hl` | Two-letter country / language codes. |
| `deviceType` | `desktop`, `mobile`. |
| `paginationToken` | Opaque cursor from the previous `pagination.nextPageToken`. |
| `sp` | Raw YouTube `sp=` token (overrides `sortBy`, `date`, `videoType`, `length`, `filters[]`). |

Response: `videoResults`, `shortsResults`, `channelResults`, `playlistResults`, `adsResults`, `sponsoredResults`, `searchInformation`, `pagination`.

Per-video result keys (verified live): `videoId`, `title`, `link`, `channel`, `description`, `length`, `views`, `viewsOriginal`, `publishedDate`, `thumbnail`, `positionOnPage`. `channel` is an object — read `.channel.name` and `.channel.link`.

## YouTube Video

```python
resp = requests.get(
    "https://api.hasdata.com/scrape/youtube/video",
    headers={"x-api-key": API_KEY},
    params={"v": "dQw4w9WgXcQ"},
    timeout=300,
)
```

| Param | Notes |
|---|---|
| `v` | **Required.** 11-character YouTube video ID — the `v=` query value. |
| `gl` / `hl` | Country / language. |
| `deviceType` | `desktop` / `mobile`. |

Top-level keys: `videoId`, `title`, `description`, `channel`, `views`, `extractedViews`, `likes`, `extractedLikes`, `lengthSeconds`, `publishedDate`, `keywords`, `captions`, `socialLinks`, `music`, `category`, `thumbnail`, `isFamilySafe`, `isUnlisted`, `relatedVideos`, `relatedShorts`, `endScreenVideos`, `requestMetadata`.

Use `extractedViews` / `extractedLikes` (integers) for math; `views` / `likes` are the formatted strings.

## YouTube Channel

```python
resp = requests.get(
    "https://api.hasdata.com/scrape/youtube/channel",
    headers={"x-api-key": API_KEY},
    params={"channelId": "@MrBeast", "tab": "videos"},
    timeout=300,
)
```

| Param | Notes |
|---|---|
| `channelId` | **Required.** `@handle`, canonical `UC…` ID, or legacy `/c/<custom>` / `/user/<name>` slug. |
| `tab` | `featured` (default), `videos`, `shorts`, `streams`, `playlists`, `posts` / `community`, `podcasts`, `releases`, `about`, `store`. |
| `paginationToken` | Cursor for tabs that paginate. |
| `gl` / `hl` / `deviceType` | Standard. |

Response: `channelInfo`, `featuredVideo`, `sections[]`.

`channelInfo` (verified live): `name`, `handle`, `channelId`, `channelUrl`, `avatar`, `banner`, `description`, `subscribers`, `extractedSubscribers`, `videosCount`, `extractedVideosCount`, `keywords[]`, `availableTabs[]`, `verified`, `websiteUrl`, `rssUrl`, `isFamilySafe`.

`sections[]` each have `title` + `items[]` — shape depends on the tab. Iterate generically:

```python
for sec in resp.json().get("sections", []):
    for item in sec.get("items", []):
        ...
```

## YouTube Transcript

```python
resp = requests.get(
    "https://api.hasdata.com/scrape/youtube/transcript",
    headers={"x-api-key": API_KEY},
    params={"v": "dQw4w9WgXcQ", "languageCode": "en"},
    timeout=300,
)
text = " ".join(seg["snippet"] for seg in resp.json().get("transcript", []))
```

| Param | Notes |
|---|---|
| `v` | **Required.** 11-character video ID. |
| `languageCode` | BCP-47 / YouTube code (`en`, `de`, `en-US`, `pt-BR`). Must exist on the video. |
| `type` | `asr` to fetch the auto-generated speech-recognition track when no human captions exist. |

Response: `transcript[]` and `availableTranscripts[]`.

Each `transcript[]` entry: `startMs`, `endMs`, `snippet`, `startTimeText` (e.g. `"0:18"`).

## Patterns

### Search → video → transcript fan-out

```python
def topic_corpus(query, k=5):
    search = requests.get(
        "https://api.hasdata.com/scrape/youtube/search",
        headers={"x-api-key": API_KEY},
        params={"q": query, "sortBy": "views"}, timeout=300,
    ).json()
    docs = []
    for v in search.get("videoResults", [])[:k]:
        tr = requests.get(
            "https://api.hasdata.com/scrape/youtube/transcript",
            headers={"x-api-key": API_KEY},
            params={"v": v["videoId"]}, timeout=300,
        ).json()
        docs.append({
            "videoId": v["videoId"],
            "title":   v["title"],
            "url":     v["link"],
            "text":    " ".join(s["snippet"] for s in tr.get("transcript", [])),
        })
    return docs
```

### Channel velocity

```python
def channel_velocity(handle):
    page = requests.get(
        "https://api.hasdata.com/scrape/youtube/channel",
        headers={"x-api-key": API_KEY},
        params={"channelId": handle, "tab": "videos"}, timeout=300,
    ).json()
    return [
        {"date": it.get("publishedDate"),
         "views": it.get("extractedViews"),
         "title": it.get("title")}
        for sec in page.get("sections", [])
        for it in sec.get("items", [])
    ]
```

### Timestamp search inside a transcript

```python
def mentions(video_id, needle):
    tr = requests.get(
        "https://api.hasdata.com/scrape/youtube/transcript",
        headers={"x-api-key": API_KEY},
        params={"v": video_id}, timeout=300,
    ).json()
    return [(s["startTimeText"], s["snippet"])
            for s in tr.get("transcript", [])
            if needle.lower() in s["snippet"].lower()]
```

## Gotchas

- **`v` is the 11-char ID, not the URL.** Strip the `v=` value first.
- **`languageCode` must exist on the video.** Inspect `availableTranscripts[]` if a fetch fails, then retry.
- **`type=asr` is required when no human-authored caption track exists.** Otherwise the API errors on videos with auto-only captions.
- **`@handle` resolves to the same channel as the canonical `UC…` ID.** Prefer handles for readability.
- **Pagination tokens are opaque** — pass them back verbatim via `paginationToken`.
- **`extractedViews` / `extractedLikes`** are integers; `views` / `likes` are formatted strings. Use the integer fields for arithmetic.
- **`channelInfo.rssUrl`** is the canonical RSS feed for the channel — use it to subscribe in podcast clients without scraping.