# Error catalogue

## How errors reach you

```text
                     ┌─ HTTP status + plain-text body      (handshake refused, no session was created)
connect() ───────────┤
                     └─ 101 Switching Protocols ──┬─ WebSocket close 3000 + CDP_* reason  (session died)
                                                  └─ CDP error {code: 1337, message}     (one command failed)
page traffic ─────────── HTTP responses; X-Error-Description header present = Oxylabs network error
```

1. **Handshake** (`connectOverCDP` / `puppeteer.connect` throws). Body is a short text: either a `CDP_*` code
   or a human sentence. Playwright's message is `WebSocket error: <full URL incl. password> <status> <status
   text>` followed by the body on the next line; Puppeteer's is `Unexpected server response: <status>`; raw
   WebSocket clients get the HTTP response. **Redact the password before logging.** No session exists,
   nothing to close.
2. **Post-connect close** (`browser.on("disconnected")`, a pending command rejects with `Target closed`,
   `Browser has been closed` or similar). The close frame carries code `3000` and a `CDP_*` reason, but only
   raw WebSocket clients can read it; Playwright and Puppeteer do not expose the close reason. Anything
   unexpected is reported as `CDP_GENERAL_ERROR`. Reconnect with a fresh session and backoff; if you pinned
   `proxy_resi_ses_id` and the disconnect happened before the first navigation, rotate the id too.
3. **In-page CDP error** with code `1337`. The session is fine; the specific command was refused.
4. **Target traffic.** Ordinary HTTP responses inside the page. If a response carries `X-Error-Description`,
   the Oxylabs network produced the error and the header explains it. Without the header, the website itself
   answered (a block page, a 403): change approach instead of retrying.

Retry policy: retry `429`, `5xx`, `CDP_SESSION_IN_USE`, `CDP_NO_BROWSERS_AVAILABLE`, `CDP_BROWSER_OVERWORKED_*`,
`CDP_BAD_PROXY`, `CDP_GENERAL_ERROR` and unexpected disconnects with exponential backoff (1 s base, 60 s cap,
jitter). Retry `409` only after the conflicting session has had time to close (30 s or more). Never retry
`400`, `401`, `403` or `1337` without changing the request.

## Handshake errors

### 401 Unauthorized

| Body | Cause | Fix |
|------|-------|-----|
| `missing auth header` | Client sent no `Authorization` header because it ignores `user:pass@` in the URL (Node's built-in `WebSocket`, some proxies) | Build `Authorization: Basic base64("user:pass")` yourself |
| `invalid auth header` | Header is not `Basic …` (e.g. `Bearer`) | Use Basic auth |
| `not a valid base64 string` | Header value is not URL-safe base64 with padding (credentials containing bytes that encode to `+` or `/`) | Encode with the URL-safe alphabet (`-`, `_`) and keep `=` padding, or change the password |
| `missing credentials` | Decoded value is not exactly `user:pass` (password contains `:`) | Change the password, or send the header manually and ensure exactly one `:` |
| `invalid credentials` | Unknown/inactive user, wrong password, incomplete username, password percent-encoded by `new URL()` | Re-read env vars; use the full username from the dashboard; concatenate the raw password |

### 400 Bad Request

| Body | Fix |
|------|-----|
| `invalid_uri, failed to decode query` | Malformed query string; check `&`/`=` and encoding of values |
| `invalid_uri, invalid query: p_device must be one of [desktop mobile]` | Use `desktop` or `mobile` |
| `session_name must be 3-36 alphanumeric or '-' characters` | Only `[A-Za-z0-9-]`, 3 to 36 chars |
| `keep_alive requires session_name` | Add `session_name` or drop `keep_alive` |
| `unsupported proxy type "x", supported: resi, dc, ddc` | Use `resi` or `dc` |
| `proxy type "x" is not enabled for this user` | Drop `proxy` (defaults to `resi`) or ask support |
| `proxy=dc does not support [p_cc …] params` | Remove geo/sticky parameters or switch to `resi` |
| `ddc proxy is not supported` | Use `resi` or `dc` |
| `invalid p_cc, supported ISO-3166 2 letter codes` | Two-letter country code |
| `proxy_resi_ses_id must be 3–36 alphanumeric characters or underscores` | Start with a letter/digit; `[A-Za-z0-9_]`, 3 to 36 chars |
| `proxy_resi_ses_time must be between 1 and 1440` | Integer minutes in range |
| `o_profile must contain only letters, numbers, hyphens, and underscores (min 3, max 36 characters)` | Fix the profile name |
| `o_profile_save requires o_profile to be set` | Add `o_profile` |
| `o_profile_save should be boolean` | `true`/`false` |
| `record should be boolean` | `true`/`false` |
| `record_name requires record=true` | Add `record=true` |
| `record_name must contain only letters, numbers, hyphens, and underscores (max 64 characters)` | Fix the name |

### 403 Forbidden

| Body | Fix |
|------|-----|
| `browser profile feature is not enabled for your account` | Remove `o_profile`; contact support to enable |
| `profile limit reached (N profiles maximum)` | Reuse a stable profile name, delete unused profiles in the dashboard, or request a higher cap |
| `recordings are not enabled for this account` | Remove `record` |
| `recording limit reached (N recordings maximum)` | Delete old recordings in the dashboard |

### 409 Conflict

| Body | Fix |
|------|-----|
| `profile is already in use by another session` | A run with `o_profile_save=true` (the setup run) still holds this profile. Wait for it to close (30 s+), then retry. Consumers must send `o_profile` without `o_profile_save`; if a consumer hit this, it is saving when it should not |

### 429 Too Many Requests

| Body | Cause | Fix |
|------|-------|-----|
| `CDP_SESSION_RATE_LIMIT_REACHED` | More than 10 new sessions within one second (account default) | Serialise launches, 150 ms apart or slower; backoff |
| `CDP_MAX_CONCURRENT_SESSIONS_REACHED` | Concurrency cap (account default 100). Sessions you did not close still count for about 20 s, named ones for 10 min | Close idle sessions; fix missing `finally { browser.close() }`; wait and retry |
| `CDP_MAX_PERSISTENT_SESSIONS_REACHED` | Cap of named sessions (default 5) | Reuse a name, or let old sessions expire |
| `CDP_SESSION_IN_USE` | Reconnecting to a named session that still has a live connection | Close the earlier connection, retry |
| `max queue sessions reached` | Too many of your connections waiting for capacity | Lower parallelism, backoff |

### 5xx

| Status / body | Fix |
|---------------|-----|
| `500 CDP_GENERAL_ERROR` | Setup failed internally. Retry with backoff; if it persists more than a few minutes, contact support with the time and parameters |
| `502`, `504` (no `CDP_*` text) | Edge/network issue. Retry with backoff |
| `503 queue timeout` | Waited about a minute for capacity. Back off 30 to 60 s, reduce parallelism, retry |

## Post-connect close (code 3000)

| Reason | Meaning | Fix |
|--------|---------|-----|
| `CDP_NO_BROWSERS_AVAILABLE` | No browser could be allocated | Retry with backoff |
| `CDP_BROWSER_OVERWORKED_1` | Allocated host is saturated | Retry with backoff |
| `CDP_BAD_PROXY` | Proxy could not be established for the requested geo / sticky id | Retry; rotate `proxy_resi_ses_id`; drop `p_city` |
| `CDP_SESSION_NOT_FOUND` | Named session expired (10 min) before reconnect | Start a new session |
| `CDP_SESSION_IN_USE` | Another connection grabbed the named session first | Close it, retry |
| `CDP_GENERAL_ERROR` | Browser died or an internal step failed | New session with backoff |

## In-page CDP errors (code 1337)

| Message | Meaning | Fix |
|---------|---------|-----|
| `Invalid target` | The navigated host is not allowed for your account (restricted category) | Do not retry. Verify the URL, then ask your account manager to unlock the category |
| `Too many targets` | Over 1000 pages in one session | Close pages after use |
| `Proxy not allowed in this context` | `Target.createBrowserContext` with `proxyServer` | Use URL parameters for proxy selection |

## Viewer (VNC) errors

| Reason | Fix |
|--------|-----|
| `CDP_VNC_MAX_CONCURRENT_SESSIONS_REACHED` | Close other viewer tabs (default cap 10) |
| `SESSION_NOT_FOUND` / `INCORRECT_SESSION_ID` | Session ended or id mistyped; fetch it again with `__session_id` |

## Decision flow for an agent

```text
connect failed?
  ├─ 401 ............ fix credentials/scheme, do not retry
  ├─ 400/403/409 .... fix the named parameter, do not retry unchanged
  ├─ 429 ............ backoff; if MAX_CONCURRENT: hunt for unclosed sessions
  └─ 5xx/503 ........ backoff, up to ~2 min total
session dropped (close 3000)?
  └─ new session with backoff; rotate sticky id on CDP_BAD_PROXY
navigate failed with 1337 Invalid target?
  └─ stop; restricted target
page shows block / 403 wall?
  ├─ X-Error-Description present .... Oxylabs network issue: backoff + retry
  └─ absent ......................... target decision: change identity, geo, device, pacing (targets.md)
```

# Examples

All examples read `OXY_UNBLOCKER_USERNAME` / `OXY_UNBLOCKER_PASSWORD` (aliases `OXY_HB_USERNAME` /
`OXY_HB_PASSWORD`) and connect to `wss://USERNAME:PASSWORD@hb.oxylabs.io`. The `endpointUrl`, `connectWithBackoff`,
`blockHeavyResources`, `watchOxylabsErrors` and `scrape` helpers are the ones from
`scripts/playwright_scrape.js`.

## Puppeteer

```javascript
const puppeteer = require("puppeteer");

const USERNAME = process.env.OXY_UNBLOCKER_USERNAME || process.env.OXY_HB_USERNAME;
const PASSWORD = process.env.OXY_UNBLOCKER_PASSWORD || process.env.OXY_HB_PASSWORD;
// Same policy as scripts/playwright_scrape.js: Puppeteer reports "Unexpected server response: <status>" without the body.
const RETRYABLE =
  /\b(429|50[0234]) [A-Z]|response: (429|50[0234])\b|CDP_SESSION_IN_USE|CDP_NO_BROWSERS_AVAILABLE|CDP_BROWSER_OVERWORKED|CDP_BAD_PROXY|CDP_GENERAL_ERROR|[Tt]imeout|ECONNRESET/;

async function connect(params, attempts = 6) {
  const q = new URLSearchParams(params).toString();
  const browserWSEndpoint = `wss://${USERNAME}:${PASSWORD}@hb.oxylabs.io${q ? "?" + q : ""}`;
  for (let i = 0; ; i++) {
    try {
      return await puppeteer.connect({ browserWSEndpoint, protocolTimeout: 60000 });
    } catch (err) {
      if (i >= attempts - 1 || !RETRYABLE.test(err.message)) throw err;
      await new Promise((r) => setTimeout(r, Math.min(60000, 1000 * 2 ** i) * (0.5 + Math.random())));
    }
  }
}

(async () => {
  const browser = await connect({ p_cc: "US" });
  try {
    const page = await browser.newPage();
    await page.setRequestInterception(true);
    page.on("request", (req) =>
      ["image", "stylesheet", "media", "font"].includes(req.resourceType()) ? req.abort() : req.continue());
    page.on("response", (resp) => {
      const d = resp.headers()["x-error-description"];
      if (d) console.warn("Oxylabs-side error", resp.status(), resp.url(), d);
    });

    await page.goto("https://example.com", { waitUntil: "domcontentloaded", timeout: 30000 });
    console.log(await page.title());
  } finally {
    await browser.close();
  }
})();
```

## Playwright (Python, async)

```python
import asyncio, os, random, re
from playwright.async_api import async_playwright

USERNAME = os.environ.get("OXY_UNBLOCKER_USERNAME") or os.environ["OXY_HB_USERNAME"]
PASSWORD = os.environ.get("OXY_UNBLOCKER_PASSWORD") or os.environ["OXY_HB_PASSWORD"]
RETRYABLE = re.compile(  # same policy as scripts/playwright_scrape.py
    r"\b(429|50[0234]) [A-Z]|response: (429|50[0234])\b|CDP_SESSION_IN_USE|CDP_NO_BROWSERS_AVAILABLE|"
    r"CDP_BROWSER_OVERWORKED|CDP_BAD_PROXY|CDP_GENERAL_ERROR|[Tt]imeout|ECONNRESET")


async def connect(pw, params, attempts=6):
    q = "&".join(f"{k}={v}" for k, v in params.items())
    url = f"wss://{USERNAME}:{PASSWORD}@hb.oxylabs.io" + (f"?{q}" if q else "")
    for i in range(attempts):
        try:
            return await pw.chromium.connect_over_cdp(url, timeout=60_000)
        except Exception as err:  # noqa: BLE001
            if i == attempts - 1 or not RETRYABLE.search(str(err)):
                raise
            await asyncio.sleep(min(60, 2 ** i) * (0.5 + random.random()))


async def main():
    async with async_playwright() as pw:
        browser = await connect(pw, {"p_cc": "US"})
        try:
            context = browser.contexts[0] if browser.contexts else await browser.new_context()
            page = await context.new_page()
            await page.route("**/*", lambda r: r.abort()
                             if r.request.resource_type in {"image", "stylesheet", "media", "font"}
                             else r.continue_())
            page.on("response", lambda r: r.headers.get("x-error-description")
                    and print("Oxylabs-side error", r.status, r.url, r.headers["x-error-description"]))

            await page.goto("https://example.com", wait_until="domcontentloaded", timeout=30_000)
            await page.screenshot(path="page.png", full_page=True)
            print(await page.title())
        finally:
            await browser.close()

asyncio.run(main())
```

## Raw CDP over WebSocket (no library)

Node's built-in `WebSocket` (and many minimal clients) silently drop `user:pass@` from the URL, which the
service reports as `401 missing auth header`. Build the header yourself. The service decodes with the
URL-safe base64 alphabet and requires exactly one `:` in the decoded value.

```javascript
const USERNAME = process.env.OXY_UNBLOCKER_USERNAME || process.env.OXY_HB_USERNAME;
const PASSWORD = process.env.OXY_UNBLOCKER_PASSWORD || process.env.OXY_HB_PASSWORD;

if (PASSWORD.includes(":")) throw new Error("password may not contain ':' for Basic auth");
const token = Buffer.from(`${USERNAME}:${PASSWORD}`).toString("base64");
if (/[+/]/.test(token)) console.warn("base64 contains + or /; use base64url encoding: " + Buffer.from(`${USERNAME}:${PASSWORD}`).toString("base64url"));

const ws = new WebSocket("wss://hb.oxylabs.io/?p_cc=US", { headers: { Authorization: `Basic ${token}` } });
let nextId = 1;
const pending = new Map();
const send = (method, params = {}, sessionId) =>
  new Promise((resolve, reject) => {
    const id = nextId++;
    pending.set(id, { resolve, reject });
    ws.send(JSON.stringify({ id, method, params, ...(sessionId && { sessionId }) }));
  });

ws.addEventListener("message", (ev) => {
  const msg = JSON.parse(ev.data);
  if (msg.id && pending.has(msg.id)) {
    const { resolve, reject } = pending.get(msg.id);
    pending.delete(msg.id);
    msg.error ? reject(new Error(`${msg.error.code} ${msg.error.message}`)) : resolve(msg.result);
  }
});
ws.addEventListener("close", (ev) => console.log("closed", ev.code, ev.reason)); // 3000 + CDP_* on failure
ws.addEventListener("open", async () => {
  try {
    const { value: sessionId } = await send("__session_id");
    console.log("session_id:", sessionId, "inspect: https://hb.oxylabs.io/novnc/?id=" + sessionId);
    const { targetId } = await send("Target.createTarget", { url: "about:blank" });
    const { sessionId: cdpSession } = await send("Target.attachToTarget", { targetId, flatten: true });
    await send("Page.enable", {}, cdpSession);
    await send("Page.navigate", { url: "https://example.com" }, cdpSession);
  } finally {
    setTimeout(() => ws.close(1000), 5000); // always close: the session holds a concurrency slot
  }
});
```

## Resume a named session

`session_name` alone keeps the remote browser alive for 10 minutes after you disconnect. What survives
depends on **how** you disconnect:

| Disconnect method | Session resumable | Open pages and their cookies kept |
|-------------------|-------------------|-----------------------------------|
| Puppeteer `browser.disconnect()` | yes | yes |
| Process exit / dropped connection | yes | yes |
| Playwright or Puppeteer `browser.close()` | yes | **no** (pages closed, cookies cleared) |

Puppeteer hand-over (process A):

```javascript
const puppeteer = require("puppeteer-core");
const browser = await puppeteer.connect({ browserWSEndpoint: endpointUrl({ session_name: "job-42", p_cc: "US" }) });
const page = await browser.newPage();
await page.goto("https://example.com/login");
// ... log in ...
await browser.disconnect(); // NOT close(): keeps the page and cookies for the next connection
```

Resume (process B, within 10 minutes, same credentials; Playwright or Puppeteer):

```javascript
const browser = await connectWithBackoff({ session_name: "job-42", p_cc: "US" }); // 429 CDP_SESSION_IN_USE while A is still attached
try {
  const ctx = browser.contexts()[0];
  const page = ctx.pages().find((p) => p.url().includes("example.com")) ?? (await ctx.newPage());
  await page.goto("https://example.com/account"); // still logged in
} finally {
  await browser.close(); // last user of the session: close for real
}
```

If every process uses Playwright, keep the first connection open until the process exits instead of calling
`close()`. For state that must outlive the 10-minute window use a profile (next example). To end a named
session early, connect once more with `keep_alive=false` and close.

## Profile setup and consumer runs

Most jobs need no profile: every session is a fresh fingerprint on a fresh IP. Use one when cookies must survive
between jobs (DataDome clearance, a login). Then split the work in two roles. **Setup** runs once per profile and is
the only connection that sends `o_profile_save=true`. **Consumers** send `o_profile` alone: read-only, no write lock,
so they can never hit `409` and can start while others run. In production these are two services: one prepares and
validates profiles, the other only uses them.

```javascript
const identity = {
  o_profile: "acme-us-01", p_cc: "US", p_device: "desktop",
  proxy_resi_ses_id: "acmeus01", proxy_resi_ses_time: "30",
};

// Setup service: run once per profile name. Earn the cookies, verify, close (closing writes the profile).
async function setupProfile(entryUrl) {
  const browser = await connectWithBackoff({ ...identity, o_profile_save: "true" });
  try {
    const page = await browser.contexts()[0].newPage();
    const response = await page.goto(entryUrl, { waitUntil: "domcontentloaded" });
    await page.waitForTimeout(5000); // let late JS and cookies settle
    if ((await classify(page, response)) !== "ok") throw new Error("setup failed: block page, pick a new profile name"); // classify() from targets.md
    // ... log in here if the target needs it ...
  } finally {
    await browser.close();
  }
}

// Consumer service: any number of runs, never o_profile_save.
await setupProfile("https://target.example/");
await scrape("https://target.example/listing/1", identity);
await scrape("https://target.example/listing/2", identity);
```

Use `browser.contexts()[0]`; a `newContext()` is not backed by the profile. Keep every geo/device value identical
across setup and consumers. When consumers start seeing block pages, do not re-save the profile from a consumer: run
`setupProfile` again under a new profile name and a new sticky id.

## Session id, live inspection and recording

```javascript
const browser = await connectWithBackoff({ p_cc: "US", record: "true", record_name: "job-42" });
try {
  const context = browser.contexts()[0];
  const page = await context.newPage();
  const cdp = await context.newCDPSession(page);
  const { value: sessionId } = await cdp.send("__session_id");
  console.log(`session_id: ${sessionId}`);
  console.log(`live view:  https://hb.oxylabs.io/novnc/?id=${sessionId}`);
  await page.goto("https://example.com");
} finally {
  await browser.close(); // the recording is finalised and appears in https://hb.oxylabs.io/dashboard
}
```

## Fan-out with launch pacing

```javascript
async function mapWithPacing(urls, params, concurrency = 8, launchGapMs = 150) {
  const results = new Array(urls.length);
  let next = 0;
  async function worker() {
    while (next < urls.length) {
      const i = next++;
      await new Promise((r) => setTimeout(r, launchGapMs)); // stay well under 10 launches/second
      try {
        results[i] = await scrape(urls[i], params);
      } catch (err) {
        results[i] = { error: err.message };
      }
    }
  }
  await Promise.all(Array.from({ length: concurrency }, worker));
  return results;
}
```

Each `scrape` call opens and closes its own session, so `concurrency` is the number of concurrent sessions
(default account cap 100).

# Connection parameters

All options are query parameters appended to the WebSocket URL:

```text
wss://USERNAME:PASSWORD@hb.oxylabs.io?p_cc=US&p_device=desktop&session_name=job-42
```

Validation happens after authentication. A failed check rejects the handshake with `400 Bad Request` and the
plain-text message listed below; nothing is retried automatically. Unknown parameters are ignored, but every
distinct parameter combination is provisioned separately, so keep the set stable across a job.

Most jobs need only `p_cc`. Sticky sessions, profiles and named sessions are opt-in for a specific need (see
`targets.md` section 2); do not add them by default.

## Session

| Parameter | Format | Default | Meaning and rules |
|-----------|--------|---------|-------------------|
| `session_name` | `^[A-Za-z0-9-]{3,36}$` | none | Names the session so it can be resumed by reconnecting with the same credentials and name. Resumable for 10 minutes after disconnect. Default cap 5 named sessions. `400 session_name must be 3-36 alphanumeric or '-' characters` |
| `keep_alive` | `true` / `false` | `true` when `session_name` is set, otherwise `false` | Keeps the browser alive after you disconnect. **Requires `session_name`**: `400 keep_alive requires session_name`. Set `false` with a name to make the named session non-resumable |

## Geo and proxy

| Parameter | Format | Default | Meaning and rules |
|-----------|--------|---------|-------------------|
| `proxy` | `resi` / `dc` | `resi` | Proxy type: residential (default) or datacenter. `dc` does not accept geo or sticky-session parameters: `400 proxy=dc does not support [p_cc …] params`. Unknown value: `400 unsupported proxy type "x", supported: resi, dc, ddc`. Not on your plan: `400 proxy type "dc" is not enabled for this user` |
| `p_cc` | ISO-3166 alpha-2, case-insensitive (`US`, `de`) | automatic | Exit country. `400 invalid p_cc, supported ISO-3166 2 letter codes` |
| `p_state` | lowercase US state, `texas` or `us_texas` | none | US state; overrides `p_cc`. The `us_` prefix is added for you if missing |
| `p_city` | lowercase, underscores for spaces (`los_angeles`) | none | City preference, best effort. Only applied when `p_cc` or `p_state` is also set; falls back to the wider area if no match |
| `proxy_resi_ses_id` | `^[a-zA-Z0-9][a-zA-Z0-9_]{2,35}$` | random per session | Opt-in. Sticky residential session: identical values on every connection keep the same exit IP. Disables automatic proxy retry (on `CDP_BAD_PROXY`, rotate the id). Residential only. `400 proxy_resi_ses_id must be 3–36 alphanumeric characters or underscores` |
| `proxy_resi_ses_time` | integer 1 to 1440 (minutes) | none | How long the sticky IP is held, counted from the first connection that used the id. Residential only. `400 proxy_resi_ses_time must be between 1 and 1440` |

## Fingerprint

| Parameter | Format | Default | Meaning and rules |
|-----------|--------|---------|-------------------|
| `p_device` | `desktop` / `mobile` | `desktop` | Device profile: viewport, touch APIs, platform headers, User-Agent and a matching residential pool. `tablet` is **not** accepted. Invalid value: `400 invalid_uri, invalid query: p_device must be one of [desktop mobile]`. Do not override viewport or device metrics yourself; the service ignores those CDP calls to keep the fingerprint coherent |

## Profiles

| Parameter | Format | Default | Meaning and rules |
|-----------|--------|---------|-------------------|
| `o_profile` | `^[a-zA-Z0-9][a-zA-Z0-9_-]{2,35}$` | none | Opt-in, for cookies that must survive between jobs. Restores cookies and localStorage saved under this name at session start. Missing profile is created empty (no error). Retained 14 days from last use. `400 o_profile must contain only letters, numbers, hyphens, and underscores (min 3, max 36 characters)`. Feature disabled: `403 browser profile feature is not enabled for your account` |
| `o_profile_save` | `true` / `false` (`1`/`0`, `on`/`off`) | `false` | Persists the profile when the session ends (also after errors, as long as the browser was closed). **Send it from exactly one setup run per profile name**; consumer runs use `o_profile` alone and never save. Requires `o_profile`: `400 o_profile_save requires o_profile to be set`. Non-boolean: `400 o_profile_save should be boolean`. Only saving sessions count toward the profile cap: `403 profile limit reached (N profiles maximum)`. The profile is write-locked while a saving session runs: `409 profile is already in use by another session` |

Use the browser's default context (`browser.contexts()[0]`) so cookies and storage are actually backed by the
profile; a fresh `newContext()` is isolated from it.

## Recording

| Parameter | Format | Default | Meaning and rules |
|-----------|--------|---------|-------------------|
| `record` | `true` / `false` | `false` | Records the session to video, viewable in the dashboard. Non-boolean: `400 record should be boolean`. Not on your plan: `403 recordings are not enabled for this account`. Cap: `403 recording limit reached (N recordings maximum)` (default 10) |
| `record_name` | `^[a-zA-Z0-9_-]{1,64}$` | none | Label for the recording. Requires `record=true`: `400 record_name requires record=true`. `400 record_name must contain only letters, numbers, hyphens, and underscores (max 64 characters)` |

## Other

| Parameter | Format | Default | Meaning and rules |
|-----------|--------|---------|-------------------|
| `bargs` | see docs | none | Chrome browser arguments (`disable-notifications`, `window-position:X,Y`, `hide-scrollbars`, `force-color-profile:<p>`, `enable-features:<f>`); repeat the key for several values. Rarely needed; do not use it to change the fingerprint |

## Combination rules

- `p_city` needs `p_cc` or `p_state`; `p_state` beats `p_cc`.
- `proxy=dc` rejects `p_cc`, `p_state`, `p_city`, `proxy_resi_ses_id`, `proxy_resi_ses_time`.
- `keep_alive=true` needs `session_name`; `o_profile_save=true` needs `o_profile`; `record_name` needs `record=true`.
- For one identity keep `p_cc`, `p_state`, `p_city`, `p_device`, `o_profile`, `proxy_resi_ses_id` and
  `proxy_resi_ses_time` identical on every connection.

---
name: headless-browser
description: Connects to Oxylabs remote headless browsers over the Chrome DevTools Protocol (CDP) with Playwright or Puppeteer. Built-in anti-detection, residential proxies, geo-targeting, persistent sessions and profiles, session recording and live VNC inspection for debugging. Use instead of WebFetch or a local browser whenever a site renders with JavaScript, blocks bots (DataDome, Cloudflare, Akamai), needs a real browser session, screenshots or PDFs. Covers connection, retries, error recovery and safe scraping of protected targets without any human help.
---

# Oxylabs Headless Browser

Remote Chrome sessions with anti-detection, proxy rotation and geo-targeting built in.
Nothing runs locally: you connect over a WebSocket, drive the browser with the CDP library you already
use, and close the session when done. This file holds the rules; the detail lives next to it:
`scripts/` (copyable templates), `parameters.md`, `errors.md`, `examples.md`, `targets.md`.

## 1. Connect

| Item | Value |
|------|-------|
| Endpoint | `wss://USERNAME:PASSWORD@hb.oxylabs.io` |
| Credentials | `OXY_UNBLOCKER_USERNAME` / `OXY_UNBLOCKER_PASSWORD` (aliases: `OXY_HB_USERNAME` / `OXY_HB_PASSWORD`) |
| Options | URL query parameters only, e.g. `?p_cc=US&session_name=job-42` (see `parameters.md`) |
| Libraries | Playwright `chromium.connectOverCDP` (recommended), Puppeteer `puppeteer.connect`, any CDP client |
| Dashboard / support | `https://hb.oxylabs.io/dashboard` · `support@oxylabs.io` |

Rules that prevent the most common `401`:

- Use `wss://`. Plain `ws://` is accepted but sends your password unencrypted.
- Build the URL by string concatenation with the **raw** password. Do not pass the finished URL through
  `new URL()` or `urllib.parse`: they percent-encode the password and authentication fails.
- Use the full username exactly as shown in the dashboard, including any suffix such as `_ab12`.
- A password containing `:` cannot be sent in the URL. Ask for a new password or send the
  `Authorization: Basic` header yourself (see `examples.md`).
- Authentication is checked before parameters: fix a `401` before looking at anything else.

## 2. Quick start

Minimal shape (Playwright, JavaScript):

```javascript
const { chromium } = require("playwright");
const url = `wss://${process.env.OXY_UNBLOCKER_USERNAME}:${process.env.OXY_UNBLOCKER_PASSWORD}@hb.oxylabs.io?p_cc=US`;
const browser = await chromium.connectOverCDP(url, { timeout: 60000 });
try {
  const page = await browser.contexts()[0].newPage(); // default context: backed by fingerprint, proxy, o_profile
  await page.goto("https://example.com", { waitUntil: "domcontentloaded", timeout: 30000 });
  console.log(await page.content());
} finally {
  await browser.close(); // always: an unclosed session keeps its concurrency slot
}
```

For real work copy `scripts/playwright_scrape.js` or `scripts/playwright_scrape.py` whole instead of
reimplementing. They add the five behaviours everything else in this file assumes:

- **Connect with backoff** (1 s base, 60 s cap, jitter, 6 attempts) only on retryable errors: `429`, `5xx`,
  `CDP_SESSION_IN_USE`, `CDP_NO_BROWSERS_AVAILABLE`, `CDP_BROWSER_OVERWORKED`, `CDP_BAD_PROXY`,
  `CDP_GENERAL_ERROR`, timeouts. `400`/`401`/`403` mean the request is wrong: fix, never retry unchanged.
- **Redact the password** from every error message before logging; Playwright embeds the connection URL in it.
- **Block `image`, `stylesheet`, `media`, `font`** by default; they cost time and are not needed for data extraction.
- **Register listeners before navigating**: the `X-Error-Description` response header marks an Oxylabs-side
  error on page traffic.
- **`browser.close()` in `finally`**, and wrap the job in an overall deadline so a wedged session still gets there.

Puppeteer, Python async, raw CDP, session hand-over, profiles, recording and fan-out: `examples.md`.

## 3. Sessions and limits

| Limit (account defaults) | Value | When exceeded |
|--------------------------|-------|---------------|
| New sessions per second | 10 | `429 CDP_SESSION_RATE_LIMIT_REACHED` (space launches >= 150 ms) |
| Concurrent sessions | 100 | `429 CDP_MAX_CONCURRENT_SESSIONS_REACHED` |
| Named (resumable) sessions | 5 | `429 CDP_MAX_PERSISTENT_SESSIONS_REACHED` |
| Stored profiles (`o_profile`) | 5 | `403 profile limit reached (5 profiles maximum)` |
| Recordings | 10 | `403 recording limit reached (10 recordings maximum)` |
| Concurrent inspection viewers | 10 | `CDP_VNC_MAX_CONCURRENT_SESSIONS_REACHED` |

- `session_name` (`^[A-Za-z0-9-]{3,36}$`) makes a session resumable for **10 minutes** after disconnect.
  `keep_alive` is implied by it; **never send `keep_alive=true` alone** (`400 keep_alive requires session_name`).
- Reconnecting while the old connection is still attached returns `429 CDP_SESSION_IN_USE`: close it first.
- Any session lives at most **1 hour**. Plan long jobs as several sessions.
- An abandoned session keeps its concurrency slot (about 20 s, or the full 10 min when named) and surfaces later
  as an unrelated `429 CDP_MAX_CONCURRENT_SESSIONS_REACHED`. Closing the Playwright/Puppeteer object is enough.
- `browser.close()` wipes open pages and cookies even though a named session stays resumable. To hand a session
  over use Puppeteer `browser.disconnect()` (see `examples.md`, "Resume a named session"). State that must
  outlive a session (logins, clearance cookies) belongs in `o_profile`, not keep-alive.
- Every distinct parameter combination is provisioned separately: keep the set stable across a job.
- Under load a connection may queue and end with `503 queue timeout` after about a minute: back off and retry.
  Higher limits via support.

## 4. Errors

Three channels. **Handshake**: HTTP status plus a short body (Playwright: `WebSocket error: <URL with password>
<status>` then the body; Puppeteer: `Unexpected server response: <status>`). **Post-connect**: the WebSocket closes
with code `3000` and a `CDP_*` reason that only raw clients see; Playwright/Puppeteer just report `Target closed`,
so treat any disconnect in the first seconds of a session as retryable. **In-page**: CDP error `1337` for one
refused command. On page traffic, a response **with** `X-Error-Description` is an Oxylabs network error (retry);
a block page **without** it is the target's decision (change approach, do not retry).

```text
connect failed?
  ├─ 401 ............ fix credentials/scheme, do not retry
  ├─ 400/403/409 .... fix the named parameter, do not retry unchanged (409: wait 30 s+ for the other session)
  ├─ 429 ............ backoff; if MAX_CONCURRENT: hunt for unclosed sessions
  └─ 5xx/503 ........ backoff, up to ~2 min total
session dropped (close 3000)?
  └─ new session with backoff; rotate sticky id on CDP_BAD_PROXY
navigate failed with 1337 Invalid target?
  └─ stop; restricted target (section 7)
page shows block / 403 wall?
  ├─ X-Error-Description present .... Oxylabs network issue: backoff + retry
  └─ absent ......................... target decision: change identity, geo, device, pacing (section 5)
```

Every message text with cause and fix: `errors.md`.

## 5. Target safety (DataDome and similar)

**Default parameter set for most jobs: `p_cc`, nothing else.** Every session already gets a fresh fingerprint
and a fresh residential IP, which is what one-shot fetches and fan-outs of independent pages need. Sticky IPs
and stored profiles are opt-in tools for a specific need, never a baseline.

**Work order for a protected target.** First write a plain script and make it pass: one fresh session per page,
the right geo and device, human pacing, then the escalation ladder below. Only when that script still fails
after the ladder do you **recommend persistent profiles to the user** (the setup/consumer pattern below, with
why it should help and what it costs: a setup step, the profile cap of 5) and implement them only on their
go-ahead. Never add a profile or sticky id on your own initiative.

| Need | Add | Not for |
|------|-----|---------|
| Several connections must look like one visitor (login, cart, a flow that outlives one session) | `proxy_resi_ses_id` + `proxy_resi_ses_time` | one page per session |
| Cookies or a login must survive between jobs (DataDome clearance, authenticated scraping) | `o_profile`, prepared once by a setup run, after the user agreed | a first attempt; targets that serve without a block |
| Resume the same browser within 10 minutes | `session_name` | everything else |

When you do use them, the combination is one identity. Keep it consistent:

```text
setup, exactly once :  ?o_profile=acme-us-01&o_profile_save=true&p_cc=US&proxy_resi_ses_id=acmeus01&proxy_resi_ses_time=30
consumers, any number:  ?o_profile=acme-us-01&p_cc=US&proxy_resi_ses_id=acmeus01&proxy_resi_ses_time=30
```

- **A profile is written by one run and read by the others.** The setup run is the only connection that ever sends
  `o_profile_save=true`: it earns the cookies (clears the entry page, logs in), verifies the page, closes. Consumer
  runs send `o_profile=<name>` alone: read-only, no write lock, no `409`. Never "top up" a profile from a consumer;
  when it stops working, run setup again under a new name. In production this is a setup service that prepares and
  validates profiles and a consumer service that only uses them (`examples.md`, "Profile setup and consumer runs").
- `proxy_resi_ses_id` + `proxy_resi_ses_time` pin the exit IP (max 1440 min). A pinned id disables automatic
  proxy retry: on `CDP_BAD_PROXY` rotate to a new id.
- **Never change `p_cc`/`p_city`/`p_state` for an identity** that has cookies. Start a new profile and sticky id.
- Match interaction to `p_device`: `mobile` = taps, small scrolls, no hover; `desktop` (default) = the opposite.
  Never set viewport or device metrics yourself; the service owns the fingerprint.
- Pace like a person: 3 to 8 s between page loads, scroll before clicking, one page at a time per identity.
  Run parallel identities, not parallel tabs.
- Escalation when blocked, one rung per fresh connection: fresh session → broader geo (drop `p_city`) →
  `p_device=mobile` → slow down → inspect (section 6) → recommend persistent profiles to the user → stop and
  report. Repeating an identical request is never a rung.

Block signatures per vendor, do/don't table and starting values for a new protected target: `targets.md`.

## 6. Operational hygiene

- **Debugging.** Two tools exist, and whenever the user asks how to debug, what the browser is doing, or why a run
  fails, tell them about both: **live inspection** (fetch the session id with the CDP command `__session_id`, open
  `https://hb.oxylabs.io/novnc/?id=<id>` and watch the session as it runs) and **recordings** (`record=true&
  record_name=<job>` saves a video of the session to replay later in `https://hb.oxylabs.io/dashboard`; cap 10,
  delete old ones there). Both are off by default. Use them yourself after **3 consecutive failures on one
  target** to confirm what the page actually shows. Snippet in `examples.md`, "Session id, live inspection and
  recording".
- **Timeouts.** Connect 60 s, navigation 30 s, plus an overall job deadline.
- **Logging.** Never log the connection URL or a raw error message; log the parameter set and session id.
- **Contexts.** Use `browser.contexts()[0]`. A `newContext()` is isolated from profile storage and fingerprint tuning.

## 7. Restricted targets

Blocked by default; access requires a short KYC via your account manager: entertainment and streaming,
banking and finance, government sites, gaming platforms, ticketing, webmail, ad networks, third-party IP
checkers. Use `https://ip.oxylabs.io/location` to verify your exit IP and geo. A blocked target fails
`Page.navigate` with CDP error `1337 Invalid target`.

See also: `scripts/` (full Playwright templates, JS and Python), `parameters.md` (every parameter and its
validation), `errors.md` (every message), `examples.md` (Puppeteer, Python async, raw CDP, reconnection,
profiles, recording, fan-out), `targets.md` (block detection, DataDome playbook).

# Protected targets playbook (DataDome and similar)

Anti-bot vendors such as DataDome, Cloudflare Bot Management and Akamai score each visitor on the consistency
of IP, TLS/browser fingerprint, cookies and behaviour over time. The service handles the fingerprint and the
proxy; the agent's job is to keep everything it controls consistent and human-paced.

## 1. Detect the block

Check these before changing anything. None of them carry `X-Error-Description`, which is how you know the
website decided, not the Oxylabs network.

| Vendor | Signature |
|--------|-----------|
| DataDome | HTTP `403`; body starts with a short HTML page containing `Contact: DataAccess@datadome.co`; cookie named `datadome` |
| Cloudflare | HTTP `403` or `503` with title `Just a moment...` or `Attention Required`; `cf-mitigated: challenge` response header; cookie `cf_clearance` after success |
| Akamai | HTTP `403` with `Access Denied` and a `Reference #` id; cookies `_abck`, `bm_sz` |
| Generic | Login wall, empty product grid, 200 with a "verify you are human" page, redirect loop to `/blocked` |

```javascript
async function classify(page, response) {
  if (response.headers()["x-error-description"]) return "oxylabs-network"; // retry per errors.md
  const status = response.status();
  const body = (await page.content()).slice(0, 4096);
  if (status === 403 && body.includes("DataAccess@datadome.co")) return "datadome";
  if ([403, 503].includes(status) && /Just a moment|Attention Required/.test(body)) return "cloudflare";
  if (status === 403 && /Access Denied/.test(body)) return "akamai-or-waf";
  return status >= 400 ? "target-error" : "ok";
}
```

`datadome`, `cloudflare` and `akamai-or-waf` all mean the site has decided against this visitor. Retrying the
same session never helps: close it and work through the escalation ladder (section 4), starting with a fresh
identity. Confirm what the page actually shows (section 4, rung 5) before spending more sessions.

## 2. Use an identity only when the target needs one

Start every target with `p_cc` only. Each session is already a new fingerprint on a new residential IP, and for
most pages that is the safest possible visitor. The order of work is fixed: first write a plain script and
drive it through the escalation ladder (section 4); only if it still fails, recommend persistent profiles to
the user and implement them after they agree. Add persistence only for a concrete reason:

- **Sticky IP** (`proxy_resi_ses_id` + `proxy_resi_ses_time`): several connections must look like one visitor
  (login, cart, a multi-page flow that spans sessions).
- **Stored profile** (`o_profile`): cookies must survive between jobs. On DataDome-class sites that is the
  clearance cookie earned on the first page; on authenticated sites it is the login.

Once you add them, the parameter set is one identity and must be identical on every connection:

```text
setup (exactly once):  ?o_profile=<name>&o_profile_save=true&p_cc=<CC>[&p_state=..][&p_city=..]&p_device=..&proxy_resi_ses_id=<id>&proxy_resi_ses_time=30..120
consumers           :  ?o_profile=<name>&p_cc=<CC>[&p_state=..][&p_city=..]&p_device=..&proxy_resi_ses_id=<id>&proxy_resi_ses_time=30..120
```

- **Setup run**: the only connection that sends `o_profile_save=true`. It loads the entry page, waits for the
  real page to render (the `datadome` or `cf_clearance` cookie appearing is a good signal), optionally logs in,
  verifies the page is the real one, then closes so the profile is written. In production this is a small service
  that prepares profiles and re-checks them before handing them out.
- **Consumer runs**: `o_profile=<name>` without `o_profile_save`. Read-only, so they never take the write lock,
  never see `409`, and can start while others run. On DataDome-class targets still keep one active consumer per
  identity and pace it (section 3); parallelism comes from more identities, not more consumers on one.
- A profile that starts drawing block pages is burned: do not re-save it from a consumer. Run setup again under a
  new name with a new sticky id.
- Sticky time is a ceiling in minutes (max 1440). If the job outlives it, the exit IP changes while cookies stay,
  which DataDome scores against you. Prefer a fresh identity over an identity with a changed IP; after a sticky
  window closed, rotate **both** the profile and the id.
- Use the browser's default context (`browser.contexts()[0]`) so the profile actually backs the cookies.

## 3. Behave consistently

| Do | Don't |
|----|-------|
| Keep `p_cc`/`p_state`/`p_city` fixed for the life of a profile | Change geo mid-job "to see if it helps" |
| Match interaction to `p_device`: mobile = taps, small scrolls, portrait flow; desktop = mouse moves, hover, wider pages | Use `p_device=mobile` and then hover, right-click, or open 10 tabs |
| Let the service own viewport and User-Agent | Call `setViewportSize`, `emulate`, `setUserAgent`, or CDP `Emulation.*` overrides |
| Use one page at a time per identity, 3 to 8 s between navigations, scroll before clicking | Fire 20 navigations in parallel from one identity |
| Wait for `domcontentloaded` plus a random 2 to 5 s before reading the DOM | Read immediately and retry in a tight loop when data is missing |
| Send `o_profile_save=true` from one setup run only; consumers use `o_profile` alone | Save the profile from consumers or on every connection (write lock, `409`, burned profiles re-saved) |

## 4. Escalation ladder

Apply one rung at a time, on a fresh connection, and stop at the first that works. Repeating the same
request with the same identity is never a rung.

1. **Fresh identity**: a new session already means a new IP and fingerprint. If you were using a profile or
   sticky id, also use a new `o_profile` name and a new `proxy_resi_ses_id`; same geo/device.
2. **Broader geo**: drop `p_city`, then `p_state`; keep the country the site serves.
3. **Device switch**: `p_device=mobile` with a mobile interaction pattern (many listing sites are more
   lenient on mobile).
4. **Slow down**: double the pauses, one identity at a time, `<= 30` pages per identity.
5. **Inspect**: after 3 consecutive failures on the same target, fetch `__session_id` and watch the session
   at `https://hb.oxylabs.io/novnc/?id=<id>`, or add `record=true&record_name=<job>`; confirm what the page
   actually shows (block page, geo-fence, login wall, empty state).
6. **Recommend persistent profiles to the user.** The plain script has now failed on every rung. Explain that the
   remaining option is to reuse a clearance cookie or login across sessions via a stored profile (setup run plus
   consumers, section 2), optionally with a sticky IP, and what it costs: a setup step, the profile cap of 5, one
   more moving part. Implement it only after the user agrees; do not add it on your own.
7. **Stop and report** which rung failed, the block signature, the parameter set and the session id.

## 5. Starting point for a new protected target

Starting values, not measured guarantees. Tune pacing down if you see block pages on more than 1 in 20 pages.

| Setting | Start with | Why |
|---------|------------|-----|
| Geo | `p_cc` of the country the site serves; `p_state`/`p_city` only if results are geo-filtered that finely | A foreign IP is the cheapest signal a vendor has |
| Device | `p_device=desktop` (default); `mobile` only if the site is mobile-first or desktop keeps failing | Must match the interaction pattern you drive |
| Identity | None. A profile (setup run + consumers, section 2) is what you recommend to the user once the plain script has failed the whole ladder, for example because the site sets a clearance cookie on the first load and blocks more than 1 in 20 pages | Fresh sessions are the safest visitor for most sites |
| Pacing | 1 page every 5 to 8 s, scroll a listing before opening a detail page, interleave detail and listing pages | Search-only or detail-only bursts are what bots do |
| Volume | <= 100 pages per identity, then a new identity | Keeps per-identity request rate ordinary |

## 6. Reporting a block

When escalation is exhausted, report: target URL, block signature from section 1, full parameter set (never
the password), session id, timestamps of each attempt, and whether `X-Error-Description` was ever present.
Send it to `support@oxylabs.io` if the block correlates with the Oxylabs network rather than the target.