# Auth keys — unlocking paid fallback strategies

The free strategies (Bluesky API, Mastodon API, HN Algolia, Reddit .json, agent-browser) work for many cases. For X / LinkedIn (specific posts) / Instagram / TikTok / Threads, paid keys unlock reliable + complete data.

## Which key unlocks what

| Key | Unlocks | Cost (approx, 2026) |
|---|---|---|
| `$SCRAPECREATORS_API_KEY` | X (tweets + threads + replies), LinkedIn posts, Instagram, TikTok, possibly Threads | Pay-as-you-go, ~$0.005–$0.02/post depending on endpoint |
| `$APIFY_API_TOKEN` | Nearly any platform via "Actors" (X, IG, TikTok, LinkedIn, Threads, even niche ones) | Per-actor pricing, often $1–$5 / 1K results |

## Setting up

### ScrapeCreators

1. Sign up at https://scrapecreators.com
2. Get API key from dashboard
3. Add to `~/.zshenv`:
   ```bash
   export SCRAPECREATORS_API_KEY="<your-key>"
   ```
4. `source ~/.zshenv` or new terminal
5. Verify: `echo $SCRAPECREATORS_API_KEY | head -c 8`

### Apify

1. Sign up at https://apify.com (free tier exists; pay-as-you-go after)
2. Get API token from Settings → Integrations → API
3. Add to `~/.zshenv`:
   ```bash
   export APIFY_API_TOKEN="<your-token>"
   ```
4. `source ~/.zshenv`
5. Verify: `echo $APIFY_API_TOKEN | head -c 8`

## Free-only mode

If neither key is set, `social-fetch` runs in free-only mode:

- **Bluesky / Mastodon / HN / Reddit** — fully functional
- **X** — preview only (text + basic engagement, no thread/replies)
- **LinkedIn profiles** — recent activity feed visible (no specific post fetch)
- **Instagram / TikTok / Threads** — Open Graph metadata only (title, description, image — no engagement, no replies)

This is fine for most one-off fetches. Paid fallback is mostly useful when:
- Building corpus (deep-research running many fetches)
- Analyzing thread structure or reply trees on X
- Inspiration-account analysis on Instagram / TikTok (where free fails entirely)

## Cost discipline

When using paid strategies:
- Always check the cache first (`~/Documents/social-fetches/_cache/` if it exists)
- Default to 24h cache TTL
- Don't auto-enrich with `--with-replies` or `--thread` unless requested — these multiply quota use
- For high-volume work (e.g., fetching last 50 posts from an inspiration account), batch via Apify actors (cheaper per item) instead of ScrapeCreators per-call

## When to set up paid

Don't set up keys preemptively. Set them up when:
- A real workflow (deep-research, inspiration analysis, competitor monitoring) is being blocked
- You've done >10 fetches and the free strategies are missing data you need
- You're starting on `swipe-save` or another skill where social capture is central

If `social-fetch` falls through to "no paid key set" on the same platform 3+ times in a session, it'll surface a one-time prompt.

# Normalized output schema

Same shape returned for every platform. Fields with no equivalent on a platform get `null`, not zero or empty string. Missing ≠ zero.

## Schema

```typescript
{
  "platform": "x" | "linkedin" | "linkedin-profile" | "instagram" | "tiktok"
    | "bluesky" | "reddit" | "mastodon" | "threads" | "hn",
  "url": string,                 // canonical URL of the post
  "fetched_at": ISO8601,
  "raw_source":                   // which strategy succeeded
    "direct-api" | "agent-browser" | "open-graph" | "nitter"
    | "wayback" | "scrapecreators" | "apify",

  "author": {
    "handle": string | null,      // @username
    "name": string | null,        // display name
    "verified": boolean | null,
    "follower_count": number | null,
    "profile_url": string | null,
    "avatar_url": string | null
  },

  "posted_at": ISO8601 | null,
  "edited_at": ISO8601 | null,

  "text": string,                 // post body, plain text. Markdown OK if source supports it.
  "html": string | null,          // original HTML if scraped (Mastodon, etc.)
  "language": string | null,      // ISO 639-1 if detected

  "media": Array<{
    "type": "image" | "video" | "gif" | "audio",
    "url": string,
    "alt": string | null,
    "width": number | null,
    "height": number | null,
    "duration_seconds": number | null    // video/audio only
  }>,

  "engagement": {
    "likes": number | null,         // tweets/posts: likes. Reddit: upvotes.
    "reposts": number | null,       // retweets, shares, reposts, reblogs
    "replies": number | null,
    "bookmarks": number | null,
    "views": number | null,
    "quotes": number | null         // quote-tweets where applicable
  },

  "links": Array<{                  // links cited in the post body
    "url": string,
    "expanded_url": string,         // resolved if t.co or similar shortener
    "title": string | null
  }>,

  "mentions": string[],             // @handles mentioned
  "hashtags": string[],

  "is_reply": boolean,
  "reply_to": {                     // if is_reply = true
    "url": string,
    "author_handle": string
  } | null,

  "is_thread": boolean,             // post is part of a multi-post thread by same author
  "thread": Array<{                 // siblings in the thread (excludes self), in order
    "url": string,
    "text": string,
    "posted_at": ISO8601
  }>,

  "replies": Array<{                // top-level replies (only if --with-replies)
    "url": string,
    "author": { "handle": string, "name": string },
    "text": string,
    "posted_at": ISO8601,
    "engagement": { "likes": number, "replies": number }
  }>,

  "raw": object | null               // raw API/scrape response (only if --raw)
}
```

## Platform field-coverage matrix

What each platform typically provides on a successful free-strategy fetch:

| Field | bluesky | mastodon | hn | reddit | x (free) | x (paid) | linkedin (free) | linkedin (paid) | ig (paid) | tiktok (paid) | threads (paid) |
|---|---|---|---|---|---|---|---|---|---|---|---|
| author.handle | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| author.verified | — | — | — | — | partial | ✅ | — | ✅ | ✅ | ✅ | ✅ |
| author.follower_count | ✅ | ✅ | — | — | — | ✅ | — | ✅ | ✅ | ✅ | ✅ |
| posted_at | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | partial | ✅ | ✅ | ✅ | ✅ |
| text | ✅ | ✅ | ✅ | ✅ | partial | ✅ | partial | ✅ | ✅ | ✅ | ✅ |
| media | ✅ | ✅ | — | ✅ | — | ✅ | — | ✅ | ✅ | ✅ | ✅ |
| engagement.likes | ✅ | ✅ | ✅ (score) | ✅ | — | ✅ | — | ✅ | ✅ | ✅ | ✅ |
| engagement.reposts | ✅ | ✅ | — | — | — | ✅ | — | ✅ | — | ✅ | ✅ |
| engagement.views | — | — | — | — | — | ✅ | — | partial | partial | ✅ | partial |
| replies | ✅ | ✅ | ✅ | ✅ | — | ✅ | — | partial | — | partial | partial |
| thread (own author) | ✅ | ✅ | n/a | n/a | — | ✅ | — | ✅ | — | — | partial |

`partial` = sometimes available depending on the post / scraping conditions.

## Example outputs (abbreviated)

### bluesky — full

```json
{
  "platform": "bluesky",
  "url": "https://bsky.app/profile/jay.bsky.team/post/3kphvqpv6xs2t",
  "raw_source": "direct-api",
  "author": {
    "handle": "@jay.bsky.team",
    "name": "Jay Graber",
    "verified": null,
    "follower_count": 500123
  },
  "posted_at": "2026-06-15T14:23:00Z",
  "text": "We're rolling out video posts.",
  "media": [],
  "engagement": {
    "likes": 1245, "reposts": 234, "replies": 89, "bookmarks": null, "views": null
  }
}
```

### x — partial (free strategy: agent-browser preview)

```json
{
  "platform": "x",
  "url": "https://x.com/example/status/1234567890",
  "raw_source": "agent-browser",
  "author": {
    "handle": "@example",
    "name": "Example User",
    "verified": true,
    "follower_count": null
  },
  "posted_at": "2026-06-17T16:53:00Z",
  "text": "The 80/20 of a useful AI second brain: ...",
  "engagement": {
    "likes": 51, "reposts": 13, "replies": 9, "bookmarks": 7, "views": 32700
  },
  "is_thread": true,
  "thread": []
}
```

Note `thread: []` despite `is_thread: true` — free strategy can detect a thread exists but can't fetch the other posts. Use paid fallback or visit each thread URL directly.

### hn — full

```json
{
  "platform": "hn",
  "url": "https://news.ycombinator.com/item?id=12345678",
  "raw_source": "direct-api",
  "author": { "handle": "pg" },
  "posted_at": "2026-06-15T18:00:00Z",
  "text": "Title of the submission",
  "links": [{"url": "https://example.com/article", "expanded_url": "https://example.com/article"}],
  "engagement": { "likes": 234, "replies": 89 },
  "replies": [
    {
      "url": "https://news.ycombinator.com/item?id=12345679",
      "author": { "handle": "user1" },
      "text": "Great article because…",
      "engagement": { "likes": 45, "replies": 3 }
    }
  ]
}
```

## Returning to the user

When the skill completes, output:

1. **One-line summary** in chat: `<author> · <platform> · <date> · "<first 80 chars>..."`
2. **Path to the JSON output** if `--save` flag set (saved to `~/Documents/social-fetches/<platform>-<id>.json`)
3. **Inline JSON** in chat for the calling skill / the user to use

If only partial data was retrievable (free strategy on X/LinkedIn/IG/TikTok), surface it clearly: *"Free strategy returned author + text + engagement. Replies + full thread require `$SCRAPECREATORS_API_KEY`. See references/auth-keys.md to set up."*

# Per-platform strategies

Each platform has 2–5 strategies tried in order. Free first, paid as fallback (and only if env keys are set).

---

## bluesky

**Reliability: high.** Public API, free.

### Strategy 1 — Public AppView API (DEFAULT)

URL: `https://bsky.app/profile/<handle>/post/<rkey>`

```bash
# Resolve handle → DID
DID=$(curl -s "https://public.api.bsky.app/xrpc/com.atproto.identity.resolveHandle?handle=<handle>" | jq -r .did)

# Build AT-URI and fetch the post + thread
URI="at://${DID}/app.bsky.feed.post/<rkey>"
curl -s "https://public.api.bsky.app/xrpc/app.bsky.feed.getPostThread?uri=${URI}" | jq .
```

Returns the post + immediate parent + first-level replies in one call. Perfect for `--with-replies`.

---

## mastodon

**Reliability: high.** Public API, free, no auth.

### Strategy 1 — Status endpoint

URL pattern: `https://<instance>/@<user>/<status-id>`

```bash
# Single status
curl -s -H "Accept: application/json" "https://<instance>/api/v1/statuses/<status-id>"

# With replies
curl -s -H "Accept: application/json" "https://<instance>/api/v1/statuses/<status-id>/context"
```

Note: instance and ID parsed from URL. Web URL `https://hachyderm.io/@user/123456` → API `https://hachyderm.io/api/v1/statuses/123456`.

---

## hn

**Reliability: high.** Free public Algolia API.

### Strategy 1 — Algolia items API

URL pattern: `https://news.ycombinator.com/item?id=<id>`

```bash
curl -s "https://hn.algolia.com/api/v1/items/<id>" | jq .
```

Returns the item + full nested comment tree. Comments are recursive — flatten or limit depth per `--with-replies` flag.

---

## reddit

**Reliability: medium.** Free .json suffix works but rate-limited per IP (~60 req/min).

### Strategy 1 — Append `.json` to URL

URL pattern: `https://www.reddit.com/r/<sub>/comments/<id>/<slug>/`

```bash
curl -s -A "Mozilla/5.0 social-fetch/0.1" "<url>.json" | jq .
```

Returns `[post, comments_tree]` as a 2-element array. Set a real User-Agent — Reddit blocks default curl UA.

### Strategy 2 — Wayback Machine fallback

If rate-limited or post deleted:

```bash
curl -s "https://archive.org/wayback/available?url=<encoded-url>" | jq -r '.archived_snapshots.closest.url'
```

Returns Wayback URL — re-fetch from there.

---

## x (twitter)

**Reliability: low without paid keys.** X aggressively blocks scraping.

### Strategy 1 — agent-browser preview (LIMITED)

For tweet preview only — body text and basic author info. No engagement counts, no replies, no thread.

```bash
agent-browser open "<url>"
sleep 3
agent-browser snapshot 2>&1 | head -50
# Parse the StaticText for tweet body
```

Often hits "Sign up to see" modal — dismiss with first interactive button if visible.

### Strategy 2 — Nitter mirror (UNRELIABLE)

Nitter instances are frequently rate-limited or down. Try if running:

```bash
# Pick a known-working instance (rotate if down)
for inst in nitter.net nitter.lacontrevoie.fr nitter.privacydev.net; do
  CODE=$(curl -s -o /dev/null -w "%{http_code}" "https://${inst}/<user>/status/<id>")
  if [ "$CODE" = "200" ]; then
    curl -s "https://${inst}/<user>/status/<id>"
    break
  fi
done
```

### Strategy 3 — Wayback Machine

```bash
curl -s "https://archive.org/wayback/available?url=https://twitter.com/<user>/status/<id>" | jq -r '.archived_snapshots.closest.url'
```

Older tweets often cached; recent ones rarely.

### Strategy 4 — ScrapeCreators API (PAID, recommended for X)

Requires `$SCRAPECREATORS_API_KEY`. If unset, skip and surface a one-time setup prompt.

```bash
curl -s "https://api.scrapecreators.com/v1/twitter/tweet?url=<encoded-url>" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY"
```

Endpoint exact path may differ — verify in ScrapeCreators docs on first use.

### Strategy 5 — Apify scraper (PAID)

Requires `$APIFY_API_TOKEN`. Use the `apify/twitter-scraper` actor or a community equivalent.

```bash
curl -X POST "https://api.apify.com/v2/acts/<actor-id>/run-sync-get-dataset-items?token=$APIFY_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"tweetUrls": ["<url>"], "maxItems": 1}'
```

---

## linkedin

**Reliability: medium.** agent-browser works for some content.

### Strategy 1 — agent-browser + dismiss modal

```bash
agent-browser open "<url>"
sleep 3
# Detect signup modal and dismiss
agent-browser snapshot -i 2>&1 | head -20
# If first interactive element is "Dismiss" button:
agent-browser click @e1
sleep 2
agent-browser snapshot 2>&1 | head -100
```

Works well for `linkedin.com/in/<handle>` profile pages (recent activity feed visible).
Specific post URLs (`linkedin.com/posts/...`) usually require login — fall through.

### Strategy 2 — ScrapeCreators API (PAID)

```bash
curl -s "https://api.scrapecreators.com/v1/linkedin/post?url=<encoded-url>" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY"
```

### Strategy 3 — Apify (PAID)

Use `apify/linkedin-profile-scraper` or `apify/linkedin-post-scraper` actor.

---

## instagram

**Reliability: low without paid.** Heavy anti-bot.

### Strategy 1 — Open Graph fallback (LIMITED — just description/image)

```bash
curl -s -A "Mozilla/5.0" "<url>" | grep -E 'og:(title|description|image)'
```

Returns metadata only. No engagement counts. Often blocked.

### Strategy 2 — ScrapeCreators API (PAID, recommended for IG)

```bash
curl -s "https://api.scrapecreators.com/v1/instagram/post?url=<encoded-url>" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY"
```

### Strategy 3 — Apify (PAID)

Use `apify/instagram-scraper` actor.

---

## tiktok

**Reliability: low without paid.**

### Strategy 1 — Open Graph fallback (LIMITED)

```bash
curl -s -A "Mozilla/5.0" "<url>" | grep -E 'og:(title|description|video)'
```

### Strategy 2 — ScrapeCreators API (PAID)

```bash
curl -s "https://api.scrapecreators.com/v1/tiktok/video?url=<encoded-url>" \
  -H "x-api-key: $SCRAPECREATORS_API_KEY"
```

### Strategy 3 — Apify (PAID)

Use `apify/tiktok-scraper` actor.

---

## threads

**Reliability: low without paid.** Meta's anti-bot, like Instagram.

### Strategy 1 — Open Graph fallback (LIMITED)

```bash
curl -s -A "Mozilla/5.0" "<url>" | grep -E 'og:(title|description)'
```

### Strategy 2 — ScrapeCreators API (PAID, if supported)

Check ScrapeCreators docs for Threads endpoint — coverage varies.

### Strategy 3 — Apify (PAID)

Use a Threads scraper actor (search Apify marketplace).

---

## Strategy chain summary

| Platform | Free strategies | Paid fallback |
|---|---|---|
| bluesky | Direct API | — |
| mastodon | Direct API | — |
| hn | Algolia API | — |
| reddit | `.json` suffix → Wayback | — |
| x | agent-browser → Nitter → Wayback | ScrapeCreators → Apify |
| linkedin | agent-browser (modal dismiss) | ScrapeCreators → Apify |
| instagram | OG tags | ScrapeCreators → Apify |
| tiktok | OG tags | ScrapeCreators → Apify |
| threads | OG tags | ScrapeCreators → Apify |