ai-infrastructure-replicate
Replicate SDK patterns for TypeScript/Node.js -- client setup, predictions, streaming, webhooks, file handling, model versioning, deployments, and training
Install
npx skills add https://github.com/agents-inc/skills/tree/main/dist/plugins/ai-infrastructure-replicate/skills/ai-infrastructure-replicate
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install agents-inc-skills@llmmart
git clone https://github.com/agents-inc/skills.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole agents-inc/skills collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Replicate SDK Patterns
Quick Guide: Use the
replicatenpm package to run open-source ML models on serverless GPUs. Usereplicate.run()for synchronous execution that returns output directly,replicate.stream()for SSE-based streaming, orreplicate.predictions.create()for async background jobs with webhook notifications. Models are referenced asowner/model(uses latest version) orowner/model:version(pinned). File outputs areFileOutputobjects implementingReadableStream. Cold starts are expected for infrequently-used models -- use deployments withmin_instancesto keep models warm.
<critical_requirements>
CRITICAL: Before Using This Skill
All code must follow project conventions in CLAUDE.md (kebab-case, named exports, import ordering,
import type, named constants)
(You MUST never hardcode API tokens -- always use environment variables via process.env.REPLICATE_API_TOKEN)
(You MUST handle FileOutput objects for models that return files -- do not assume outputs are plain strings or URLs)
(You MUST validate webhooks using validateWebhook() from the replicate package -- never trust unverified webhook payloads)
(You MUST account for cold starts when running infrequently-used models -- use deployments for latency-sensitive applications)
(You MUST specify model versions (owner/model:version) in production to ensure reproducible results -- unversioned references use the latest, which can change)
</critical_requirements>
Auto-detection: Replicate, replicate, replicate.run, replicate.stream, replicate.predictions, replicate.deployments, replicate.trainings, replicate.models, FileOutput, validateWebhook, REPLICATE_API_TOKEN, serverless GPU, cold start, webhook_events_filter
When to use:
- Running open-source ML models (Llama, Stable Diffusion, Whisper, etc.) without managing GPU infrastructure
- Generating images, transcribing audio, running LLMs, or any ML inference via API
- Streaming LLM output in real-time with server-sent events
- Processing predictions asynchronously with webhook notifications
- Fine-tuning models with custom training data
- Running models on dedicated hardware with custom scaling via deployments
Key patterns covered:
- Client initialization and configuration (auth, user agent, file encoding)
- Running predictions (
replicate.run(),replicate.predictions.create(),replicate.wait()) - Streaming output (
replicate.stream()with SSE events) - Model versioning (
owner/modelvsowner/model:version) - File input/output handling (
FileOutput, file uploads,Bufferinputs) - Webhooks (setup, event filtering, signature validation)
- Deployments (custom hardware, scaling, keeping models warm)
- Training / fine-tuning
When NOT to use:
- You need a unified multi-provider LLM SDK (OpenAI, Anthropic, Google) -- use a provider-agnostic SDK
- You want to run models locally -- Replicate is a cloud-only serverless platform
- You need sub-second latency guarantees without deployments -- cold starts can take minutes
Examples Index
- Core: Setup, Predictions & Files -- Client init, run(), predictions.create(), wait(), file I/O, error handling
- Streaming & Webhooks -- stream(), SSE events, webhook setup, signature validation
- Deployments & Training -- Custom hardware, scaling, fine-tuning, model management
- Quick API Reference -- Method signatures, constructor options, error types, model reference format
<decision_framework>
Decision Framework
Which Execution Method to Use
Is this a user-facing LLM response?
+-- YES -> Use replicate.stream() for real-time SSE output
+-- NO -> Do you need the result immediately?
+-- YES -> Use replicate.run() (blocks until complete)
+-- NO -> Use replicate.predictions.create() + webhook
+-- Need to poll instead? -> Use replicate.wait(prediction)
Model Reference Format
Are you in development/prototyping?
+-- YES -> Use owner/model (latest version, convenient)
+-- NO -> Are you in production?
+-- YES -> Use owner/model:version_hash (pinned, reproducible)
+-- Does the model change frequently?
+-- YES -> Pin version, test updates explicitly
+-- NO -> Either format works, prefer pinned
Deployments vs Direct API
Do you need consistent low latency?
+-- YES -> Create a deployment with min_instances >= 1
+-- NO -> Do you need custom hardware (A100, H100)?
+-- YES -> Create a deployment with specific hardware
+-- NO -> Use replicate.run() / replicate.stream() directly
(Replicate auto-allocates hardware)
When to Use This SDK vs Other AI SDKs
Are you running open-source models on serverless GPUs?
+-- YES -> Use Replicate SDK
+-- NO -> Are you calling proprietary APIs (OpenAI, Anthropic)?
+-- YES -> Not this skill's scope -- use provider-specific SDKs
+-- NO -> Do you need to switch between multiple providers?
+-- YES -> Not this skill's scope -- use a unified provider SDK
+-- NO -> Do you want to self-host models?
+-- YES -> Not this skill's scope -- consider Cog or vLLM
+-- NO -> Replicate SDK is appropriate
</decision_framework>
<red_flags>
RED FLAGS
High Priority Issues:
- Hardcoding
REPLICATE_API_TOKENin source code (security breach risk) - Treating
FileOutputas a string (it is aReadableStreamobject -- use.url()or.blob()) - Not validating webhook signatures with
validateWebhook()(allows forged webhook payloads) - Using
replicate.run()for long-running models in request handlers (blocks the response, can timeout)
Medium Priority Issues:
- Not pinning model versions in production (
owner/modeluses latest, which can change without notice) - Relying solely on default retry behavior for production (5 retries with exponential backoff may be too aggressive for some use cases)
- Uploading large files as
Bufferinstead of hosting them at a URL (100 MiB limit on uploads) - Ignoring cold start latency for infrequently-used models (first request can take minutes)
Common Mistakes:
- Confusing
replicate.run()(returns output directly) withreplicate.predictions.create()(returns a prediction object with status/id) - Destructuring image output incorrectly:
const output = await replicate.run(...)instead ofconst [output] = await replicate.run(...)(image models return arrays) - Using
replicate.stream()with models that do not support streaming (only language models with SSE support) - Forgetting that
replicate.predictions.create()accepts either aversionhash or amodelstring (owner/model) -- useversionfor pinned reproducibility,modelfor latest-version convenience - Not consuming the async iterator from
replicate.stream()(events are lost)
Gotchas & Edge Cases:
- Prediction inputs and outputs are automatically deleted after one hour -- persist outputs via webhooks or download immediately
- The SDK auto-retries on 429 (rate limit) and 5xx errors -- 5 retries by default with exponential backoff. GET requests retry on 429 and 5xx; non-GET requests retry only on 429
replicate.stream()returnsServerSentEventobjects with.event("output","error","done") and.data(string) properties- File uploads are limited to 100 MiB -- for larger files, host them at a URL and pass the URL as input
- Browser usage is not supported -- the SDK requires a server-side environment (Node.js 18+, Bun, Deno, Cloudflare Workers)
webhook_events_filteraccepts["start", "output", "logs", "completed"]-- use["completed"]unless you need intermediate status updates- The
Prefer: waitheader enables sync mode on the HTTP API (up to 60s), butreplicate.run()already handles this automatically - Community models may disappear or change without warning -- pin versions and maintain fallbacks for critical workflows
replicate.wait()polls the API until the prediction completes -- use webhooks for production to avoid polling overheadFileOutput.url()returns the underlying URL, but these URLs are temporary -- download or persist the file before it expires
</red_flags>
<critical_reminders>
CRITICAL REMINDERS
All code must follow project conventions in CLAUDE.md (kebab-case, named exports, import ordering,
import type, named constants)
(You MUST never hardcode API tokens -- always use environment variables via process.env.REPLICATE_API_TOKEN)
(You MUST handle FileOutput objects for models that return files -- do not assume outputs are plain strings or URLs)
(You MUST validate webhooks using validateWebhook() from the replicate package -- never trust unverified webhook payloads)
(You MUST account for cold starts when running infrequently-used models -- use deployments for latency-sensitive applications)
(You MUST specify model versions (owner/model:version) in production to ensure reproducible results -- unversioned references use the latest, which can change)
Failure to follow these rules will produce insecure, unreliable, or unpredictable AI integrations.
</critical_reminders>
Files (skills)
-
examples
-
core.md 8.2 KB
# Replicate SDK -- Setup, Predictions & File Handling Examples > Client initialization, production config, running predictions, file I/O, model versioning, and error handling. See [SKILL.md](../SKILL.md) for core patterns. **Related examples:** - [streaming-webhooks.md](streaming-webhooks.md) -- Streaming output, SSE events, webhooks - [deployments-training.md](deployments-training.md) -- Deployments, training, model management --- ## Basic Client Setup ```typescript // lib/replicate.ts import Replicate from "replicate"; // Reads REPLICATE_API_TOKEN from env automatically const replicate = new Replicate(); export { replicate }; ``` --- ## Production Configuration ```typescript // lib/replicate.ts import Replicate from "replicate"; const replicate = new Replicate({ auth: process.env.REPLICATE_API_TOKEN, userAgent: "my-app/1.0.0", }); export { replicate }; ``` ### Constructor Options ```typescript new Replicate({ auth: process.env.REPLICATE_API_TOKEN, // Auto-reads from env if omitted baseUrl: "https://api.replicate.com/v1", // Override for proxies userAgent: "my-app/1.0.0", // Custom user agent string fetch: globalThis.fetch, // Custom fetch implementation fileEncodingStrategy: "default", // "default" | "upload" | "data-uri" useFileOutput: true, // Return FileOutput objects (default: true) }); ``` --- ## Running Predictions with replicate.run() ### Image Generation ```typescript import Replicate from "replicate"; const replicate = new Replicate(); // Image models return arrays of FileOutput objects const [output] = await replicate.run("black-forest-labs/flux-schnell", { input: { prompt: "a serene mountain landscape at sunset, photorealistic", aspect_ratio: "16:9", num_outputs: 1, }, }); console.log(output.url()); // Temporary URL to generated image ``` ### Text Generation (LLM) ```typescript const output = await replicate.run("meta/meta-llama-3-70b-instruct", { input: { prompt: "Explain the difference between TCP and UDP in 3 sentences.", max_tokens: 256, temperature: 0.7, }, }); // LLM output is a string (or array of strings for some models) console.log(output); ``` ### Audio Transcription ```typescript const output = await replicate.run("openai/whisper", { input: { audio: "https://example.com/audio-file.mp3", model: "large-v3", language: "en", }, }); // Output contains transcription text and segments console.log(output.text); ``` --- ## Version Pinning ```typescript // Development: latest version (convenient but unpredictable) const output = await replicate.run("stability-ai/sdxl", { input: { prompt: "a cat" }, }); // Production: pin to a specific version hash const SDXL_VERSION = "39ed52f2a78e934b3ba6e2a89f5b1c712de7dfea535525255b1aa35c5565e08b"; const output = await replicate.run(`stability-ai/sdxl:${SDXL_VERSION}`, { input: { prompt: "a cat" }, }); ``` ### Listing Model Versions ```typescript // Get all versions of a model const versions = await replicate.models.versions.list("stability-ai", "sdxl"); for (const version of versions.results) { console.log(version.id, version.created_at); } // Get a specific version const version = await replicate.models.versions.get( "stability-ai", "sdxl", "39ed52f2a78e934b3ba6e2a89f5b1c712de7dfea535525255b1aa35c5565e08b", ); console.log(version.openapi_schema); // Model's input/output schema ``` --- ## Async Predictions with predictions.create() ```typescript // Create a prediction (returns immediately, does not wait for completion) // Use `version` for pinned reproducibility, or `model` for latest-version convenience const prediction = await replicate.predictions.create({ model: "owner/model", // OR version: "sha256hash" for pinned version input: { prompt: "a painting of a cat in the style of Van Gogh", }, webhook: "https://my.app/webhooks/replicate", webhook_events_filter: ["completed"], }); console.log(prediction.id); // "abc123xyz" console.log(prediction.status); // "starting" // Option 1: Poll until complete const result = await replicate.wait(prediction); console.log(result.output); console.log(result.status); // "succeeded" | "failed" | "canceled" // Option 2: Check status manually const updated = await replicate.predictions.get(prediction.id); console.log(updated.status); // Option 3: Cancel if no longer needed await replicate.predictions.cancel(prediction.id); ``` --- ## Progress Callbacks ```typescript import type { Prediction } from "replicate"; function onProgress(prediction: Prediction): void { console.log(`Status: ${prediction.status}`); if (prediction.logs) { console.log(prediction.logs); } } const output = await replicate.run( "black-forest-labs/flux-schnell", { input: { prompt: "a sunset" } }, onProgress, ); ``` --- ## File Input Handling ### URL Input (Recommended for Large Files) ```typescript // Pass a URL -- no upload needed, no size limit const output = await replicate.run("owner/image-upscaler", { input: { image: "https://example.com/photo.jpg", scale: 4, }, }); ``` ### Buffer Input (Auto-Uploaded, Max 100 MiB) ```typescript import { readFile } from "node:fs/promises"; const imageBuffer = await readFile("./input.png"); const output = await replicate.run("owner/image-upscaler", { input: { image: imageBuffer, // Automatically uploaded to Replicate scale: 4, }, }); ``` ### Explicit File Upload via Files API ```typescript // Upload a file and get a persistent reference const file = await replicate.files.create( await readFile("./training-data.zip"), { filename: "training-data.zip" }, ); console.log(file.id); // File ID for later reference console.log(file.urls); // Temporary download URLs ``` --- ## File Output Handling ```typescript import { writeFile } from "node:fs/promises"; const [output] = await replicate.run("black-forest-labs/flux-schnell", { input: { prompt: "a beautiful garden" }, }); // FileOutput implements ReadableStream // Method 1: Get the URL const url = output.url(); console.log(url); // Temporary URL (expires -- download promptly) // Method 2: Get as Blob, then save const blob = await output.blob(); const buffer = Buffer.from(await blob.arrayBuffer()); await writeFile("./output.png", buffer); // Method 3: Write directly using Node.js stream utilities await writeFile("./output.png", output); ``` --- ## Error Handling ```typescript import Replicate from "replicate"; const replicate = new Replicate(); async function safePrediction(prompt: string): Promise<string | null> { try { const output = await replicate.run("meta/meta-llama-3-70b-instruct", { input: { prompt, max_tokens: 512 }, }); return String(output); } catch (error) { if (!(error instanceof Error)) { throw error; } console.error(`Replicate error: ${error.message}`); // Check for HTTP status codes if ("status" in error) { const status = (error as { status: number }).status; switch (status) { case 401: throw new Error("Invalid API token. Check REPLICATE_API_TOKEN."); case 404: console.error("Model not found. Check the model identifier."); return null; case 422: console.error("Invalid input parameters for this model."); return null; case 429: console.error("Rate limited. All retries exhausted."); return null; } // 5xx errors -- SDK auto-retried, all attempts failed if (status >= 500) { console.error("Server error after all retries."); return null; } } throw error; // Re-throw unexpected errors } } const result = await safePrediction("Hello!"); if (result) { console.log(result); } ``` --- ## Listing Predictions ```typescript // List recent predictions (paginated) const page = await replicate.predictions.list(); for (const prediction of page.results) { console.log(prediction.id, prediction.status, prediction.model); } // Get next page if (page.next) { const nextPage = await replicate.predictions.list({ cursor: page.next }); } ``` --- _For streaming and webhooks, see [streaming-webhooks.md](streaming-webhooks.md). For deployments and training, see [deployments-training.md](deployments-training.md). For API reference tables, see [reference.md](../reference.md)._ -
deployments-training.md 6.5 KB
# Replicate SDK -- Deployments & Training Examples > Deployment management, custom hardware, scaling configuration, fine-tuning models, and model CRUD operations. See [SKILL.md](../SKILL.md) for core patterns. **Related examples:** - [core.md](core.md) -- Client setup, predictions, file handling - [streaming-webhooks.md](streaming-webhooks.md) -- Streaming output, webhooks --- ## Deployments ### Why Use Deployments Deployments give you a **private, fixed API endpoint** with control over: - **Hardware** -- Choose specific GPU types (e.g., A40, A100) - **Scaling** -- Set `min_instances` to keep models warm (eliminates cold starts) - **Versioning** -- Pin to a model version independently of the public model page ### Creating a Deployment ```typescript const deployment = await replicate.deployments.create({ name: "my-llama-deployment", model: "meta/meta-llama-3-70b-instruct", version: "abc123...", hardware: "gpu-a100-large", min_instances: 1, // Always warm -- no cold starts max_instances: 5, // Auto-scale up to 5 instances }); console.log(deployment.name); console.log(deployment.current_release); ``` ### Running Predictions on a Deployment ```typescript // Use the deployment name instead of model identifier const prediction = await replicate.deployments.predictions.create( "my-org/my-llama-deployment", { input: { prompt: "Summarize this article...", max_tokens: 1024, }, }, ); // Wait for result const result = await replicate.wait(prediction); console.log(result.output); ``` ### Streaming from a Deployment ```typescript const stream = replicate.stream("my-org/my-llama-deployment", { input: { prompt: "Hello world" }, }); for await (const event of stream) { if (event.event === "output") { process.stdout.write(event.data); } } ``` ### Managing Deployments ```typescript // List all deployments const deployments = await replicate.deployments.list(); for (const d of deployments.results) { console.log(d.name, d.current_release?.model); } // Get a specific deployment const deployment = await replicate.deployments.get( "my-org/my-llama-deployment", ); // Update deployment (change hardware, version, or scaling) await replicate.deployments.update("my-org/my-llama-deployment", { version: "new-version-hash", hardware: "gpu-a100-large", min_instances: 2, max_instances: 10, }); // Delete a deployment await replicate.deployments.delete("my-org/my-llama-deployment"); ``` ### Available Hardware ```typescript const hardware = await replicate.hardware.list(); for (const hw of hardware) { console.log(hw.name, hw.sku); } ``` See [reference.md](../reference.md) for the full hardware SKU table. --- ## Training (Fine-Tuning) ### Starting a Training Job ```typescript const training = await replicate.trainings.create( "owner", "model-name", "version-hash", { input: { train_data: "https://example.com/my-training-data.zip", num_train_epochs: 4, learning_rate: 0.0001, }, destination: "my-org/my-fine-tuned-model", webhook: "https://my.app/webhooks/training", webhook_events_filter: ["completed"], }, ); console.log(training.id); console.log(training.status); // "starting" ``` ### Monitoring Training Progress ```typescript // Poll for training status const training = await replicate.trainings.get(training.id); console.log(training.status); // "starting" | "processing" | "succeeded" | "failed" console.log(training.logs); // Training logs // Wait for completion const result = await replicate.wait(training); if (result.status === "succeeded") { console.log("Training complete!"); console.log("New model version:", result.output?.version); } else { console.error("Training failed:", result.error); } ``` ### Listing and Canceling Trainings ```typescript // List all trainings const trainings = await replicate.trainings.list(); for (const t of trainings.results) { console.log(t.id, t.status, t.model); } // Cancel a running training await replicate.trainings.cancel(training.id); ``` --- ## Model Management ### Getting Model Info ```typescript const model = await replicate.models.get("stability-ai", "sdxl"); console.log(model.owner); console.log(model.name); console.log(model.description); console.log(model.visibility); // "public" | "private" console.log(model.latest_version?.id); ``` ### Creating a Model ```typescript // Create a new model (for training destinations or custom models) const model = await replicate.models.create("my-org", "my-custom-model", { description: "A fine-tuned image generation model", visibility: "private", hardware: "gpu-a40-large", }); ``` ### Listing Models ```typescript // List your models const models = await replicate.models.list(); for (const m of models.results) { console.log(m.owner, m.name, m.run_count); } // Search public models const results = await replicate.models.search("text to image"); for (const m of results.results) { console.log(m.owner, m.name, m.description); } ``` ### Model Versions ```typescript // List all versions const versions = await replicate.models.versions.list("stability-ai", "sdxl"); for (const v of versions.results) { console.log(v.id, v.created_at); } // Get a specific version (includes input/output schema) const version = await replicate.models.versions.get( "stability-ai", "sdxl", "39ed52f2a78e934b3ba6e2a89f5b1c712de7dfea535525255b1aa35c5565e08b", ); // Access input/output schema from OpenAPI spec console.log(version.openapi_schema); ``` --- ## Collections ```typescript // List model collections const collections = await replicate.collections.list(); for (const c of collections.results) { console.log(c.slug, c.name, c.description); } // Get a specific collection const collection = await replicate.collections.get("text-to-image"); for (const model of collection.models) { console.log(model.owner, model.name); } ``` --- ## Files API ```typescript import { readFile } from "node:fs/promises"; // Upload a file const file = await replicate.files.create(await readFile("./data.zip"), { filename: "data.zip", }); console.log(file.id); // List files const files = await replicate.files.list(); for (const f of files.results) { console.log(f.id, f.name, f.size); } // Get file info const fileInfo = await replicate.files.get(file.id); // Delete a file await replicate.files.delete(file.id); ``` --- _For client setup and predictions, see [core.md](core.md). For streaming and webhooks, see [streaming-webhooks.md](streaming-webhooks.md). For API reference tables, see [reference.md](../reference.md)._ -
streaming-webhooks.md 7.2 KB
# Replicate SDK -- Streaming & Webhooks Examples > Streaming output with SSE events, webhook setup, event filtering, and signature validation. See [SKILL.md](../SKILL.md) for core patterns. **Related examples:** - [core.md](core.md) -- Client setup, predictions, file handling - [deployments-training.md](deployments-training.md) -- Deployments, training, model management --- ## Basic Streaming with replicate.stream() ```typescript import Replicate from "replicate"; const replicate = new Replicate(); const stream = replicate.stream("meta/meta-llama-3-70b-instruct", { input: { prompt: "Explain async/await in TypeScript.", max_tokens: 512, }, }); for await (const event of stream) { if (event.event === "output") { process.stdout.write(event.data); } } console.log(); // newline ``` --- ## Handling All SSE Event Types ```typescript const stream = replicate.stream("meta/meta-llama-3-70b-instruct", { input: { prompt: "Tell me a story." }, }); const chunks: string[] = []; for await (const event of stream) { switch (event.event) { case "output": // Progressive text output chunks.push(event.data); process.stdout.write(event.data); break; case "error": // Prediction error (event.data is JSON with details) console.error("Stream error:", event.data); break; case "done": // Prediction complete (event.data may contain reason) console.log("\nStream complete"); break; } } const fullOutput = chunks.join(""); console.log("Total length:", fullOutput.length); ``` ### SSE Event Types Reference | Event | Data Format | Description | | -------- | ----------- | ---------------------------------------------------------- | | `output` | Plain text | New model output chunk | | `error` | JSON string | Error details (e.g., `{"detail": "..."}`) | | `done` | JSON string | Completion signal (e.g., `{}` or `{"reason": "canceled"}`) | --- ## Streaming with Error Handling ```typescript try { const stream = replicate.stream("meta/meta-llama-3-70b-instruct", { input: { prompt: "Hello" }, }); for await (const event of stream) { if (event.event === "output") { process.stdout.write(event.data); } if (event.event === "error") { throw new Error(`Prediction error: ${event.data}`); } } } catch (error) { if (error instanceof Error) { console.error("Stream failed:", error.message); } throw error; } ``` --- ## Streaming on Deployments ```typescript // Stream from a deployment (consistent latency if min_instances > 0) const stream = replicate.stream("my-org/my-deployment", { input: { prompt: "Summarize this document..." }, }); for await (const event of stream) { if (event.event === "output") { process.stdout.write(event.data); } } ``` --- ## Webhook Setup ### Creating a Prediction with Webhooks ```typescript const prediction = await replicate.predictions.create({ version: "27b93a2413e7f36cd83da926f3656280b2931564ff050bf9575f1fdf9bcd7478", input: { prompt: "a painting of a sunset" }, webhook: "https://my.app/webhooks/replicate", webhook_events_filter: ["completed"], }); console.log(prediction.id); // Track via ID ``` ### Webhook Events Filter Options | Event | Description | When to Use | | ----------- | ----------------------------------------- | ---------------------------------------- | | `start` | Prediction has started processing | Track cold start / queue time | | `output` | New output is available (intermediate) | Progressive updates for long predictions | | `logs` | New log output from the model | Debugging, monitoring | | `completed` | Prediction finished (succeeded or failed) | Most common -- final result notification | ```typescript // Receive all events (verbose, useful for debugging) const prediction = await replicate.predictions.create({ version: "abc123...", input: { prompt: "hello" }, webhook: "https://my.app/webhooks/replicate", webhook_events_filter: ["start", "output", "logs", "completed"], }); // Receive only the final result (recommended for production) const prediction = await replicate.predictions.create({ version: "abc123...", input: { prompt: "hello" }, webhook: "https://my.app/webhooks/replicate", webhook_events_filter: ["completed"], }); ``` --- ## Webhook Signature Validation ### Using the Built-in Validator ```typescript import { validateWebhook } from "replicate"; async function handleWebhook(request: Request): Promise<Response> { const secret = process.env.REPLICATE_WEBHOOK_SIGNING_SECRET; if (!secret) { throw new Error("REPLICATE_WEBHOOK_SIGNING_SECRET not set"); } const isValid = await validateWebhook(request, secret); if (!isValid) { return new Response("Invalid webhook signature", { status: 401 }); } const prediction = await request.json(); // Safe to process -- signature verified console.log("Prediction status:", prediction.status); console.log("Prediction output:", prediction.output); if (prediction.status === "succeeded") { // Handle successful prediction await processOutput(prediction.output); } else if (prediction.status === "failed") { console.error("Prediction failed:", prediction.error); } return new Response("OK", { status: 200 }); } ``` ### Manual Signature Validation ```typescript import { validateWebhook } from "replicate"; // When you have raw request data instead of a Request object const isValid = await validateWebhook({ id: request.headers["webhook-id"], timestamp: request.headers["webhook-timestamp"], signature: request.headers["webhook-signature"], body: rawBody, // string | ArrayBuffer | ReadableStream secret: process.env.REPLICATE_WEBHOOK_SIGNING_SECRET, }); ``` --- ## Webhook Payload Structure ```typescript // The webhook POST body is the same as the prediction object interface WebhookPayload { id: string; model: string; version: string; status: "starting" | "processing" | "succeeded" | "failed" | "canceled"; input: Record<string, unknown>; output: unknown; // Model-specific output error: string | null; logs: string; created_at: string; started_at: string | null; completed_at: string | null; urls: { get: string; cancel: string; stream?: string; }; } ``` --- ## Polling as Alternative to Webhooks ```typescript // For environments where webhooks are not feasible const prediction = await replicate.predictions.create({ version: "abc123...", input: { prompt: "hello" }, }); // replicate.wait() polls until the prediction completes const result = await replicate.wait(prediction); if (result.status === "succeeded") { console.log(result.output); } else if (result.status === "failed") { console.error("Failed:", result.error); } else if (result.status === "canceled") { console.log("Prediction was canceled"); } ``` --- _For client setup and file handling, see [core.md](core.md). For deployments and training, see [deployments-training.md](deployments-training.md). For API reference tables, see [reference.md](../reference.md)._
-
-
reference.md 9.1 KB
# Replicate SDK Quick Reference > Constructor options, API methods, error types, model reference format, and webhook events. See [SKILL.md](SKILL.md) for core concepts and [examples/](examples/) for code examples. --- ## Package Installation ```bash npm install replicate ``` --- ## Client Configuration ```typescript import Replicate from "replicate"; const replicate = new Replicate({ auth: process.env.REPLICATE_API_TOKEN, // Auto-reads from env if omitted baseUrl: "https://api.replicate.com/v1", // Override for proxies userAgent: "my-app/1.0.0", // Custom user agent string fetch: globalThis.fetch, // Custom fetch implementation (Node.js 18+) fileEncodingStrategy: "default", // "default" | "upload" | "data-uri" useFileOutput: true, // Return FileOutput objects for file outputs (default: true) }); ``` ### Environment Variables | Variable | Purpose | | ---------------------------------- | ----------------------------------- | | `REPLICATE_API_TOKEN` | API token (auto-detected) | | `REPLICATE_WEBHOOK_SIGNING_SECRET` | Webhook signature validation secret | --- ## Model Reference Format ``` owner/model -- Uses latest version (convenient, not reproducible) owner/model:sha256hash -- Pinned to a specific version (production use) ``` ### Examples | Reference | Type | | -------------------------------- | ------------------------ | | `meta/meta-llama-3-70b-instruct` | Latest version | | `black-forest-labs/flux-schnell` | Latest version | | `stability-ai/sdxl:39ed52f2...` | Pinned version (SHA-256) | --- ## API Methods Reference ### Core Methods ```typescript // Run a model synchronously (waits for completion) const output = await replicate.run( "owner/model", // or "owner/model:version" { input: { prompt: "..." }, // Model-specific input parameters }, onProgress?, // Optional: (prediction: Prediction) => void ); // Stream model output as SSE events const stream = replicate.stream( "owner/model", // or deployment name { input: { prompt: "..." }, // Model-specific input parameters }, ); // Returns AsyncGenerator<ServerSentEvent> // Wait for a prediction to complete (polling) const result = await replicate.wait(prediction); ``` ### Predictions ```typescript // Create (async, returns immediately) // Specify EITHER model OR version (not both) const prediction = await replicate.predictions.create({ model: "owner/name", // Use for latest version // OR: version: "sha256hash", // Use for pinned reproducibility input: { prompt: "..." }, // Required: model inputs webhook?: "https://...", // Optional: webhook URL webhook_events_filter?: ["completed"], // Optional: event filter }); // Get prediction status const prediction = await replicate.predictions.get(prediction.id); // List predictions (paginated) const page = await replicate.predictions.list(); // Cancel a running prediction await replicate.predictions.cancel(prediction.id); ``` ### Models ```typescript // Get model info const model = await replicate.models.get("owner", "name"); // List models (paginated) const models = await replicate.models.list(); // Search models const results = await replicate.models.search("query"); // Create a model await replicate.models.create("owner", "name", { description?: string, visibility: "public" | "private", hardware?: string, }); // List model versions const versions = await replicate.models.versions.list("owner", "name"); // Get a specific version const version = await replicate.models.versions.get("owner", "name", "version_id"); ``` ### Deployments ```typescript // Create a deployment await replicate.deployments.create({ name: string, model: string, version: string, hardware: string, min_instances?: number, max_instances?: number, }); // Run prediction on deployment await replicate.deployments.predictions.create("owner/deployment", { input: { ... }, webhook?: string, }); // Get deployment info await replicate.deployments.get("owner/deployment"); // List deployments await replicate.deployments.list(); // Update deployment await replicate.deployments.update("owner/deployment", { ... }); // Delete deployment await replicate.deployments.delete("owner/deployment"); ``` ### Trainings ```typescript // Start training await replicate.trainings.create("owner", "model", "version", { input: { train_data: "https://...", ... }, destination: "owner/new-model", webhook?: string, webhook_events_filter?: string[], }); // Get training status await replicate.trainings.get(training.id); // List trainings await replicate.trainings.list(); // Cancel training await replicate.trainings.cancel(training.id); ``` ### Files ```typescript // Upload file await replicate.files.create(buffer, { filename: "data.zip" }); // Get file info await replicate.files.get(file.id); // List files await replicate.files.list(); // Delete file await replicate.files.delete(file.id); ``` ### Hardware ```typescript const hardware = await replicate.hardware.list(); // Returns: Array<{ name: string, sku: string }> ``` ### Collections ```typescript await replicate.collections.get("collection-slug"); await replicate.collections.list(); ``` --- ## Prediction Status Values | Status | Description | | ------------ | ------------------------------------ | | `starting` | Prediction is queued / model booting | | `processing` | Model is running | | `succeeded` | Prediction completed successfully | | `failed` | Prediction encountered an error | | `canceled` | Prediction was canceled by the user | --- ## SSE Event Types (Streaming) | Event | Data Format | Description | | -------- | ----------- | ---------------------------------------------------------- | | `output` | Plain text | New model output chunk | | `error` | JSON string | Error details (e.g., `{"detail": "Something went wrong"}`) | | `done` | JSON string | Completion signal (e.g., `{}` or `{"reason": "canceled"}`) | --- ## Webhook Events | Event | Description | | ----------- | ----------------------------------------- | | `start` | Prediction has started processing | | `output` | New output is available (intermediate) | | `logs` | New log output from the model | | `completed` | Prediction finished (succeeded or failed) | Webhook payload is the same as the prediction object (JSON POST to your URL). --- ## Webhook Signature Validation ```typescript import { validateWebhook } from "replicate"; // With a Request object const isValid = await validateWebhook(request, secret); // With raw data const isValid = await validateWebhook({ id: headers["webhook-id"], timestamp: headers["webhook-timestamp"], signature: headers["webhook-signature"], body: rawBody, secret: process.env.REPLICATE_WEBHOOK_SIGNING_SECRET, }); ``` --- ## FileOutput Object ```typescript // Returned by replicate.run() for models that output files interface FileOutput extends ReadableStream { url(): URL; // Get the underlying URL object (temporary -- download promptly) blob(): Promise<Blob>; // Get as Blob toString(): string; // String representation } ``` --- ## Error Types The SDK throws errors with HTTP status codes for API failures: | Status | Description | Auto-Retried? | | ------ | -------------------- | ------------- | | 400 | Bad Request | No | | 401 | Authentication Error | No | | 403 | Permission Denied | No | | 404 | Not Found | No | | 422 | Unprocessable Entity | No | | 429 | Rate Limit Exceeded | Yes | | >= 500 | Server Error | Yes | The SDK automatically retries on 429 and 5xx errors (5 retries by default with exponential backoff). GET requests retry on 429 and 5xx; non-GET requests retry only on 429. --- ## Hardware SKUs | SKU | Description | | ---------------- | ------------------ | | `cpu` | CPU only | | `gpu-t4-nano` | Nvidia T4 (small) | | `gpu-t4-small` | Nvidia T4 | | `gpu-a40-small` | Nvidia A40 (small) | | `gpu-a40-large` | Nvidia A40 (large) | | `gpu-a100-large` | Nvidia A100 (80GB) | --- ## Platform Support | Platform | Minimum Version | | ------------------- | ---------------------------------------- | | Node.js | 18+ | | Bun | 1.0+ | | Deno | 1.28+ | | Cloudflare Workers | Supported | | Vercel Edge Runtime | Supported | | AWS Lambda | Supported | | **Browsers** | **Not supported** -- use a backend proxy | -
SKILL.md 19.6 KB
--- name: ai-infrastructure-replicate description: Replicate SDK patterns for TypeScript/Node.js -- client setup, predictions, streaming, webhooks, file handling, model versioning, deployments, and training --- # Replicate SDK Patterns > **Quick Guide:** Use the `replicate` npm package to run open-source ML models on serverless GPUs. Use `replicate.run()` for synchronous execution that returns output directly, `replicate.stream()` for SSE-based streaming, or `replicate.predictions.create()` for async background jobs with webhook notifications. Models are referenced as `owner/model` (uses latest version) or `owner/model:version` (pinned). File outputs are `FileOutput` objects implementing `ReadableStream`. Cold starts are expected for infrequently-used models -- use deployments with `min_instances` to keep models warm. --- <critical_requirements> ## CRITICAL: Before Using This Skill > **All code must follow project conventions in CLAUDE.md** (kebab-case, named exports, import ordering, `import type`, named constants) **(You MUST never hardcode API tokens -- always use environment variables via `process.env.REPLICATE_API_TOKEN`)** **(You MUST handle `FileOutput` objects for models that return files -- do not assume outputs are plain strings or URLs)** **(You MUST validate webhooks using `validateWebhook()` from the `replicate` package -- never trust unverified webhook payloads)** **(You MUST account for cold starts when running infrequently-used models -- use deployments for latency-sensitive applications)** **(You MUST specify model versions (`owner/model:version`) in production to ensure reproducible results -- unversioned references use the latest, which can change)** </critical_requirements> --- **Auto-detection:** Replicate, replicate, replicate.run, replicate.stream, replicate.predictions, replicate.deployments, replicate.trainings, replicate.models, FileOutput, validateWebhook, REPLICATE_API_TOKEN, serverless GPU, cold start, webhook_events_filter **When to use:** - Running open-source ML models (Llama, Stable Diffusion, Whisper, etc.) without managing GPU infrastructure - Generating images, transcribing audio, running LLMs, or any ML inference via API - Streaming LLM output in real-time with server-sent events - Processing predictions asynchronously with webhook notifications - Fine-tuning models with custom training data - Running models on dedicated hardware with custom scaling via deployments **Key patterns covered:** - Client initialization and configuration (auth, user agent, file encoding) - Running predictions (`replicate.run()`, `replicate.predictions.create()`, `replicate.wait()`) - Streaming output (`replicate.stream()` with SSE events) - Model versioning (`owner/model` vs `owner/model:version`) - File input/output handling (`FileOutput`, file uploads, `Buffer` inputs) - Webhooks (setup, event filtering, signature validation) - Deployments (custom hardware, scaling, keeping models warm) - Training / fine-tuning **When NOT to use:** - You need a unified multi-provider LLM SDK (OpenAI, Anthropic, Google) -- use a provider-agnostic SDK - You want to run models locally -- Replicate is a cloud-only serverless platform - You need sub-second latency guarantees without deployments -- cold starts can take minutes --- ## Examples Index - [Core: Setup, Predictions & Files](examples/core.md) -- Client init, run(), predictions.create(), wait(), file I/O, error handling - [Streaming & Webhooks](examples/streaming-webhooks.md) -- stream(), SSE events, webhook setup, signature validation - [Deployments & Training](examples/deployments-training.md) -- Custom hardware, scaling, fine-tuning, model management - [Quick API Reference](reference.md) -- Method signatures, constructor options, error types, model reference format --- <philosophy> ## Philosophy Replicate provides **serverless GPU infrastructure** for running open-source ML models. You send inputs, Replicate allocates GPU hardware, runs the model, and returns outputs. No Docker, no CUDA drivers, no GPU provisioning. **Core principles:** 1. **Serverless execution** -- Models run on-demand on Replicate's infrastructure. You pay only for compute time. Cold starts are a trade-off for not maintaining always-on GPUs. 2. **Model marketplace** -- Thousands of community and official models available at `replicate.com/explore`. Run any public model with just its identifier. 3. **Version pinning for reproducibility** -- Models are versioned with SHA-256 hashes. Pin to a version in production (`owner/model:abc123...`) to guarantee identical behavior across deploys. 4. **Three execution modes** -- `replicate.run()` for synchronous wait, `replicate.stream()` for real-time SSE output, `replicate.predictions.create()` for fire-and-forget with webhooks. 5. **File-first I/O** -- Many models accept and produce files (images, audio, video). The SDK handles file uploads automatically and returns `FileOutput` objects for file outputs. </philosophy> --- <patterns> ## Core Patterns ### Pattern 1: Client Setup Initialize the Replicate client. It auto-reads `REPLICATE_API_TOKEN` from the environment. ```typescript // lib/replicate.ts -- basic setup import Replicate from "replicate"; const replicate = new Replicate(); export { replicate }; ``` ```typescript // lib/replicate.ts -- explicit auth + custom user agent import Replicate from "replicate"; const replicate = new Replicate({ auth: process.env.REPLICATE_API_TOKEN, // Auto-reads from env if omitted userAgent: "my-app/1.0.0", }); export { replicate }; ``` **Why good:** Minimal setup, env var auto-detected, explicit auth optional but useful for clarity ```typescript // BAD: Hardcoded token const replicate = new Replicate({ auth: "r8_abc123...", }); ``` **Why bad:** Hardcoded API token is a security risk, will leak in version control **See:** [examples/core.md](examples/core.md) for full constructor options, error handling patterns --- ### Pattern 2: Running Predictions Use `replicate.run()` for synchronous execution. Returns the model output directly. ```typescript // Run an image generation model const [output] = await replicate.run("black-forest-labs/flux-schnell", { input: { prompt: "a serene mountain landscape at sunset", }, }); // output is a FileOutput object for image models console.log(output.url()); // URL of generated image ``` ```typescript // Run an LLM -- output is a string for text models const output = await replicate.run("meta/meta-llama-3-70b-instruct", { input: { prompt: "Explain TypeScript generics in 3 sentences.", max_tokens: 512, }, }); console.log(output); // Text response ``` **Why good:** Simple API, returns output directly, destructuring works for array outputs (images) ```typescript // BAD: Not pinning version in production const output = await replicate.run("community-user/experimental-model", { input: { prompt: "hello" }, }); ``` **Why bad:** Community models without version pinning can change behavior unexpectedly when authors push updates **See:** [examples/core.md](examples/core.md) for version pinning, `predictions.create()` + `wait()`, and progress callbacks --- ### Pattern 3: Streaming Use `replicate.stream()` for real-time SSE output from language models. ```typescript const stream = replicate.stream("meta/meta-llama-3-70b-instruct", { input: { prompt: "Write a short poem about TypeScript.", max_tokens: 512, }, }); for await (const event of stream) { if (event.event === "output") { process.stdout.write(event.data); } } ``` **Why good:** Progressive output for better UX, event-based with typed `event` and `data` fields ```typescript // BAD: Using replicate.run() for user-facing LLM output const output = await replicate.run("meta/meta-llama-3-70b-instruct", { input: { prompt: "Write a long essay..." }, }); // User waits for entire generation to complete before seeing anything ``` **Why bad:** No progressive feedback, user sees a blank screen for seconds **See:** [examples/streaming-webhooks.md](examples/streaming-webhooks.md) for event types, error handling, cancellation --- ### Pattern 4: Model Versioning Models are referenced as `owner/model` (latest version) or `owner/model:sha256hash` (pinned version). ```typescript // Development: use latest version for convenience const output = await replicate.run("stability-ai/sdxl", { input: { prompt: "a cat" }, }); // Production: pin to a specific version for reproducibility const VERSION_HASH = "39ed52f2a78e934b3ba6e2a89f5b1c712de7dfea535525255b1aa35c5565e08b"; const output = await replicate.run(`stability-ai/sdxl:${VERSION_HASH}`, { input: { prompt: "a cat" }, }); ``` **Why good:** Pinned version guarantees identical behavior, hash is immutable **See:** [examples/core.md](examples/core.md) for listing model versions, getting version details --- ### Pattern 5: File Handling Models that output files return `FileOutput` objects implementing `ReadableStream`. ```typescript import { writeFile } from "node:fs/promises"; const [output] = await replicate.run("black-forest-labs/flux-schnell", { input: { prompt: "a sunset over mountains" }, }); // FileOutput has .url() and .blob() methods console.log(output.url()); // Underlying URL // Save to disk const blob = await output.blob(); const buffer = Buffer.from(await blob.arrayBuffer()); await writeFile("./output.png", buffer); ``` ```typescript // File inputs: pass URLs, Buffers, or ReadStreams import { readFile } from "node:fs/promises"; const imageBuffer = await readFile("./input.png"); const output = await replicate.run("some-user/image-model", { input: { image: imageBuffer, // Auto-uploaded (max 100 MiB) }, }); ``` **Why good:** `FileOutput` is a `ReadableStream`, works with Node.js stream APIs, `.url()` for the underlying URL ```typescript // BAD: Treating file output as a plain URL string const [output] = await replicate.run("black-forest-labs/flux-schnell", { input: { prompt: "hello" }, }); const url = output; // WRONG: output is a FileOutput object, not a string ``` **Why bad:** `FileOutput` is an object, not a string -- use `.url()` to get the URL **See:** [examples/core.md](examples/core.md) for file uploads, large file handling, encoding strategies --- ### Pattern 6: Async Predictions with Webhooks Use `replicate.predictions.create()` for background jobs with webhook notifications. ```typescript const prediction = await replicate.predictions.create({ model: "owner/model", // OR version: "sha256hash" for pinned version input: { prompt: "a painting of a cat" }, webhook: "https://my.app/webhooks/replicate", webhook_events_filter: ["completed"], }); console.log(prediction.id); // Use to track status console.log(prediction.status); // "starting" ``` ```typescript // Webhook signature validation (CRITICAL for security) import { validateWebhook } from "replicate"; async function handleWebhook(request: Request): Promise<Response> { const secret = process.env.REPLICATE_WEBHOOK_SIGNING_SECRET; const isValid = await validateWebhook(request, secret); if (!isValid) { return new Response("Invalid signature", { status: 401 }); } const prediction = await request.json(); // Process prediction.output safely return new Response("OK", { status: 200 }); } ``` **Why good:** Decoupled processing, secure signature validation, filtered events reduce noise **See:** [examples/streaming-webhooks.md](examples/streaming-webhooks.md) for webhook event types, polling alternative --- ### Pattern 7: Deployments Deployments give you a private, fixed endpoint with custom hardware and scaling. ```typescript // Create a prediction on a deployment (no cold start if min_instances > 0) const prediction = await replicate.deployments.predictions.create( "my-org/my-deployment", { input: { prompt: "hello world" }, }, ); const result = await replicate.wait(prediction); console.log(result.output); ``` **Why good:** Predictable latency with `min_instances`, private endpoint, custom hardware selection **See:** [examples/deployments-training.md](examples/deployments-training.md) for creating/managing deployments, training API --- ### Pattern 8: Error Handling Catch API errors with status codes. The SDK auto-retries on 429 and 5xx errors (5 retries by default with exponential backoff). ```typescript try { const output = await replicate.run("owner/model", { input: { prompt: "hello" }, }); } catch (error) { if (error instanceof Error) { console.error(`Replicate error: ${error.message}`); // Check for specific HTTP status codes in the error if ("status" in error) { const status = (error as { status: number }).status; if (status === 401) { throw new Error("Invalid API token. Check REPLICATE_API_TOKEN."); } if (status === 422) { console.error("Invalid input parameters"); } if (status === 429) { console.error( "Rate limited -- SDK auto-retries (5 attempts) exhausted", ); } } } throw error; } ``` **Why good:** Checks error type, handles specific status codes, re-throws unexpected errors **See:** [examples/core.md](examples/core.md) for full error handling example with status code handling </patterns> --- <performance> ## Performance Optimization ### Cold Start Mitigation ``` Frequent model with varying load -> Use deployments with min_instances >= 1 One-off batch jobs -> Use predictions.create() with webhooks (no waiting) Popular public models -> Usually warm, replicate.run() is fine Custom/niche models -> Expect 30s-5min cold start on first run ``` ### Key Optimization Patterns - **Use deployments** for latency-sensitive applications -- set `min_instances: 1` to eliminate cold starts - **Use webhooks** instead of polling for async jobs -- reduces API calls and latency - **Batch file inputs as URLs** instead of uploading buffers -- avoids 100 MiB upload limit and is faster - **Pin model versions** in production -- avoids unexpected behavior changes and enables caching - **Use `replicate.stream()`** for LLMs -- progressive output feels faster than waiting for full completion - **Cancel unneeded predictions** with `replicate.predictions.cancel()` -- stops billing immediately </performance> --- <decision_framework> ## Decision Framework ### Which Execution Method to Use ``` Is this a user-facing LLM response? +-- YES -> Use replicate.stream() for real-time SSE output +-- NO -> Do you need the result immediately? +-- YES -> Use replicate.run() (blocks until complete) +-- NO -> Use replicate.predictions.create() + webhook +-- Need to poll instead? -> Use replicate.wait(prediction) ``` ### Model Reference Format ``` Are you in development/prototyping? +-- YES -> Use owner/model (latest version, convenient) +-- NO -> Are you in production? +-- YES -> Use owner/model:version_hash (pinned, reproducible) +-- Does the model change frequently? +-- YES -> Pin version, test updates explicitly +-- NO -> Either format works, prefer pinned ``` ### Deployments vs Direct API ``` Do you need consistent low latency? +-- YES -> Create a deployment with min_instances >= 1 +-- NO -> Do you need custom hardware (A100, H100)? +-- YES -> Create a deployment with specific hardware +-- NO -> Use replicate.run() / replicate.stream() directly (Replicate auto-allocates hardware) ``` ### When to Use This SDK vs Other AI SDKs ``` Are you running open-source models on serverless GPUs? +-- YES -> Use Replicate SDK +-- NO -> Are you calling proprietary APIs (OpenAI, Anthropic)? +-- YES -> Not this skill's scope -- use provider-specific SDKs +-- NO -> Do you need to switch between multiple providers? +-- YES -> Not this skill's scope -- use a unified provider SDK +-- NO -> Do you want to self-host models? +-- YES -> Not this skill's scope -- consider Cog or vLLM +-- NO -> Replicate SDK is appropriate ``` </decision_framework> --- <red_flags> ## RED FLAGS **High Priority Issues:** - Hardcoding `REPLICATE_API_TOKEN` in source code (security breach risk) - Treating `FileOutput` as a string (it is a `ReadableStream` object -- use `.url()` or `.blob()`) - Not validating webhook signatures with `validateWebhook()` (allows forged webhook payloads) - Using `replicate.run()` for long-running models in request handlers (blocks the response, can timeout) **Medium Priority Issues:** - Not pinning model versions in production (`owner/model` uses latest, which can change without notice) - Relying solely on default retry behavior for production (5 retries with exponential backoff may be too aggressive for some use cases) - Uploading large files as `Buffer` instead of hosting them at a URL (100 MiB limit on uploads) - Ignoring cold start latency for infrequently-used models (first request can take minutes) **Common Mistakes:** - Confusing `replicate.run()` (returns output directly) with `replicate.predictions.create()` (returns a prediction object with status/id) - Destructuring image output incorrectly: `const output = await replicate.run(...)` instead of `const [output] = await replicate.run(...)` (image models return arrays) - Using `replicate.stream()` with models that do not support streaming (only language models with SSE support) - Forgetting that `replicate.predictions.create()` accepts either a `version` hash or a `model` string (`owner/model`) -- use `version` for pinned reproducibility, `model` for latest-version convenience - Not consuming the async iterator from `replicate.stream()` (events are lost) **Gotchas & Edge Cases:** - Prediction inputs and outputs are automatically deleted after one hour -- persist outputs via webhooks or download immediately - The SDK auto-retries on 429 (rate limit) and 5xx errors -- 5 retries by default with exponential backoff. GET requests retry on 429 and 5xx; non-GET requests retry only on 429 - `replicate.stream()` returns `ServerSentEvent` objects with `.event` (`"output"`, `"error"`, `"done"`) and `.data` (string) properties - File uploads are limited to 100 MiB -- for larger files, host them at a URL and pass the URL as input - Browser usage is not supported -- the SDK requires a server-side environment (Node.js 18+, Bun, Deno, Cloudflare Workers) - `webhook_events_filter` accepts `["start", "output", "logs", "completed"]` -- use `["completed"]` unless you need intermediate status updates - The `Prefer: wait` header enables sync mode on the HTTP API (up to 60s), but `replicate.run()` already handles this automatically - Community models may disappear or change without warning -- pin versions and maintain fallbacks for critical workflows - `replicate.wait()` polls the API until the prediction completes -- use webhooks for production to avoid polling overhead - `FileOutput.url()` returns the underlying URL, but these URLs are temporary -- download or persist the file before it expires </red_flags> --- <critical_reminders> ## CRITICAL REMINDERS > **All code must follow project conventions in CLAUDE.md** (kebab-case, named exports, import ordering, `import type`, named constants) **(You MUST never hardcode API tokens -- always use environment variables via `process.env.REPLICATE_API_TOKEN`)** **(You MUST handle `FileOutput` objects for models that return files -- do not assume outputs are plain strings or URLs)** **(You MUST validate webhooks using `validateWebhook()` from the `replicate` package -- never trust unverified webhook payloads)** **(You MUST account for cold starts when running infrequently-used models -- use deployments for latency-sensitive applications)** **(You MUST specify model versions (`owner/model:version`) in production to ensure reproducible results -- unversioned references use the latest, which can change)** **Failure to follow these rules will produce insecure, unreliable, or unpredictable AI integrations.** </critical_reminders>
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.