api-messaging-webhooks
Webhook patterns — receiving, sending, signature verification, and retry logic
Install
npx skills add https://github.com/agents-inc/skills/tree/main/dist/plugins/api-messaging-webhooks/skills/api-messaging-webhooks
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install agents-inc-skills@llmmart
git clone https://github.com/agents-inc/skills.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole agents-inc/skills collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
Webhook Patterns
Quick Guide: Verify signatures with HMAC-SHA256 using
crypto.createHmac+crypto.timingSafeEqualon the raw body bytes -- never parsed JSON. Enforce idempotency by storing processed webhook IDs. Protect against replay attacks with timestamp validation. Return 200 immediately, process asynchronously. When sending, use exponential backoff with jitter and move exhausted retries to a dead letter queue.
<critical_requirements>
CRITICAL: Before Using This Skill
All code must follow project conventions in CLAUDE.md (kebab-case, named exports, import ordering,
import type, named constants)
(You MUST verify signatures against the RAW request body -- never parsed/re-serialized JSON)
(You MUST use crypto.timingSafeEqual for signature comparison -- never === which leaks timing information)
(You MUST return 2xx immediately and process webhooks asynchronously -- synchronous processing causes timeouts and duplicate deliveries)
(You MUST enforce idempotency by checking a stored webhook ID before processing -- retries WILL send the same event multiple times)
</critical_requirements>
Auto-detection: webhook, webhooks, HMAC, signature verification, createHmac, timingSafeEqual, webhook-signature, webhook-id, webhook-timestamp, idempotency key, replay attack, exponential backoff, dead letter queue, event routing, webhook handler, webhook endpoint, webhook delivery, webhook retry
When to use:
- Receiving webhooks from external providers (payment processors, version control, messaging platforms)
- Building a webhook-sending system to notify external consumers of events
- Implementing signature verification for incoming webhook payloads
- Adding retry logic with exponential backoff for outbound webhook delivery
- Routing webhook events to type-safe handlers by event type
When NOT to use:
- Real-time bidirectional communication (use WebSockets or Server-Sent Events)
- Internal service-to-service communication where both sides are trusted and co-deployed
- Simple polling scenarios where the consumer controls the fetch timing
Key patterns covered:
- HMAC-SHA256 signature verification with timing-safe comparison
- Replay attack protection with timestamp validation
- Idempotency via stored webhook IDs with TTL
- Type-safe event routing with discriminated unions
- Outbound webhook delivery with exponential backoff and jitter
- Dead letter queues for exhausted retries
- Raw body handling to preserve signature integrity
Detailed Resources:
- examples/core.md - Receiving, signature verification, idempotency, event routing
- examples/sending.md - Sending webhooks, retry logic, delivery tracking
- reference.md - Decision frameworks, header conventions, status code handling
<decision_framework>
Decision Framework
Receiving Webhooks
Incoming webhook request:
|
+-> Is the raw body available (not parsed)?
| +-> NO -> Fix your middleware to preserve raw body FIRST
| +-> YES -> Continue
|
+-> Does the provider send a signature header?
| +-> YES -> Verify HMAC signature with timing-safe comparison
| +-> NO -> Use IP allowlisting or mutual TLS instead
|
+-> Does the provider send a timestamp?
| +-> YES -> Validate timestamp within tolerance (e.g., 5 min)
| +-> NO -> Skip replay protection (rely on idempotency)
|
+-> Does the provider send a unique event ID?
| +-> YES -> Check idempotency store, skip if duplicate
| +-> NO -> Generate a hash of the payload as a dedup key
|
+-> Process asynchronously, return 200 immediately
Sending Webhooks
Need to notify external consumers of events?
|
+-> Sign every payload with HMAC-SHA256
+-> Include: webhook-id, webhook-timestamp, webhook-signature headers
+-> Deliver with retry on failure:
|
+-> Is it a 2xx response?
| +-> YES -> Delivery succeeded, done
|
+-> Is it 429, 503, or a network error?
| +-> YES -> Retry with exponential backoff + jitter
|
+-> Is it 400, 401, 404, 422?
| +-> YES -> Move to dead letter queue immediately (not retriable)
|
+-> Max retries exhausted?
+-> YES -> Move to dead letter queue
Status Code Quick Reference
| Status | Meaning | Action |
|---|---|---|
| 200-299 | Success | Mark delivered |
| 400 | Bad request | DLQ immediately |
| 401 | Unauthorized | DLQ immediately |
| 404 | Not found | DLQ immediately |
| 410 | Gone | Disable endpoint |
| 422 | Validation error | DLQ immediately |
| 429 | Rate limited | Retry with Retry-After |
| 500 | Server error | Retry with backoff |
| 502/503 | Unavailable | Retry with backoff |
| Timeout | No response | Retry with backoff |
| Connection refused | Unreachable | Retry with backoff |
</decision_framework>
<red_flags>
RED FLAGS
High Priority Issues:
- Verifying signature against parsed/re-serialized JSON instead of raw body -- JSON serialization is not deterministic (key order, whitespace), so the signature will fail
- Using
===for signature comparison instead ofcrypto.timingSafeEqual-- leaks timing information that allows attackers to incrementally guess signatures byte by byte - Processing webhooks synchronously before responding -- causes timeouts which trigger retries which cause duplicate processing
- No idempotency check -- retries from the producer will process the same event multiple times, causing duplicate orders, payments, etc.
- Retrying on 400/401/404 status codes -- these are permanent failures that retrying will never fix, wastes resources and delays DLQ investigation
Medium Priority Issues:
- No timestamp validation when the provider sends one -- enables replay attacks with captured payloads
- No maximum retry limit -- infinite retries on a dead endpoint waste resources permanently
- Retrying without jitter -- all failed deliveries retry at the same time (thundering herd)
- Using in-memory idempotency store in a multi-instance deployment -- each instance has a separate store, duplicates still occur
- Wildcard CORS on webhook endpoints -- webhook endpoints should not serve browser requests at all
Gotchas & Edge Cases:
crypto.timingSafeEqualthrows if buffers have different lengths -- always check.lengthequality first or convert both to the same encoding before comparison- Webhook middleware order matters -- raw body capture middleware must run BEFORE any JSON parsing middleware, or the raw bytes are lost
- Some providers prefix signatures (e.g.,
sha256=abc123) -- strip the prefix before comparison - Clock skew between producer and consumer can cause valid webhooks to fail timestamp validation -- use
Math.abs()for the time difference and allow 5 minutes tolerance - 410 Gone from a consumer means the endpoint is permanently removed -- disable the subscription instead of retrying
- Exponential backoff without a cap grows to astronomical delays -- always cap at a reasonable maximum (e.g., 1 hour)
- Webhook IDs should have a TTL in the idempotency store -- without TTL, storage grows unbounded
</red_flags>
<critical_reminders>
CRITICAL REMINDERS
All code must follow project conventions in CLAUDE.md
(You MUST verify signatures against the RAW request body -- never parsed/re-serialized JSON)
(You MUST use crypto.timingSafeEqual for signature comparison -- never === which leaks timing information)
(You MUST return 2xx immediately and process webhooks asynchronously -- synchronous processing causes timeouts and duplicate deliveries)
(You MUST enforce idempotency by checking a stored webhook ID before processing -- retries WILL send the same event multiple times)
Failure to follow these rules will cause signature verification failures, timing attack vulnerabilities, duplicate event processing, and lost webhook events.
</critical_reminders>
Files (skills)
-
examples
-
core.md 12.9 KB
# Webhooks - Core Receiving Patterns > Essential patterns for receiving webhooks securely. See [SKILL.md](../SKILL.md) for decision frameworks and [reference.md](../reference.md) for quick reference. **Additional Examples:** - [sending.md](sending.md) - Outbound delivery, retry logic, dead letter queues --- ## Pattern 1: HMAC-SHA256 Signature Verification ### Good Example - Timing-safe verification on raw body ```typescript import { createHmac, timingSafeEqual } from "node:crypto"; const SIGNATURE_ALGORITHM = "sha256"; const SIGNATURE_ENCODING = "hex"; /** * Verify an HMAC-SHA256 signature against the raw request body. * Uses timing-safe comparison to prevent timing attacks. */ function verifyWebhookSignature( rawBody: string, signatureHeader: string, secret: string, ): boolean { // Strip common prefixes (e.g., "sha256=" from GitHub, "v1=" from Stripe) const receivedSignature = signatureHeader.replace(/^(sha256=|v1=)/, ""); const expected = createHmac(SIGNATURE_ALGORITHM, secret) .update(rawBody) .digest(SIGNATURE_ENCODING); const expectedBuffer = Buffer.from(expected, "utf8"); const receivedBuffer = Buffer.from(receivedSignature, "utf8"); // timingSafeEqual throws if lengths differ -- check first if (expectedBuffer.length !== receivedBuffer.length) { return false; } return timingSafeEqual(expectedBuffer, receivedBuffer); } export { verifyWebhookSignature }; ``` **Why good:** operates on raw bytes (not parsed JSON), uses `timingSafeEqual` to prevent timing attacks, handles common signature prefixes, length check prevents `timingSafeEqual` from throwing ### Bad Example - Naive string comparison on parsed body ```typescript import { createHmac } from "node:crypto"; function verifySignature( parsedBody: object, signature: string, secret: string, ): boolean { // BAD: Re-serializing parsed JSON -- key order and whitespace may differ from original const body = JSON.stringify(parsedBody); const expected = createHmac("sha256", secret).update(body).digest("hex"); // BAD: String equality leaks timing information return expected === signature; } ``` **Why bad:** `JSON.stringify` does not guarantee the same byte sequence as the original payload (key order, whitespace, encoding), `===` comparison leaks timing information allowing attackers to guess signatures incrementally --- ## Pattern 2: Timestamp-Inclusive Signature Verification Many providers (Stripe, Standard Webhooks) include a timestamp in the signed content to prevent replay attacks. The signature covers `timestamp.payload`. ### Good Example - Signing timestamp + payload ```typescript import { createHmac, timingSafeEqual } from "node:crypto"; const SIGNATURE_ALGORITHM = "sha256"; const SIGNATURE_ENCODING = "hex"; const MAX_TIMESTAMP_AGE_SECONDS = 300; // 5 minutes const MS_PER_SECOND = 1000; interface VerificationResult { valid: boolean; reason?: string; } function verifyWebhookWithTimestamp( rawBody: string, signatureHeader: string, timestampHeader: string, secret: string, ): VerificationResult { // 1. Validate timestamp is recent const timestamp = parseInt(timestampHeader, 10); if (Number.isNaN(timestamp)) { return { valid: false, reason: "Invalid timestamp" }; } const currentTime = Math.floor(Date.now() / MS_PER_SECOND); if (Math.abs(currentTime - timestamp) > MAX_TIMESTAMP_AGE_SECONDS) { return { valid: false, reason: "Timestamp outside tolerance window" }; } // 2. Verify signature over "timestamp.body" (standard convention) const signedContent = `${timestampHeader}.${rawBody}`; const expected = createHmac(SIGNATURE_ALGORITHM, secret) .update(signedContent) .digest(SIGNATURE_ENCODING); const expectedBuffer = Buffer.from(expected, "utf8"); const receivedBuffer = Buffer.from(signatureHeader, "utf8"); if (expectedBuffer.length !== receivedBuffer.length) { return { valid: false, reason: "Signature mismatch" }; } if (!timingSafeEqual(expectedBuffer, receivedBuffer)) { return { valid: false, reason: "Signature mismatch" }; } return { valid: true }; } export { verifyWebhookWithTimestamp }; export type { VerificationResult }; ``` **Why good:** timestamp validation prevents replay attacks, signature covers the timestamp so an attacker cannot swap timestamps on old payloads, `Math.abs` handles clock skew in both directions, returns structured result with reason for logging --- ## Pattern 3: Raw Body Extraction Webhook signature verification requires the exact bytes sent by the producer. JSON parsing middleware destroys this by re-serializing with potentially different formatting. ### Good Example - Framework-agnostic raw body from Request ```typescript /** * Extract raw body from a standard Web API Request. * Works with any framework that uses the Request/Response standard * (most modern frameworks do). */ async function extractRawBody(request: Request): Promise<string> { return await request.text(); } ``` **Why good:** `request.text()` returns the body as the exact UTF-8 string received, no parsing or transformation ### Good Example - Preserving raw body alongside parsed JSON When you need both the raw body (for verification) and parsed JSON (for processing): ```typescript async function handleWebhook(request: Request): Promise<Response> { // Read raw body FIRST -- before any JSON parsing const rawBody = await request.text(); // Verify signature against raw body const signature = request.headers.get("webhook-signature") ?? ""; const timestamp = request.headers.get("webhook-timestamp") ?? ""; const secret = getWebhookSecret(); const result = verifyWebhookWithTimestamp( rawBody, signature, timestamp, secret, ); if (!result.valid) { return new Response(JSON.stringify({ error: result.reason }), { status: 401, }); } // Parse JSON only AFTER verification succeeds const event = JSON.parse(rawBody) as WebhookEvent; // Return 200 immediately, process asynchronously queueForProcessing(event); return new Response(null, { status: 200 }); } ``` **Why good:** raw body read before any parsing, verification before JSON parse (avoids wasted work on invalid payloads), immediate 200 response with async processing --- ## Pattern 4: Idempotent Webhook Processing Producers retry failed deliveries, so the same event arrives multiple times. Use the webhook ID as an idempotency key. ### Good Example - Idempotency with storage abstraction ```typescript const IDEMPOTENCY_TTL_DAYS = 7; const HOURS_PER_DAY = 24; const MINUTES_PER_HOUR = 60; const SECONDS_PER_MINUTE = 60; const MS_PER_SECOND = 1000; const IDEMPOTENCY_TTL_MS = IDEMPOTENCY_TTL_DAYS * HOURS_PER_DAY * MINUTES_PER_HOUR * SECONDS_PER_MINUTE * MS_PER_SECOND; // Storage interface -- implement with your chosen data store interface IdempotencyStore { has(key: string): Promise<boolean>; set(key: string, ttlMs: number): Promise<void>; } async function processWebhookIdempotently( webhookId: string, store: IdempotencyStore, handler: () => Promise<void>, ): Promise<{ status: "processed" | "duplicate" }> { // Check BEFORE processing const alreadyProcessed = await store.has(webhookId); if (alreadyProcessed) { return { status: "duplicate" }; } await handler(); // Mark AFTER successful processing await store.set(webhookId, IDEMPOTENCY_TTL_MS); return { status: "processed" }; } export { processWebhookIdempotently }; export type { IdempotencyStore }; ``` **Why good:** storage abstracted behind an interface (works with any data store), TTL prevents unbounded growth, checks before processing, marks after success, returns status for monitoring ### Bad Example - No idempotency, in-memory store in multi-instance deployment ```typescript const processed = new Set<string>(); async function handleWebhook(event: WebhookEvent): Promise<void> { // BAD: In-memory set is lost on restart and not shared across instances if (processed.has(event.id)) return; await processEvent(event); processed.add(event.id); // BAD: No TTL -- set grows unbounded forever } ``` **Why bad:** in-memory set is not shared across instances (each instance processes the same event), no TTL means unbounded memory growth, lost on restart --- ## Pattern 5: Type-Safe Event Routing Route events to typed handlers using a handler map. Adding a new event type to the union causes a compile error until a handler is added. ### Good Example - Discriminated union with handler map ```typescript import { z } from "zod"; // Define event schemas const OrderCreatedSchema = z.object({ type: z.literal("order.created"), data: z.object({ orderId: z.string(), total: z.number() }), }); const OrderCancelledSchema = z.object({ type: z.literal("order.cancelled"), data: z.object({ orderId: z.string(), reason: z.string() }), }); const PaymentCompletedSchema = z.object({ type: z.literal("payment.completed"), data: z.object({ paymentId: z.string(), amount: z.number() }), }); const WebhookEventSchema = z.discriminatedUnion("type", [ OrderCreatedSchema, OrderCancelledSchema, PaymentCompletedSchema, ]); type WebhookEvent = z.infer<typeof WebhookEventSchema>; // Handler map -- compile error if a case is missing type EventHandlerMap = { [E in WebhookEvent as E["type"]]: (data: E["data"]) => Promise<void>; }; const handlers: EventHandlerMap = { "order.created": async (data) => { // data is typed as { orderId: string; total: number } }, "order.cancelled": async (data) => { // data is typed as { orderId: string; reason: string } }, "payment.completed": async (data) => { // data is typed as { paymentId: string; amount: number } }, }; async function routeWebhookEvent(rawPayload: string): Promise<void> { const parsed = WebhookEventSchema.safeParse(JSON.parse(rawPayload)); if (!parsed.success) { // Unknown or malformed event -- log and skip (don't throw) console.warn("Unknown webhook event type, skipping:", parsed.error.message); return; } const event = parsed.data; const handler = handlers[event.type] as ( data: WebhookEvent["data"], ) => Promise<void>; await handler(event.data); } export { routeWebhookEvent, WebhookEventSchema }; ``` **Why good:** Zod validates the raw payload before routing, discriminated union ensures exhaustive handler coverage, each handler receives correctly typed `data`, unknown events are logged and skipped (not thrown) ### Bad Example - Switch statement without exhaustiveness ```typescript // BAD: No type safety, easy to forget a case async function routeEvent(event: { type: string; data: unknown; }): Promise<void> { switch (event.type) { case "order.created": await handleOrderCreated(event.data); break; case "order.cancelled": await handleOrderCancelled(event.data); break; // BAD: "payment.completed" handler forgotten -- silently dropped default: break; } } ``` **Why bad:** no compile-time enforcement of exhaustive handling, `data` is `unknown` requiring manual casting in every handler, adding a new event type does not cause a compile error --- ## Pattern 6: Complete Webhook Receiving Handler Combining all patterns into a single cohesive handler. ### Good Example - Full receiving pipeline ```typescript import type { IdempotencyStore } from "./idempotency.js"; const WEBHOOK_SECRET_ENV = "WEBHOOK_SECRET"; async function handleIncomingWebhook( request: Request, idempotencyStore: IdempotencyStore, ): Promise<Response> { // 1. Extract raw body before any parsing const rawBody = await request.text(); // 2. Extract headers const webhookId = request.headers.get("webhook-id"); const signature = request.headers.get("webhook-signature"); const timestamp = request.headers.get("webhook-timestamp"); if (!webhookId || !signature || !timestamp) { return new Response( JSON.stringify({ error: "Missing required webhook headers" }), { status: 400 }, ); } // 3. Verify signature (includes timestamp validation) const secret = process.env[WEBHOOK_SECRET_ENV]; if (!secret) throw new Error(`Missing ${WEBHOOK_SECRET_ENV} environment variable`); const verification = verifyWebhookWithTimestamp( rawBody, signature, timestamp, secret, ); if (!verification.valid) { return new Response(JSON.stringify({ error: "Invalid signature" }), { status: 401, }); } // 4. Idempotency check const result = await processWebhookIdempotently( webhookId, idempotencyStore, async () => { // 5. Route to typed handler await routeWebhookEvent(rawBody); }, ); if (result.status === "duplicate") { // Return 200 for duplicates -- the producer should stop retrying return new Response(null, { status: 200 }); } return new Response(null, { status: 200 }); } export { handleIncomingWebhook }; ``` **Why good:** follows the correct order (raw body -> headers -> verify -> idempotency -> route), returns 200 for duplicates (so the producer stops retrying), returns 400/401 for invalid requests, 200 for success -
sending.md 10 KB
# Webhooks - Sending Patterns > Patterns for sending webhooks with signing, retry, and delivery tracking. See [SKILL.md](../SKILL.md) for decision frameworks and [core.md](core.md) for receiving patterns. --- ## Pattern 1: Signing Outbound Webhooks Sign every outbound webhook with HMAC-SHA256. Include a unique ID and timestamp in the headers so the consumer can verify authenticity and prevent replays. ### Good Example - Signing with Standard Webhooks headers ```typescript import { createHmac, randomUUID } from "node:crypto"; const SIGNATURE_ALGORITHM = "sha256"; const SIGNATURE_ENCODING = "hex"; const MS_PER_SECOND = 1000; interface SignedWebhook { headers: Record<string, string>; body: string; } function signWebhookPayload(payload: object, secret: string): SignedWebhook { const body = JSON.stringify(payload); const webhookId = randomUUID(); const timestamp = Math.floor(Date.now() / MS_PER_SECOND).toString(); // Sign "timestamp.body" -- consumer verifies the same way const signedContent = `${timestamp}.${body}`; const signature = createHmac(SIGNATURE_ALGORITHM, secret) .update(signedContent) .digest(SIGNATURE_ENCODING); return { headers: { "webhook-id": webhookId, "webhook-timestamp": timestamp, "webhook-signature": signature, "content-type": "application/json", }, body, }; } export { signWebhookPayload }; export type { SignedWebhook }; ``` **Why good:** follows Standard Webhooks header convention (`webhook-id`, `webhook-timestamp`, `webhook-signature`), signs `timestamp.body` for replay protection, unique ID enables idempotency on the consumer side, JSON serialized once (same bytes for signing and sending) ### Bad Example - No signing, no metadata headers ```typescript // BAD: No authentication -- consumer cannot verify the request came from you async function sendWebhook(url: string, payload: object): Promise<void> { await fetch(url, { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify(payload), }); } ``` **Why bad:** no signature means the consumer cannot verify authenticity, no webhook-id means the consumer cannot deduplicate, no timestamp means no replay protection --- ## Pattern 2: Exponential Backoff with Jitter Retry failed deliveries with increasing delays. Jitter prevents all retries from hitting the server at the same instant. ### Good Example - Configurable retry with backoff ```typescript const DEFAULT_MAX_RETRIES = 5; const BASE_DELAY_MS = 1000; const MAX_DELAY_MS = 3_600_000; // 1 hour cap interface RetryConfig { maxRetries: number; baseDelayMs: number; maxDelayMs: number; } const DEFAULT_RETRY_CONFIG: RetryConfig = { maxRetries: DEFAULT_MAX_RETRIES, baseDelayMs: BASE_DELAY_MS, maxDelayMs: MAX_DELAY_MS, }; /** * Calculate delay for a given attempt number. * Formula: min(baseDelay * 2^attempt, maxDelay) + random jitter */ function calculateRetryDelay(attempt: number, config: RetryConfig): number { const exponential = config.baseDelayMs * Math.pow(2, attempt); const capped = Math.min(exponential, config.maxDelayMs); const jitter = Math.random() * config.baseDelayMs; return capped + jitter; } export { calculateRetryDelay, DEFAULT_RETRY_CONFIG }; export type { RetryConfig }; ``` **Why good:** exponential growth gives failing endpoints recovery time, cap prevents delays growing to days, jitter spreads retries across time (prevents thundering herd), configurable per-subscriber ### Retry Schedule Example With default config (`baseDelayMs: 1000`): | Attempt | Base Delay | + Jitter (0-1s) | Total Range | | ------- | ---------- | --------------- | ----------- | | 0 | 1s | 0-1s | 1-2s | | 1 | 2s | 0-1s | 2-3s | | 2 | 4s | 0-1s | 4-5s | | 3 | 8s | 0-1s | 8-9s | | 4 | 16s | 0-1s | 16-17s | --- ## Pattern 3: Status Code Classification Not all failures should be retried. Classify HTTP responses to decide the correct action. ### Good Example - Retriable vs permanent failure ```typescript type DeliveryAction = "success" | "retry" | "dead-letter" | "disable-endpoint"; const HTTP_OK_MIN = 200; const HTTP_OK_MAX = 299; const HTTP_BAD_REQUEST = 400; const HTTP_UNAUTHORIZED = 401; const HTTP_NOT_FOUND = 404; const HTTP_GONE = 410; const HTTP_UNPROCESSABLE = 422; const HTTP_RATE_LIMITED = 429; function classifyDeliveryResult(statusCode: number): DeliveryAction { // 2xx -- success if (statusCode >= HTTP_OK_MIN && statusCode <= HTTP_OK_MAX) { return "success"; } // Permanent client errors -- don't retry, move to DLQ const permanentFailures = [ HTTP_BAD_REQUEST, HTTP_UNAUTHORIZED, HTTP_NOT_FOUND, HTTP_UNPROCESSABLE, ]; if (permanentFailures.includes(statusCode)) { return "dead-letter"; } // 410 Gone -- the endpoint is permanently removed if (statusCode === HTTP_GONE) { return "disable-endpoint"; } // Everything else (429, 5xx, timeouts) -- retry with backoff return "retry"; } export { classifyDeliveryResult }; export type { DeliveryAction }; ``` **Why good:** separates retriable from permanent failures, 410 triggers endpoint disabling (not just DLQ), avoids wasting retries on requests that will never succeed --- ## Pattern 4: Complete Webhook Delivery Pipeline Combining signing, delivery, retry, and dead letter queue into a delivery function. ### Good Example - Full delivery pipeline ```typescript import type { RetryConfig } from "./retry.js"; import type { DeliveryAction } from "./status.js"; const DELIVERY_TIMEOUT_MS = 30_000; // 30 seconds interface DeliveryResult { webhookId: string; delivered: boolean; attempts: number; lastStatusCode?: number; lastError?: string; } interface DeadLetterEntry { webhookId: string; targetUrl: string; payload: string; signature: string; lastAttempt: Date; totalAttempts: number; lastStatusCode?: number; lastError: string; } // Storage interfaces -- implement with your chosen data store interface DeliveryStore { recordDelivery(result: DeliveryResult): Promise<void>; } interface DeadLetterStore { enqueue(entry: DeadLetterEntry): Promise<void>; } async function deliverWebhook( targetUrl: string, payload: object, secret: string, config: RetryConfig, deliveryStore: DeliveryStore, deadLetterStore: DeadLetterStore, ): Promise<DeliveryResult> { const signed = signWebhookPayload(payload, secret); const webhookId = signed.headers["webhook-id"]; let lastStatusCode: number | undefined; let lastError: string | undefined; for (let attempt = 0; attempt <= config.maxRetries; attempt++) { // Wait before retry (not on first attempt) if (attempt > 0) { const delay = calculateRetryDelay(attempt - 1, config); await sleep(delay); } try { const controller = new AbortController(); const timeout = setTimeout(() => controller.abort(), DELIVERY_TIMEOUT_MS); const response = await fetch(targetUrl, { method: "POST", headers: signed.headers, body: signed.body, signal: controller.signal, }); clearTimeout(timeout); lastStatusCode = response.status; const action = classifyDeliveryResult(response.status); if (action === "success") { const result: DeliveryResult = { webhookId, delivered: true, attempts: attempt + 1, lastStatusCode, }; await deliveryStore.recordDelivery(result); return result; } if (action === "dead-letter" || action === "disable-endpoint") { // Permanent failure -- stop retrying break; } // action === "retry" -- continue loop } catch (error) { lastError = error instanceof Error ? error.message : "Unknown error"; } } // All retries exhausted or permanent failure -- dead letter queue await deadLetterStore.enqueue({ webhookId, targetUrl, payload: signed.body, signature: signed.headers["webhook-signature"], lastAttempt: new Date(), totalAttempts: config.maxRetries + 1, lastStatusCode, lastError: lastError ?? `HTTP ${lastStatusCode}`, }); const result: DeliveryResult = { webhookId, delivered: false, attempts: config.maxRetries + 1, lastStatusCode, lastError, }; await deliveryStore.recordDelivery(result); return result; } function sleep(ms: number): Promise<void> { return new Promise((resolve) => setTimeout(resolve, ms)); } export { deliverWebhook }; export type { DeliveryResult, DeadLetterEntry, DeliveryStore, DeadLetterStore }; ``` **Why good:** complete pipeline with signing, timeout, retry classification, exponential backoff, dead letter queue, and delivery tracking. Storage abstracted behind interfaces. Permanent failures break out of the retry loop immediately. --- ## Pattern 5: Handling Rate Limits (429) When the consumer sends `Retry-After`, respect that value instead of using your own backoff. ### Good Example - Respecting Retry-After header ```typescript const FALLBACK_RETRY_AFTER_SECONDS = 60; const MS_PER_SECOND = 1000; /** * Extract retry delay from a 429 response. * Supports both "seconds" and "HTTP-date" formats per RFC 7231. */ function getRetryAfterMs(response: Response): number { const retryAfter = response.headers.get("retry-after"); if (!retryAfter) { return FALLBACK_RETRY_AFTER_SECONDS * MS_PER_SECOND; } // Try as integer (seconds) const seconds = parseInt(retryAfter, 10); if (!Number.isNaN(seconds)) { return seconds * MS_PER_SECOND; } // Try as HTTP-date const date = new Date(retryAfter); if (!Number.isNaN(date.getTime())) { const delayMs = date.getTime() - Date.now(); return Math.max(0, delayMs); } return FALLBACK_RETRY_AFTER_SECONDS * MS_PER_SECOND; } export { getRetryAfterMs }; ``` **Why good:** handles both RFC 7231 formats (seconds and HTTP-date), provides a sensible fallback when header is missing, `Math.max(0, ...)` prevents negative delays if the date is in the past
-
-
reference.md 2.5 KB
# Webhooks Quick Reference > Header conventions and implementation checklists. Referenced from [SKILL.md](SKILL.md). --- ## Webhook Header Conventions The [Standard Webhooks](https://github.com/standard-webhooks/standard-webhooks) specification defines these headers: | Header | Purpose | Example | | ------------------- | ---------------------------------------------- | ------------------ | | `webhook-id` | Unique event identifier for idempotency | `evt_1234abcd` | | `webhook-timestamp` | Unix timestamp (seconds) for replay protection | `1714556800` | | `webhook-signature` | HMAC-SHA256 signature of `timestamp.body` | `a1b2c3d4...` | | `content-type` | Always `application/json` | `application/json` | **Provider-specific variants:** | Provider | Signature Header | Timestamp | Format | | -------- | ----------------------- | ------------------- | -------------------------- | | Stripe | `stripe-signature` | In header (`t=...`) | `t=timestamp,v1=signature` | | GitHub | `x-hub-signature-256` | N/A | `sha256=signature` | | Shopify | `x-shopify-hmac-sha256` | N/A | Base64-encoded HMAC | | Twilio | `x-twilio-signature` | N/A | Base64-encoded HMAC | --- ## Security Checklist - [ ] Signatures verified against raw body (not parsed JSON) - [ ] Using `crypto.timingSafeEqual` (not `===`) - [ ] Buffer length checked before `timingSafeEqual` - [ ] Timestamp validated within tolerance window (5 min default) - [ ] Webhook secret stored in environment variable (not hardcoded) - [ ] Webhook endpoints not exposed via wildcard CORS ## Reliability Checklist - [ ] Idempotency enforced via stored webhook ID - [ ] Idempotency store has TTL (7-30 days) - [ ] 200 returned immediately, processing is async - [ ] Duplicate webhooks return 200 (not error) - [ ] Unknown event types logged and skipped (not thrown) ## Sending Checklist - [ ] Every payload signed with HMAC-SHA256 - [ ] `webhook-id`, `webhook-timestamp`, `webhook-signature` headers included - [ ] Retry with exponential backoff + jitter - [ ] Max retry cap (5-10 attempts) - [ ] Delay cap (e.g., 1 hour max) - [ ] 4xx permanent failures go to DLQ immediately - [ ] 429 respects `Retry-After` header - [ ] 410 disables the endpoint subscription - [ ] Dead letter queue preserves full event context -
SKILL.md 15.5 KB
--- name: api-messaging-webhooks description: Webhook patterns — receiving, sending, signature verification, and retry logic --- # Webhook Patterns > **Quick Guide:** Verify signatures with HMAC-SHA256 using `crypto.createHmac` + `crypto.timingSafeEqual` on the **raw body bytes** -- never parsed JSON. Enforce idempotency by storing processed webhook IDs. Protect against replay attacks with timestamp validation. Return 200 immediately, process asynchronously. When sending, use exponential backoff with jitter and move exhausted retries to a dead letter queue. --- <critical_requirements> ## CRITICAL: Before Using This Skill > **All code must follow project conventions in CLAUDE.md** (kebab-case, named exports, import ordering, `import type`, named constants) **(You MUST verify signatures against the RAW request body -- never parsed/re-serialized JSON)** **(You MUST use `crypto.timingSafeEqual` for signature comparison -- never `===` which leaks timing information)** **(You MUST return 2xx immediately and process webhooks asynchronously -- synchronous processing causes timeouts and duplicate deliveries)** **(You MUST enforce idempotency by checking a stored webhook ID before processing -- retries WILL send the same event multiple times)** </critical_requirements> --- **Auto-detection:** webhook, webhooks, HMAC, signature verification, createHmac, timingSafeEqual, webhook-signature, webhook-id, webhook-timestamp, idempotency key, replay attack, exponential backoff, dead letter queue, event routing, webhook handler, webhook endpoint, webhook delivery, webhook retry **When to use:** - Receiving webhooks from external providers (payment processors, version control, messaging platforms) - Building a webhook-sending system to notify external consumers of events - Implementing signature verification for incoming webhook payloads - Adding retry logic with exponential backoff for outbound webhook delivery - Routing webhook events to type-safe handlers by event type **When NOT to use:** - Real-time bidirectional communication (use WebSockets or Server-Sent Events) - Internal service-to-service communication where both sides are trusted and co-deployed - Simple polling scenarios where the consumer controls the fetch timing **Key patterns covered:** - HMAC-SHA256 signature verification with timing-safe comparison - Replay attack protection with timestamp validation - Idempotency via stored webhook IDs with TTL - Type-safe event routing with discriminated unions - Outbound webhook delivery with exponential backoff and jitter - Dead letter queues for exhausted retries - Raw body handling to preserve signature integrity **Detailed Resources:** - [examples/core.md](examples/core.md) - Receiving, signature verification, idempotency, event routing - [examples/sending.md](examples/sending.md) - Sending webhooks, retry logic, delivery tracking - [reference.md](reference.md) - Decision frameworks, header conventions, status code handling --- <philosophy> ## Philosophy Webhooks are HTTP callbacks -- a producer POSTs a payload to a consumer's URL when an event occurs. The fundamental challenge is **trust and reliability**: the consumer must verify the payload is authentic (signature verification), not replayed (timestamp validation), and not processed twice (idempotency). The producer must handle delivery failures gracefully (retries with backoff) and not lose events permanently (dead letter queues). **Core security principle:** The signature is computed over the exact bytes transmitted. Any transformation -- JSON parsing, re-serialization, whitespace normalization -- invalidates the signature. Always verify against the raw body. **Core reliability principle:** Networks are unreliable. The consumer should acknowledge receipt immediately (return 2xx) and process asynchronously. The producer should retry with exponential backoff and eventually move to a dead letter queue. **When to implement webhooks:** - Notifying external systems of events in near-real-time - Replacing polling for event-driven integrations - Building platform APIs that external developers consume **When NOT to implement webhooks:** - When polling is simpler and latency requirements are relaxed (minutes, not seconds) - For internal pub/sub where a message broker is more appropriate - When the consumer cannot expose a public HTTP endpoint </philosophy> --- <patterns> ## Core Patterns ### Pattern 1: HMAC-SHA256 Signature Verification The foundation of webhook security. The producer signs the payload with a shared secret; the consumer recomputes the signature and compares using timing-safe equality. ```typescript import { createHmac, timingSafeEqual } from "node:crypto"; const SIGNATURE_ALGORITHM = "sha256"; const SIGNATURE_ENCODING = "hex"; function verifySignature( rawBody: string, signature: string, secret: string, ): boolean { const expected = createHmac(SIGNATURE_ALGORITHM, secret) .update(rawBody) .digest(SIGNATURE_ENCODING); const expectedBuffer = Buffer.from(expected, "utf8"); const receivedBuffer = Buffer.from(signature, "utf8"); if (expectedBuffer.length !== receivedBuffer.length) return false; return timingSafeEqual(expectedBuffer, receivedBuffer); } ``` **Why good:** uses `timingSafeEqual` to prevent timing attacks, operates on raw body bytes, length check before comparison prevents `timingSafeEqual` throwing on mismatched lengths See [examples/core.md](examples/core.md) for the full handler with raw body extraction and error responses. --- ### Pattern 2: Replay Attack Protection Timestamp validation prevents attackers from re-sending captured webhook payloads. Reject payloads older than a tolerance window. ```typescript const MAX_TIMESTAMP_AGE_SECONDS = 300; // 5 minutes const MS_PER_SECOND = 1000; function isTimestampValid(timestampHeader: string): boolean { const webhookTime = parseInt(timestampHeader, 10); if (Number.isNaN(webhookTime)) return false; const currentTime = Math.floor(Date.now() / MS_PER_SECOND); return Math.abs(currentTime - webhookTime) <= MAX_TIMESTAMP_AGE_SECONDS; } ``` **Why good:** uses absolute difference to handle minor clock skew in both directions, rejects `NaN` timestamps, named constants for tolerance window **When to use:** When the webhook producer includes a timestamp header (most major providers do). Combine with signature verification -- sign the `timestamp.payload` concatenation so the timestamp itself is authenticated. See [examples/core.md](examples/core.md) for timestamp-inclusive signature verification. --- ### Pattern 3: Idempotency Webhook producers retry on failure, sending the same event multiple times. Store processed webhook IDs and skip duplicates. ```typescript const IDEMPOTENCY_TTL_DAYS = 7; async function processWebhookIdempotently( webhookId: string, handler: () => Promise<void>, ): Promise<{ status: "processed" | "duplicate" }> { const alreadyProcessed = await hasBeenProcessed(webhookId); if (alreadyProcessed) return { status: "duplicate" }; await handler(); await markAsProcessed(webhookId, IDEMPOTENCY_TTL_DAYS); return { status: "processed" }; } ``` **Why good:** checks before processing, stores with TTL to prevent unbounded storage growth, returns status for logging/monitoring **Storage options:** Any key-value store with TTL support works -- in-memory cache, database table, or dedicated cache service. See [examples/core.md](examples/core.md) for the complete idempotent webhook handler. --- ### Pattern 4: Type-Safe Event Routing Route webhook events to handlers using a discriminated union on the event type. The type system enforces exhaustive handling. ```typescript type WebhookEvent = | { type: "order.created"; data: { orderId: string; total: number } } | { type: "order.cancelled"; data: { orderId: string; reason: string } } | { type: "payment.completed"; data: { paymentId: string; amount: number } }; type EventHandlerMap = { [E in WebhookEvent as E["type"]]: (data: E["data"]) => Promise<void>; }; const handlers: EventHandlerMap = { "order.created": async (data) => { /* handle */ }, "order.cancelled": async (data) => { /* handle */ }, "payment.completed": async (data) => { /* handle */ }, }; async function routeEvent(event: WebhookEvent): Promise<void> { const handler = handlers[event.type] as ( data: WebhookEvent["data"], ) => Promise<void>; await handler(event.data); } ``` **Why good:** adding a new event type to the union causes a compile error until a handler is added, each handler receives correctly typed `data` See [examples/core.md](examples/core.md) for full event routing with Zod validation and unknown event handling. --- ### Pattern 5: Outbound Webhook Delivery with Retry When sending webhooks, sign the payload and deliver with exponential backoff. Classify failures by HTTP status to decide whether to retry. ```typescript const MAX_RETRIES = 5; const BASE_DELAY_MS = 1000; const MAX_DELAY_MS = 3600000; // 1 hour function calculateDelay(attempt: number): number { const exponential = BASE_DELAY_MS * Math.pow(2, attempt); const capped = Math.min(exponential, MAX_DELAY_MS); const jitter = Math.random() * BASE_DELAY_MS; return capped + jitter; } ``` **Why good:** exponential backoff gives failing endpoints recovery time, cap prevents absurd delays, jitter prevents thundering herd when many deliveries retry simultaneously See [examples/sending.md](examples/sending.md) for complete delivery with signing, status classification, and dead letter queues. --- ### Pattern 6: Dead Letter Queue After all retry attempts are exhausted, preserve the event for manual investigation rather than dropping it silently. ```typescript interface DeadLetterEntry { webhookId: string; targetUrl: string; payload: string; lastAttempt: Date; attempts: number; lastError: string; } ``` **Why good:** preserves full context for debugging, enables manual replay after the target recovers **When to move to DLQ immediately (no retry):** 400 Bad Request, 401 Unauthorized, 404 Not Found, 422 Unprocessable Entity -- these indicate a configuration or payload problem that retrying won't fix. Exception: 429 Too Many Requests should be retried with the `Retry-After` header value. See [examples/sending.md](examples/sending.md) for dead letter queue implementation and status code classification. </patterns> --- <decision_framework> ## Decision Framework ### Receiving Webhooks ``` Incoming webhook request: | +-> Is the raw body available (not parsed)? | +-> NO -> Fix your middleware to preserve raw body FIRST | +-> YES -> Continue | +-> Does the provider send a signature header? | +-> YES -> Verify HMAC signature with timing-safe comparison | +-> NO -> Use IP allowlisting or mutual TLS instead | +-> Does the provider send a timestamp? | +-> YES -> Validate timestamp within tolerance (e.g., 5 min) | +-> NO -> Skip replay protection (rely on idempotency) | +-> Does the provider send a unique event ID? | +-> YES -> Check idempotency store, skip if duplicate | +-> NO -> Generate a hash of the payload as a dedup key | +-> Process asynchronously, return 200 immediately ``` ### Sending Webhooks ``` Need to notify external consumers of events? | +-> Sign every payload with HMAC-SHA256 +-> Include: webhook-id, webhook-timestamp, webhook-signature headers +-> Deliver with retry on failure: | +-> Is it a 2xx response? | +-> YES -> Delivery succeeded, done | +-> Is it 429, 503, or a network error? | +-> YES -> Retry with exponential backoff + jitter | +-> Is it 400, 401, 404, 422? | +-> YES -> Move to dead letter queue immediately (not retriable) | +-> Max retries exhausted? +-> YES -> Move to dead letter queue ``` ### Status Code Quick Reference | Status | Meaning | Action | | ------------------ | ---------------- | ------------------------ | | 200-299 | Success | Mark delivered | | 400 | Bad request | DLQ immediately | | 401 | Unauthorized | DLQ immediately | | 404 | Not found | DLQ immediately | | 410 | Gone | Disable endpoint | | 422 | Validation error | DLQ immediately | | 429 | Rate limited | Retry with `Retry-After` | | 500 | Server error | Retry with backoff | | 502/503 | Unavailable | Retry with backoff | | Timeout | No response | Retry with backoff | | Connection refused | Unreachable | Retry with backoff | </decision_framework> --- <red_flags> ## RED FLAGS **High Priority Issues:** - Verifying signature against parsed/re-serialized JSON instead of raw body -- JSON serialization is not deterministic (key order, whitespace), so the signature will fail - Using `===` for signature comparison instead of `crypto.timingSafeEqual` -- leaks timing information that allows attackers to incrementally guess signatures byte by byte - Processing webhooks synchronously before responding -- causes timeouts which trigger retries which cause duplicate processing - No idempotency check -- retries from the producer will process the same event multiple times, causing duplicate orders, payments, etc. - Retrying on 400/401/404 status codes -- these are permanent failures that retrying will never fix, wastes resources and delays DLQ investigation **Medium Priority Issues:** - No timestamp validation when the provider sends one -- enables replay attacks with captured payloads - No maximum retry limit -- infinite retries on a dead endpoint waste resources permanently - Retrying without jitter -- all failed deliveries retry at the same time (thundering herd) - Using in-memory idempotency store in a multi-instance deployment -- each instance has a separate store, duplicates still occur - Wildcard CORS on webhook endpoints -- webhook endpoints should not serve browser requests at all **Gotchas & Edge Cases:** - `crypto.timingSafeEqual` throws if buffers have different lengths -- always check `.length` equality first or convert both to the same encoding before comparison - Webhook middleware order matters -- raw body capture middleware must run BEFORE any JSON parsing middleware, or the raw bytes are lost - Some providers prefix signatures (e.g., `sha256=abc123`) -- strip the prefix before comparison - Clock skew between producer and consumer can cause valid webhooks to fail timestamp validation -- use `Math.abs()` for the time difference and allow 5 minutes tolerance - 410 Gone from a consumer means the endpoint is permanently removed -- disable the subscription instead of retrying - Exponential backoff without a cap grows to astronomical delays -- always cap at a reasonable maximum (e.g., 1 hour) - Webhook IDs should have a TTL in the idempotency store -- without TTL, storage grows unbounded </red_flags> --- <critical_reminders> ## CRITICAL REMINDERS > **All code must follow project conventions in CLAUDE.md** **(You MUST verify signatures against the RAW request body -- never parsed/re-serialized JSON)** **(You MUST use `crypto.timingSafeEqual` for signature comparison -- never `===` which leaks timing information)** **(You MUST return 2xx immediately and process webhooks asynchronously -- synchronous processing causes timeouts and duplicate deliveries)** **(You MUST enforce idempotency by checking a stored webhook ID before processing -- retries WILL send the same event multiple times)** **Failure to follow these rules will cause signature verification failures, timing attack vulnerabilities, duplicate event processing, and lost webhook events.** </critical_reminders>
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.