# Observability - Axiom Monitors and Debugging

> APL queries for monitoring, alerting, and debugging. Back to [SKILL.md](../SKILL.md) | See [core.md](core.md) for foundational patterns.

**Prerequisites**: Understand [correlation-ids.md](correlation-ids.md) for request tracing context.

---

## Pattern: Axiom Monitors

### Error Rate Monitor

```apl
// Axiom APL - Error rate alert (> 1% errors)
['myapp-prod']
| where _time > ago(5m)
| summarize
    total = count(),
    errors = countif(level == "error")
| extend error_rate = todouble(errors) / todouble(total) * 100
| where error_rate > 1
```

### Latency Monitor

```apl
// Axiom APL - P95 latency alert (> 2 seconds)
['myapp-prod']
| where _time > ago(5m)
| where isnotnull(duration)
| summarize p95 = percentile(duration, 95)
| where p95 > 2000
```

### Specific Error Monitor

```apl
// Axiom APL - Database connection errors
['myapp-prod']
| where _time > ago(5m)
| where level == "error"
| where message contains "database" or message contains "connection"
| summarize count() by bin(_time, 1m)
| where count_ > 5
```

---

## Pattern: Monitor Configuration in Axiom

1. **Error Rate Alert**
   - Query: Error rate > 1% over 5 minutes
   - Severity: Warning
   - Notification: Slack #alerts channel

2. **High Latency Alert**
   - Query: P95 > 2000ms over 5 minutes
   - Severity: Warning
   - Notification: Slack #alerts channel

3. **Database Errors Alert**
   - Query: > 5 database errors in 1 minute
   - Severity: Critical
   - Notification: PagerDuty + Slack

---

## Pattern: Debugging Guide - Tracing a Request

### Step 1: Get Correlation ID

Find the correlation ID from:

- Response headers: `x-correlation-id`
- Error reports in Sentry
- User-reported issue (if client shows correlation ID)

### Step 2: Search Axiom

```apl
// Find all logs for a specific request
['myapp-prod']
| where correlationId == "abc-123-def-456"
| order by _time asc
```

### Step 3: Analyze Request Flow

```apl
// See request timeline with duration
['myapp-prod']
| where correlationId == "abc-123-def-456"
| project _time, level, operation, message, duration
| order by _time asc
```

### Step 4: Find Related Errors

```apl
// Get error details and stack traces
['myapp-prod']
| where correlationId == "abc-123-def-456"
| where level == "error"
| project _time, message, error, stack
```

### Step 5: Check Sentry

If an error was captured:

1. Search Sentry by correlation ID tag
2. Review stack trace with source maps
3. Check user context and breadcrumbs
4. View session replay if available

---

## Debugging Checklist

- [ ] Get correlation ID from error/user report
- [ ] Search Axiom for request timeline
- [ ] Identify where request failed
- [ ] Check error details and stack trace
- [ ] Review related database queries
- [ ] Check external service calls
- [ ] Verify user context in Sentry

---

_See [core.md](core.md) for foundational patterns: Log Levels, Structured Logging._

# Observability Core Examples

> Essential logging patterns for everyday use. Reference from [SKILL.md](../SKILL.md).

**Extended examples:**

- [correlation-ids.md](correlation-ids.md) - Middleware for request tracing
- [tracing.md](tracing.md) - OpenTelemetry spans and custom instrumentation
- [error-boundaries.md](error-boundaries.md) - React error boundaries with Sentry
- [sentry-config.md](sentry-config.md) - User context and error filtering
- [axiom.md](axiom.md) - Monitors, alerts, and debugging queries
- [performance.md](performance.md) - Query and API call tracking

---

## Pattern 1: Log Levels

### Good Example - Appropriate Log Levels

```typescript
import { logger } from "./logger";

// debug: Development-only, filtered in production
logger.debug({ userId, query }, "Search query parameters");

// info: Normal operation completed
logger.info({ userId, jobId }, "Job application submitted successfully");

// warn: Something unexpected but handled
logger.warn({ userId, retryCount: attempt }, "Payment retry attempt");

// error: Something that needs attention
logger.error({ userId, error: err.message }, "Payment processing failed");
```

**Why good:** Each level has a clear purpose, makes log filtering effective, alerts only trigger on actual issues

### Bad Example - Wrong Log Levels

```typescript
// Using error for non-errors
logger.error({ userId }, "User logged in"); // This is info, not error!

// Using info for debugging
logger.info({ allUserData }, "Debugging user state"); // This is debug

// No level distinction
console.log("Something happened"); // No structured data, no level
```

**Why bad:** Wrong levels make filtering useless, error alerts trigger for normal events, no structured data prevents searching

---

## Pattern 2: Structured Logging

### Good Example - Structured Logging with Context

```typescript
import { logger } from "./logger";

const OPERATION_CREATE_JOB = "job.create";
const OPERATION_SEARCH_JOBS = "job.search";

// Create child logger with request context
const log = logger.child({
  correlationId: req.headers["x-correlation-id"],
  service: "api",
  userId: req.user?.id,
});

// Log operation start
log.info({ operation: OPERATION_CREATE_JOB, jobTitle }, "Creating job listing");

// Log operation completion with duration
const startTime = performance.now();
const job = await createJob(data);
const duration = Math.round(performance.now() - startTime);

log.info(
  {
    operation: OPERATION_CREATE_JOB,
    jobId: job.id,
    duration,
  },
  "Job listing created successfully",
);
```

**Why good:** Child logger inherits context (no repetition), named constants for operations enable consistent filtering, duration tracking enables performance monitoring, correlationId links all logs from same request

### Bad Example - Unstructured Logging

```typescript
console.log("Creating job for user " + userId);
console.log("Job created: " + jobId);
```

**Why bad:** No structured data means can't search or filter, no correlation ID means can't trace request flow, string concatenation is not searchable, no duration tracking

---

_Extended examples: [correlation-ids.md](correlation-ids.md) | [tracing.md](tracing.md) | [error-boundaries.md](error-boundaries.md) | [sentry-config.md](sentry-config.md) | [axiom.md](axiom.md) | [performance.md](performance.md)_

# Observability - Correlation IDs

> Middleware patterns for request tracing across services. Back to [SKILL.md](../SKILL.md) | See [core.md](core.md) for foundational patterns.

**Prerequisites**: Understand [Pattern 2: Structured Logging](core.md#pattern-2-structured-logging) from core examples first.

---

## Pattern: Correlation ID Middleware

**Key logic (adapt to your HTTP framework's middleware API):**

```typescript
import { randomUUID } from "crypto";

const CORRELATION_ID_HEADER = "x-correlation-id";

/**
 * Middleware: extract or generate correlation ID per request.
 * Store it on the request context and set the response header.
 */
function correlationIdMiddleware(
  req: Request,
  res: Response,
  next: () => void,
) {
  const correlationId = req.headers.get(CORRELATION_ID_HEADER) ?? randomUUID();

  // Store on request context (mechanism varies by framework)
  req.correlationId = correlationId;

  // Echo back in response for client tracing
  res.headers.set(CORRELATION_ID_HEADER, correlationId);

  next();
}

// Helper to retrieve correlation ID in route handlers
function getCorrelationId(req: Request): string {
  return req.correlationId ?? "unknown";
}
```

---

## Pattern: Request Logger Middleware

```typescript
import { logger } from "./logger";

const HTTP_STATUS_BAD_REQUEST = 400;
const HTTP_STATUS_INTERNAL_ERROR = 500;

/**
 * Middleware: create a request-scoped child logger with correlation context,
 * then log start/completion with appropriate level and duration.
 */
function requestLoggerMiddleware(
  req: Request,
  res: Response,
  next: () => void,
) {
  const startTime = performance.now();

  // Create request-scoped logger
  const log = logger.child({
    correlationId: req.correlationId,
    method: req.method,
    path: new URL(req.url).pathname,
  });

  log.info("Request started");

  next();

  const duration = Math.round(performance.now() - startTime);
  const status = res.status;
  const logData = { duration, status };

  if (status >= HTTP_STATUS_INTERNAL_ERROR) {
    log.error(logData, "Request failed with server error");
  } else if (status >= HTTP_STATUS_BAD_REQUEST) {
    log.warn(logData, "Request completed with client error");
  } else {
    log.info(logData, "Request completed successfully");
  }
}
```

**Why good:** Correlation ID propagated through entire request lifecycle, child logger prevents repeating context, appropriate log level based on response status, duration tracking built-in

---

## Pattern: Using in Route Handlers

```typescript
import { logger } from "./logger";

// Inside a route handler (adapt to your framework)
async function createJobHandler(req: Request) {
  const log = logger.child({
    correlationId: req.correlationId,
    operation: "job.create",
  });

  log.info("Processing job creation request");

  const job = await createJob(data);

  log.info({ jobId: job.id }, "Job created successfully");
  return Response.json(job);
}
```

---

## Pattern: AsyncLocalStorage with Mixin (Modern Alternative)

For applications where you need automatic context injection without manually creating child loggers in every handler, use AsyncLocalStorage with Pino's `mixin` option.

**async-context.ts** - Shared storage for request context:

```typescript
import { AsyncLocalStorage } from "async_hooks";

interface RequestContext {
  correlationId: string;
  userId?: string;
  service?: string;
}

export const asyncContext = new AsyncLocalStorage<RequestContext>();

export const getRequestContext = (): RequestContext | undefined => {
  return asyncContext.getStore();
};
```

**logger.ts** - Logger with automatic context injection via mixin:

```typescript
import pino from "pino";
import { getRequestContext } from "./async-context";

export const logger = pino({
  level: process.env.LOG_LEVEL || "info",
  // Mixin automatically injects context into EVERY log call
  mixin() {
    const context = getRequestContext();
    if (context) {
      return {
        correlationId: context.correlationId,
        userId: context.userId,
        service: context.service,
      };
    }
    return {};
  },
});
```

**context-middleware.ts** - Middleware that wraps requests in async context:

```typescript
import { randomUUID } from "crypto";
import { asyncContext } from "./async-context";

const CORRELATION_ID_HEADER = "x-correlation-id";

// Adapt to your framework's middleware signature
function contextMiddleware(req: Request, res: Response, next: () => void) {
  const correlationId = req.headers.get(CORRELATION_ID_HEADER) ?? randomUUID();
  res.headers.set(CORRELATION_ID_HEADER, correlationId);

  // Run the rest of the request inside the async context
  asyncContext.run({ correlationId, service: "api" }, () => next());
}
```

**Usage (no child logger needed):**

```typescript
import { logger } from "./logger";

// Inside a route handler - correlationId is AUTOMATICALLY included via mixin
async function createJobHandler(req: Request) {
  logger.info({ operation: "job.create" }, "Processing job creation request");

  const job = await createJob(data);

  logger.info({ jobId: job.id }, "Job created successfully");
  return Response.json(job);
}
```

**Why good:** No manual child logger creation in every handler, context automatically propagates through entire async call chain (including nested function calls and database operations), cleaner code with less boilerplate, works across your entire codebase without passing logger instances

**When to use AsyncLocalStorage + mixin vs child loggers:**

- **AsyncLocalStorage + mixin**: When you want automatic context injection everywhere, larger applications with many nested function calls
- **Child loggers**: When you need explicit control, simpler applications, or want to add operation-specific context

---

_See [core.md](core.md) for foundational patterns: Log Levels, Structured Logging._

# Observability - Sentry Error Boundaries

> React error boundary patterns with Sentry integration. Back to [SKILL.md](../SKILL.md) | See [core.md](core.md) for foundational patterns.

**Related**: See [sentry-config.md](sentry-config.md) for user context and error filtering.

---

## Pattern: Sentry Built-in ErrorBoundary (Recommended)

Use Sentry's built-in ErrorBoundary component for automatic error capturing.

```typescript
"use client";

// Import from your framework-specific Sentry package (@sentry/react, @sentry/nextjs, etc.)
import * as Sentry from "@sentry/react";

export function JobsPage() {
  return (
    <div>
      <h1>Available Jobs</h1>
      <Sentry.ErrorBoundary
        fallback={({ error, resetError }) => (
          <div role="alert">
            <p>Failed to load jobs: {error.message}</p>
            <button onClick={resetError}>Retry</button>
          </div>
        )}
      >
        <JobList />
      </Sentry.ErrorBoundary>
    </div>
  );
}
```

**Why good:** Automatic error reporting to Sentry, built-in reset capability, properly typed fallback props

---

## Pattern: React 19+ Error Hooks (v8.6.0+)

React 19 exposes error hooks on `createRoot` and `hydrateRoot`. Use `Sentry.reactErrorHandler()` to capture errors at the root level.

```typescript
import { createRoot } from "react-dom/client";
import * as Sentry from "@sentry/react";

const container = document.getElementById("app");

const root = createRoot(container!, {
  // Errors NOT caught by any ErrorBoundary
  onUncaughtError: Sentry.reactErrorHandler((error, errorInfo) => {
    console.warn("Uncaught error", error, errorInfo.componentStack);
  }),
  // Errors caught by an ErrorBoundary
  onCaughtError: Sentry.reactErrorHandler(),
  // Automatic recovery errors
  onRecoverableError: Sentry.reactErrorHandler(),
});

root.render(<App />);
```

**Why good:** Captures errors at React root level before they propagate, works with React 19's new error handling model, provides centralized error processing

**Note:** For finer-grained control, use only `onUncaughtError` and `onRecoverableError` at root level, then use ErrorBoundary components for caught errors.

---

## Pattern: Custom Error Boundary with captureReactException (v9.8.0+)

For custom error boundaries, use `Sentry.captureReactException` instead of `captureException` to get proper React component stack traces.

```typescript
"use client";

import React from "react";
// Import from your framework-specific Sentry package
import * as Sentry from "@sentry/react";

import type { ReactNode } from "react";

interface ErrorBoundaryProps {
  children: ReactNode;
  fallback?: (props: { error: Error; reset: () => void }) => ReactNode;
}

interface ErrorBoundaryState {
  hasError: boolean;
  error: Error | null;
}

export class ErrorBoundary extends React.Component<ErrorBoundaryProps, ErrorBoundaryState> {
  constructor(props: ErrorBoundaryProps) {
    super(props);
    this.state = { hasError: false, error: null };
  }

  static getDerivedStateFromError(error: Error): ErrorBoundaryState {
    return { hasError: true, error };
  }

  componentDidCatch(error: Error, errorInfo: React.ErrorInfo) {
    // v9.8.0+: Use captureReactException for proper component stack
    Sentry.captureReactException(error, errorInfo);
  }

  reset = () => {
    this.setState({ hasError: false, error: null });
  };

  render() {
    if (this.state.hasError && this.state.error) {
      if (this.props.fallback) {
        return this.props.fallback({
          error: this.state.error,
          reset: this.reset,
        });
      }

      return <DefaultErrorFallback error={this.state.error} reset={this.reset} />;
    }

    return this.props.children;
  }
}

// Default fallback component
function DefaultErrorFallback({ error, reset }: { error: Error; reset: () => void }) {
  return (
    <div role="alert" className="error-boundary">
      <h2>Something went wrong</h2>
      <pre>{error.message}</pre>
      <button onClick={reset}>Try again</button>
    </div>
  );
}
```

**Why good:** `captureReactException` (v9.8.0+) provides better React-specific error context than generic `captureException`, properly captures component stack for debugging

---

## Pattern: Global Error Handler (SSR Framework)

Global error pages in SSR frameworks (e.g., `global-error.tsx`) can capture errors to Sentry.

```typescript
"use client";

import { useEffect } from "react";
// Import from your framework-specific Sentry package
import * as Sentry from "@sentry/react";

// Note: Some SSR frameworks require default exports for error pages
export default function GlobalError({
  error,
  reset,
}: {
  error: Error & { digest?: string };
  reset: () => void;
}) {
  useEffect(() => {
    // Report to Sentry
    Sentry.captureException(error);
  }, [error]);

  return (
    <html>
      <body>
        <div className="global-error">
          <h1>Something went wrong!</h1>
          <p>We've been notified and are working on a fix.</p>
          <button onClick={reset}>Try again</button>
        </div>
      </body>
    </html>
  );
}
```

**Why good:** Catches root-level errors that escape individual ErrorBoundary components, reports to Sentry automatically, provides user-facing recovery via reset button

---

## Pattern: Using Error Boundaries

```typescript
import { ErrorBoundary } from "./error-boundary";
import { JobList } from "./job-list";

export function JobsPage() {
  return (
    <div>
      <h1>Available Jobs</h1>
      <ErrorBoundary
        fallback={({ error, reset }) => (
          <div>
            <p>Failed to load jobs: {error.message}</p>
            <button onClick={reset}>Retry</button>
          </div>
        )}
      >
        <JobList />
      </ErrorBoundary>
    </div>
  );
}
```

---

## Sentry SDK Version Reference

| Feature                   | Minimum Version | Notes                          |
| ------------------------- | --------------- | ------------------------------ |
| `Sentry.ErrorBoundary`    | v7.x            | Built-in component             |
| `reactErrorHandler()`     | v8.6.0          | For React 19 hooks             |
| `captureReactException()` | v9.8.0          | For custom boundaries          |
| v10 baseline              | v10.0.0         | OTel v2, FID removed, see docs |

---

_See [core.md](core.md) for foundational patterns: Log Levels, Structured Logging._

# Observability - Performance Monitoring

> Tracking utilities for database queries and external API calls. Back to [SKILL.md](../SKILL.md) | See [core.md](core.md) for foundational patterns.

**Prerequisites**: Understand [tracing.md](tracing.md) for OpenTelemetry integration.

---

## Pattern: Performance Tracking Utilities

```typescript
import { logger } from "./logger";

const SLOW_QUERY_THRESHOLD_MS = 1000;
const SLOW_API_CALL_THRESHOLD_MS = 3000;

/**
 * Wrap database queries with performance tracking
 */
export async function trackedQuery<T>(
  operation: string,
  query: () => Promise<T>,
  log: ReturnType<typeof logger.child>,
): Promise<T> {
  const startTime = performance.now();

  try {
    const result = await query();
    const duration = Math.round(performance.now() - startTime);

    // Log slow queries as warnings
    if (duration > SLOW_QUERY_THRESHOLD_MS) {
      log.warn({ operation, duration }, "Slow database query detected");
    } else {
      log.debug({ operation, duration }, "Query completed");
    }

    return result;
  } catch (error) {
    const duration = Math.round(performance.now() - startTime);
    log.error(
      { operation, duration, error: (error as Error).message },
      "Query failed",
    );
    throw error;
  }
}

/**
 * Wrap external API calls with performance tracking
 */
export async function trackedApiCall<T>(
  service: string,
  endpoint: string,
  apiCall: () => Promise<T>,
  log: ReturnType<typeof logger.child>,
): Promise<T> {
  const startTime = performance.now();

  try {
    const result = await apiCall();
    const duration = Math.round(performance.now() - startTime);

    if (duration > SLOW_API_CALL_THRESHOLD_MS) {
      log.warn({ service, endpoint, duration }, "Slow external API call");
    } else {
      log.info({ service, endpoint, duration }, "External API call completed");
    }

    return result;
  } catch (error) {
    const duration = Math.round(performance.now() - startTime);
    log.error(
      { service, endpoint, duration, error: (error as Error).message },
      "External API call failed",
    );
    throw error;
  }
}
```

---

## Pattern: Using Performance Wrappers

```typescript
import { trackedQuery, trackedApiCall } from "./performance";

const MAX_JOBS = 100;

async function getJobsWithExternalData(log: Logger) {
  // Track database query
  const dbJobs = await trackedQuery(
    "jobs.findMany",
    () => db.query.jobs.findMany({ limit: MAX_JOBS }),
    log,
  );

  // Track external API call
  const enrichedData = await trackedApiCall(
    "enrichment-api",
    "/api/v1/enrich",
    () =>
      fetch("https://api.enrichment.com/enrich", {
        body: JSON.stringify(dbJobs),
      }),
    log,
  );

  return enrichedData;
}
```

**Why good:** Named constants for thresholds, automatic slow operation detection, consistent error logging, duration tracking for all operations

---

_See [core.md](core.md) for foundational patterns: Log Levels, Structured Logging._

# Observability - Sentry Configuration

> User context, error filtering, and client configuration. Back to [SKILL.md](../SKILL.md) | See [core.md](core.md) for foundational patterns.

**Related**: See [error-boundaries.md](error-boundaries.md) for React error boundary components.

**SDK Version Notes:**

- v9.x: `captureUserFeedback()` renamed to `captureFeedback()`, `comments` field renamed to `message`
- v9.x: `enableTracing` option removed, use `tracesSampleRate` directly (no longer needs boolean flag)
- v9.x: `getCurrentHub()` removed, use `Sentry.getClient()`, `Sentry.getCurrentScope()`, or top-level functions
- v9.x: `beforeSendSpan` cannot return `null` to drop spans, use integrations instead
- v9.x: `hideSourceMaps` option removed (SDK emits hidden source maps by default)
- v9.x: Metrics API completely removed (was deprecated in v8)
- v9.x: Minimum browser support raised to ES2020 (Chrome 80+, Safari 14+, Firefox 74+)
- v10.x: `BaseClient` removed, use `Client`; `hasTracingEnabled()` renamed `hasSpansEnabled()`
- v10.x: OpenTelemetry dependencies bumped to v2; FID web vital reporting removed (use INP)
- v10.x: `_experiments.enableLogs` moved to top-level `enableLogs`
- v10.x: IP address inference now gated by `sendDefaultPii` option

---

## Pattern: Setting User Context

```typescript
// Import from your framework-specific Sentry package
import * as Sentry from "@sentry/react";

import type { User } from "../types/user";

/**
 * Call after successful authentication
 */
export function setSentryUser(user: User) {
  Sentry.setUser({
    id: user.id,
    email: user.email,
    username: user.name,
    // Add custom attributes
    subscription: user.subscriptionTier,
  });
}

/**
 * Call on logout
 */
export function clearSentryUser() {
  Sentry.setUser(null);
}

/**
 * Add additional context for the current scope
 */
export function setSentryContext(key: string, data: Record<string, unknown>) {
  Sentry.setContext(key, data);
}
```

---

## Pattern: Using in Auth Flow

```typescript
import { useEffect } from "react";
import { useSession } from "./auth";
import { setSentryUser, clearSentryUser } from "./sentry-user";

export function AuthProvider({ children }: { children: React.ReactNode }) {
  const { user, isAuthenticated } = useSession();

  useEffect(() => {
    if (isAuthenticated && user) {
      setSentryUser(user);
    } else {
      clearSentryUser();
    }
  }, [isAuthenticated, user]);

  return children;
}
```

**Why good:** User context helps identify affected users when debugging, cleared on logout for privacy, additional context can be added per-feature

---

## Pattern: Error Filtering Configuration

```typescript
// Import from your framework-specific Sentry package
import * as Sentry from "@sentry/react";

const IGNORED_ERROR_PATTERNS = [
  // User-initiated cancellations
  "AbortError",
  "cancelled",
  "user aborted",

  // Network issues (user's problem, not ours)
  "Failed to fetch",
  "NetworkError",
  "Load failed",

  // Expected authentication errors
  "Unauthorized",
  "Session expired",

  // Browser extensions causing issues
  "ResizeObserver loop",
  "Script error.",
];

const HTTP_STATUS_NOT_FOUND = 404;
const HTTP_STATUS_UNAUTHORIZED = 401;
const HTTP_STATUS_FORBIDDEN = 403;

Sentry.init({
  dsn: process.env.SENTRY_DSN,

  beforeSend(event, hint) {
    const error = hint.originalException;

    // Skip expected errors by message pattern
    if (error instanceof Error) {
      const isIgnored = IGNORED_ERROR_PATTERNS.some((pattern) =>
        error.message.toLowerCase().includes(pattern.toLowerCase()),
      );

      if (isIgnored) {
        return null;
      }
    }

    // Skip expected HTTP errors
    if (event.contexts?.response?.status_code) {
      const status = event.contexts.response.status_code;
      if (
        status === HTTP_STATUS_NOT_FOUND ||
        status === HTTP_STATUS_UNAUTHORIZED ||
        status === HTTP_STATUS_FORBIDDEN
      ) {
        return null;
      }
    }

    return event;
  },

  // Filter breadcrumbs to reduce noise
  beforeBreadcrumb(breadcrumb) {
    // Skip console.log breadcrumbs (too noisy)
    if (breadcrumb.category === "console" && breadcrumb.level === "log") {
      return null;
    }
    return breadcrumb;
  },
});
```

**Why good:** Named constants for HTTP status codes, configurable ignore patterns, filters both errors and noisy breadcrumbs, preserves unexpected errors for alerting

---

## Anti-Pattern: No Filtering

```typescript
Sentry.init({
  dsn: process.env.SENTRY_DSN,
  // No beforeSend - every error goes to Sentry
});
```

**Why bad:** Expected errors (404s, user cancellations) waste Sentry quota, alerts trigger for non-issues, real errors buried in noise

---

_See [core.md](core.md) for foundational patterns: Log Levels, Structured Logging._

# Observability - OpenTelemetry Tracing

> Custom spans and instrumentation for performance debugging. Back to [SKILL.md](../SKILL.md) | See [core.md](core.md) for foundational patterns.

**Prerequisites**: Understand [correlation-ids.md](correlation-ids.md) for request context propagation.

---

## Pattern: Tracing Utilities

```typescript
import { trace, SpanStatusCode, type Span } from "@opentelemetry/api";

const tracer = trace.getTracer("api");

/**
 * Wrap an async operation in a traced span
 */
export async function withSpan<T>(
  spanName: string,
  attributes: Record<string, string | number | boolean>,
  fn: (span: Span) => Promise<T>,
): Promise<T> {
  return tracer.startActiveSpan(spanName, { attributes }, async (span) => {
    try {
      const result = await fn(span);
      span.setStatus({ code: SpanStatusCode.OK });
      return result;
    } catch (error) {
      span.setStatus({
        code: SpanStatusCode.ERROR,
        message: error instanceof Error ? error.message : "Unknown error",
      });
      span.recordException(error as Error);
      throw error;
    } finally {
      span.end();
    }
  });
}

/**
 * Create a simple span for synchronous operations
 */
export function createSpan(
  spanName: string,
  attributes?: Record<string, string | number | boolean>,
): Span {
  return tracer.startSpan(spanName, { attributes });
}
```

---

## Pattern: Using Custom Spans

```typescript
import { withSpan } from "./tracing";

const OPERATION_DB_QUERY = "db.query";

async function getJobWithCompany(jobId: string) {
  return withSpan(
    OPERATION_DB_QUERY,
    {
      "db.operation": "findFirst",
      "db.table": "jobs",
      "job.id": jobId,
    },
    async (span) => {
      // Use your ORM/query builder
      const result = await db.query.jobs.findFirst({ where: { id: jobId } });

      span.setAttribute("db.rows_returned", result ? 1 : 0);
      return result;
    },
  );
}
```

**Why good:** Custom spans provide detailed performance breakdown, attributes enable filtering in Axiom, error handling records exceptions for debugging, spans automatically inherit parent context

---

_See [core.md](core.md) for foundational patterns: Log Levels, Structured Logging._