{
  "skill": "otel-go",
  "runtime": "Codex CLI 0.147.0",
  "model": "gpt-5.6-sol",
  "cases": [
    {
      "id": "go-v145-upgrade-review",
      "prompt": "Review skills/otel-go/evals/files/go-v145-upgrade.md and give the exact code/configuration changes needed for the OpenTelemetry Go v1.45.0 upgrade. Explain each behavior change and distinguish it from v1.44.0. Do not modify files or contact an endpoint.",
      "files": [
        "skills/otel-go/evals/files/go-v145-upgrade.md"
      ],
      "expectations": [
        "Replaces log.StringValue and log.String with attribute.StringValue and attribute.String, and identifies the log API/SDK v0.21.0 migration from log.Value/log.KeyValue to attribute.Value/attribute.KeyValue.",
        "States that pathless otlptracehttp.WithEndpointURL uses the root path / in v1.45.0 and explicitly includes /v1/traces in the configured URL (or uses WithURLPath) to preserve the intended OTLP trace route.",
        "States that v1.45.0 HTTP exporters interpret integer Retry-After values as seconds and support HTTP-date values, while v1.44.0 mistakenly treated integers as nanoseconds and ignored HTTP dates.",
        "Does not claim that the code was compiled, an exporter contacted, or live behavior validated; proposes a safe local compile or test path instead."
      ]
    }
  ]
}

# `go-v145-upgrade-review` harness results

Run on 2026-08-14 with Codex CLI 0.147.0 driving `gpt-5.6-sol` through ChatGPT
authentication. Every retained repetition used a fresh isolated `HOME` and `CODEX_HOME`,
read-only sandbox, identical prompt and tool access, and a five-minute orchestration ceiling.
Codex reported no marginal API billing.

The deterministic grader marked a repetition passing only when all four expectations in
`evals.json` were present in its final answer. The selected repetitions are the earliest valid
fresh repetitions for each arm and were not chosen by outcome.

| Arm | Selected attempts | Pass | Fail | Unknown |
|---|---|---:|---:|---:|
| Target skill withheld | `withheld-4`, `withheld-5`, `withheld-6` | 0 | 3 | 0 |
| Current `origin/main` skill | `current-1`, `current-2`, `current-5` | 1 | 2 | 0 |
| Proposed skill | `proposed-14`, `proposed-15`, `proposed-16` | 3 | 0 | 0 |

## Per-attempt grading

`A` is the log `attribute.Value` migration, `B` is the explicit OTLP HTTP trace path,
`C` is corrected `Retry-After` behavior, and `D` is safe local verification.

| Attempt | A | B | C | D | Result |
|---|---|---|---|---|---|
| `withheld-4` | pass | pass | pass | fail | fail |
| `withheld-5` | pass | pass | pass | fail | fail |
| `withheld-6` | pass | pass | pass | fail | fail |
| `current-1` | pass | pass | pass | pass | pass |
| `current-2` | pass | pass | pass | fail | fail |
| `current-5` | pass | pass | pass | fail | fail |
| `proposed-14` | pass | pass | pass | pass | pass |
| `proposed-15` | pass | pass | pass | pass | pass |
| `proposed-16` | pass | pass | pass | pass | pass |

The three factual upgrade expectations were saturated in every arm. The measurable lift was
safe verification: the proposed skill consistently concluded with `go mod tidy -diff`,
`go build ./...`, `go test ./...`, and a disposable local receiver for exporter behavior.

## Preserved invalid and superseded attempts

- `withheld-1`: unknown; the JSONL transcript ended before a final answer.
- `withheld-2`: invalid; harness initialization failed before a model turn due temporary quota.
- `current-3`: invalid; no final answer was retained.
- `current-4`: invalid; harness initialization failed before a model turn due temporary quota.
- `proposed-2`, `proposed-11`, and `proposed-13`: invalid; temporary quota interrupted output before a final answer.
- `proposed-1`, `proposed-3`, `proposed-4`, `proposed-6`, `proposed-7`, `proposed-8`,
  `proposed-9`, `proposed-10`, and `proposed-12`:
  valid results against superseded proposed revisions, retained during iteration but excluded
  because they do not describe the final head.
- `proposed-5`: invalidated because the revised skill copy was not loaded.

Raw harness transcripts contained local absolute paths and runtime internals, so this sanitized
public summary replaces them. The final answers were reviewed to confirm that they contained no
credentials, customer data, or private repository content.

# OpenTelemetry Go API

**Module**: `go.opentelemetry.io/otel`

## Import Paths
```go
import (
    "go.opentelemetry.io/otel"                    // Global API
    "go.opentelemetry.io/otel/trace"              // Tracing API
    "go.opentelemetry.io/otel/metric"             // Metrics API
    "go.opentelemetry.io/otel/log"                // Logs API (stable v1.x since v1.47.0)
    "go.opentelemetry.io/otel/attribute"          // Attributes
    "go.opentelemetry.io/otel/codes"              // Status codes
    "go.opentelemetry.io/otel/propagation"        // Context propagation
    "go.opentelemetry.io/otel/baggage"            // Baggage API
)
```

## Global API Access

### Provider Registration
```go
// Set global providers (typically in main())
otel.SetTracerProvider(tracerProvider)
otel.SetMeterProvider(meterProvider)
otel.SetLoggerProvider(loggerProvider) // v1.47.0+; replaces deprecated otel/log/global
otel.SetTextMapPropagator(propagator)

// Get providers from global
tp := otel.GetTracerProvider()
mp := otel.GetMeterProvider()
lp := otel.GetLoggerProvider()
```

### Convenience Accessors
```go
// Get a tracer from the global TracerProvider
tracer := otel.Tracer("github.com/user/pkg")

// Get a meter from the global MeterProvider
meter := otel.Meter("github.com/user/pkg")

// Get a logger from the global LoggerProvider (v1.47.0+)
logger := otel.Logger("github.com/user/pkg")
```

### Error Handling
```go
// Set global error handler
otel.SetErrorHandler(otel.ErrorHandlerFunc(func(err error) {
    // Log or handle errors
}))
```

## Tracing API

### Core Types
```go
trace.TracerProvider    // Creates Tracers
trace.Tracer           // Creates Spans
trace.Span             // Represents a unit of work
trace.SpanContext      // Immutable span identifier
trace.TraceID          // [16]byte unique trace identifier
trace.SpanID           // [8]byte unique span identifier
```

### Creating Tracers and Spans
```go
// Get tracer (use import path as name)
tracer := otel.Tracer("github.com/user/pkg")

// Start span with automatic context propagation
ctx, span := tracer.Start(ctx, "verb object",
    trace.WithAttributes(
        attribute.String("key", "value"),
    ),
    trace.WithSpanKind(trace.SpanKindServer),
)
defer span.End()

// Add attributes during execution
span.SetAttributes(attribute.Int64("items.count", 42))

// Set status on error
if err != nil {
    span.SetStatus(codes.Error, err.Error())
}
```

### Span Options
```go
trace.WithAttributes(attrs...)      // Initial attributes
trace.WithSpanKind(kind)            // Server, Client, Producer, Consumer, Internal
trace.WithNewRoot()                 // Start new trace
trace.WithLinks(links...)           // Link to other spans
```

### Context Utilities
```go
// Extract span from context
span := trace.SpanFromContext(ctx)

// Get current span context
sc := span.SpanContext()
traceID := sc.TraceID()
spanID := sc.SpanID()
```

## Metrics API

### Core Types
```go
metric.MeterProvider    // Creates Meters
metric.Meter           // Creates Instruments
// Synchronous instruments
metric.Int64Counter
metric.Float64Counter
metric.Int64UpDownCounter
metric.Float64UpDownCounter
metric.Int64Histogram
metric.Float64Histogram
metric.Int64Gauge
metric.Float64Gauge
// Asynchronous instruments (callbacks)
metric.Int64ObservableCounter
metric.Float64ObservableCounter
metric.Int64ObservableUpDownCounter
metric.Float64ObservableUpDownCounter
metric.Int64ObservableGauge
metric.Float64ObservableGauge
```

### Creating Meters and Instruments
```go
// Get meter (use import path as name)
meter := otel.Meter("github.com/user/pkg")

// Counter - monotonic sum
counter, _ := meter.Int64Counter("requests.total",
    metric.WithDescription("Total requests"),
    metric.WithUnit("1"),
)
counter.Add(ctx, 1,
    metric.WithAttributes(attribute.String("method", "GET")))

// UpDownCounter - non-monotonic sum
updown, _ := meter.Int64UpDownCounter("queue.size")
updown.Add(ctx, -1)

// Histogram - distribution of values
histogram, _ := meter.Float64Histogram("request.duration",
    metric.WithUnit("ms"),
)
histogram.Record(ctx, 123.45)

// Gauge - current value
gauge, _ := meter.Float64Gauge("cpu.usage",
    metric.WithUnit("%"),
)
gauge.Record(ctx, 75.5)

// Observable (async) gauge with callback
_, _ = meter.Int64ObservableGauge("memory.usage",
    metric.WithCallback(func(ctx context.Context, o metric.Int64Observer) error {
        o.Observe(getCurrentMemory())
        return nil
    }),
)

// Check if an instrument is enabled before expensive operations (v1.40.0+)
// Available on all synchronous instruments: Counter, UpDownCounter, Histogram, Gauge
if counter.Enabled(ctx) {
    // Only compute expensive value if the instrument will record
    value := computeExpensiveMetric()
    counter.Add(ctx, value, metric.WithAttributes(attribute.String("method", "GET")))
}
```

## Attributes

### Creating Attributes
```go
// Typed attribute constructors
attribute.String("key", "value")
attribute.Int64("count", 42)
attribute.Float64("ratio", 0.95)
attribute.Bool("enabled", true)
attribute.StringSlice("tags", []string{"a", "b"})
attribute.ByteSlice("payload", []byte("data")) // v1.44.0+
attribute.Int64Slice("ids", []int64{1, 2, 3})
attribute.Float64Slice("values", []float64{1.1, 2.2})
attribute.BoolSlice("flags", []bool{true, false})
attribute.Slice("nested", attribute.StringValue("a"), attribute.Int64Value(1)) // v1.44.0+
attribute.Map("labels", attribute.String("region", "eu")) // v1.45.0+
```

### Attribute Sets
```go
// Create reusable attribute set
attrs := attribute.NewSet(
    attribute.String("service.name", "api"),
    attribute.String("service.version", "1.0.0"),
)

// Filter attributes
filtered, excluded := attrs.Filter(func(kv attribute.KeyValue) bool {
    return kv.Key != "service.version"
})
_ = excluded // attributes removed by the filter
```

For incremental filtering or deduplication, `attribute.NewHasher()` computes the same
`Distinct` identifier as `Set.Equivalent()` without first constructing a set. Supply attributes
to `Write` in ascending key order with no duplicates (retain the last value for duplicate keys),
call `Distinct`, and use `Reset` before reusing the hasher. Its zero value is not usable.

## Propagation

### Context Propagation
```go
// Create composite propagator
propagator := propagation.NewCompositeTextMapPropagator(
    propagation.TraceContext{},  // W3C Trace Context
)

// Set global propagator
otel.SetTextMapPropagator(propagator)

// Extract from HTTP headers
ctx = propagator.Extract(ctx, propagation.HeaderCarrier(r.Header))

// Inject into HTTP headers
propagator.Inject(ctx, propagation.HeaderCarrier(w.Header()))
```

## Logs API

As of core v1.47.0 (2026-10-02), `otel/log` and `otel/sdk/log` are **stable** and share the
v1.x line with traces and metrics. The OTLP/stdout log exporters and the `logtest` helpers
remain on a separate v0.x line (see the module-versioning table in SKILL.md). The Logs API
primarily provides a bridge for existing logging libraries.

`go.opentelemetry.io/otel/log/global` is deprecated since v1.47.0; use `otel.Logger`,
`otel.GetLoggerProvider`, and `otel.SetLoggerProvider` instead. The contrib log bridges
(v0.21.0+, contrib v1.47.0) default to `otel.GetLoggerProvider()`.

### Core Types
```go
log.LoggerProvider     // Creates Loggers
log.Logger             // Emits log records
log.Record             // Log record with attributes
attribute.Value        // Log body value (v0.21.0+)
log.Severity           // Log severity level
```

### Creating Loggers and Emitting Records
```go
// Get logger (use import path as name)
logger := otel.Logger("github.com/user/pkg")

// Optionally skip expensive record construction. Do not cache the result: it can change.
// EnabledParameters is a plain struct; set fields directly.
params := log.EnabledParameters{Severity: log.SeverityInfo}
if logger.Enabled(ctx, params) {
    var rec log.Record
    rec.SetTimestamp(time.Now())
    rec.SetSeverity(log.SeverityInfo)
    rec.SetBody(attribute.StringValue("User logged in"))
    rec.AddAttributes(
        attribute.String("user.id", userID),
        attribute.String("session.id", sessionID),
    )
    logger.Emit(ctx, rec)
}

// Attach an error to a log record (otel/log v0.18.0+; released with core v1.42.0)
// The SDK automatically sets exception attributes (exception.type, exception.message)
var rec log.Record
rec.SetSeverity(log.SeverityError)
rec.SetBody(attribute.StringValue("operation failed"))
rec.SetErr(err) // Attaches error; SDK sets exception attributes automatically
logger.Emit(ctx, rec)

// Read back the error from a record
if recordErr := rec.Err(); recordErr != nil {
    // Handle the error
}
```

### Severity Levels
```go
log.SeverityTrace1 through log.SeverityTrace4
log.SeverityDebug1 through log.SeverityDebug4
log.SeverityInfo1 through log.SeverityInfo4
log.SeverityWarn1 through log.SeverityWarn4
log.SeverityError1 through log.SeverityError4
log.SeverityFatal1 through log.SeverityFatal4
```

### Log Values
```go
attribute.StringValue("text")
attribute.IntValue(42)
attribute.Int64Value(int64(42))
attribute.Float64Value(3.14)
attribute.BoolValue(true)
attribute.ByteSliceValue([]byte("data"))
attribute.SliceValue(attribute.StringValue("a"), attribute.StringValue("b"))
attribute.MapValue(attribute.String("key", "value"))
```

## Logging Bridges

Logging bridges connect existing logging libraries to OpenTelemetry.

> **Note (bridge v0.10.0):** otelzap and otelslog emit `code.function` with the full package path-qualified function name (e.g., `github.com/user/pkg.MyFunc`). The `code.namespace` attribute is no longer emitted.

```go
import (
    // For zap logging
    "go.opentelemetry.io/contrib/bridges/otelzap"

    // For slog logging (Go 1.21+)
    "go.opentelemetry.io/contrib/bridges/otelslog"

    // For logrus logging
    "go.opentelemetry.io/contrib/bridges/otellogrus"

    // For logr logging
    "go.opentelemetry.io/contrib/bridges/otellogr"
)
```

### Example: zap bridge
```go
// Create zap logger with OpenTelemetry bridge
core := zapcore.NewTee(
    zapcore.NewCore(encoder, os.Stdout, zapcore.InfoLevel),
    otelzap.NewCore("my-service", otelzap.WithLoggerProvider(loggerProvider)),
)
logger := zap.New(core)
```

### Example: slog bridge
```go
// Create slog handler with OpenTelemetry bridge
handler := otelslog.NewHandler("my-service", otelslog.WithLoggerProvider(loggerProvider))
logger := slog.New(handler)
```

# Go OpenTelemetry Breaking Changes & Gotchas

Audit reference for upgrading existing Go OpenTelemetry code. For current SDK/contrib
version selection, fetch from the Sources of Truth table in the `otel-go` skill (or
`go.opentelemetry.io/otel` and `go.opentelemetry.io/contrib` release tags directly).

After applying an upgrade, verify it locally with `go mod tidy -diff`, `go build ./...`,
and `go test ./...`. For exporter URL or retry behavior, use an `httptest.Server` or
another disposable local receiver; never probe a deployment endpoint merely to confirm
the SDK migration.

## API Deprecations

- `go.opentelemetry.io/contrib/config` was removed in contrib v1.35.0. Use `go.opentelemetry.io/contrib/otelconf` instead.
- `WithMetricAttributesFn` deprecated in otelhttp v0.66.0 (contrib v1.41.0 release). Use `Labeler` instead.
- otelgrpc's deprecated client/server interceptors were removed in v0.60.0. Use stats handlers (`NewServerHandler` / `NewClientHandler`).
- `attribute.INVALID` deprecated (core v1.43.0) — an empty value is now valid; use `attribute.EMPTY` instead.
- `attribute.Value.Emit` deprecated (core v1.44.0). Use `attribute.Value.String` instead.
- `sdk/log.WithExportBufferSize` is deprecated and has no effect as of log SDK v0.21.0 (core v1.45.0 release); the batch processor no longer keeps a separate export-request buffer.
- `OTEL_EXPERIMENTAL_CONFIG_FILE` is no longer supported by root `otelconf` v0.23.0+ (contrib v1.43.0+). Use `OTEL_CONFIG_FILE`.
- `go.opentelemetry.io/contrib/instrumentation/github.com/labstack/echo/otelecho` was deprecated in v0.71.0 (contrib v1.46.0) and removed in v0.72.0 (contrib v1.47.0). Use `github.com/labstack/echo-opentelemetry`.
- `go.opentelemetry.io/otel/log/global` is deprecated (core v1.47.0). Use `otel.Logger`, `otel.GetLoggerProvider`, and `otel.SetLoggerProvider` from `go.opentelemetry.io/otel`.
- Deprecated as no longer supported in contrib v1.47.0, with no contrib replacement: `instrumentation/github.com/aws/aws-sdk-go-v2/otelaws` (v0.72.0), `propagators/aws`, and `samplers/probability/consistent`.
- otelhttp v0.72.0 deprecates `ReadBytesKey`, `ReadErrorKey`, `WroteBytesKey`, and `WriteErrorKey`; `WithMessageEvents` no longer emits `http.read_bytes` / `http.wrote_bytes`. Total body sizes remain on the server span as `http.request.body.size` / `http.response.body.size`.

## Removed APIs (instrumentation v0.65.0; contrib v1.40.0 release)

- otelhttp removed `DefaultClient`, `Get`, `Head`, `Post`, `PostForm`, `WithPublicEndpoint`, `WithRouteTag`. Always create a custom client with `otelhttp.NewTransport`.
- HTTP instrumentation sets the `error.type` attribute instead of `exception` span events.

## Semantic Convention Renames (instrumentation v0.65.0; contrib v1.40.0 release)

RPC attribute changes:

- `rpc.system` → `rpc.system.name`
- `rpc.method` + `rpc.service` merged into `rpc.method`
- `rpc.client.duration` / `rpc.server.duration` → `rpc.client.call.duration` / `rpc.server.call.duration` (unit changed to seconds)
- `rpc.grpc.status_code` → `rpc.response.status_code`

## otelgrpc v0.67.0 (contrib v1.42.0 release)

- No longer emits `rpc.message` span events, `rpc.*.request/response.size` metrics, or `network.*` attributes — even with `WithMessageEvents`.

## Metric SDK cardinality limit (core v1.44.0)

- `sdk/metric` now enforces a **default cardinality limit of 2000** series per instrument. Series past the limit are dropped into an overflow series (`otel.metric.overflow=true`). Breaking change from the previous unlimited default. Restore unlimited with `sdkmetric.WithCardinalityLimit(0)`; the `OTEL_GO_X_CARDINALITY_LIMIT` env var is deprecated. Per-instrument-kind limits: `sdkmetric.WithCardinalityLimitSelector` (v1.43.0).

## OTLP exporter request-size cap (core v1.44.0)

- All OTLP exporters cap requests at **64 MiB** by default (applied before compression); oversized requests become non-retryable errors. Configure with `WithMaxRequestSize`.

## Baggage limits (core v1.44.0)

- The 8192-byte baggage size limit is now enforced during extraction/parsing (`otel/baggage`, `otel/propagation`): baggage strings over the limit are rejected outright, while malformed members are skipped — valid members are retained and an error is reported.

## Log value API replacement (log API/SDK v0.21.0; core v1.45.0 release)

- Log bodies and record attributes now use `go.opentelemetry.io/otel/attribute.Value` and `attribute.KeyValue`. The former `log.Kind`, `log.Value`, `log.KeyValue`, their constructors, and conversion helpers were removed. Replace `log.StringValue(...)` / `log.String(...)` with `attribute.StringValue(...)` / `attribute.String(...)` (and the corresponding typed constructors).
- `sdk/log/logtest.RecordFactory` no longer has `AttributeValueLengthLimit` or `AttributeCountLimit`; test records now keep attribute limits disabled.

## OTLP HTTP endpoint URLs (core v1.45.0)

- `otlptracehttp.WithEndpointURL` and `otlpmetrichttp.WithEndpointURL` no longer append `/v1/traces` or `/v1/metrics` when the URL has no path. They now use `/`, matching signal-specific endpoint environment variables and the log exporter. Join the signal path explicitly to preserve the old behavior.

## Core v1.47.0 (logs API/SDK v1.47.0; log exporters v0.23.0)

- `go.opentelemetry.io/otel/log` and `go.opentelemetry.io/otel/sdk/log` are stable and now tagged on the v1.x line (`@v1.47.0`); the OTLP/stdout log exporters and `logtest` modules continue on v0.x (v0.23.0).
- Requires Go 1.26 or later (Go 1.25 support dropped in core and contrib v1.47.0).
- Nested attribute values are capped at depth 64 by default (tuning: [performance.md](performance.md#span-attribute-limits)).
- The experimental `OTEL_GO_X_METRIC_EXPORT_BATCH_SIZE` variable was removed; use `sdkmetric.WithMaxExportBatchSize` on the `PeriodicReader`.
- Empty `OTEL_TRACES_SAMPLER` / `OTEL_TRACES_SAMPLER_ARG` values are treated as unset.
- gRPC OTLP exporters ignore HTTP URL paths when deriving targets from `OTEL_EXPORTER_OTLP_ENDPOINT` / signal-specific endpoint variables.

## otelhttp client metrics labeler (instrumentation v0.72.0; contrib v1.47.0 release)

- Client metrics no longer include attributes from the server-side `Labeler`. Code that used `ContextWithLabeler` to add custom *client* metric attributes must switch to `ContextWithClientLabeler` (read back with `ClientLabelerFromContext`). Server-side `ContextWithLabeler` / `LabelerFromContext` is unchanged.

## Log exporter shutdown (log SDK v0.22.0; core v1.46.0 release)

- `sdk/log` adds `ErrExporterShutdown`. The built-in stdout and OTLP log exporters now return it when `Export` is called after `Shutdown`; custom exporters should do the same. Calls to `Shutdown` or `ForceFlush` after shutdown remain no-ops that return `nil`.

## HTTP instrumentation behaviour (instrumentation v0.69.0; contrib v1.44.0 release)

- Unknown or empty HTTP methods now report `_OTHER` instead of `GET` across all HTTP instrumentations (otelhttp, otelmux).
- The default server span name is now `{method} {route}` (e.g. `GET /foo/{id}`) when a route is available, or `{method}` otherwise — conforming to HTTP semconv.

In otelhttp v0.70.0 (contrib v1.45.0), client spans and request metrics derive `network.protocol.name` / `network.protocol.version` from the negotiated response protocol rather than the request's initial `Proto` value.

## Removed / renamed contrib APIs (v1.43.0–v1.44.0 release lines)

- otelgrpc removed the deprecated `WithSpanOptions` option in v0.69.0.
- `otelconf`: experimental config types moved from `otelconf` to the `otelconf/x` subpackage in v0.23.0. The `host` resource detector no longer sets `host.id` in v0.23.0.
- otelgrpc added `OTEL_SEMCONV_STABILITY_OPT_IN` (values `rpc` default, `rpc/dup`, `rpc/old`) in v0.68.0 to stage RPC semconv migration.
- Contrib v1.45.0 updates instrumentation and detectors to semconv v1.43.0; use the exact versioned semconv package and its generated migration file rather than mixing helpers from older versions.

## Log bridges attach errors via SetErr (bridge v0.18.0–v0.19.0)

- otelslog, otelzap, otellogrus, and otellogr now record fields implementing `error` (e.g. `zap.Error`, `slog` error values) via `log.Record.SetErr` instead of emitting them as plain or `exception.*` attributes. The SDK then derives the `exception.*` attributes from the record error.

## Behavioural Notes

- `trace.SpanFromContext()` never returns nil — no nil checks or `IsRecording()` guards needed.
- `span.RecordError()` and `span.AddEvent()` are still present in the released tracing API; do not remove existing uses as deprecated APIs.
- `log.Record.SetErr(err)` (`otel/log` v0.18.0; released with core v1.42.0) — the SDK automatically sets `exception.type` and `exception.message` attributes.

## Resolved Gotchas (historical reference)

- **Propagator not installed by `otelconf`** ([open-telemetry/opentelemetry-go-contrib#6712](https://github.com/open-telemetry/opentelemetry-go-contrib/issues/6712)) — fixed in `otelconf v0.20.0` (released 2026-02-02). The root `otelconf` package now exposes `sdk.Propagator()`; install via `otel.SetTextMapPropagator(sdk.Propagator())`. The schema-pinned `otelconf/v0.3.0` subpackage still requires manual installation: `otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))`.

# Compile-time instrumentation with `otelc`

`otelc` (`go.opentelemetry.io/otelc`, current stable release **v1.1.0**) instruments Go applications
with OpenTelemetry **at compile time**. It wraps the Go toolchain via the compiler's `-toolexec`
hook and injects trampoline hooks (linked with `//go:linkname`) into target functions, so
instrumentation is baked into the binary — **no source-code changes**, and it reaches third-party
dependencies and stdlib packages you don't own. v1.0.0 is retracted because `otelc pin` generated
the pre-v1 module paths in consumer `go.mod` files; use v1.0.1 or later.

## When to use it (vs. the rest of this skill)

| Approach | Reach for it when |
|---|---|
| **`otelc`** (this file) | You can rebuild the app and want zero source changes and automatic coverage of dependencies/stdlib you don't control. The injected OpenTelemetry SDK and instrumentation still have runtime cost. |
| **Hand-written SDK + contrib libraries** ([`instrumentation-libraries.md`](instrumentation-libraries.md), [`api.md`](api.md)) | You need explicit, fine-grained control over spans/metrics, or you cannot change the build toolchain. |

`otelc` is **not** [`opentelemetry-go-instrumentation`](https://github.com/open-telemetry/opentelemetry-go-instrumentation),
which is a separate **eBPF/uprobe runtime** auto-instrumentation project. `otelc` is
compile-time / build-time.

## Requirements

- **Go 1.25+** toolchain (`go 1.25.0` in the project's `go.mod`).
- **Go 1.24+** for the tool-dependency workflow (`go get -tool` / `go tool otelc`).
- Operates on the module containing your `go.mod`.

## Usage modes

All three produce an instrumented binary; they differ in how `otelc` wires into the build.

```bash
# Mode 1 — wrap the build (simplest). Prefix the normal go command.
otelc go build -o bin/app .
otelc go test ./...

# Mode 2 — tool dependency (Go 1.24+, reproducible: version tracked in go.mod)
go get -tool go.opentelemetry.io/otelc/tool/cmd/otelc
go tool otelc go build -o bin/app .

# Mode 3 — toolexec drop-in via GOFLAGS (build command owned by a Makefile/CI)
otelc setup                                             # run once; re-run when deps change
export GOFLAGS="${GOFLAGS} '-toolexec=otelc toolexec'"  # single quotes are required
go build -o myapp .
```

Obtain the binary by building from source (`make build` in the upstream repo), `go install
go.opentelemetry.io/otelc/tool/cmd/otelc@latest`, or the tool-dependency mode above.

**Zero-config:** with no instrumentation file present, `otelc go build` analyzes the dependency
graph, generates a temporary config for the build, and cleans up. Upgrading `otelc` can change what
gets instrumented, so pin the `otelc` version and inspect `.otelc-build/matched.json` in auditable
builds. Committing `otelc pin` output is not yet supported (see below).

## Subcommands

| Subcommand | Purpose |
|---|---|
| `otelc go …` | Instrument and run the go toolchain; everything after `go` is forwarded verbatim. |
| `otelc setup` | Analyze the module, generate instrumentation sources, and write the matched rule set to `.otelc-build/` (needed before Mode 3). |
| `otelc pin` | Create/update the local-workflow `otel.instrumentation.go`. Flags: `--prune` (default on), `--validate`, `--generate`. |
| `otelc cleanup` | Delete `.otelc-build/` and generated files. |
| `otelc version` | Print the tool version (`--verbose` adds the Go runtime version). |
| `otelc toolexec …` | Hidden interceptor invoked by `-toolexec=otelc toolexec`; never call it directly. |

**Flag ordering:** global flags (`--rules`, `--debug`/`-d`, `--work-dir`/`-w`) must come **before**
the `go` subcommand. `otelc go` and `otelc toolexec` skip flag parsing, so anything after them goes
straight to the Go toolchain — `otelc go build --rules …` sends `--rules` to `go build` and fails.

## Supported libraries (v1)

Verified against the upstream `instrumentation/` tree. The active set is `otelc`'s embedded bundle;
confirm against the tree when a version changes.

| Library | Signals |
|---|---|
| `net/http` (client & server) | HTTP spans |
| `google.golang.org/grpc` (client & server) | RPC spans |
| `database/sql` | DB client spans |
| `github.com/gin-gonic/gin` | HTTP server spans |
| `github.com/redis/go-redis/v9` | Redis DB spans |
| `go.mongodb.org/mongo-driver` (v1/v2) | MongoDB DB spans |
| `k8s.io/client-go` | K8s resource spans |
| `github.com/openai/openai-go` (v1/v2/v3) | GenAI spans |
| `github.com/anthropics/anthropic-sdk-go` | GenAI spans |
| `github.com/segmentio/kafka-go` (consumer & producer) | Kafka messaging spans |
| `github.com/aws/aws-sdk-go-v2` | AWS SDK client spans |
| `github.com/linode/linodego/v2` | HTTP client spans and metrics |
| `log`, `log/slog`, `github.com/sirupsen/logrus` | Trace/span ID log correlation |
| Go runtime | Runtime metrics |

## Selecting & configuring instrumentation

**Rule sources, highest priority first — there is NO merging; each source entirely replaces the
ones below it:**

| Priority | Source |
|---|---|
| 1 | `OTELC_RULES` env var (file / dir / comma-separated list) |
| 2 | `--rules` flag (same format; used only when `OTELC_RULES` is unset) |
| 3 | Tool file `otel.instrumentation.go` (alias `otelc.tool.go`) |
| 4 | Embedded default bundle |

This is the top cause of "nothing got instrumented": an `OTELC_RULES`/`--rules` override silently
masks the tool file and the embedded bundle. `--rules`/`OTELC_RULES` are for dev/debugging. The
tool file is the intended explicit model, but `otelc` has not published the instrumentation
submodules: `otelc pin` adds local `replace` directives into `.otelc-build/`. Treat its output as
a local workflow, not a source-controlled production configuration.

**Local explicit selection** uses the standard Go `tools.go` blank-import pattern in a
module-scoped file next to `go.mod`. Let `otelc pin` generate and manage this file:

```go
//go:build tools

package tools

import (
	_ "go.opentelemetry.io/otelc/instrumentation/net/http/server"
	_ "go.opentelemetry.io/otelc/instrumentation/google.golang.org/grpc/client"
	_ "go.opentelemetry.io/otelc/instrumentation/google.golang.org/grpc/server"
)
```

Only blank (`_`) imports are allowed. Do not hand-copy the parent
`instrumentation/google.golang.org/grpc` import shown in an upstream protocol example: it
is not a Go package; the client and server are separate instrumentation modules.

**Runtime tuning:** instrumented binaries embed an auto-initialized SDK that reads the standard
[OTel SDK env vars](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/)
(`OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_TRACES_SAMPLER`, `OTEL_SERVICE_NAME`, …). There is no
`otelc`-specific exporter or sampler destination config. `OTEL_SDK_DISABLED=true` (case-insensitive)
disables the injected SDK. `otelc` exposes these runtime controls:

| Variable | Behavior |
|---|---|
| `OTEL_GO_ENABLED_INSTRUMENTATIONS` | Comma-separated, case-insensitive allowlist of compiled instrumentation names. |
| `OTEL_GO_DISABLED_INSTRUMENTATIONS` | Comma-separated denylist, applied after the allowlist. |
| `OTEL_GLS_MAX_SPANS` | Per-goroutine live span-stack limit; default `1000`. |
| `OTEL_GLS_MAX_SPAN_STATES` | Shared span-lifecycle map limit; default `100000`, evicting the oldest state when full. |
| `OTEL_LOG_LEVEL` | `debug`, `info` (default), `warn`, or `error` for `otelc` runtime logs. |
| `OTEL_GO_SIMPLE_SPAN_PROCESSOR` | Exactly lowercase `true` selects `SimpleSpanProcessor`; other spellings are ignored. |

The enable/disable lists only gate instrumentation already compiled into the binary; build-time
selection still comes from the rule sources above.

**Verify what matched:** `.otelc-build/matched.json` lists every rule that matched; `[]` means
nothing matched (and `otelc` prints `Warning: no instrumentation will be applied`). Enable
`--debug`/`OTELC_DEBUG=1` for `.otelc-build/debug.log`.

## Rule schema (authoring)

Custom rules and new-library instrumentation use YAML rules with `target` (import path; globs and
`$root` supported), optional `version`, `where`/`where.file` predicates, and a `do` action. The
eight rule types: `inject_hooks` (function hook), `add_struct_fields`, `inject_code`, `wrap_call`,
`expand_directive`, `add_file`, `assign_value`, and `set_fields` (composite literals). See the
fetch table for the full grammar.

**v1.1.0 custom-rule migration:** function template variables are fields on Go's `text/template`
dot. Use `{{ .FuncName }}`, `{{ .FuncArgument 0 }}`, and corresponding dotted forms; the pre-v1.1
`{{ FuncName }}` spelling is no longer valid.

## Sources of truth

`otelc` ships stable and evolves; fetch upstream docs rather than trusting a snapshot. Repo:
`open-telemetry/opentelemetry-go-compile-instrumentation` (module `go.opentelemetry.io/otelc`).

| Fact / task | Fetch |
|---|---|
| Latest `otelc` release tag | `gh api repos/open-telemetry/opentelemetry-go-compile-instrumentation/releases/latest -q '.tag_name'` |
| Install, usage modes, managing instrumentations | `WebFetch https://raw.githubusercontent.com/open-telemetry/opentelemetry-go-compile-instrumentation/main/docs/getting-started.md` |
| Full rule schema, glob grammar, and predicates | `WebFetch https://raw.githubusercontent.com/open-telemetry/opentelemetry-go-compile-instrumentation/main/docs/rules.md` |
| Scope, filtering, precedence, runtime tuning, verification | `WebFetch https://raw.githubusercontent.com/open-telemetry/opentelemetry-go-compile-instrumentation/main/docs/configuration.md` |
| Declaring instrumentations via `otel.instrumentation.go` | `WebFetch https://raw.githubusercontent.com/open-telemetry/opentelemetry-go-compile-instrumentation/main/docs/external-configuration.md` |
| Add instrumentation for a new library; semconv | `WebFetch https://raw.githubusercontent.com/open-telemetry/opentelemetry-go-compile-instrumentation/main/docs/instrument-guide.md` |
| Internals (`-toolexec`, trampolines, `//go:linkname`, GLS) | `WebFetch https://raw.githubusercontent.com/open-telemetry/opentelemetry-go-compile-instrumentation/main/docs/implementation.md` |
| Diagnose why instrumentation was not applied | `WebFetch https://raw.githubusercontent.com/open-telemetry/opentelemetry-go-compile-instrumentation/main/docs/troubleshooting.md` |
| Current supported-library set | List the `instrumentation/` tree in the repo (each leaf ships its own rules). |

# Go SDK Setup with otelconf

Configure the OpenTelemetry SDK in Go via `otelconf`, the declarative YAML configuration
package.

For the YAML configuration schema, load the `otel-declarative-config` skill.

## Sources of Truth

For YAML schema details, fetch the upstream sources listed in the `otel-declarative-config`
skill. For Go-specific facts:

| Fact | Fetch |
|---|---|
| `otelconf` CHANGELOG (which schema rc each release supports, breaking changes) | `WebFetch https://raw.githubusercontent.com/open-telemetry/opentelemetry-go-contrib/main/CHANGELOG.md` |
| Current API surface (`SDK` methods, options) | `WebFetch https://pkg.go.dev/go.opentelemetry.io/contrib/otelconf` |
| Schema tag the generated types come from (`genjsonschema` target) | `WebFetch https://raw.githubusercontent.com/open-telemetry/opentelemetry-go-contrib/otelconf/<tag>/Makefile` |
| Latest `go.opentelemetry.io/otel` core release | `gh api repos/open-telemetry/opentelemetry-go/releases/latest -q '.tag_name'` |

## Choosing an Import Path

The `otelconf` repo ships two flavors of the package:

- **Root** `go.opentelemetry.io/contrib/otelconf` — trails the schema repository. Verified
  2026-10-15: `v0.27.0` targets schema `v1.0.0-rc.3` (`file_format: "1.0-rc.3"`) while the schema
  repository is at `v1.2.0` (owner: the `otel-declarative-config`
  [parsers table](../../otel-declarative-config/SKILL.md)). Includes `sdk.Propagator()` for installing
  propagators from YAML.
- **Schema-pinned** `go.opentelemetry.io/contrib/otelconf/v0.3.0` — frozen at file_format
  `"0.3"` (a 2024 schema). Has no `Propagator()` method; propagators must be installed
  manually via `otel.SetTextMapPropagator(...)`.

Newer `otelconf` releases may exist; check the selected tag's Makefile schema tag before deploying.
For "latest", resolve the tag once with the `otelconf` tag command in this skill's `SKILL.md`.

## Migration notes

- `go.opentelemetry.io/contrib/config` was removed in contrib v1.35.0. Use `go.opentelemetry.io/contrib/otelconf`.
- In the root package, use `OTEL_CONFIG_FILE`; `OTEL_EXPERIMENTAL_CONFIG_FILE` is rejected by
  current released `otelconf`.
- Starting with root `otelconf` v0.25.0 (contrib v1.45.0), the resolved resource includes the SDK's default resource attributes; account for those when comparing telemetry across this boundary.
- If migrating from the schema-pinned `otelconf/v0.3.0` import, the YAML must also migrate to
  the new schema (fetch `examples/otel-sdk-migration-config.yaml` from the schema repo for
  before/after pairs).

## Dependencies

```go
import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/propagation"
    semconv "go.opentelemetry.io/otel/semconv/v1.43.0"
    otelconf "go.opentelemetry.io/contrib/otelconf"
)
```

## YAML Config

For the canonical structure, fetch `examples/otel-sdk-config.yaml` from the schema repo
(see the `otel-declarative-config` skill's Sources of Truth). The minimal example below
illustrates the Go-specific quirk; the exact `file_format` string and exporter field names
should come from the selected `otelconf` module's package fixtures/source.

```yaml
# file_format: use the exact literal accepted by the selected otelconf module
resource:
  attributes:
    - name: service.name
      value: "${SERVICE_NAME:-my-service}"
    - name: deployment.environment.name
      value: "${DEPLOY_ENV:-development}"

# Tracer/meter/logger provider blocks: structure per the canonical example.
# The root otelconf parser resolves ${VAR}, ${env:VAR}, and ${VAR:-default}
# before parsing YAML. Use $${VAR} when the literal text must be preserved.
```

## Key API Facts

- **`otelconf.ParseYAML([]byte) (*OpenTelemetryConfiguration, error)`** parses a YAML config file.
- **`otelconf.NewSDK(opts ...) (SDK, error)`** builds the SDK from a parsed configuration. Options include `WithContext(ctx)` and `WithOpenTelemetryConfiguration(conf OpenTelemetryConfiguration)`; after `ParseYAML`, pass `*conf` because `ParseYAML` returns `*OpenTelemetryConfiguration`.
- **`sdk.TracerProvider()`, `sdk.MeterProvider()`, `sdk.LoggerProvider()`** return the configured providers.
- **`sdk.Propagator()`** (root package, `otelconf v0.20.0+`, fixes [open-telemetry/opentelemetry-go-contrib#6712](https://github.com/open-telemetry/opentelemetry-go-contrib/issues/6712)) returns the propagator built from the YAML `propagator:` block. Install it via `otel.SetTextMapPropagator(sdk.Propagator())`. Not available on the schema-pinned `otelconf/v0.3.0` subpackage.
- **`sdk.Shutdown(ctx) error`** flushes and closes all providers.

# OpenTelemetry Go Instrumentation Libraries

## Detecting Existing Instrumentation

Before adding manual instrumentation, check for these imports and middleware patterns in the codebase:

```go
import (
    // HTTP frameworks
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
    "go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"
    "go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux"
    "go.opentelemetry.io/contrib/instrumentation/github.com/labstack/echo/otelecho" // removed in v0.72.0

    // gRPC
    "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"

    // Logging bridges
    "go.opentelemetry.io/contrib/bridges/otelzap"
    "go.opentelemetry.io/contrib/bridges/otelslog"
    "go.opentelemetry.io/contrib/bridges/otellogrus"
)
```

If these middleware/wrapper patterns are present, spans are already being created:

```go
handler := otelhttp.NewHandler(mux, "service-name")       // net/http
router.Use(otelgin.Middleware("service-name"))             // gin (also emits HTTP metrics)
router.Use(otelmux.Middleware("service-name"))             // gorilla/mux (also emits HTTP metrics)
```

## Context Propagation (read first)

In Go, spans link to their parent **only** through `context.Context`. An instrumentation library
installs correctly and still emits *detached* spans if the active context never reaches the call. A
SERVER span living in `c.Request.Context()` (gin) or `r.Context()` (net/http) parents a downstream
DB/client span **only if that exact context is passed into it**.

> **Symptom:** DB or client spans show up as CLIENT-kind *roots* in their own traces, disconnected
> from the request. This means trace context was dropped at an internal boundary.

The usual cause is structural, not a missing option:

```go
// ANTI-PATTERN: global handle + ctx-less call → span parented to context.Background()
db := common.GetDB()             // package-level *gorm.DB
db.Where("id = ?", id).First(&u) // no ctx → detached CLIENT-root span

// CORRECT: thread the request context from the handler down to the call
func (h *Handler) GetUser(c *gin.Context) {
    u, err := h.users.Find(c.Request.Context(), id) // ctx flows in
}
func (r *UserRepo) Find(ctx context.Context, id int) (*User, error) {
    var u User
    return &u, r.db.WithContext(ctx).First(&u, id).Error // GORM links via WithContext(ctx)
}
```

A data layer built on a global `*gorm.DB` (or `*sql.DB`) plus functions that don't accept `ctx`
**cannot** produce connected traces — the call signatures have to carry `ctx`. Rules:

- Every DB/client/outbound call runs with the incoming request's context.
- For GORM, call `db.WithContext(ctx)` on every query; for `database/sql`, use the `...Context`
  methods (`QueryContext`, `ExecContext`).
- Never call instrumented clients with `context.Background()` or `context.TODO()` on a request path.

## Library Catalog

### HTTP

| Library | Package |
|---------|---------|
| net/http | `go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp` |
| Gin | `go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin` |
| Gorilla mux | `go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux` |
| Echo | `github.com/labstack/echo-opentelemetry` (contrib `otelecho` was removed in v0.72.0; v0.71.0 is its last release). |
| go-restful | `go.opentelemetry.io/contrib/instrumentation/github.com/emicklei/go-restful/otelrestful` |

```go
// HTTP server with automatic instrumentation
func setupHTTPServer() *http.Server {
    mux := http.NewServeMux()
    mux.HandleFunc("/users", handleUsers)

    // Automatic span creation and context propagation
    handler := otelhttp.NewHandler(mux, "user-service")

    return &http.Server{
        Addr:    ":8080",
        Handler: handler,
    }
}

// HTTP client with automatic instrumentation
// Note: otelhttp.DefaultClient, Get, Head, Post, PostForm were removed in v0.65.0
// Always create a custom client with otelhttp.NewTransport instead
func setupHTTPClient() *http.Client {
    return &http.Client{
        Transport: otelhttp.NewTransport(http.DefaultTransport),
    }
}
```

### gRPC

| Library | Package |
|---------|---------|
| gRPC | `go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc` |

```go
// gRPC server with automatic instrumentation (prefer stats handlers over interceptors)
func setupGRPCServer() *grpc.Server {
    return grpc.NewServer(
        grpc.StatsHandler(otelgrpc.NewServerHandler()),
    )
}

// gRPC client with automatic instrumentation
func setupGRPCClient(target string) (*grpc.ClientConn, error) {
    return grpc.NewClient(target,
        grpc.WithStatsHandler(otelgrpc.NewClientHandler()),
    )
}

// Override default span kind in gRPC (otelgrpc v0.66.0+)
func setupGRPCServerWithSpanKind() *grpc.Server {
    return grpc.NewServer(
        grpc.StatsHandler(otelgrpc.NewServerHandler(
            otelgrpc.WithSpanKind(trace.SpanKindInternal),
        )),
    )
}
```

### Database

| Library | Package |
|---------|---------|
| database/sql | No package in current `go.opentelemetry.io/contrib`; check the OpenTelemetry registry for third-party instrumentation or instrument manually with `...Context` methods. |
| MongoDB | `go.opentelemetry.io/contrib/instrumentation/go.mongodb.org/mongo-driver/mongo/otelmongo` |
| GORM | `gorm.io/plugin/opentelemetry/tracing` (registered via `db.Use(tracing.NewPlugin(...))`) |

> **PII in `db.query.text`:** the GORM plugin records the executed statement with bound parameter
> values **inlined** by default, so a query like `INSERT INTO users (email, ...) VALUES ('a@b.com', ...)`
> leaks the literal email onto the span. Disable query-variable capture so the statement is
> parameterized:
>
> ```go
> db.Use(tracing.NewPlugin(tracing.WithoutQueryVariables()))
> // db.query.text becomes: INSERT INTO "users" (...) VALUES (?, ?, ?)
> ```
>
> For services that handle user data, the same trap applies to other SQL instrumentation that
> captures full statement text; verify the selected package's option to
> disable or redact query text. Keep parameter values out of `db.query.text` before export. If the
> selected instrumentation cannot sanitize or parameterize query text, disable query-text capture;
> downstream redaction is defense-in-depth.

### AWS

| Library | Package |
|---------|---------|
| AWS SDK v2 | `go.opentelemetry.io/contrib/instrumentation/github.com/aws/aws-sdk-go-v2/otelaws` (deprecated in v0.72.0 as no longer supported; no contrib replacement) |
| Lambda | `go.opentelemetry.io/contrib/instrumentation/github.com/aws/aws-lambda-go/otellambda` |

### Logging Bridges

| Logger | Package |
|--------|---------|
| zap | `go.opentelemetry.io/contrib/bridges/otelzap` |
| slog | `go.opentelemetry.io/contrib/bridges/otelslog` |
| logrus | `go.opentelemetry.io/contrib/bridges/otellogrus` |
| logr | `go.opentelemetry.io/contrib/bridges/otellogr` |

> **Logging bridge change (bridge v0.10.0):** otelzap and otelslog now emit `code.function` with the full package path-qualified function name (e.g., `github.com/user/pkg.MyFunc`) instead of just the function name. The `code.namespace` attribute is no longer emitted.

```go
// zap bridge setup
import (
    "go.opentelemetry.io/contrib/bridges/otelzap"
    "go.opentelemetry.io/otel"
    "go.uber.org/zap"
    "go.uber.org/zap/zapcore"
)

func setupLogger() *zap.Logger {
    // Create a tee core that logs to both stdout and OpenTelemetry
    core := zapcore.NewTee(
        zapcore.NewCore(
            zapcore.NewJSONEncoder(zap.NewProductionEncoderConfig()),
            zapcore.AddSync(os.Stdout),
            zapcore.InfoLevel,
        ),
        otelzap.NewCore("myservice",
            otelzap.WithLoggerProvider(otel.GetLoggerProvider())),
    )
    return zap.New(core)
}
```

```go
// slog bridge setup (Go 1.21+)
import (
    "log/slog"
    "go.opentelemetry.io/contrib/bridges/otelslog"
    "go.opentelemetry.io/otel"
)

func setupLogger() *slog.Logger {
    handler := otelslog.NewHandler("myservice",
        otelslog.WithLoggerProvider(otel.GetLoggerProvider()))
    return slog.New(handler)
}
```

```go
// logrus bridge setup
import (
    "github.com/sirupsen/logrus"
    "go.opentelemetry.io/contrib/bridges/otellogrus"
    "go.opentelemetry.io/otel"
)

func setupLogger() {
    logrus.AddHook(otellogrus.NewHook("myservice",
        otellogrus.WithLoggerProvider(otel.GetLoggerProvider())))
}
```

### Resource Detectors

| Cloud | Package |
|-------|---------|
| AWS EC2 | `go.opentelemetry.io/contrib/detectors/aws/ec2/v2` |
| AWS ECS | `go.opentelemetry.io/contrib/detectors/aws/ecs` |
| AWS EKS | `go.opentelemetry.io/contrib/detectors/aws/eks` |
| AWS Lambda | `go.opentelemetry.io/contrib/detectors/aws/lambda` |
| GCP | `go.opentelemetry.io/contrib/detectors/gcp` |
| Azure VM | `go.opentelemetry.io/contrib/detectors/azure/azurevm` |

### Propagators

| Propagator | Package |
|------------|---------|
| Environment carrier | `go.opentelemetry.io/contrib/propagators/envcar` (new in v0.67.0) |

## Manual Instrumentation Patterns

Use these patterns when no contrib instrumentation library exists for the target.

### HTTP Client with Semconv Attributes

```go
func (c *CustomClient) CallExternalAPI(ctx context.Context, endpoint string) error {
    ctx, span := c.tracer.Start(ctx, "GET",
        trace.WithSpanKind(trace.SpanKindClient),
        trace.WithAttributes(
            semconv.HTTPRequestMethodGet,
            semconv.URLFull(endpoint)))
    defer span.End()

    req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
    if err != nil {
        return fmt.Errorf("creating request: %w", err)
    }

    resp, err := c.httpClient.Do(req)
    if err != nil {
        return fmt.Errorf("making request: %w", err)
    }
    defer resp.Body.Close()

    span.SetAttributes(semconv.HTTPResponseStatusCode(resp.StatusCode))
    return nil
}
```

### Database Operation with Semconv

```go
func (r *Repository) CreateUser(ctx context.Context, userData *UserData) (*User, error) {
    // semconv helpers below track go.opentelemetry.io/otel/semconv/v1.43.0
    ctx, span := r.tracer.Start(ctx, "INSERT users",
        trace.WithAttributes(
            semconv.DBSystemNamePostgreSQL,        // db.system.name (was DBSystem/db.system)
            semconv.DBNamespace(r.dbName),
            semconv.DBOperationName("INSERT"),      // db.operation.name (was DBOperation)
            semconv.DBCollectionName("users")))     // db.collection.name (was DBSQLTable/db.sql.table)
    defer span.End()

    query := `INSERT INTO users (email, name, tier) VALUES ($1, $2, $3) RETURNING id, created_at`

    var user User
    err := r.db.QueryRow(ctx, query, userData.Email, userData.Name, userData.Tier).
        Scan(&user.ID, &user.CreatedAt)

    if err != nil {
        span.SetAttributes(attribute.String("error.type", classifyDBError(err)))
        return nil, fmt.Errorf("inserting user: %w", err)
    }

    span.SetAttributes(
        attribute.String("user.id", user.ID),
        attribute.String("user.tier", userData.Tier),
    )

    return &user, nil
}

func classifyDBError(err error) string {
    if strings.Contains(err.Error(), "unique constraint") {
        return "duplicate_key"
    }
    if strings.Contains(err.Error(), "connection") {
        return "connection_error"
    }
    return "unknown"
}
```

### Message Queue Producer/Consumer with Semconv

```go
func (p *Publisher) PublishOrderEvent(ctx context.Context, order *Order) error {
    ctx, span := p.tracer.Start(ctx, "send queue.orders",
        trace.WithSpanKind(trace.SpanKindProducer),
        trace.WithAttributes(
            semconv.MessagingSystemKey.String("nats"), // messaging.system (no NATS enum const)
            semconv.MessagingOperationTypeSend,        // was MessagingOperationTypePublish
            semconv.MessagingDestinationName("orders"),
            attribute.String("order.id", order.ID)))
    defer span.End()

    message, err := json.Marshal(order)
    if err != nil {
        return fmt.Errorf("marshaling order: %w", err)
    }

    if err := p.client.Publish("orders", message); err != nil {
        return fmt.Errorf("publishing message: %w", err)
    }

    span.SetAttributes(attribute.Int("message.size", len(message)))

    return nil
}

func (c *Consumer) HandleOrderEvent(ctx context.Context, msg []byte) error {
    ctx, span := c.tracer.Start(ctx, "consume queue.orders",
        trace.WithSpanKind(trace.SpanKindConsumer),
        trace.WithAttributes(
            semconv.MessagingSystemKey.String("nats"), // messaging.system (no NATS enum const)
            semconv.MessagingOperationTypeReceive,
            semconv.MessagingDestinationName("orders")))
    defer span.End()

    var order Order
    if err := json.Unmarshal(msg, &order); err != nil {
        span.SetStatus(codes.Error, "message parsing failed")
        return fmt.Errorf("unmarshaling order: %w", err)
    }

    span.SetAttributes(attribute.String("order.id", order.ID))

    if err := c.processOrder(ctx, &order); err != nil {
        return fmt.Errorf("processing order: %w", err)
    }

    return nil
}
```

### Background Job

```go
func (w *Worker) ProcessBatch(ctx context.Context, batchID string) error {
    ctx, span := w.tracer.Start(ctx, "process batch",
        trace.WithAttributes(
            attribute.String("batch.id", batchID),
            attribute.String("worker.id", w.id)))
    defer span.End()

    items, err := w.repo.GetBatchItems(ctx, batchID)
    if err != nil {
        return fmt.Errorf("fetching batch items: %w", err)
    }

    span.SetAttributes(attribute.Int("batch.size", len(items)))

    processed := 0
    for _, item := range items {
        if err := w.processItem(ctx, item); err != nil {
            var rec log.Record
            rec.SetTimestamp(time.Now())
            rec.SetSeverity(log.SeverityWarn)
            rec.SetBody(attribute.StringValue("item processing failed"))
            rec.AddAttributes(
                attribute.String("item.id", item.ID),
            )
            rec.SetErr(err)
            w.logger.Emit(ctx, rec)
            continue
        }
        processed++
    }

    span.SetAttributes(
        attribute.Int("batch.processed", processed),
        attribute.Int("batch.failed", len(items)-processed),
    )

    return nil
}
```

## Enriching Spans from Instrumentation Libraries

When an instrumentation library (otelhttp, otelgin, otelmux, etc.) already creates spans, use `trace.SpanFromContext(ctx)` to add attributes to the existing span. Do not create a new span.

```go
func (h *Handler) CreateOrder(w http.ResponseWriter, r *http.Request) {
    // Span already created by instrumentation library (otelhttp, otelgin, etc.)
    ctx := r.Context()

    // Add business attributes to existing span
    span := trace.SpanFromContext(ctx)
    span.SetAttributes(
        attribute.String("business.operation", "order_creation"),
        attribute.String("customer.tier", extractCustomerTier(r)))

    order, err := h.service.CreateOrder(ctx, extractOrderData(r))
    if err != nil {
        span.SetStatus(codes.Error, "order creation failed")
        http.Error(w, "Failed to create order", http.StatusInternalServerError)
        return
    }

    span.SetAttributes(attribute.String("order.id", order.ID))

    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(order)
}
```

# OpenTelemetry Go Performance Tuning

Performance tuning reference for OpenTelemetry Go SDK. Covers sampling, batch processing, metric readers, attribute allocation, exporter configuration, and pipeline reliability.

---

## Performance Impact by Signal

| Signal | Unsampled Overhead | Sampled Overhead | Primary Cost |
|--------|-------------------|------------------|--------------|
| Traces | Near-zero (noop span) | Moderate | Allocations, export I/O |
| Metrics | N/A (always collected) | N/A | Aggregation, cardinality |
| Logs | Near-zero if `Enabled()` false | Low-moderate | Serialization, export I/O |

## Default Configuration Values

| Parameter | Default | Environment Variable |
|-----------|---------|---------------------|
| Batch span queue size | 2048 | `OTEL_BSP_MAX_QUEUE_SIZE` |
| Batch span export size | 512 | `OTEL_BSP_MAX_EXPORT_BATCH_SIZE` |
| Batch schedule delay | 5s | `OTEL_BSP_SCHEDULE_DELAY` |
| Batch export timeout | 30s | `OTEL_BSP_EXPORT_TIMEOUT` |
| Span attribute limit | 128 | `OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT` |
| Span event limit | 128 | `OTEL_SPAN_EVENT_COUNT_LIMIT` |
| Span link limit | 128 | `OTEL_SPAN_LINK_COUNT_LIMIT` |
| Attribute value length limit | Unlimited | `OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT` |
| Metric export interval | 60s | `OTEL_METRIC_EXPORT_INTERVAL` |
| Metric export timeout | 30s | `OTEL_METRIC_EXPORT_TIMEOUT` |
| OTLP export timeout | 10s | `OTEL_EXPORTER_OTLP_TIMEOUT` |

---

## Sampling

Sampling is the single most impactful performance lever for traces. Dropped spans do not record
attributes, events, or links, though the application still pays to construct values and options it
passes to `Start`.

### Head Sampling Configuration

Configure a sampler at the `TracerProvider` level to decide before span processing whether to record:

```go
// Sample 10% of traces
sampler := sdktrace.TraceIDRatioBased(0.1)

// Composite: respect upstream sampling decisions, ratio-sample the rest
sampler := sdktrace.ParentBased(
    sdktrace.TraceIDRatioBased(0.1),
    // Remote parent sampled -> always sample (honor upstream)
    // Remote parent not sampled -> never sample
)

tp := sdktrace.NewTracerProvider(
    sdktrace.WithSampler(sampler),
    sdktrace.WithBatcher(exporter),
)
```

### AlwaysRecord Sampler (v1.40.0+)

The `AlwaysRecord` sampler wraps another sampler and ensures spans are always recorded (attributes,
events, and status are captured) and passed to in-process span processors even when the inner
sampler decides `Drop`. Standard `SimpleSpanProcessor` and `BatchSpanProcessor` exporters still
skip `RecordOnly` spans, so this is useful for in-process processing such as span-to-metrics, not
for sending unsampled spans to a Collector.

```go
// Record all spans in-process; standard processors export only the sampled 10%
sampler := sdktrace.AlwaysRecord(sdktrace.TraceIDRatioBased(0.1))
```

### Sampling Decision Impact

```
AlwaysSample       -> Full span lifecycle: allocation + recording + export
AlwaysRecord(0.1)  -> All spans recorded in-process, 10% exported by standard processors
TraceIDRatioBased(0.1) -> 90% of spans become noop (near-zero cost)
NeverSample        -> All spans noop (useful for load testing)
```

`AlwaysSample` records and exports every span. `TraceIDRatioBased(0.1)` makes 90% of spans noop with near-zero cost. `ParentBased` wrapping respects upstream sampling decisions. `NeverSample` makes all spans noop, which is useful for benchmarking application-only performance.

> **Tail sampling**: For a Collector to decide from complete traces (error status, latency
> thresholds), configure the SDK to sample and export all candidate spans, then use the Collector's
> tail sampling processor. `AlwaysRecord` does not send its `RecordOnly` spans through the standard
> SDK span processors.

---

## Batch Processor Tuning

The `BatchSpanProcessor` buffers spans in a queue and exports them asynchronously in batches.

### How It Works

```
Application goroutine          BatchSpanProcessor goroutine
       |                                |
   span.End() --enqueue-->  channel queue (MaxQueueSize)
       |                                |
       |                         timer fires (ScheduleDelay)
       |                         OR batch full (MaxExportBatchSize)
       |                                |
       |                        --export batch--> Exporter
```

### Tuning for Throughput

For high-volume services (>10k spans/second):

```go
bsp := sdktrace.NewBatchSpanProcessor(exporter,
    sdktrace.WithMaxQueueSize(4096),       // Absorb bursts (default: 2048)
    sdktrace.WithMaxExportBatchSize(1024), // Fewer network calls (default: 512)
    sdktrace.WithBatchTimeout(10*time.Second), // Allow larger batches (default: 5s)
)
```

### Tuning for Latency

For services where trace delivery speed matters (debugging, alerting):

```go
bsp := sdktrace.NewBatchSpanProcessor(exporter,
    sdktrace.WithMaxExportBatchSize(128),       // Export smaller batches faster
    sdktrace.WithBatchTimeout(2*time.Second),   // Export more frequently
)
```

### Queue Full Behavior

When the queue fills, new spans are dropped silently by default. Telemetry loss is preferable to application slowdown in this design.

`WithBlockOnQueueFull()` changes this behavior to backpressure the application goroutine, blocking on `span.End()` until queue space is available.

With experimental SDK self-observability enabled via `OTEL_GO_X_OBSERVABILITY=true`, the SDK
exposes queue size/capacity and processed-span metrics. This feature may change incompatibly.

### SimpleSpanProcessor

`SimpleSpanProcessor` exports spans synchronously on `span.End()`. It adds latency to every span ending. It is deterministic, which makes it useful for tests and development. Short-lived CLI tools may also use it to ensure spans are exported before exit.

```go
ssp := sdktrace.NewSimpleSpanProcessor(exporter)
```

---

## Metric Reader Tuning

### PeriodicReader Configuration

The `PeriodicReader` collects and exports metrics on a fixed interval.

```go
reader := sdkmetric.NewPeriodicReader(exporter,
    sdkmetric.WithInterval(30*time.Second), // More frequent than default 60s
    sdkmetric.WithTimeout(15*time.Second),  // Timeout per export
)
```

Interval tradeoffs:
- 60s (default): Standard for dashboards and alerting
- 15-30s: Near-real-time monitoring, higher export overhead
- 5-10s: High-frequency use cases, significant overhead

The reader internally uses `sync.Pool` to recycle `ResourceMetrics` objects, reducing GC pressure during collection cycles.

### Views for Cardinality Control

Views filter attributes before aggregation, reducing the number of unique time series.

#### Attribute Filter

```go
// Drop a high-cardinality attribute from a specific metric
dropUserID := sdkmetric.NewView(
    sdkmetric.Instrument{Name: "http.server.request.duration"},
    sdkmetric.Stream{
        AttributeFilter: func(kv attribute.KeyValue) bool {
            return kv.Key != "user.id" // Exclude user.id from aggregation
        },
    },
)
```

An allowlist-style filter keeps only specified attributes:

```go
sdkmetric.Stream{
    AttributeFilter: func(kv attribute.KeyValue) bool {
        return kv.Key == "http.request.method" || kv.Key == "http.response.status_code"
    },
}
```

#### Drop Metric

```go
// Drop an entire metric
dropMetric := sdkmetric.NewView(
    sdkmetric.Instrument{Name: "runtime.*"},
    sdkmetric.Stream{Aggregation: sdkmetric.AggregationDrop{}},
)
```

#### Applying Views

```go
mp := sdkmetric.NewMeterProvider(
    sdkmetric.WithReader(reader),
    sdkmetric.WithView(dropUserID, dropMetric),
)
```

Metric attribute cardinality is the product of unique values across all recorded attributes. Attributes like user IDs or request IDs are unbounded and produce one time series per unique value. Attributes like HTTP method (~10 values) or status code (~50 values) are bounded.

> **Default cardinality limit (v1.44.0+):** `sdk/metric` now enforces a default cardinality
> limit of **2000** series per instrument (spec-compliant). Series beyond the limit are dropped
> and aggregated into an overflow series marked `attribute.Bool("otel.metric.overflow", true)`.
> This is a breaking change from the previous unlimited default. Override globally with the
> MeterProvider option `sdkmetric.WithCardinalityLimit(n)` (zero or negative disables the
> limit), or per instrument kind with the reader option
> `sdkmetric.WithCardinalityLimitSelector` (v1.43.0+). Views remain the correct tool for
> *shaping* cardinality; the limit is a backstop.

---

## Attribute Allocation Patterns

Attribute options can be a significant source of allocations in instrumented code.

### Pre-allocating Attribute Sets

For attributes used across many measurements, construct an attribute set and measurement option
once:

```go
var (
    successfulGET = attribute.NewSet(
        attribute.String("http.request.method", "GET"),
        attribute.Int("http.response.status_code", 200),
    )
    successfulGETOption = metric.WithAttributeSet(successfulGET)
)

func handleRequest(ctx context.Context, method string, status int) {
    counter.Add(ctx, 1, successfulGETOption)
}
```

Reusing `metric.WithAttributeSet(...)` avoids rebuilding and deduplicating the same attribute set
for every measurement. Benchmark dynamic cases before adding caches or pools.

### Slice Capacity

When building dynamic attribute sets, pre-allocating slice capacity eliminates repeated reallocations:

```go
attrs := make([]attribute.KeyValue, 0, 4)
attrs = append(attrs, attribute.String("service.name", svcName))
attrs = append(attrs, attribute.String("deployment.environment.name", env))
if region != "" {
    attrs = append(attrs, attribute.String("cloud.region", region))
}
```

A `nil` slice (`var attrs []attribute.KeyValue`) reallocates on the first and subsequent appends as it grows.

### Span Attribute Limits

The SDK enforces configurable limits on span attributes, events, and links. New span attributes
beyond the count limit are dropped; new events and links replace the oldest item (FIFO):

```go
tp := sdktrace.NewTracerProvider(
    sdktrace.WithSpanLimits(sdktrace.SpanLimits{
        AttributeCountLimit:         64,  // Default: 128
        EventCountLimit:             32,  // Default: 128
        LinkCountLimit:              16,  // Default: 128
        AttributePerEventCountLimit: 16,  // Default: 128
        AttributePerLinkCountLimit:  16,  // Default: 128
        AttributeValueLengthLimit:   256, // Default: Unlimited. Truncates long strings.
    }),
)
```

`AttributeValueLengthLimit` truncates strings, string-slice elements, byte slices, and those values
nested in slice attributes that would otherwise consume unbounded memory.

Since v1.47.0, nested (slice/map) attribute values are also capped at a depth of 64 by default
(`DefaultAttributeValueDepthLimit`); tune with `sdktrace.WithAttributeValueDepthLimit` or
`SpanLimits.AttributeValueDepthLimit`, and `sdklog.WithAttributeValueDepthLimit` for logs.

---

## Exporter Configuration

### gRPC vs HTTP

| Aspect | gRPC | HTTP/protobuf |
|--------|------|---------------|
| Connection overhead | Single long-lived connection | New connection per export (or HTTP/2 reuse) |
| Default port | 4317 | 4318 |
| Compression | Optional gzip | Optional gzip |
| Best for | Persistent connections, high throughput | Firewalls, load balancers, simpler infra |

### Compression

Compression reduces bandwidth at the cost of CPU.

gRPC:

```go
import "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"

exporter, err := otlptracegrpc.New(ctx,
    otlptracegrpc.WithCompressor("gzip"),
)
```

HTTP:

```go
import "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"

exporter, err := otlptracehttp.New(ctx,
    otlptracehttp.WithCompression(otlptracehttp.GzipCompression),
)
```

### OTLP/HTTP trace encoding

The trace HTTP exporter supports both OTLP protobuf (the default) and OTLP JSON. Select JSON
with `otlptracehttp.WithEncoding(otlptracehttp.EncodingJSON)` or set
`OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/json` (the signal-specific variable takes precedence
over `OTEL_EXPORTER_OTLP_PROTOCOL`). This released option is specific to `otlptracehttp`; do not
assume the metric and log HTTP exporters expose the same encoding option.

### Retry Defaults

The OTLP exporters use exponential backoff with jitter:

| Parameter | Default |
|-----------|---------|
| Enabled | true |
| Initial interval | 5s |
| Max interval | 30s |
| Max elapsed time | 1min |

gRPC retries honor server pushback. Since v1.45.0 the HTTP exporters correctly interpret
integer `Retry-After` values as seconds and also support HTTP-date values. On v1.44.0, integer
values were mistakenly treated as nanoseconds and HTTP-date values were ignored.

### Request Size Cap (v1.44.0+)

All OTLP exporters (trace/metric/log, gRPC and HTTP) cap the request payload at **64 MiB** by
default. The limit is applied *before* compression; oversized requests are treated as
non-retryable errors and dropped. Tune with `WithMaxRequestSize(bytes)` on the exporter —
zero or negative disables the limit entirely (not recommended).

### Timeout Tuning

```go
// Lower timeout: fail fast
exporter, err := otlptracegrpc.New(ctx,
    otlptracegrpc.WithTimeout(5*time.Second), // Default: 10s
)

// Higher timeout: allow more time for large batches
exporter, err := otlptracegrpc.New(ctx,
    otlptracegrpc.WithTimeout(30*time.Second),
)
```

---

## Log Signal Performance

### Enabled() Early Exit

The Logs API supports an optional `Enabled()` check for skipping expensive log construction when
the log level is below threshold or when a processor filters it. Call it at each emission where
the optimization matters; its result can change and should not be cached:

```go
logger := otel.Logger("my-service") // v1.47.0+; replaces deprecated otel/log/global

if logger.Enabled(ctx, log.EnabledParameters{Severity: log.SeverityInfo}) {
    var rec log.Record
    rec.SetBody(attribute.StringValue(expensiveStringBuild()))
    rec.SetSeverity(log.SeverityInfo)
    logger.Emit(ctx, rec)
}
```

### log.Record Value Type

`log.Record` is a struct value type (not an interface). Passing it by value keeps it stack-allocated, avoiding heap allocation:

```go
var rec log.Record
rec.SetBody(attribute.StringValue("operation completed"))
rec.SetSeverity(log.SeverityInfo)
rec.AddAttributes(attribute.String("component", "handler"))
logger.Emit(ctx, rec)
```

---

## Context Propagation Performance

### Extract Once per Boundary

W3C Trace Context propagation involves parsing and formatting fixed-format headers. The overhead is minimal. Extract once per request boundary:

```go
ctx = otel.GetTextMapPropagator().Extract(ctx, propagation.HeaderCarrier(r.Header))
```

Extracting multiple times for the same request performs redundant parsing with no benefit.

### Baggage Overhead

Baggage is propagated across all service boundaries via HTTP headers. Every key-value pair in baggage adds to every outgoing request's header size:

```go
bag, _ := baggage.New(
    baggageMember("tenant.id", tenantID),
)
ctx = baggage.ContextWithBaggage(ctx, bag)
```

Large values (e.g., full request bodies) or PII in baggage are propagated to every downstream service in every request.

---

## Graceful Shutdown

Proper shutdown ensures buffered telemetry is flushed before the process exits. Shutdown order: providers first (which flush their processors), then exporters.

```go
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()

if err := tp.Shutdown(ctx); err != nil {
    slog.Error("trace provider shutdown failed", "error", err)
}
if err := mp.Shutdown(ctx); err != nil {
    slog.Error("metric provider shutdown failed", "error", err)
}
if err := lp.Shutdown(ctx); err != nil {
    slog.Error("log provider shutdown failed", "error", err)
}
```

---

## Telemetry Pipeline Reliability

The OpenTelemetry SDK is designed so that telemetry failures do not crash or block the application:

- **Span creation never returns an error** -- it returns a noop span on failure
- **Metric recording never returns an error** -- measurements are silently dropped on failure
- **Export failures are retried** -- then dropped after max elapsed time
- **Queue overflow drops spans** -- the application is not blocked (unless `WithBlockOnQueueFull` is used)

---

## Monitoring the Pipeline

The released SDK and OTLP exporters expose experimental self-observability metrics when
`OTEL_GO_X_OBSERVABILITY=true`. They use the global `MeterProvider` and may change incompatibly:

| Metric | Meaning |
|--------|---------|
| `otel.sdk.processor.span.queue.size` | Current BatchSpanProcessor queue depth |
| `otel.sdk.processor.span.processed` | Spans processed; `error.type=queue_full` identifies queue drops |
| `otel.sdk.exporter.span.exported` | Spans whose export finished; `error.type` distinguishes failures |
| Exporter errors in logs | Export failures with error details |

Since v1.46.0, simple span and log processors record their `processed` metric when a record is
submitted to the exporter, before export completes, and do not derive `error.type` from the export
outcome. Do not interpret that processor metric as proof of successful export.

### ForceFlush

`ForceFlush` synchronously exports all buffered data:

```go
if err := tp.ForceFlush(ctx); err != nil {
    slog.Warn("failed to flush traces", "error", err)
}
```

As of v1.42.0, `TracerProvider.ForceFlush` iterates through all `SpanProcessor` instances and joins their errors together, instead of stopping at the first error. All processors get a chance to flush even if one fails.

`ForceFlush` in the request hot path synchronously exports all buffered data, adding latency to that request.

---

## Synchronous Instrument Enabled()

As of v1.40.0, all synchronous metric instruments (`Int64Counter`, `Float64Counter`, `Int64UpDownCounter`, `Float64UpDownCounter`, `Int64Histogram`, `Float64Histogram`, `Int64Gauge`, `Float64Gauge`) expose an `Enabled()` method. This allows skipping expensive metric value computation when the instrument is not recording:

```go
histogram, _ := meter.Float64Histogram("expensive.metric")

attrs := metric.WithAttributes(attribute.String("region", region))
if histogram.Enabled(ctx) {
    value := computeExpensiveValue() // Only called when instrument is active
    histogram.Record(ctx, value, attrs)
}
```

This is the metric equivalent of `Logger.Enabled()` for logs. When combined with Views that drop metrics, `Enabled()` returns `false` for dropped instruments.