# OpenTelemetry Python API

**Sources of Truth:** [opentelemetry-python](https://github.com/open-telemetry/opentelemetry-python) · [opentelemetry-python-contrib](https://github.com/open-telemetry/opentelemetry-python-contrib) · [Python API docs](https://opentelemetry-python.readthedocs.io/)

## Import Paths

### API packages (no SDK dependency)
```python
from opentelemetry import trace               # TracerProvider, get_tracer
from opentelemetry import metrics             # MeterProvider, get_meter
from opentelemetry._logs import get_logger_provider  # LoggerProvider access
from opentelemetry import baggage
from opentelemetry import context
from opentelemetry import propagate           # inject / extract
```

### SDK packages (implementation, required at runtime)
```python
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk._logs import LoggerProvider
from opentelemetry.sdk.resources import Resource
```

SDK packages ship in `opentelemetry-sdk`. The API packages ship in `opentelemetry-api`. Application code should import from the API; SDK imports belong in bootstrap/setup code.
The trace and metrics API surfaces are stable. The Logs API remains under active
stabilization as of 1.45.0, even though it is distributed in the stable-versioned
`opentelemetry-api` package.

## Global API Access

### Provider registration (bootstrap code)
```python
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk._logs import LoggerProvider
from opentelemetry._logs import set_logger_provider

tracer_provider = TracerProvider(resource=resource)
trace.set_tracer_provider(tracer_provider)

meter_provider = MeterProvider(resource=resource)
metrics.set_meter_provider(meter_provider)

logger_provider = LoggerProvider(resource=resource)
set_logger_provider(logger_provider)
```

### Provider lifecycle additions in 1.43.0 / 1.44.0

`MeterProvider.add_metric_reader()` and `remove_metric_reader()` are public as
of SDK 1.43.0, so a reader can be attached after provider construction. Removing
a registered reader detaches and shuts it down:

```python
meter_provider.add_metric_reader(reader)
# ... collect/export through reader ...
meter_provider.remove_metric_reader(reader)
```

SDK 1.44.0 also publicly exports `SynchronousMultiLogRecordProcessor` and
`ConcurrentMultiLogRecordProcessor` from `opentelemetry.sdk._logs`. Pass one as
`LoggerProvider(multi_log_record_processor=...)` when processor fan-out needs an
explicit sequential or parallel strategy; the default is synchronous.

On platforms with `os.register_at_fork`, the tracer, meter, and logger providers
refresh process-dependent resource attributes after a fork. Since 1.44.0 this
includes `service.instance.id`; the refreshed value overrides an explicitly set
value in the child process.

### Obtaining instances (application code)
```python
tracer = trace.get_tracer("my.library")
meter  = metrics.get_meter("my.library")
logger = get_logger_provider().get_logger("my.library")
```

Use the instrumenting library's import path or package name as the scope name. Pass `schema_url` to align with a semantic conventions version — see `otel-semantic-conventions`.

## Tracing API

### Creating spans
```python
tracer = trace.get_tracer("my.library")

# Context-manager form — preferred
with tracer.start_as_current_span("operation.name") as span:
    span.set_attribute("key", "value")
    do_work()

# Manual form — use when you need the span across call boundaries
span = tracer.start_span("operation.name")
token = context.attach(trace.set_span_in_context(span))
try:
    do_work()
finally:
    context.detach(token)
    span.end()
```

### Span attributes and status
```python
from opentelemetry.trace import StatusCode

span.set_attribute("http.request.method", "GET")   # prefer semantic conventions
span.set_attributes({"key1": "v1", "key2": 42})

# Record an exception and mark the span as failed
try:
    risky()
except Exception as exc:
    span.record_exception(exc)
    span.set_status(StatusCode.ERROR, description=str(exc))
    raise
```

Attribute names: follow [OpenTelemetry Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/). Cross-reference the `otel-semantic-conventions` skill.

### Span kind
```python
from opentelemetry.trace import SpanKind

tracer.start_as_current_span("http.request", kind=SpanKind.CLIENT)
# Values: INTERNAL (default), SERVER, CLIENT, PRODUCER, CONSUMER
```

### Context utilities
```python
# Get the active span
span = trace.get_current_span()

# Extract trace/span IDs
ctx = span.get_span_context()
trace_id = format(ctx.trace_id, "032x")
span_id  = format(ctx.span_id, "016x")

# Run code in a specific span's context
with trace.use_span(span):
    child_span = tracer.start_span("child")
```

## Metrics API

### Synchronous instruments
```python
meter = metrics.get_meter("my.library")

# Counter — monotonic, additive
counter = meter.create_counter(
    "requests.total",
    description="Total HTTP requests",
    unit="1",
)
counter.add(1, {"http.request.method": "GET", "http.response.status_code": 200})

# UpDownCounter — non-monotonic, additive (e.g., queue depth)
queue_size = meter.create_up_down_counter("queue.size", unit="1")
queue_size.add(1)   # enqueue
queue_size.add(-1)  # dequeue

# Histogram — distribution of values
duration = meter.create_histogram("request.duration", unit="ms")
duration.record(42.5, {"http.request.method": "GET"})
```

### Asynchronous (observable) instruments
```python
import psutil

def cpu_observer(options):
    yield metrics.Observation(psutil.cpu_percent(), {"cpu": "total"})

meter.create_observable_gauge(
    "system.cpu.utilization",
    callbacks=[cpu_observer],
    unit="%",
)

# Also: create_observable_counter, create_observable_up_down_counter
```

The callback receives an `options` argument and must `yield` (or return) `Observation` objects. Callbacks are invoked by the SDK on each collection cycle.

## Attributes

Since API/SDK 1.45.0, `AttributeValue` is the spec's `AnyValue`: `str | bool | int | float | bytes`,
`None`, (possibly heterogeneous or nested) sequences, and string-keyed mappings of those. `bytes`
values are passed through as bytes instead of being UTF-8 decoded. Before 1.45.0 only primitives
and homogeneous primitive sequences were accepted. Exporters and backends may not support
complex values.

```python
span.set_attribute("service.name", "api")
span.set_attribute("http.response.status_code", 200)
span.set_attribute("db.query.parameter.0", "alice")

# Metric attributes are plain dicts
counter.add(1, {"http.request.method": "GET"})
```

Attribute keys must follow semantic conventions where applicable — see `otel-semantic-conventions`.

## Propagation

`opentelemetry.propagate` delegates to the globally registered `TextMapPropagator` (default: W3C TraceContext + Baggage).

```python
from opentelemetry import propagate

# Inject into outgoing headers (e.g., requests dict)
headers = {}
propagate.inject(headers)

# Extract from incoming headers (e.g., WSGI environ or dict)
ctx = propagate.extract(carrier=request.headers)

# Run handler in extracted context
token = context.attach(ctx)
try:
    handle_request()
finally:
    context.detach(token)
```

### Custom / additional propagators
```python
from opentelemetry.propagators.composite import CompositePropagator
from opentelemetry.propagators.b3 import B3MultiFormat
from opentelemetry import propagate

propagate.set_global_textmap(
    CompositePropagator([B3MultiFormat()])
)
```

## Logs / Python Logging Bridge

The OTel Python SDK does not replace `logging` — it bridges it. The bridge converts stdlib `logging.LogRecord` objects into OTel log records and forwards them to the configured `LoggerProvider`.

### Manual wiring

Package required: `opentelemetry-instrumentation-logging`

```python
import logging
from opentelemetry._logs import get_logger_provider
from opentelemetry.instrumentation.logging.handler import LoggingHandler

# Attach to the root logger so all loggers inherit it
logging.getLogger().addHandler(
    LoggingHandler(logger_provider=get_logger_provider())
)
logging.getLogger().setLevel(logging.INFO)

# Now use stdlib logging normally
log = logging.getLogger("my.module")
log.info("handled request")  # emitted as OTel log record with trace context
```

The handler automatically injects the active span's `trace_id` and `span_id` into each log record, correlating logs to traces.
As of contrib 0.65b0, attributes that a configured stdlib formatter adds to the
`LogRecord` are also included in the exported OTel log attributes.

> **Deprecated path:** `opentelemetry.sdk._logs.LoggingHandler` is deprecated as of SDK 1.40.0/0.61b0. Always import from `opentelemetry.instrumentation.logging.handler`.

### `LoggingInstrumentor` behavior

`LoggingInstrumentor().instrument()` (also reachable via the `opentelemetry-instrument` CLI when the logging instrumentation is installed) has two separate behaviors:

- It installs the contrib `LoggingHandler` by default, routing stdlib log records through the global `LoggerProvider`. Disable that with `OTEL_PYTHON_LOG_AUTO_INSTRUMENTATION=false` or `enable_log_auto_instrumentation=False`.
- It can inject `otelTraceID`, `otelSpanID`, `otelTraceSampled`, and `otelServiceName` into stdlib `LogRecord`s for text log correlation. Use `inject_trace_context=True` to add those fields without changing the logging format, or `set_logging_format=True` / `OTEL_PYTHON_LOG_CORRELATION=true` to also call `logging.basicConfig` with a format that prints them.

```python
# Add correlation fields to LogRecord without changing logging.basicConfig
from opentelemetry.instrumentation.logging import LoggingInstrumentor
LoggingInstrumentor().instrument(inject_trace_context=True)
```

# Python OpenTelemetry Breaking Changes — Audit Workflow

Use this reference when upgrading existing Python OpenTelemetry code. It teaches how to find
deprecated or renamed APIs from upstream CHANGELOGs rather than embedding a static list that
rots with each release.

## Step 1: Fetch the CHANGELOGs

Fetch both released upstream sources before reviewing any code. As of
2026-10-03, the released ceiling is core **1.45.0** and contrib **0.66b0**:

```
WebFetch https://raw.githubusercontent.com/open-telemetry/opentelemetry-python/v1.45.0/CHANGELOG.md
WebFetch https://raw.githubusercontent.com/open-telemetry/opentelemetry-python-contrib/v0.66b0/CHANGELOG.md
```

Scan for entries marked **Deprecated**, **Removed**, or **Breaking** in the version range you are
crossing. Contrib and SDK carry independent version numbers but are released in lock-step (see
Step 2); check both even for SDK-only upgrades, because contrib instrumentation libraries depend
on the SDK ABI.

## Step 2: Understand the Dual Versioning Scheme

Python OpenTelemetry uses two independent version tracks:

| Track | Packages | Example |
|---|---|---|
| Stable (SemVer 1.x) | `opentelemetry-api`, `opentelemetry-sdk` | `1.45.0` |
| Beta (`0.Yb`-suffix) | `opentelemetry-instrumentation-*`, contrib packages, and experimental packages such as `opentelemetry-configuration` (OTLP exporters track stable 1.x) | `0.66b0` |

The two tracks move in lock-step: SDK `1.45.x` pairs with contrib `0.66bx`. When the CHANGELOG
entry says "version 0.66b0", look up the paired SDK version before concluding which SDK release
introduced the change.

The definitive indicator of which track a given package follows is its version string, not its
name: a `b` suffix (e.g. `0.63b0`) means beta track; a plain SemVer (e.g. `1.42.0`) means stable.
Check with `pip show <pkg>` or the PyPI `.info.version` field; independently released packages
can follow their own version tracks.

Fetch current versions from PyPI to orient yourself:

```
WebFetch https://pypi.org/pypi/opentelemetry-sdk/json          # .info.version → stable track
WebFetch https://pypi.org/pypi/opentelemetry-distro/json       # .info.version → beta track
```

## Step 3: Find Deprecated and Renamed APIs

In the CHANGELOG output, search for:

- `Deprecated` — the old symbol still exists but will be removed in a future release.
- `Removed` — the symbol is gone; code referencing it raises `ImportError` at startup.
- `Renamed` — import path or class name changed.

**Example deprecation an audit would catch:** The SDK's `opentelemetry.sdk._logs.LoggingHandler`
is deprecated; the class emits a `DeprecationWarning` at construction reading "`LoggingHandler` in
`opentelemetry-sdk` is deprecated. Use the handler from `opentelemetry-instrumentation-logging`
instead." The replacement is the `LoggingHandler` class from the contrib
`opentelemetry-instrumentation-logging` package, imported as a drop-in replacement:
```python
from opentelemetry.instrumentation.logging.handler import LoggingHandler
```
Code using the old SDK handler keeps working until removal, but the deprecation entry marks the
migration window. Add the contrib package to `requirements.txt`. See `references/api.md` for wiring
details. (Note: `LoggingInstrumentor` can install the contrib handler during instrumentation and can
also inject trace context into text logs; check the current `opentelemetry-instrumentation-logging`
behavior before treating it as only a formatting helper.)

For each finding, grep the codebase for the old symbol before concluding impact:

```bash
grep -r "LoggingHandler" .  # adjust pattern per finding
```

### Released 1.44.0 / 0.65b0 audit points

When crossing this release boundary, explicitly check for:

- Imports from `opentelemetry.sdk._configuration.file`; declarative setup moved
  to the public, experimental `opentelemetry.configuration` namespace. The old
  SDK extra remains only as an install alias.
- The removed Events API/SDK; emit a `LogRecord` with `event_name` instead.
- Custom environment-carrier code: `EnvironmentGetter.get` now requires the
  source mapping as its first argument rather than using a cached environment
  snapshot.
- Custom log exporters that read `ReadableLogRecord.context`; records buffered
  by `BatchLogRecordProcessor` now carry an empty `Context()` to reduce retained
  memory.
- `ProcessResourceDetector` consumers expecting `process.command_args` or
  `process.command_line`; these privacy-sensitive attributes are now omitted by
  default and require `include_command_args=True`.
- Pre-fork servers that explicitly set `service.instance.id`; tracer, meter,
  and logger providers now refresh process-dependent resource attributes in
  the child, and the regenerated `service.instance.id` overrides the configured
  value there.
- In-place mutation of `opentelemetry.context.Context`; inherited `dict`
  mutation methods no longer modify it. Use the context API to create updated
  contexts.
- Histogram configurations with non-finite or non-increasing explicit bucket
  boundaries; creating the metric stream's aggregation now raises `ValueError`.
  NaN and Inf measurements are dropped at the instrument boundary.
- `opentelemetry-instrumentation-elasticsearch`; contrib removed the package in
  0.65b0 because supported Elasticsearch clients provide native OTel
  instrumentation.

### Released 1.45.0 / 0.66b0 audit points

When crossing this release boundary, explicitly check for:

- Custom `opentelemetry._logs.Logger` subclasses; they must now implement
  `enabled()` (the Logger API, SDK, and `LogRecordProcessor` gained
  `enabled()` support).
- Code or exporters that assume primitive-only attribute values; `AttributeValue`
  is now `AnyValue` — see [api.md](api.md#attributes).
- OTLP HTTP exporters: new `urllib3` default transport and a `max_request_size`
  limit (default 64 MiB; larger requests are dropped; `0` disables) — see [performance.md](performance.md#exporter-configuration).
- `SimpleSpanProcessor` now drops spans ended after `shutdown()`; SDK
  self-metrics `otel.sdk.processor.{span,log}.processed` now count at
  submission and no longer carry exporter failures as `error.type`.
- Metric `View` instrument-name matching is now case-insensitive, and
  tracer/meter/logger configurator scope-name matching is case-sensitive, on
  every platform.
- Declarative config: an unset `${VAR}` without default no longer raises — see
  [declarative-setup.md](declarative-setup.md#yaml-config).
- Contrib: `opentelemetry-instrumentation-kafka-python` dropped
  `kafka-python-ng`; `AsyncioInstrumentor.trace_to_thread` is renamed to
  `wrap_to_thread_func` (old name kept as an alias); the `anthropic`, `claude-agent-sdk`, `langchain`, and
  `weaviate` instrumentations moved to the
  [opentelemetry-python-genai](https://github.com/open-telemetry/opentelemetry-python-genai)
  repository.

## Step 4: Audit Semantic Convention Renames

Instrumentation libraries adopt updated semconv attribute names on their own schedule; the SDK
CHANGELOG and contrib CHANGELOG both carry these entries. Common patterns include:

- HTTP: `http.method` → `http.request.method`, `http.url` → `url.full`
- DB: `db.sql.table` → `db.collection.name`
- RPC: `rpc.system` values and metric names shift with each stable semconv release

For the canonical attribute-level mapping, cross-reference the **`otel-semantic-conventions`**
skill, which fetches the upstream semconv CHANGELOG and spec. Do not rely on memory for attribute
names — fetch the spec.

Core 1.45.0 updates `opentelemetry-semantic-conventions` to the 1.44.0
semantic-conventions release. Re-run this audit even when your application API
imports did not change.

If your code or dashboards query raw attribute names (e.g. Prometheus label names, Datadog facets,
log filters), a semconv rename is a silent breaking change: telemetry keeps flowing but queries
return no data. Treat semconv entries in the CHANGELOG with the same urgency as API removals.

## Step 5: Check Declarative Configuration

In 1.44.0 / 0.65b0, declarative configuration moved from the private
`opentelemetry.sdk._configuration.file` module to the separate
`opentelemetry-configuration` package and public `opentelemetry.configuration`
namespace. The new package remains experimental, so audit it on each upgrade.

Before upgrading, fetch the README/schema for the release you are upgrading to. As of the
latest released SDK (1.45.0), the declarative configuration README and schema are:

```text
WebFetch https://raw.githubusercontent.com/open-telemetry/opentelemetry-python/v1.45.0/opentelemetry-configuration/README.rst
WebFetch https://raw.githubusercontent.com/open-telemetry/opentelemetry-python/v1.45.0/opentelemetry-configuration/src/opentelemetry/configuration/schema.json
```

Install `opentelemetry-configuration` directly. The old
`opentelemetry-sdk[file-configuration]` extra is a deprecated alias that still
installs it for compatibility.

Cross-reference the **`references/declarative-setup.md`** reference in this skill for the current
YAML schema and activation API. If a release changes the package, that reference is the
authoritative source of the updated usage pattern.

## Step 6: Verify Installed Versions Match

After identifying changes, confirm the running environment matches the target version:

```bash
pip show opentelemetry-sdk opentelemetry-api opentelemetry-configuration \
  opentelemetry-distro opentelemetry-instrumentation
```

A mismatch between `opentelemetry-api` and `opentelemetry-sdk` versions is a common source of
subtle breakage; they must be pinned to the same stable-track version.

## Cross-References

- Attribute renames and semconv mapping: **`otel-semantic-conventions`** skill.
- Declarative config schema and activation: **`references/declarative-setup.md`** in this skill.
- Version selection across languages: **`otel-sdk-versions`** skill.

# Python SDK Setup with Declarative File Configuration

Configure the OpenTelemetry SDK in Python via a YAML file processed by
the `opentelemetry-configuration` package (`opentelemetry.configuration`).

> **Experimental public package.** Release 1.44.0 / 0.65b0 moved declarative
> configuration from the private `opentelemetry.sdk._configuration.*` modules
> into the public `opentelemetry-configuration` package. The package is still
> experimental (Development Status: Alpha), so its API and models may change
> between minor releases. Pin and audit the changelog when upgrading.

For the YAML configuration schema, load the `otel-declarative-config` skill.

## Sources of Truth

For YAML schema details, fetch the upstream sources listed in the
`otel-declarative-config` skill. For Python-specific facts:

| Fact | Fetch |
|---|---|
| Latest `opentelemetry-sdk` / `opentelemetry-api` | `WebFetch https://pypi.org/pypi/opentelemetry-sdk/json` (`.info.version`) |
| Latest `opentelemetry-configuration` | `WebFetch https://pypi.org/pypi/opentelemetry-configuration/json` |
| Latest `opentelemetry-distro` | `WebFetch https://pypi.org/pypi/opentelemetry-distro/json` |
| Latest OTLP exporter | `WebFetch https://pypi.org/pypi/opentelemetry-exporter-otlp/json` |
| Declarative config support (released 1.45.0) | `WebFetch https://raw.githubusercontent.com/open-telemetry/opentelemetry-python/v1.45.0/opentelemetry-configuration/README.rst` |
| Vendored schema (released 1.45.0) | `WebFetch https://raw.githubusercontent.com/open-telemetry/opentelemetry-python/v1.45.0/opentelemetry-configuration/src/opentelemetry/configuration/schema.json` |
| SDK CHANGELOG through 1.45.0 | `WebFetch https://raw.githubusercontent.com/open-telemetry/opentelemetry-python/v1.45.0/CHANGELOG.md` |

## Install

Install the separate experimental package directly. It installs matching
`opentelemetry-api` and `opentelemetry-sdk` versions plus the YAML/schema
dependencies:

```bash
pip install opentelemetry-configuration
```

`opentelemetry-sdk[file-configuration]` remains as a deprecated compatibility
alias that installs `opentelemetry-configuration`; prefer the direct package for
new setups.

Add exporters and instrumentation libraries as needed:

```bash
pip install opentelemetry-exporter-otlp opentelemetry-distro \
            opentelemetry-instrumentation-fastapi \
            opentelemetry-instrumentation-logging
```

## Activation

As of SDK **1.45.0 / 0.66b0** there are two supported activation paths.

### Zero-code: `OTEL_CONFIG_FILE`

Point the SDK configurator at a config file and it loads it in place of the
env-var init path — no bootstrap module required. This works with the
`opentelemetry-instrument` CLI (and any distro that runs the SDK configurator):

```bash
OTEL_CONFIG_FILE=otel.yaml opentelemetry-instrument python -m uvicorn app:app
```

When `OTEL_CONFIG_FILE` is set, the file is the **sole** source of SDK
construction: spec-defined `OTEL_*` variables that have schema equivalents are
ignored. Env vars are still read via `${VAR}` / `${VAR:-default}` substitution
inside the file and by components the file enables (e.g. resource detectors).
Python-specific `OTEL_PYTHON_*` configuration extensions are also bypassed
because the regular env-var initialization path is skipped; express equivalent
behavior in the file or in code.

### Programmatic: `configure_sdk`

`configure_sdk` is the single entry point that takes a parsed
`OpenTelemetryConfiguration`, builds the resource, and applies the
tracer/meter/logger providers and propagator globally — honoring the top-level
`disabled` flag. A bootstrap module must be imported before any application code.

```python
"""bootstrap.py — import before any application code."""
from opentelemetry.configuration import configure_sdk, load_config_file

configure_sdk(load_config_file("otel.yaml"))
```

`load_config_file` parses YAML/JSON, performs `${VAR}` and `${VAR:-default}` substitution,
validates against the vendored schema, and returns a **fully-typed**
`OpenTelemetryConfiguration` — nested fields (`resource`, `tracer_provider`,
`meter_provider`, `logger_provider`) are typed dataclasses, not raw dicts. (Prior
to 1.43.0 the loader returned raw dicts for nested fields and callers had to
build the dataclass tree by hand.) Both `load_config_file` and `configure_sdk`
are exported from `opentelemetry.configuration`.

For finer control, `opentelemetry.configuration.file` exports paired
`create_tracer_provider` / `configure_tracer_provider`,
`create_meter_provider` / `configure_meter_provider`, and
`create_logger_provider` / `configure_logger_provider` functions, plus
`create_propagator`, `configure_propagator`, and `create_resource`. The
`create_*` functions return objects without registering them globally; the
`configure_*` functions set the corresponding global.

### Application entry point

```python
import bootstrap  # noqa: F401 — must be first import

from opentelemetry import trace, metrics
# ... rest of app imports
```

The import order matters: telemetry emitted before `bootstrap` is imported (or
before the CLI activates the config) uses the no-op defaults. API objects such
as proxy tracers obtained before setup can bind to a provider installed later,
but spans started before setup are not retroactively recorded.

## YAML Config

`file_format` is required and must be a string version. The loader validates it
per the configuration spec: unsupported major versions are rejected; newer minor
versions with the same major version are accepted with a warning. Verified
2026-10-15: `opentelemetry-configuration` 0.66b1 (SDK 1.45.1) vendors schema
v1.1.0 (as does the 1.45.0 pinned above) and its loader targets 1.1, so `"1.0"` and `"1.1"` load without a warning
and `"1.2"` warns (1.44.0 targeted 1.0 and warned on `"1.1"`).

Minimal verified skeleton (all three signals, console exporters):

```yaml
file_format: "1.0"
resource:
  attributes:
    - name: service.name
      value: "${SERVICE_NAME:-my-service}"
    - name: service.version
      value: "0.1.0"
tracer_provider:
  processors:
    - simple:
        exporter:
          console: {}
meter_provider:
  readers:
    - periodic:
        interval: 5000
        exporter:
          console: {}
logger_provider:
  processors:
    - simple:
        exporter:
          console: {}
```

For OTLP exporters and the full schema, load the `otel-declarative-config` skill.

Env-var substitution uses `${VAR}` and `${VAR:-default}` syntax; use `$$` for a
literal dollar sign. Since 1.45.0, `load_config_file` substitutes only scalar
values after parsing (never keys or comments), and an unset variable with no
default becomes an empty value (YAML null) instead of an error; a resource
attribute that resolves to null is skipped with a warning.

## Instrumentor Activation

Since 1.44.0, the loader can activate installed contrib instrumentors from the
`instrumentation/development.python` section. Keys are
`opentelemetry_instrumentor` entry-point names; `enabled: false` skips one:

```yaml
instrumentation/development:
  python:
    requests:
      enabled: true
    urllib3:
      enabled: false
```

Additional keys are passed to `instrument()`. If an instrumentor publishes a
dataclass in its `configuration` attribute, those values are type-coerced and
validated first. Already-active instrumentors are skipped to avoid double
instrumentation; unknown or failing instrumentors are logged and do not stop
the remaining entries.

## Key Facts

- **`configure_sdk` / `configure_*` set globals.** `configure_sdk` applies every
  signal; individually `configure_tracer_provider` calls
  `trace.set_tracer_provider(...)`, `configure_meter_provider` calls
  `metrics.set_meter_provider(...)`, and `configure_logger_provider` calls
  `set_logger_provider(...)`. After the bootstrap runs, `trace.get_tracer(...)` etc.
  return providers backed by the configured SDK.

- **Absent provider section ⇒ provider global left unset.** An absent
  `tracer_provider`, `meter_provider`, or `logger_provider` section leaves that
  signal's global untouched. Propagators differ: when configuration is enabled,
  `configure_sdk` always applies the propagator configuration, and an absent
  propagator section installs an empty `CompositePropagator`.

- **Configured ID generator is applied.** Since 1.44.0 the
  `tracer_provider.id_generator` configuration is wired into `TracerProvider`;
  resolve the supported shape from the vendored schema.

- **Fields wired in 1.45.0.** Top-level `log_level` sets the level of the
  `opentelemetry` Python logger (SDK self-diagnostics); top-level
  `attribute_limits` applies to the tracer and logger providers; the
  experimental `tracer_configurator/development`,
  `meter_configurator/development`, and `logger_configurator/development`
  per-scope `enabled` overrides are applied (earlier releases parsed and
  discarded them). Logger-configurator `minimum_severity` / `trace_based` are
  ignored with a warning.

- **`LoggingHandler` import.** The deprecated
  `opentelemetry.sdk._logs.LoggingHandler` (deprecated in 1.40.0/0.61b0) should be
  replaced with `opentelemetry.instrumentation.logging.handler.LoggingHandler`
  (from `opentelemetry-instrumentation-logging`).

- **Env substitution** is handled by `load_config_file`; see `otel-declarative-config`
  for the substitution syntax reference.

# OpenTelemetry Python Instrumentation Libraries

## Detecting Existing Instrumentation

Before adding instrumentation, check whether it is already present:

```bash
# Check installed instrumentation packages
pip show opentelemetry-distro opentelemetry-instrumentation 2>/dev/null
pip list | grep opentelemetry-instrumentation-

# Check if the zero-code CLI wrapper is in use
grep -r "opentelemetry-instrument" Dockerfile docker-compose.yml Makefile .env* 2>/dev/null

# Check for per-app instrumentation in source
grep -rn "Instrumentor\|instrument_app\|instrument(" --include="*.py" .
```

If you find `opentelemetry-instrument` in the run command, the zero-code path is active and instrumentors are injected automatically. If you find `*Instrumentor.instrument_app(...)` calls in source, the per-app path is in use.

## Zero-Code Path

The zero-code path wraps the Python process with the `opentelemetry-instrument` CLI. No source changes are required.

### Install

```bash
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install   # detects installed libs; installs matching instrumentors
```

`opentelemetry-bootstrap` reads `pip list`, maps installed packages to known instrumentors, and installs them. Re-run after adding new dependencies.

### Run

```bash
opentelemetry-instrument \
  --service_name my-service \
  python -m uvicorn app:app
```

Without `OTEL_CONFIG_FILE`, configuration comes from environment variables
(`OTEL_SERVICE_NAME`, `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_TRACES_SAMPLER`,
etc.) or distro defaults. With `opentelemetry-configuration` installed, the
same CLI can activate declarative YAML through `OTEL_CONFIG_FILE`; see
`declarative-setup.md`.

### Common env vars

| Variable | Example |
|----------|---------|
| `OTEL_SERVICE_NAME` | `my-service` |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://localhost:4318` |
| `OTEL_TRACES_EXPORTER` | `otlp` |
| `OTEL_METRICS_EXPORTER` | `otlp` |
| `OTEL_LOGS_EXPORTER` | `otlp` |
| `OTEL_PROPAGATORS` | `tracecontext,baggage` |
| `OTEL_PYTHON_DISABLED_INSTRUMENTATIONS` | `requests,logging` |
| `OTEL_SEMCONV_STABILITY_OPT_IN` | `http,database` |
| `OTEL_PYTHON_LOG_AUTO_INSTRUMENTATION` | `false` |

In contrib 0.65b0, the aiopg, asyncpg, Cassandra, pymemcache, pymongo, and
Tortoise ORM instrumentors added database semantic-convention migration
support. Their behavior now follows `OTEL_SEMCONV_STABILITY_OPT_IN`; audit
attributes and downstream queries before enabling `database` or
`database/dup`.

## Per-App Instrumentation

Use this form when you need fine-grained control, or when the CLI wrapper is not suitable (e.g., serverless, custom startup code, or the declarative config path).

### FastAPI (verified)

```python
from fastapi import FastAPI
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

app = FastAPI()
FastAPIInstrumentor.instrument_app(app)   # attaches ASGI middleware to this app instance
```

`instrument_app` is the per-app variant: it attaches the ASGI middleware to a specific app instance and produces `SpanKind.SERVER` spans for incoming requests.

> The API shape differs across instrumentors. FastAPI and Flask define static
> `instrument_app(app)` methods (the upstream Flask usage commonly calls it
> through `FlaskInstrumentor()`); Django uses the inherited instance method
> `DjangoInstrumentor().instrument()` with no app argument. Check the contrib
> package for the exact form.

### Flask

```python
from flask import Flask
from opentelemetry.instrumentation.flask import FlaskInstrumentor

app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)
```

### Django

For Django, call `instrument()` at startup (e.g., in `manage.py` or `wsgi.py`) before the WSGI app is created:

```python
from opentelemetry.instrumentation.django import DjangoInstrumentor

DjangoInstrumentor().instrument()
```

### Requests (HTTP client)

```python
from opentelemetry.instrumentation.requests import RequestsInstrumentor

RequestsInstrumentor().instrument()   # patches the requests library globally
```

## Contrib Catalog

Source of truth: `instrumentation/` directory in the [opentelemetry-python-contrib](https://github.com/open-telemetry/opentelemetry-python-contrib) repo. Do not pin versions in configuration — fetch the exact version from PyPI at install time.

### Web Frameworks

| Library | Package |
|---------|---------|
| FastAPI | `opentelemetry-instrumentation-fastapi` |
| Flask | `opentelemetry-instrumentation-flask` |
| Django | `opentelemetry-instrumentation-django` |
| Starlette | `opentelemetry-instrumentation-starlette` |
| Tornado | `opentelemetry-instrumentation-tornado` |
| Falcon | `opentelemetry-instrumentation-falcon` |
| Pyramid | `opentelemetry-instrumentation-pyramid` |
| ASGI (generic) | `opentelemetry-instrumentation-asgi` |
| WSGI (generic) | `opentelemetry-instrumentation-wsgi` |

### HTTP Clients

| Library | Package |
|---------|---------|
| requests | `opentelemetry-instrumentation-requests` |
| httpx | `opentelemetry-instrumentation-httpx` |
| urllib3 | `opentelemetry-instrumentation-urllib3` |
| urllib (stdlib) | `opentelemetry-instrumentation-urllib` |
| aiohttp client | `opentelemetry-instrumentation-aiohttp-client` |
| aiohttp server | `opentelemetry-instrumentation-aiohttp-server` |

### Databases

| Library | Package |
|---------|---------|
| psycopg (v3) | `opentelemetry-instrumentation-psycopg` |
| psycopg2 | `opentelemetry-instrumentation-psycopg2` |
| SQLAlchemy | `opentelemetry-instrumentation-sqlalchemy` |
| MySQL (mysql-connector) | `opentelemetry-instrumentation-mysql` |
| mysqlclient | `opentelemetry-instrumentation-mysqlclient` |
| PyMySQL | `opentelemetry-instrumentation-pymysql` |
| pymssql | `opentelemetry-instrumentation-pymssql` |
| SQLite3 | `opentelemetry-instrumentation-sqlite3` |
| asyncpg | `opentelemetry-instrumentation-asyncpg` |
| aiopg | `opentelemetry-instrumentation-aiopg` |
| pymongo | `opentelemetry-instrumentation-pymongo` |
| Redis | `opentelemetry-instrumentation-redis` |
| pymemcache | `opentelemetry-instrumentation-pymemcache` |
| Cassandra | `opentelemetry-instrumentation-cassandra` |
| Tortoise ORM | `opentelemetry-instrumentation-tortoiseorm` |
| DB-API 2.0 (generic) | `opentelemetry-instrumentation-dbapi` |

`opentelemetry-instrumentation-elasticsearch` was removed from contrib in
0.65b0 because the supported Elasticsearch client versions provide native OTel
instrumentation. Remove that package when upgrading and follow the client's
instrumentation guidance.

### Messaging

| Library | Package |
|---------|---------|
| Celery | `opentelemetry-instrumentation-celery` |
| kafka-python | `opentelemetry-instrumentation-kafka-python` |
| confluent-kafka | `opentelemetry-instrumentation-confluent-kafka` |
| aiokafka | `opentelemetry-instrumentation-aiokafka` |
| pika (RabbitMQ) | `opentelemetry-instrumentation-pika` |
| aio-pika (RabbitMQ async) | `opentelemetry-instrumentation-aio-pika` |
| boto3 SQS | `opentelemetry-instrumentation-boto3sqs` |
| botocore | `opentelemetry-instrumentation-botocore` |
| Remoulade | `opentelemetry-instrumentation-remoulade` |

### Logging / System

| Library | Package |
|---------|---------|
| logging (stdlib) | `opentelemetry-instrumentation-logging` |
| structlog | `opentelemetry-instrumentation-structlog` |
| uncaught exceptions (process/thread/asyncio → logs) | `opentelemetry-instrumentation-exceptions` |
| system-metrics | `opentelemetry-instrumentation-system-metrics` |
| Jinja2 | `opentelemetry-instrumentation-jinja2` |
| asyncio | `opentelemetry-instrumentation-asyncio` |
| threading | `opentelemetry-instrumentation-threading` |
| click | `opentelemetry-instrumentation-click` |
| asyncclick | `opentelemetry-instrumentation-asyncclick` |
| AWS Lambda | `opentelemetry-instrumentation-aws-lambda` |
| gRPC | `opentelemetry-instrumentation-grpc` |

## Manual Instrumentation Patterns

Use these when no contrib instrumentor covers the target. Follow semconv — see the `otel-semantic-conventions` skill for attribute names.

Use the current stable attribute constants from `opentelemetry.semconv.attributes` (e.g. `http.request.method`, `url.full`, `db.collection.name`). The older `opentelemetry.semconv.trace.SpanAttributes.HTTP_*` / `DB_*` names (`http.method`, `http.url`, `db.sql.table`) are deprecated. During contrib's migration window, instrumentors can still emit legacy names by default; use `OTEL_SEMCONV_STABILITY_OPT_IN` deliberately and audit downstream queries so manual and auto-instrumented spans stay aligned.

### HTTP Client Call

```python
import requests
from opentelemetry import trace
from opentelemetry.semconv.attributes import http_attributes, url_attributes

tracer = trace.get_tracer(__name__)

def call_external_api(url: str) -> dict:
    with tracer.start_as_current_span(
        "GET",
        kind=trace.SpanKind.CLIENT,
        attributes={
            http_attributes.HTTP_REQUEST_METHOD: "GET",
            url_attributes.URL_FULL: url,
        },
    ) as span:
        resp = requests.get(url)
        span.set_attribute(
            http_attributes.HTTP_RESPONSE_STATUS_CODE, resp.status_code
        )
        return resp.json()
```

### Database Call

```python
from opentelemetry import trace
from opentelemetry.semconv.attributes import db_attributes

tracer = trace.get_tracer(__name__)

def fetch_user(conn, user_id: str) -> dict:
    with tracer.start_as_current_span(
        "SELECT users",
        kind=trace.SpanKind.CLIENT,
        attributes={
            db_attributes.DB_SYSTEM_NAME: "postgresql",
            db_attributes.DB_OPERATION_NAME: "SELECT",
            db_attributes.DB_COLLECTION_NAME: "users",
        },
    ) as span:
        row = conn.execute("SELECT * FROM users WHERE id = %s", (user_id,)).fetchone()
        span.set_attribute("db.rows_returned", 1 if row else 0)
        return row
```

### Background Job

```python
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer(__name__)

def process_batch(batch_id: str, items: list) -> None:
    with tracer.start_as_current_span("process_batch") as span:
        span.set_attribute("batch.id", batch_id)
        span.set_attribute("batch.size", len(items))
        processed = 0
        for item in items:
            try:
                handle_item(item)
                processed += 1
            except Exception as exc:
                span.record_exception(exc)
        span.set_attribute("batch.processed", processed)
        span.set_attribute("batch.failed", len(items) - processed)
        if processed < len(items):
            span.set_status(Status(StatusCode.ERROR, "some items failed"))
```

## Enriching Auto-Instrumented Spans

When a contrib instrumentor already creates a span (e.g., FastAPI's `GET /work` span), retrieve it from context and add attributes — do not create a new span.

```python
from opentelemetry import trace

@app.get("/orders")
def list_orders(customer_id: str):
    # Span created by FastAPIInstrumentor; enrich it instead of creating a child
    span = trace.get_current_span()
    span.set_attribute("customer.id", customer_id)
    span.set_attribute("business.operation", "list_orders")

    orders = db.query_orders(customer_id)
    span.set_attribute("result.count", len(orders))
    return orders
```

`trace.get_current_span()` returns the active span or a no-op span if none is active — it is always safe to call.

# OpenTelemetry Python Performance Tuning

Performance tuning reference for the OpenTelemetry Python SDK. Covers sampling, batch processing, metric readers, cardinality control via Views, asyncio context propagation, exporter configuration, log-handler cost, and graceful shutdown.

---

## Performance Impact by Signal

| Signal | Unsampled Overhead | Sampled Overhead | Primary Cost |
|--------|-------------------|------------------|--------------|
| Traces | Lower (non-recording span) | Moderate | Context/sampling or recording/export I/O |
| Metrics | N/A (always collected) | N/A | Aggregation, cardinality |
| Logs | Low if handler level filtered | Low-moderate | Serialization, export I/O |

---

## Default Configuration Values

The SDK reads defaults from environment variables; check the [released OpenTelemetry Python SDK changelog](https://github.com/open-telemetry/opentelemetry-python/blob/v1.45.0/CHANGELOG.md) or env-var spec for current values — do not treat any number here as authoritative.

| Parameter | Environment Variable | Note |
|-----------|---------------------|------|
| BSP max queue size | `OTEL_BSP_MAX_QUEUE_SIZE` | Check SDK default |
| BSP max export batch size | `OTEL_BSP_MAX_EXPORT_BATCH_SIZE` | Check SDK default |
| BSP schedule delay (ms) | `OTEL_BSP_SCHEDULE_DELAY` | Check SDK default |
| BSP export timeout (ms) | `OTEL_BSP_EXPORT_TIMEOUT` | Not applied — see [BatchSpanProcessor Tuning](#constructor-arguments-and-env-vars) |
| Span attribute count limit | `OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT` | Check SDK default |
| Span event count limit | `OTEL_SPAN_EVENT_COUNT_LIMIT` | Check SDK default |
| Span link count limit | `OTEL_SPAN_LINK_COUNT_LIMIT` | Check SDK default |
| Attribute value length limit | `OTEL_SPAN_ATTRIBUTE_VALUE_LENGTH_LIMIT` | Unlimited if unset |
| Log record attribute count limit | `OTEL_LOGRECORD_ATTRIBUTE_COUNT_LIMIT` | Falls back to `OTEL_ATTRIBUTE_COUNT_LIMIT` |
| Log record attribute value length limit | `OTEL_LOGRECORD_ATTRIBUTE_VALUE_LENGTH_LIMIT` | Falls back to `OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT` |
| Metric export interval (ms) | `OTEL_METRIC_EXPORT_INTERVAL` | Check SDK default |
| Metric export timeout (ms) | `OTEL_METRIC_EXPORT_TIMEOUT` | Check SDK default |
| OTLP export timeout (s) | `OTEL_EXPORTER_OTLP_TIMEOUT` | Python reads this as **seconds** (default `10`), deviating from the spec, which defines it in milliseconds |
| Traces sampler | `OTEL_TRACES_SAMPLER` | `parentbased_always_on` if unset |
| Traces sampler arg | `OTEL_TRACES_SAMPLER_ARG` | Ratio for ratio-based samplers |

---

## Sampling

Sampling is the most impactful performance lever for traces. Unsampled spans
are non-recording: they skip attribute/event storage and export, but still incur
context propagation, ID generation, and sampling-decision work.

### Head Sampling Configuration

Configure a sampler on the `TracerProvider`:

```python
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.sampling import (
    TraceIdRatioBased,
    ParentBased,
    ALWAYS_ON,
    ALWAYS_OFF,
)

# Sample 10% of root traces
sampler = TraceIdRatioBased(0.1)

# Respect upstream decisions; ratio-sample new roots
sampler = ParentBased(root=TraceIdRatioBased(0.1))

provider = TracerProvider(sampler=sampler)
```

Via environment variables (no code change):

```bash
OTEL_TRACES_SAMPLER=traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1
```

Supported `OTEL_TRACES_SAMPLER` values: `always_on`, `always_off`, `traceidratio`, `parentbased_always_on`, `parentbased_always_off`, `parentbased_traceidratio`.

### Sampling Decision Impact

```
ALWAYS_ON                   -> Full span lifecycle: allocation + recording + export
ParentBased(ALWAYS_ON)      -> Same, but respects upstream not-sampled decisions
TraceIdRatioBased(0.1)      -> ~90% of root spans become non-recording
ALWAYS_OFF                  -> All spans are non-recording
```

The SDK default is `ParentBased(ALWAYS_ON)`: it honors upstream sampling decisions propagated via W3C TraceContext and samples new root traces.

> **Tail sampling**: A Collector `tail_sampling` processor can decide from
> complete traces that reach it (for example by error status or latency), but
> it cannot recover spans discarded by SDK head sampling. Keep that interaction
> in mind when combining both stages.

---

## BatchSpanProcessor Tuning

`BatchSpanProcessor` buffers completed spans and exports them asynchronously in batches.

```
from opentelemetry.sdk.trace.export import BatchSpanProcessor
```

### How It Works

```
Application thread              Background thread
      |                                |
  span.end() --enqueue-->  queue (max_queue_size)
      |                                |
      |                      schedule_delay_millis elapses
      |                      OR batch reaches max_export_batch_size
      |                                |
      |                       --export batch--> Exporter
```

### Constructor Arguments and Env Vars

| Constructor arg | Environment variable | Purpose |
|----------------|---------------------|---------|
| `max_queue_size` | `OTEL_BSP_MAX_QUEUE_SIZE` | In-memory queue capacity |
| `max_export_batch_size` | `OTEL_BSP_MAX_EXPORT_BATCH_SIZE` | Spans per export call |
| `schedule_delay_millis` | `OTEL_BSP_SCHEDULE_DELAY` | Max wait before export |
| `export_timeout_millis` | `OTEL_BSP_EXPORT_TIMEOUT` | Stored but not applied by `BatchSpanProcessor` as of SDK 1.45.0 |

### Tuning for Throughput

For high-volume services:

```python
from opentelemetry.sdk.trace.export import BatchSpanProcessor

bsp = BatchSpanProcessor(
    exporter,
    max_queue_size=8192,          # Absorb bursts
    max_export_batch_size=1024,   # Fewer network calls per flush
    schedule_delay_millis=10_000, # Allow larger batches to accumulate
)
```

### Tuning for Latency

For services where trace delivery speed matters (live debugging, alerting):

```python
bsp = BatchSpanProcessor(
    exporter,
    max_export_batch_size=128,   # Export smaller batches sooner
    schedule_delay_millis=2_000, # Export more frequently
)
```

### Queue-Full Behavior

When the queue is full, adding a new span evicts the oldest queued span and logs
`Queue full, dropping Span.` The SDK prioritizes application throughput over
telemetry completeness. Monitor queue-full warnings and the queue capacity set
by `OTEL_BSP_MAX_QUEUE_SIZE`.

### SimpleSpanProcessor

`SimpleSpanProcessor` exports spans synchronously on `span.end()`, adding exporter latency to every span boundary. Use it for:

- Tests and development (deterministic, no background thread)
- Short-lived CLI tools that must export before exit

```python
from opentelemetry.sdk.trace.export import SimpleSpanProcessor

provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(exporter))
```

Avoid `SimpleSpanProcessor` in production services — it blocks the calling thread on every `span.end()`.

---

## Metric Reader Tuning

### PeriodicExportingMetricReader

`PeriodicExportingMetricReader` collects and exports metrics on a fixed interval.

```python
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader

reader = PeriodicExportingMetricReader(
    exporter,
    export_interval_millis=30_000,  # More frequent than default
    export_timeout_millis=15_000,   # Timeout per export attempt
)
```

SDK 1.43.0 added public APIs for changing readers after construction:

```python
meter_provider.add_metric_reader(reader)
meter_provider.remove_metric_reader(reader)  # also shuts down reader
```

Adding the same reader twice or removing an unregistered reader logs a warning
and returns. Readers remain independent and collect separate metric streams.

Or via environment variables:

```bash
OTEL_METRIC_EXPORT_INTERVAL=30000
OTEL_METRIC_EXPORT_TIMEOUT=15000
```

Interval tradeoffs:

- Longer interval (default): lower overhead, suitable for dashboards and alerting
- 15–30s: near-real-time monitoring, moderately higher overhead
- 5–10s: high-frequency use cases, significant CPU and network overhead

---

## Views for Cardinality Control

Cardinality is the number of unique label-value combinations across all recorded attributes. Unbounded attributes (user IDs, request IDs, trace IDs) produce one time series per unique value and are a common source of memory and storage blowup.

`View` instances filter or transform metrics before they reach storage.

```python
from opentelemetry.sdk.metrics.view import View, DropAggregation
from opentelemetry.sdk.metrics import MeterProvider
```

### Allowlist Attribute Filter

Keep only specific attributes on a metric, dropping everything else:

```python
view = View(
    instrument_name="http.server.request.duration",
    attribute_keys={"http.request.method", "http.response.status_code"},
)
```

`attribute_keys` is a `set[str]`. Any recorded attribute not in the set is discarded before aggregation. An empty set `set()` drops all attributes, producing a single time series for that metric.

### Drop an Entire Metric

```python
drop_view = View(
    instrument_name="debug.*",  # Wildcard matching supported
    aggregation=DropAggregation(),
)
```

### Applying Views

```python
mp = MeterProvider(
    metric_readers=[reader],
    views=[view, drop_view],
)
```

Every matching View creates a metric stream, with two exceptions: a View using
`DropAggregation` matches but produces no stream (it discards the instrument's
measurements), and a View that is incompatible with the instrument (for example,
an explicit-bucket histogram on an asynchronous instrument) is skipped with a
warning. Ordering does not make the first match win. Overlapping Views can
therefore produce multiple streams; when their metric identities conflict the SDK
logs a warning but still emits the conflicting streams. A metric with no matching
View uses the default aggregation.

Attributes like HTTP method (~10 values) or response status code (~50 values) are bounded. Attributes like `user.id`, `request.id`, or `session.id` are unbounded and should be filtered out with Views unless you specifically intend per-user metrics.

---

## asyncio Context Propagation

The Python SDK uses `contextvars.ContextVar` (via the `opentelemetry-api` `Context` type) to propagate the active span and baggage. `asyncio` tasks automatically inherit a copy of the current `contextvars` context when created with `asyncio.create_task()` or `loop.create_task()`.

### Correct Pattern in asyncio

```python
import asyncio
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

async def handle_request():
    with tracer.start_as_current_span("handle_request") as span:
        # Child tasks created here inherit this span's context
        await asyncio.gather(
            fetch_data(),
            fetch_config(),
        )

async def fetch_data():
    # The active span from the parent coroutine is visible here
    with tracer.start_as_current_span("fetch_data"):
        ...
```

### Pitfall: Crossing into Threads

`asyncio.to_thread()` copies the current `contextvars` context into the worker thread, so the active span is available there. Mutations to `ContextVar` inside the thread do not propagate back to the asyncio event loop:

```python
import asyncio
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def blocking_work():
    # The span from the calling coroutine IS visible here (context was copied in)
    current_span = trace.get_current_span()
    # Any span started here is local to this thread's context copy
    with tracer.start_as_current_span("blocking_work"):
        ...
    # This span does NOT become the active span back in the event loop

async def main():
    with tracer.start_as_current_span("main"):
        await asyncio.to_thread(blocking_work)
        # Active span here is still "main" — the thread's mutations are gone
```

`loop.run_in_executor(...)` and manually spawned threads may not automatically copy the current context. Prefer `asyncio.to_thread()` when possible; otherwise pass the context explicitly for deterministic behavior:

```python
import asyncio
import threading
from contextvars import copy_context
from opentelemetry import context

# run_in_executor: wrap the callable in the copied context
async def main():
    loop = asyncio.get_running_loop()
    ctxvars = copy_context()
    await loop.run_in_executor(None, ctxvars.run, blocking_work)

# manual thread: attach the captured OTel context
ctx = context.get_current()  # Capture context in the calling thread

def worker():
    token = context.attach(ctx)  # Attach the captured context
    try:
        with tracer.start_as_current_span("worker"):
            ...
    finally:
        context.detach(token)

t = threading.Thread(target=worker)
t.start()
```

### Pitfall: Context Lost Across `await` with Manual Context Setting

Avoid manually calling `context.attach()` across `await` boundaries without matching `context.detach()`. Unmatched attaches accumulate context frames and create subtle propagation bugs. Prefer `start_as_current_span()` as a context manager, which handles attach/detach automatically.

---

## Exporter Configuration

### gRPC vs HTTP

| Aspect | gRPC (`otlp.proto.grpc`) | HTTP (`otlp.proto.http`) |
|--------|--------------------------|--------------------------|
| Default port | 4317 | 4318 |
| Connection model | Persistent, multiplexed | HTTP/1.1 (`urllib3` transport by default since 1.45.0; `requests` only when a `requests.Session` is supplied, e.g. `session=`; install the `[requests]` extra) |
| Compression | Optional gzip | Optional gzip |
| Best for | High throughput, stable connections | Firewalls, load balancers, simpler setup |

```python
# gRPC exporter
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter(
    endpoint="http://localhost:4317",
    compression=Compression.Gzip,  # from grpc import Compression
)

# HTTP exporter
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

exporter = OTLPSpanExporter(
    endpoint="http://localhost:4318/v1/traces",
)
```

### Compression

Enable gzip compression to reduce bandwidth at the cost of CPU:

```bash
OTEL_EXPORTER_OTLP_COMPRESSION=gzip
```

Or programmatically for gRPC:

```python
from grpc import Compression

exporter = OTLPSpanExporter(compression=Compression.Gzip)
```

### Retry and Timeout

The OTLP exporters retry on transient errors (connection refused, 5xx responses). Check the current released retry behavior in the [exporter source](https://github.com/open-telemetry/opentelemetry-python/tree/v1.45.0/exporter) — do not assume specific backoff intervals.

Configure timeout via environment variable or constructor:

```bash
OTEL_EXPORTER_OTLP_TIMEOUT=5   # seconds
```

```python
exporter = OTLPSpanExporter(timeout=5)  # seconds in constructor
```

Note: Python interprets `OTEL_EXPORTER_OTLP_TIMEOUT` and the `timeout=` constructor argument in **seconds** (default `10`). This deviates from the OpenTelemetry specification, which defines the variable in milliseconds (default `10000`). A value of `5` is a 5-second timeout in Python, not 5 milliseconds. This is a known, tracked deviation ([opentelemetry-python#4044](https://github.com/open-telemetry/opentelemetry-python/issues/4044)); revisit this guidance if that issue is resolved.

Since 1.45.0 the OTLP HTTP exporters also take `max_request_size` (default
64 MiB, measured before compression; `0` disables): larger serialized requests
are dropped instead of sent.

Lower timeout: fail fast and free the batch processor for the next export cycle.
Higher timeout: accommodate large batches or slow backends.

---

## Log Handler Cost

The OpenTelemetry Python SDK integrates with Python's standard `logging` module via `LoggingHandler`. Log records emitted through this handler are converted to OTel log records and processed by the `LoggerProvider`.

```python
import logging
from opentelemetry.sdk._logs import LoggerProvider
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.instrumentation.logging.handler import LoggingHandler

logger_provider = LoggerProvider()
logger_provider.add_log_record_processor(
    BatchLogRecordProcessor(log_exporter)
)

handler = LoggingHandler(level=logging.WARNING, logger_provider=logger_provider)
logging.getLogger().addHandler(handler)
```

Performance considerations:

- **Level filter first**: Set `LoggingHandler(level=logging.WARNING)` to avoid processing DEBUG/INFO records entirely. Python's standard `logging` level check happens before the OTel handler is invoked.
- **BatchLogRecordProcessor vs SimpleLogRecordProcessor**: Prefer `BatchLogRecordProcessor` in production to avoid blocking the logging call on export I/O. `SimpleLogRecordProcessor` exports synchronously on every log record.
- **Body construction cost**: Avoid constructing expensive log message bodies (e.g. via `%`-formatting with heavy objects) at the call site when the level is filtered. Python's `logging` lazy `%`-formatting applies only if the record passes the level filter.

---

## Graceful Shutdown

Proper shutdown ensures buffered telemetry is flushed before the process exits.

```python
import signal
import sys

def shutdown(sig=None, frame=None):
    # Shutdown order: providers flush their processors, then exporters close
    tracer_provider.shutdown()
    meter_provider.shutdown()
    logger_provider.shutdown()
    sys.exit(0)

signal.signal(signal.SIGTERM, shutdown)
signal.signal(signal.SIGINT, shutdown)
```

`shutdown()` blocks until all buffered data is exported or the processor's timeout elapses.

### force_flush()

`force_flush()` synchronously exports all buffered data immediately:

```python
# Force-flush before a checkpoint or test assertion
tracer_provider.force_flush()
meter_provider.force_flush()
logger_provider.force_flush()
```

`force_flush()` is synchronous and adds latency at the call site. Do not call it in the request hot path. Use it for:

- Pre-fork checkpoints (e.g. before `os.fork()`)
- Test teardown (ensure spans are exported before assertions)
- Graceful drain during rolling deploys

### asyncio Shutdown

In an asyncio application, wrap provider shutdown in a coroutine or run it from the event loop's cleanup:

```python
import asyncio

async def shutdown_providers():
    loop = asyncio.get_running_loop()
    # Providers are synchronous; run in executor to avoid blocking the loop
    await loop.run_in_executor(None, tracer_provider.shutdown)
    await loop.run_in_executor(None, meter_provider.shutdown)
    await loop.run_in_executor(None, logger_provider.shutdown)
```

---

## Telemetry Pipeline Reliability

Built-in processors isolate exporter failures from application code, but the
exact behavior depends on the processor and exporter:

- **Batch processors decouple export I/O** from application threads; simple
  processors perform export synchronously and therefore add exporter latency
- **Built-in span processors catch exporter exceptions** and log them; custom
  samplers, processors, and exporters must honor the same non-throwing contract
- **Invalid metric measurements** such as NaN and Inf are logged and dropped at
  the instrument boundary (SDK 1.44.0+)
- **OTLP exporters retry eligible failures** according to their transport-specific
  retry behavior; other exporters may not retry
- **Queue overflow drops spans** — the application is not blocked

---

## Monitoring the Pipeline

Watch for SDK-emitted warnings in application logs. The `BatchSpanProcessor` logs when spans are dropped due to queue overflow. Enable SDK debug logging during load testing:

```python
import logging
logging.getLogger("opentelemetry").setLevel(logging.DEBUG)
```

Key signals to watch:

| Indicator | Meaning |
|-----------|---------|
| `Queue full, dropping Span.` warnings | Queue overflow — increase `max_queue_size` or reduce `schedule_delay_millis` |
| Export timeout errors | Backend/exporter too slow or batch too large — tune the exporter timeout or `max_export_batch_size` (BSP `export_timeout_millis` is [not applied](#constructor-arguments-and-env-vars)) |
| High memory growth | Metric cardinality explosion — add Views to filter attributes |