# `attributes`: advanced use-cases

## Scope with include / exclude

Limit the actions to a subset so you don't touch unrelated telemetry. `include` is evaluated first; `exclude` then trims that set.

```yaml
processors:
  attributes/scoped:
    include:
      match_type: strict
      services: ["auth-service"]
    exclude:
      match_type: strict
      attributes:
        - key: test_request
          value: true
    actions:
      - key: sensitive_data
        action: delete
      - key: user.email
        action: hash
```

Only spans from `auth-service` are considered, and test requests within that set are skipped. For metrics, only `metric_names` and `resources` are valid match fields — see [Configuration](configuration.md#valid-fields-per-signal).

## Hash PII instead of deleting

Hashing keeps a stable, joinable token without exposing the raw value. Hash by key or by regex `pattern`:

```yaml
actions:
  - key: user.email
    action: hash
  - pattern: "^pii_.*"
    action: hash
```

## Extract structured fields from a string

`extract` runs a regex with **named** capture groups against a string attribute and creates one attribute per group. The source value is left in place.

```yaml
actions:
  - key: http.url
    pattern: '^(?P<http_scheme>https?):\/\/(?P<http_host>[^\/]+)(?P<http_path>.*)'
    action: extract
  # http.url = "https://api.example.com/v1/users"
  # -> http_scheme="https", http_host="api.example.com", http_path="/v1/users"
```

## Convert types

Coerce an attribute to `int`, `double`, or `string` for downstream compatibility:

```yaml
actions:
  - key: http.status_code
    action: convert
    converted_type: int
  - key: response_time_ms
    action: convert
    converted_type: double
```

## Ordered-action pipelines

Actions run top to bottom, so you can stage a transform: copy through a temporary key, then clean up.

```yaml
actions:
  - key: operation        # 1. backfill a default
    value: default
    action: insert
  - key: svc.operation    # 2. copy to the standardized key
    from_attribute: operation
    action: upsert
  - key: operation        # 3. drop the old key
    action: delete
```

## Inject request context

Pull values from transport metadata or an authenticator (requires `include_metadata: true` on the receiver, or an authenticator extension — see [Configuration](configuration.md#context-values)).

```yaml
actions:
  - key: client.ip
    from_context: client.address
    action: insert
  - key: enduser.id
    from_context: auth.subject
    action: insert
```

## Named instances

Configure the same type more than once with `type/name` and reference each in the pipeline that needs it:

```yaml
processors:
  attributes/redact:
    actions:
      - key: user.email
        action: hash
  attributes/enrich:
    actions:
      - key: region
        value: us-east-1
        action: upsert

service:
  pipelines:
    traces:
      processors: [attributes/redact, attributes/enrich]
```

## Combining with `resource` and `filter`

- Pair with **`resource`** when you also need to edit resource-level attributes (`attributes` cannot — it only sees span/log/datapoint attributes).
- Run **before `filter`** so the attribute you want to filter on exists, or has been normalized, by the time `filter`'s condition is evaluated.

```yaml
processors:
  attributes:
    actions:
      - key: env
        from_attribute: deployment.env
        action: upsert
  filter:
    traces:
      span:
        - 'attributes["env"] == "test"'   # drop test spans normalized above

service:
  pipelines:
    traces:
      processors: [attributes, filter]
```

# `attributes`: configuration

The processor operates on **span / log-record / metric-datapoint attributes** — not resource attributes (use the `resource` processor for those). It has two parts: a required ordered list of `actions`, and an optional `include`/`exclude` matching block that scopes which records the actions apply to.

## Root configuration

| Field | Type | Default | Required | Description |
|-------|------|---------|----------|-------------|
| `actions` | list | none | Yes | Ordered list of attribute actions. At least one is required; applied in the order specified. |
| `include` | object | none | No | Match criteria; only matching records are processed. |
| `exclude` | object | none | No | Match criteria; matching records are skipped. When both are set, `include` is evaluated first, then `exclude` filters the included set. |

## Action fields

Each entry in `actions` is one operation:

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `key` | string | Conditional | Attribute key to act on. Required for all actions except `delete`/`hash` when only a `pattern` is given. |
| `action` | string | Yes | One of `insert`, `update`, `upsert`, `delete`, `hash`, `extract`, `convert` (case-insensitive). |
| `value` | any | Conditional | Value to set (string/int/double/bool). Supports env-var expansion (`${env:VAR}`); an unset reference falls back to `default_value`. Required for `insert`/`update`/`upsert` unless `from_attribute`, `from_context`, or `default_value` is used. Mutually exclusive with `from_attribute`/`from_context`. |
| `from_attribute` | string | Conditional | Copy the value from another attribute on the same record. Mutually exclusive with `value` and `from_context`. |
| `from_context` | string | Conditional | Pull the value from request context. Mutually exclusive with `value` and `from_attribute`. See [Context values](#context-values). |
| `default_value` | any | No | Fallback value when the `value`/`from_attribute`/`from_context` source is missing (`insert`/`update`/`upsert` only; v0.152.0). Prevents the action from being skipped; ignored by other actions. |
| `pattern` | string | Conditional | Regex (Go syntax). For `delete`/`hash`, matches attribute **keys** to act on. For `extract`, the regex applied to the value (must use named capture groups). |
| `converted_type` | string | Conditional | Target type for `convert`: `int`, `double`, or `string`. Required for `convert`. |

### Action types

| Action | Requires | Behavior |
|--------|----------|----------|
| `insert` | `key` + one of `value`/`from_attribute`/`from_context` | Adds the attribute only if the key does **not** already exist. No-op if the key exists, or if the source is missing and no `default_value` is set. |
| `update` | `key` + one of `value`/`from_attribute`/`from_context` | Replaces the value only if the key **already** exists. No-op if the key is absent, or if the source is missing and no `default_value` is set. |
| `upsert` | `key` + one of `value`/`from_attribute`/`from_context` | Insert if absent, update if present (insert + update). No-op only if the source value is missing and no `default_value` is set. |
| `delete` | `key` and/or `pattern` | Removes the named key and/or every key matching `pattern`. |
| `hash` | `key` and/or `pattern` | Replaces the value(s) of the named key and/or keys matching `pattern` with their SHA-256 hash (hex string). |
| `extract` | `key` + `pattern` (named groups) | Applies the regex to the **string** value of `key` and creates one attribute per named capture group. Source value is left unchanged. Only acts on string values; overwrites existing attributes when group names collide. |
| `convert` | `key` + `converted_type` | Converts the existing value of `key` to `int`/`double`/`string`. No-op if the key is absent; on a failed conversion the original value is kept (logged at debug). |

Example covering several actions:

```yaml
processors:
  attributes:
    actions:
      - key: environment        # default value if absent
        value: production
        action: insert
      - key: db.statement       # redact in place
        value: "[REDACTED]"
        action: update
      - key: user.email         # hash PII
        action: hash
      - pattern: "^temp_.*"     # bulk delete by key regex
        action: delete
      - key: http.status_code   # coerce type
        action: convert
        converted_type: int
```

## Include / exclude matching

`include` and `exclude` share the same structure. Both are optional; if neither is set, every record is processed.

| Field | Type | Description |
|-------|------|-------------|
| `match_type` | string | `strict` (exact, case-sensitive) or `regexp` (Go regexp). Required when `include`/`exclude` is present. |
| `regexp` | object | Regex tuning, only valid with `match_type: regexp`. See [Regexp options](#regexp-options). |
| `services` | list[string] | Match by service name (traces and logs only). |
| `span_names` | list[string] | Match by span name (traces only). |
| `span_kinds` | list[string] | Match by span kind, e.g. `SPAN_KIND_INTERNAL` (traces only). |
| `log_bodies` | list[string] | Match by log body, **string bodies only** (logs only). |
| `log_severity_texts` | list[string] | Match by log severity text (logs only). |
| `log_severity_number` | object | Match by numeric severity (logs only). See below. |
| `metric_names` | list[string] | Match by metric name (metrics only). |
| `attributes` | list[object] | Match by attribute: each entry has `key` and optional `value`. Key-only matches any value. |
| `resources` | list[object] | Match by resource attribute: `key` and optional `value`. |
| `libraries` | list[object] | Match by instrumentation library: `name` and optional `version`. |

`log_severity_number` takes `min` (inclusive minimum severity number — TRACE 1-4, DEBUG 5-8, INFO 9-12, WARN 13-16, ERROR 17-20, FATAL 21-24) and `match_undefined` (when `true`, records with no severity number also match; default `false`).

### Valid fields per signal

- **Traces:** `services`, `span_names`, `span_kinds`, `attributes`, `resources`, `libraries`.
- **Logs:** `log_bodies` (string bodies), `log_severity_texts`, `log_severity_number`, `attributes`, `resources`, `libraries`.
- **Metrics:** `metric_names`, `resources`. (No `services`, `attributes`, `span_*`, or `log_*` fields.)

### Regexp options

When `match_type: regexp`, a nested `regexp` block can tune an LRU cache of match results for high-throughput pipelines:

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `cacheenabled` | bool | `false` | Cache compiled-regex match results. |
| `cachemaxnumentries` | int | unlimited | Cap on cache entries when caching is enabled. |

```yaml
include:
  match_type: regexp
  regexp:
    cacheenabled: true
    cachemaxnumentries: 1000
  services: ["^auth-.*", "^payment-.*"]
```

## Context values

`from_context` pulls a value from request context. Three source kinds:

- **`client.address`** — the client IP (plain key, no prefix).
- **`metadata.*`** — gRPC metadata / HTTP headers, e.g. `metadata.x-request-id`. Requires the receiver to set `include_metadata: true`.
- **`auth.*`** — data from a server authenticator extension, e.g. `auth.subject`. Requires an authenticator on the receiver; available keys depend on the authenticator.

If a context key has multiple values, they are joined with `;`.

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        include_metadata: true   # required for metadata.* keys

processors:
  attributes/context:
    actions:
      - key: client.ip
        from_context: client.address
        action: insert
      - key: request_id
        from_context: metadata.x-request-id
        action: insert
      - key: enduser.id
        from_context: auth.subject
        action: insert
```

# `attributes`: known quirks

## Action order matters

`actions` run strictly in the order listed, and each sees the result of the previous ones. A `delete` before a `from_attribute` copy removes the source first; an `insert` of a default before an `upsert` copy changes what gets copied. Order cheap actions (`delete`, `insert`) before expensive ones (`extract`, `convert`) when possible. If related actions are split across named processor instances, their order in `service.pipelines.<signal>.processors` is likewise load-bearing: the later instance sees changes from the earlier one only when it is listed after it.

## It does NOT touch resource attributes

This is the most common confusion. `attributes` operates on **span / log-record / metric-datapoint** attributes only. To add, rename, hash, or delete **resource** attributes (e.g. `service.name`, `host.name`, `deployment.environment`), use the **`resource`** processor — it has the same `actions` model but resource scope. Note that include/exclude *matching* can read resource attributes (the `resources` match field), but the actions still only write to non-resource attributes.

## insert vs update vs upsert

- `insert` — writes only if the key is **absent**. Silently no-ops on an existing key. Use for defaults/backfill.
- `update` — writes only if the key is **present**. Silently no-ops on a missing key. Use to rewrite/redact known fields.
- `upsert` — always writes (insert if absent, update if present). The safe choice when you don't care about prior state; also the right tool for debugging "why didn't my action apply" (swap to `upsert` to rule out existence checks).

A frequent bug: using `insert` and seeing nothing change because the key already existed, or `update` and seeing nothing because the key was absent.

## `extract` quirks

- Requires **named** capture groups (`(?P<name>...)`); a pattern with unnamed groups is a config error.
- Only acts on **string** values. A non-string source (or a missing key) is a no-op.
- **Overwrites** existing attributes when a capture-group name collides with an existing key — there is no insert-only mode for `extract`.
- The source `key`'s own value is left unchanged; `extract` only adds the captured fields.

## Regexp performance

`match_type: regexp` filters and `pattern`-based `delete`/`hash` compile and run Go regexes per record. On high-throughput pipelines enable the LRU cache (`regexp.cacheenabled: true`, bound with `cachemaxnumentries`) and keep patterns simple — anchored prefixes (`^foo_.*`) let the engine reject non-matches early, while unanchored patterns scan the whole value. (Go's `regexp` is RE2-based and linear-time, so there's no catastrophic backtracking; the cost is per-record CPU and cache pressure, not exponential blow-up.) Remember `.` matches any character; use `\.` for a literal dot.

## Metric support caveats (identity conflict)

For metrics, **adding new** data-point attributes is safe, but **modifying or deleting existing** data-point attributes can create an **identity conflict**: a metric's identity includes its data-point attributes, and this processor does **not** re-aggregate. Changing them can split or duplicate time series and corrupt cardinality. If you must change existing metric attributes, re-aggregate downstream (a connector, or the `metricstransform` processor), or fix it at the source. This warning does **not** apply to traces or logs — attributes there can be freely modified. Also note `attributes` does not rename metrics or change metric names — that's `metricstransform`.

## Common config errors

| Symptom | Cause | Fix |
|---------|-------|-----|
| Action silently does nothing | `insert` on an existing key, or `update` on a missing key | Use `upsert`, or check the include/exclude filter isn't blocking the record. |
| Config validation error | More than one of `value` / `from_attribute` / `from_context` set on one action | Use exactly one value source per action. |
| Config validation error | `extract` pattern has unnamed groups | Use `(?P<name>...)` named groups. |
| Convert no-op | Value can't be parsed to the target type (e.g. `"abc"` → int), or key absent | Pre-validate; conversion failures keep the original value (logged at debug). |
| `from_context` empty | Receiver missing `include_metadata: true`, or no authenticator for `auth.*` | Configure the receiver/authenticator (see configuration.md). |
| Wrong-signal field rejected | e.g. `span_names` under a metrics pipeline | Use only the match fields valid for the signal. |

## Stability

All three signals (traces, logs, metrics) are **Beta** — production viable, breaking changes rare. Hashing uses SHA-256 on modern versions; very old collectors used SHA-1, so hashes are not comparable across a version boundary. `default_value` on an action is newer (v0.152.0) — confirm your collector version before relying on it.

# `attributes` processor

| | |
|-|-|
| Kind | processor |
| Type | `attributes` |
| Signals | traces (Beta), logs (Beta), metrics (Beta) |
| Distributions | core, contrib, k8s |
| Go module | `github.com/open-telemetry/opentelemetry-collector-contrib/processor/attributesprocessor` |
| Upstream README | <https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/attributesprocessor> |

## Description

Modifies the **attributes** of spans, log records, and metric data points through an ordered list of `actions`. Each action names an attribute `key` and one of `insert`, `update`, `upsert`, `delete`, `hash`, `extract`, or `convert`; actions run in the order listed, so later actions see the effect of earlier ones. Values can be literal (`value`), copied from another attribute (`from_attribute`), or pulled from request context (`from_context` — client IP, transport metadata, or authenticator data).

An optional `include`/`exclude` matching block scopes the processor to a subset of telemetry (by service, span name, attribute, resource attribute, instrumentation library, log body, log severity, metric name, …) so only matching records are touched. This processor operates on **span/log/metric-datapoint attributes only** — it does **not** touch resource attributes; use the `resource` processor for those.

## Main use-cases

Use it when:
- You want to standardize, backfill, or rename attribute keys across services (`insert` defaults, `upsert` to copy with `from_attribute`).
- You need to redact or hash sensitive attribute values (`hash` PII, `delete` credentials) by key or regex `pattern`.
- You want to extract structured fields from a string attribute (`extract` with named capture groups) or coerce types (`convert`).
- You want to inject request context — client IP, headers, authenticated subject — onto telemetry (`from_context`).

Avoid it when:
- You need to edit **resource** attributes — use `resource` (same action set, resource scope).
- You need arbitrary conditional logic, math, or cross-field transforms — use `transform` (OTTL is a superset).
- You only need to drop records — use `filter`.
- You need a fail-closed allow list for egress governance — use `redaction`.

## Related components

- `resource` — same `actions` model but applied to **resource** attributes instead of span/log/datapoint attributes.
- `transform` — OTTL-based mutation; a superset of what `attributes` can do, at the cost of more verbosity.
- `redaction` — allow/block-list masking with a fail-closed allow list and URL/DB sanitizers.
- `filter` — drops whole records that match a condition (run `attributes` before it to shape the matchable attributes).

## Details

- [Configuration](configuration.md) — the `actions` list (every action type and its `key`/`value`/`pattern`/`from_attribute`/`from_context`/`converted_type` fields), the `include`/`exclude` matching block, and context-value sources.
- [Verification](verification.md) — telemetrygen recipe that adds/overwrites an attribute and confirms it in `debug` output.
- [Advanced use-cases](advanced.md) — include/exclude scoping, hashing PII, regex `extract`, type conversion, ordered-action pipelines, named instances, and combining with `resource`/`filter`.
- [Known quirks](quirks.md) — action order, the resource-attribute confusion, `insert`/`update`/`upsert` semantics, `extract` overwrites, regexp performance, metric identity-conflict caveats, and stability.

# `attributes`: verification

See [Verification harness](../../SKILL.md#verification-harness) for how to run this end-to-end.

`attributes` ships in the `core`, `contrib`, and `k8s` distributions.

Config (`attributes-verify.yaml`) — an `upsert` action that sets (or overwrites) an `env` attribute on every span:

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
processors:
  attributes:
    actions:
      - key: env
        value: prod
        action: upsert
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [attributes]
      exporters: [debug]
```

Send a known number of traces, each carrying an `env` attribute with a *different* value so the `upsert` overwrite is observable. Use `--workers 1` so the `--traces` count (which is **per worker**) is exact:

```bash
telemetrygen traces --otlp-insecure --otlp-endpoint localhost:4317 \
  --workers 1 --traces 5 --telemetry-attributes 'env="staging"'
```

Flags used, all confirmed against the `otel-telemetrygen` skill (`references/flags.md`):

- `--otlp-insecure` — disable transport security (common flag).
- `--otlp-endpoint` — destination, defaults to `localhost:4317` for gRPC (common flag).
- `--workers` — concurrent workers; `1` keeps the count exact (common flag).
- `--traces` — traces **per worker** (trace-specific flag).
- `--telemetry-attributes` — span-level attribute as `key="value"` (must be quoted). Resource-level `--otlp-attributes` would attach to the Resource, which the `attributes` processor does **not** touch.

**What proves it worked:** in the `debug` exporter output, every span shows `env: Str(prod)` in its span attributes — the value sent (`staging`) has been overwritten by the `upsert` action. To prove the **insert** half of `upsert`, drop `--telemetry-attributes` and send plain traces; `env: Str(prod)` still appears because the key was absent and got inserted. (Use `insert` instead of `upsert` to confirm it leaves a pre-existing `env` untouched, or `update` to confirm it only writes when the key already exists.)

You can run the same recipe with `telemetrygen logs ... --logs 5` and a `logs` pipeline to confirm log-record attributes; `--logs` is likewise per-worker, so keep `--workers 1`.

# `drain`: advanced use-cases

## Persist the tree across restarts

Without persistence the parse tree is rebuilt from scratch on every restart, so templates reset and warm up again. Back the processor with a storage extension and snapshot periodically:

```yaml
extensions:
  file_storage:
    directory: /var/lib/otelcol/drain

processors:
  drain:
    storage: file_storage
    save_interval: 5m

service:
  extensions: [file_storage]
```

On startup a valid snapshot is loaded (skipping `seed_templates`/`seed_logs`, and warmup if the loaded tree already satisfies `warmup_min_clusters`); `save_interval > 0` snapshots periodically; a final snapshot is always saved on shutdown. With a shared backend (Redis, a database) new instances inherit a tree trained by running instances, avoiding cold starts.

## Tune cluster granularity

```yaml
processors:
  drain:
    tree_depth: 5          # higher → more specific templates
    merge_threshold: 0.5   # higher → more specific templates
```

Raise `tree_depth`/`merge_threshold` for **more specific** templates (less over-merging); lower them for **more general** ones. Add `extra_delimiters` when tokens you want split are joined by non-whitespace characters such as `:` or `,`.

## Suppress warmup before annotating

```yaml
processors:
  drain:
    warmup_min_clusters: 20
```

With `warmup_min_clusters > 0`, the processor trains on every record but does **not** write the template attribute until that many distinct clusters exist. Records still pass through immediately — no buffering, no added latency — they simply arrive unannotated during the warmup window (and are not retroactively annotated). This keeps unstable, half-trained templates from reaching downstream `filter`/`routing` rules.

## Split structured tokens with `extra_delimiters`

```yaml
processors:
  drain:
    extra_delimiters: [":", ","]
```

Useful when values you want abstracted are glued to neighboring tokens by punctuation rather than whitespace.

## Mask named values and emit remaining wildcards

Since v0.158.0, `masking_rules` extracts selected regex matches under stable names, while `emit_wildcards` exposes any remaining Drain `<*>` positions in template order:

```yaml
processors:
  drain:
    masking_rules:
      - name: ip
        pattern: '\b(?:\d{1,3}\.){3}\d{1,3}\b'
    emit_wildcards: true
```

For body `user alice logged in from 10.0.0.1`, this can produce template `user <*> logged in from <ip>`, `log.record.template.parameter.ip: 10.0.0.1`, and `log.record.template.wildcards: ["alice"]`. A fully literal or fully named template produces no wildcard attribute.

## Combine templates with downstream components

Feed the derived template into a `filter` to drop a noisy class, a `routing` connector to fan log classes into separate pipelines, or a `log_dedup` processor to aggregate identical records with a count:

```yaml
processors:
  drain:
  filter/drop_noisy:
    error_mode: ignore
    logs:
      log_record:
        - attributes["log.record.template"] == "heartbeat ping <*>"
        - attributes["log.record.template"] == "connected to <*>"

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [drain, filter/drop_noisy]
      exporters: [otlp]
```

# `drain`: configuration

The Drain algorithm tokenizes each log line and walks a **fixed-depth parse tree** (depth set by `tree_depth`). Lines with similar token structure land in the same **cluster**; each cluster carries a template. As more logs arrive, the template is refined — tokens that vary across a cluster's members become `<*>` wildcards while stable tokens are kept verbatim. The algorithm is deterministic: given the same configuration and a representative sample of log forms, the same token structure always yields the same template. Because of this, independent collector instances converge on identical templates once each has seen enough lines.

## Typical config

```yaml
processors:
  drain:
    template_attribute: log.record.template

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [memory_limiter, drain]
      exporters: [otlphttp]
```

## Configuration reference

| Key | Type | Default | Notes |
|-----|------|---------|-------|
| `tree_depth` | int | `4` | Max depth of the Drain parse tree (`depth` in the paper). Higher = more specific templates. **Minimum: 3.** |
| `merge_threshold` | float | `0.4` | Minimum fraction of tokens that must match an existing cluster for a line to merge into it rather than form a new cluster (`st` in the paper). Range `[0.0, 1.0]`. |
| `max_node_children` | int | `100` | Maximum children per internal parse-tree node (`maxChild` in the paper). Bounds memory on high-cardinality token positions. |
| `max_clusters` | int | `0` | Maximum clusters tracked; when exceeded, the least-recently-used cluster is evicted. `0` = unlimited. |
| `extra_delimiters` | []string | `[]` | Additional token delimiters beyond whitespace (e.g. `[",", ":"]`). |
| `body_field` | string | `""` | If set and the body is a map, the value of this **single top-level key** is templated instead of the full body. Full OTTL paths are not supported. `""` = use the full body string. |
| `template_attribute` | string | `"log.record.template"` | Attribute key the derived template string is written to. |
| `masking_rules` | list of `{name, pattern}` | `[]` | Ordered RE2 substitutions applied before clustering. Each whole match becomes `<name>` and is emitted at `<parameter_key_prefix>.<name>`. |
| `parameter_key_prefix` | string | `"log.record.template.parameter"` | Prefix for dynamically named attributes produced by `masking_rules`. |
| `emit_wildcards` | bool | `false` | Emit body tokens occupying Drain-generated `<*>` positions as a positional string slice. |
| `wildcards_attribute` | string | `"log.record.template.wildcards"` | Attribute key for the wildcard slice. Consulted only when `emit_wildcards: true`. |
| `warmup_min_clusters` | int | `0` | Distinct clusters that must be observed before annotation starts. `0` disables warmup. |
| `storage` | component ID | `""` | ID of a storage extension used to persist the tree across restarts. `""` = disabled. |
| `save_interval` | duration | `0s` | Interval between periodic snapshot saves. `0s` = save on shutdown only. Requires `storage`. |
| `seed_templates` | []string | `[]` | Template strings pre-loaded at startup. Empty/whitespace-only entries are skipped. |
| `seed_logs` | []string | `[]` | Raw example log lines trained on at startup. Empty/whitespace-only entries are skipped. |

## Validation rules

- `tree_depth` must be `>= 3`.
- `merge_threshold` must be in `[0.0, 1.0]`.
- `warmup_min_clusters` must be `>= 0`.
- `save_interval` must be `>= 0`.
- `save_interval > 0` requires `storage` to be set.
- Every masking-rule `name` must be non-empty, cannot be `*`, and cannot contain angle brackets or whitespace.
- Every masking-rule `pattern` must be a non-empty valid Go RE2 regular expression.

## Structured (map) body behavior

If the body is a map, the full serialized form is templated by default, which yields low-value templates. Set `body_field` to the **single top-level** message key (full OTTL paths like `body["event"]["message"]` are not supported), or promote that field to a plain string body upstream (e.g. a `move` operator in the filelog receiver). The full body is used unchanged if the key is absent or the body is not a map.

## Named masking and positional wildcard emission

Masking rules run in declaration order on a working copy of the body. Each whole regex match becomes a named token in the template and its original value is emitted as an attribute:

```yaml
processors:
  drain:
    masking_rules:
      - name: ip
        pattern: '\b(?:\d{1,3}\.){3}\d{1,3}\b'
    parameter_key_prefix: log.record.template.parameter
    emit_wildcards: true
    wildcards_attribute: log.record.template.wildcards
```

For body `user alice logged in from 10.0.0.1`, masking can produce template `user <*> logged in from <ip>`, attribute `log.record.template.parameter.ip: 10.0.0.1`, and wildcard slice `["alice"]`. If a name matches multiple template positions, the first value wins and `otelcol_processor_drain_masks_duplicates` increments once per record and duplicated name. Wildcard emission uses the same tokenization as the tree and writes no slice when no `<*>` remains. During warmup suppression none of these output attributes is written.

The v0.157.0-only keys `extract_parameters` and `params_attribute` were removed in v0.158.0. Replace them with `emit_wildcards` and `wildcards_attribute`; the default output key changed from `log.record.template.params` to `log.record.template.wildcards`.

# `drain`: known quirks

## Unbounded memory with `max_clusters: 0`

The default `max_clusters: 0` means *unlimited* — active clusters and tree nodes grow with unique log patterns. On high-diversity streams this can grow without bound. Bound it with `max_clusters` (enables LRU eviction) and cap fan-out with `max_node_children`. Track `otelcol_processor_drain_clusters_active` for growth.

## Warmup withholds templates

When `warmup_min_clusters > 0`, the template attribute does not appear until that many clusters have been observed. If the attribute "never shows up", check `otelcol_processor_drain_clusters_active` against the threshold, or lower `warmup_min_clusters`. The warmup window is observable as `otelcol_processor_incoming_items - otelcol_processor_drain_log_records_annotated`.

Warmup also withholds named-mask and wildcard attributes. Even after warmup, a template with no `<*>` positions emits no wildcard slice; this is not an extraction failure.

## Templates reset on restart without persistence

The tree lives in memory. Without a `storage` extension, templates are rebuilt (and re-warmed) on every restart, and independent instances may disagree on a template during early training on low-volume or highly variable streams. Mitigate with `seed_templates`/`seed_logs`, `warmup_min_clusters`, or shared `storage` (see [Advanced use-cases](advanced.md)).

## Structured (map) bodies

If the body is a map, the full serialized form is templated by default, which yields low-value templates. Set `body_field` to the **single top-level** message key (full OTTL paths like `body["event"]["message"]` are not supported), or promote that field to a plain string body upstream (e.g. a `move` operator in the filelog receiver). The full body is used unchanged if the key is absent or the body is not a map.

## Alpha stability and provisional attribute name

The processor is **Alpha** — keys and behavior may change. This happened in v0.158.0: the v0.157.0-only `extract_parameters` / `params_attribute` keys became `emit_wildcards` / `wildcards_attribute`. The default `log.record.template` tracks a *proposed* (not yet adopted) semantic convention ([semantic-conventions#1283](https://github.com/open-telemetry/semantic-conventions/issues/1283), [#2064](https://github.com/open-telemetry/semantic-conventions/issues/2064)); pin output keys explicitly if you depend on them. The internal `otelcol_processor_drain_*` metrics are Development stability and may change or be removed.

## Drain does not reduce volume

It only annotates. Expecting `drain` alone to drop or aggregate logs is a common mistake — always add a `filter`/`routing` (or `log_dedup`) component downstream to act on the template.

# `drain` processor

| | |
|-|-|
| Kind | processor |
| Type | `drain` |
| Signals | logs |
| Stability | Alpha (processor); Development (telemetry metrics `otelcol_processor_drain_*`) |
| Distributions | contrib, k8s |
| Go module | `github.com/open-telemetry/opentelemetry-collector-contrib/processor/drainprocessor` |
| Upstream README | <https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/drainprocessor> |

Logs support is Alpha since v0.151.0; configuration keys and behavior may change between releases.

## Description

Applies the [Drain](https://jiemingzhu.github.io/pub/pjhe_icws2017.pdf) log-clustering algorithm to each log record, derives a **template string** (e.g. `user <*> logged in from <*>`), and writes it to a configurable attribute (`log.record.template` by default). The template becomes a stable key for grouping, filtering, routing, or sampling whole classes of logs. Since v0.158.0, ordered regex masking rules can turn high-cardinality values into named template tokens and attributes; optional wildcard emission writes Drain's remaining `<*>` values to a string-slice attribute.

Drain tokenizes each log line and walks a **fixed-depth parse tree** (depth set by `tree_depth`). Lines with similar token structure land in the same cluster, and each cluster carries a template whose varying tokens become `<*>` wildcards while stable tokens are kept verbatim. The processor **annotates only** — it does not drop, aggregate, reorder, or otherwise modify the flow of logs.

## Main use-cases

Use it when:
- You want to group high-volume, semi-structured logs into a small number of stable patterns.
- You need a stable key for downstream filtering, routing, or sampling of log classes.
- You need named regex-matched values or the remaining positional wildcard values as structured attributes.
- Your log bodies are free-form strings (or a structured body with a clear message field).
- You want to identify the noisiest log patterns before deciding what to drop.

Avoid it when:
- Your logs are already fully structured with a reliable event identifier — use that attribute directly.
- You need exact, hand-authored templates with no early-training variance — seed them, or template upstream.
- Bodies are large JSON blobs with no single message field — templating the serialized map produces low-value templates (set `body_field`, or promote the message field upstream).

## Related components

- `filter` — drop or keep logs by matching the derived template.
- `routing` (connector) — route log classes to different pipelines by template.
- `log_dedup` — aggregate identical logs with a count; complements template-based grouping.
- `transform` — OTTL-based per-record transformations.
- Storage extensions (e.g. `file_storage`) — back `storage` for snapshot persistence.

## Details

- [Configuration](configuration.md) — config keys, defaults, named masking, wildcard emission, validation rules, and structured-body behavior. Open when wiring up `drain` or tuning what gets templated.
- [Verification](verification.md) — telemetrygen recipe to confirm templates are written. Open when checking the processor is bundled and working end-to-end.
- [Advanced use-cases](advanced.md) — persistence, granularity tuning, warmup, named masking, wildcard emission, delimiters, and combining with downstream components. Open when going beyond a single stock instance.
- [Known quirks](quirks.md) — unbounded memory, warmup withholding, restart resets, map bodies, Alpha stability, and the "does not reduce volume" trap. Open when something behaves unexpectedly.

# `drain`: verification

See [Verification harness](../../SKILL.md#verification-harness) for how to run this end-to-end.

Confirm `drain` is in your distribution (check the component's metadata); build via OCB if it is not yet bundled.

Config (`drain-verify.yaml`):

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
processors:
  drain:
    template_attribute: log.record.template
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [drain]
      exporters: [debug]
```

Send log records that share structure but differ in values (see the `otel-telemetrygen` skill):

```bash
telemetrygen logs --otlp-insecure --otlp-endpoint localhost:4317 \
  --logs 100 --body "user alice logged in from 10.0.0.1"
```

(Don't add `--duration`: telemetrygen ignores `--logs` when `--duration` is set.) telemetrygen sends a **constant** body per invocation (`--body` defaults to `"the message"`), so a single run produces one template with no wildcards. To watch several distinct values collapse to one template (e.g. `user <*> logged in from <*>`), run the command a few times with different `--body` values — for example `"user bob logged in from 192.168.1.1"` and `"user carol logged in from 172.16.0.9"` — or point the pipeline at a real log source.

**What proves it worked:** the `debug` exporter shows each record annotated with a `log.record.template` attribute holding the clustered template string (e.g. `user <*> logged in from <*>`).

# `file_log` receiver: advanced use-cases

## Multiline entries

By default an entry is one line. The `multiline` block makes the **file reader** split the byte
stream on a pattern instead — for stack traces, pretty-printed payloads, or any record that spans
lines. It must contain **exactly one** of `line_start_pattern` or `line_end_pattern` (regexes);
`omit_pattern: true` strips the matched delimiter from each entry.

```yaml
receivers:
  file_log:
    include: [ /var/log/example/multiline.log ]
    multiline:
      line_start_pattern: ^Exception     # a new entry begins at each "Exception" line
```

Everything between two `^Exception` lines becomes a single record. For rules that can't be
expressed as one start/end regex (e.g. "combine until the next line has a timestamp"), use the
[`recombine`](operators.md#general-purpose-operators) operator instead — more flexible, more cost.

## Durable offsets with `storage`

By default read offsets live only in memory, so a collector restart re-reads from `start_at`
(typically losing position or re-ingesting). Point `storage` at a `file_storage` extension to
persist them:

```yaml
extensions:
  file_storage:
    directory: /var/lib/otelcol/file_storage

receivers:
  file_log:
    include: [ /var/log/myapp/*.log ]
    storage: file_storage          # persists Fingerprint + Offset + FileAttributes per file

service:
  extensions: [file_storage]
  pipelines:
    logs:
      receivers: [file_log]
      exporters: [debug]
```

Stored per tracked file: the fingerprint (first bytes), the byte offset, and the file attributes.
Note this guarantees the *file* is read accurately; records can still be lost further downstream —
for full durability also configure exporter persistent queues. With `polls_to_archive` (>0,
experimental) plus `storage`, offsets of readers older than three poll cycles are archived to disk
rather than purged, which helps avoid re-ingestion when used with `exclude_older_than`.

## Log rotation and `on_truncate`

The receiver follows files across rotation by inode + fingerprint, so it keeps reading even when a
rotated file no longer matches `include`. Two strategies are handled:

- **move/create** — handled transparently; reading continues on the renamed file.
- **copy/truncate** — the file shrinks in place. `on_truncate` controls the response when a
  same-fingerprint file is seen smaller than the stored offset:
  - `ignore` (default) — keep the offset; read nothing until the file grows past it. Avoids
    duplicates.
  - `read_whole_file` — reset offset to 0 and re-read everything. No data loss, possible
    duplicates.
  - `read_new` — set offset to the current (post-truncate) size; read only new appends.

```yaml
receivers:
  file_log:
    include: [ /var/log/myapp/*.log ]
    on_truncate: read_whole_file
```

**File-attribute behavior during rotation:** if the rotated name no longer matches `include`, the
*original* `log.file.name`/`log.file.path` are preserved; if it still matches, the *new* rotated
name is reported (prevents duplicate per-file metrics).

## Header metadata parsing

Read attributes from a file's header lines and stamp them onto every record from that file.
Requires the `filelog.allowHeaderMetadataParsing` feature gate and `start_at: beginning`.

```yaml
receivers:
  file_log:
    include: [ /var/log/myapp/*.log ]
    start_at: beginning
    header:
      pattern: '^#.*'                       # every header line matches this
      metadata_operators:
        - type: regex_parser
          regex: '^#Service: (?P<service>.*)$'
```

Each header line is run through `metadata_operators`; the resulting attributes are merged (upsert)
and attached to every emitted record. Header lines themselves are not emitted.

## Compression

Read gzip-compressed files directly:

```yaml
receivers:
  file_log:
    include: [ /var/log/example/*.log.gz ]
    compression: gzip              # or `auto` to auto-detect gzip by header
```

`auto` is useful for a mix of compressed and plain files in one receiver. Compressed inputs must
be **appended** to (not rewritten) if they keep growing.

## File ordering

When many rotated files match `include` but you only want the newest tracked, `ordering_criteria`
groups and sorts matches and keeps the top N:

```yaml
receivers:
  file_log:
    include: [ /var/log/myapp/app-*.log ]
    ordering_criteria:
      regex: 'app-(?P<num>\d+)\.log'
      top_n: 1
      sort_by:
        - sort_type: numeric
          regex_key: num
          ascending: false         # track only the highest-numbered file
```

An explicit `top_n: 0` means "match all files." Omitting `top_n` still falls
back to the legacy value `1` unless the alpha `filelog.requireExplicitTopN` gate is enabled; with
that gate, a sorted configuration must state either `0` or a positive limit.

## Kubernetes pod logs

The canonical node-level collection path: tail the kubelet's per-pod log files and parse the
container runtime's wrapper format with the [`container`](operators.md#parsers) operator, which
handles `docker`, `cri-o`, and `containerd` and lifts the runtime metadata (stream, timestamp)
out of the line. Pair with [`k8s_attributes`](../k8s_attributes/README.md) downstream for
namespace/pod/label enrichment.

```yaml
receivers:
  file_log:
    include: [ /var/log/pods/*/*/*.log ]
    start_at: end                  # live tail; don't backfill node history on restart
    include_file_path: true        # k8s_attributes can associate by log.file.path
    operators:
      - type: container            # auto-detects docker/cri-o/containerd
```

## Named instances

Run the receiver more than once (e.g. different parsing per log source) with the `type/name` form:

```yaml
receivers:
  file_log/app:
    include: [ /var/log/app/*.log ]
    operators: [ { type: json_parser } ]
  file_log/access:
    include: [ /var/log/nginx/access.log ]
    operators: [ { type: regex_parser, regex: '(?P<msg>.*)' } ]

service:
  pipelines:
    logs:
      receivers: [file_log/app, file_log/access]
      exporters: [debug]
```

# `file_log` receiver: configuration

Every key below traces to the contrib v0.160.0 source (`receiver/filelogreceiver/README.md`, generated from the `fileconsumer` config). The receiver wraps the stanza `file_input` operator, so most keys are file-discovery and file-reading knobs; the parsing pipeline is `operators` (see [operators.md](operators.md)).

Only `include` is required.

## File discovery

| Key | Type | Default | Notes |
|-----|------|---------|-------|
| `include` | `[]string` | **required** | Glob patterns of file paths to read. |
| `exclude` | `[]string` | `[]` | Glob patterns to exclude, applied against the `include` matches. |
| `exclude_older_than` | duration | — | Exclude files whose modification time is older than this age. |
| `start_at` | `beginning` \| `end` | `end` | Where to start reading **newly discovered** files. Default `end` means a pre-existing, idle file yields nothing — see [quirks.md](quirks.md). |
| `poll_interval` | duration | `200ms` | Interval between filesystem polls for new data / new files. |
| `max_concurrent_files` | int | `1024` | Max files read concurrently; excess files are processed in batches. |
| `max_batches` | int | `0` | When batching is needed to respect `max_concurrent_files`, caps batches per poll. `0` = no limit. |

## File reading & sizing

| Key | Type | Default | Notes |
|-----|------|---------|-------|
| `encoding` | string | `utf-8` | File encoding. See [supported encodings](#supported-encodings). |
| `fingerprint_size` | bytes | `1000` | Bytes from the start of a file used to uniquely identify it. Minimum `16`. Decreasing it re-ingests files larger than the new size. |
| `initial_buffer_size` | bytes | `16KiB` | Initial read-buffer size for headers and logs; grows as needed. |
| `max_log_size` | bytes | `1MiB` | Max size of a single log entry. Behavior on overflow set by `max_log_size_behavior`. |
| `max_log_size_behavior` | `split` \| `truncate` | `split` | `split` emits the oversized entry as multiple records; `truncate` keeps the head and drops the rest. |
| `force_flush_period` | duration | `500ms` | Time since last new data after which a partial trailing line may be emitted. `0` disables (waits for a delimiter). |
| `preserve_leading_whitespaces` | bool | `false` | Keep leading whitespace on each entry. |
| `preserve_trailing_whitespaces` | bool | `false` | Keep trailing whitespace on each entry. |
| `compression` | `` \| `gzip` \| `auto` | `` | Read compressed files. `auto` auto-detects gzip by header. Appended (not rewritten) compressed files only. |

## Emitted file attributes

Each toggles a `log.file.*` attribute on every record from that file. `*_owner_*` and `*_permissions` are **not supported on Windows**.

| Key | Type | Default | Attribute added |
|-----|------|---------|-----------------|
| `include_file_name` | bool | `true` | `log.file.name` |
| `include_file_path` | bool | `false` | `log.file.path` |
| `include_file_name_resolved` | bool | `false` | `log.file.name_resolved` (after symlink resolution) |
| `include_file_path_resolved` | bool | `false` | `log.file.path_resolved` (after symlink resolution) |
| `include_file_owner_name` | bool | `false` | `log.file.owner.name` (Unix only) |
| `include_file_owner_group_name` | bool | `false` | `log.file.owner.group.name` (Unix only) |
| `include_file_permissions` | bool | `false` | `log.file.permissions`, 3-digit octal e.g. `755` (Unix only) |
| `include_file_record_number` | bool | `false` | `log.file.record_number` |
| `include_file_record_offset` | bool | `false` | `log.file.record_offset` |

## Static attributes & resource

| Key | Type | Default | Notes |
|-----|------|---------|-------|
| `attributes` | map | `{}` | `key: value` pairs added to each entry's attributes. Values are strings or stanza [expressions](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/docs/types/expression.md) returning a string. |
| `resource` | map | `{}` | `key: value` pairs added to each entry's resource. Same value rules as `attributes`. |
| `operators` | `[]operator` | `[]` | The stanza parsing pipeline. See [operators.md](operators.md). |
| `multiline` | block | — | Split entries on a pattern instead of newlines. See [advanced.md](advanced.md#multiline-entries). |

## Durability & rotation

| Key | Type | Default | Notes |
|-----|------|---------|-------|
| `storage` | component ID | none (in-memory) | A `file_storage` (or other storage) extension ID for persisting offsets across restarts. See [advanced.md](advanced.md#durable-offsets-with-storage). |
| `delete_after_read` | bool | `false` | Read then immediately delete each file. Requires feature gate `filelog.allowFileDeletion`; must be `false` when `start_at: end`. |
| `acquire_fs_lock` | bool | `false` | Acquire a filesystem lock before reading (Unix only). |
| `file_cache_advise` | bool | `false` | Hint the OS to release cached pages after read (Linux only); helps page-cache pressure on large sequential reads. |
| `on_truncate` | `ignore` \| `read_whole_file` \| `read_new` | `ignore` | Behavior when a same-fingerprint file shrinks (copytruncate rotation). See [advanced.md](advanced.md#log-rotation-and-on_truncate). |
| `skip_unmodified_files` | bool | `false` | Skip opening and fingerprinting a tracked file when its path and modification time are unchanged since the previous poll. Added in v0.160.0. |
| `polls_to_archive` | int | `0` | **Experimental.** With `storage` set, retain offsets of old readers on disk for this many poll cycles instead of purging after 3 generations. |

## Header metadata parsing

Requires feature gate `filelog.allowHeaderMetadataParsing` and `start_at: beginning`. See [advanced.md](advanced.md#header-metadata-parsing).

| Key | Type | Default | Notes |
|-----|------|---------|-------|
| `header.pattern` | regex | required (when `header` set) | Regex that every header line matches. |
| `header.metadata_operators` | `[]operator` | required (when `header` set) | Operators that parse attributes out of the header lines. Merged onto every record from the file; header lines themselves are not emitted. |

## Retry on downstream failure

| Key | Type | Default | Notes |
|-----|------|---------|-------|
| `retry_on_failure.enabled` | bool | `false` | Pause reading a file and resend the current batch on a downstream error. |
| `retry_on_failure.initial_interval` | duration | `1s` | Backoff before the first retry. |
| `retry_on_failure.max_interval` | duration | `30s` | Upper bound on backoff. |
| `retry_on_failure.max_elapsed_time` | duration | `5m` | Total time (incl. retries) before data is dropped. `0` = retry forever. |

## File ordering (`ordering_criteria`)

Tracks only the top-N files after grouping and sorting — useful when many rotated files match but you want a deterministic read order.

| Key | Type | Default | Notes |
|-----|------|---------|-------|
| `ordering_criteria.regex` | regex | — | Must contain named capture groups referenced by `sort_by.regex_key`. |
| `ordering_criteria.group_by` | regex | — | Named-capture regex for pre-sort grouping. |
| `ordering_criteria.top_n` | int | unset → `1` | Number of files to track after ordering. Explicit `0` means match all files; negative values are rejected. With the alpha `filelog.requireExplicitTopN` gate enabled, omitting this key while `sort_by` is set is a startup error. Added in v0.159.0. |
| `ordering_criteria.sort_by.regex_key` | string | — | Named group (from `ordering_criteria.regex`) to sort on. |
| `ordering_criteria.sort_by.sort_type` | `numeric` \| `alphabetical` \| `timestamp` \| `mtime` | — | Sort strategy. `mtime` requires the `filelog.mtimeSortType` feature gate (Beta, on by default since v0.161.0; Alpha off before). |
| `ordering_criteria.sort_by.location` | string | — | Timestamp location (when `sort_type: timestamp`). |
| `ordering_criteria.sort_by.layout` | strptime | — | Timestamp format, strptime (when `sort_type: timestamp`). ⚠️ The upstream README documents this key as `format`, but the actual field (matcher source + `config.schema.yaml`) is `layout` — `format:` is rejected as an invalid key (verified on v0.156.0). |
| `ordering_criteria.sort_by.ascending` | bool | — | Sort direction. |

## Supported encodings

`nop` (raw bytes, no validation), `utf-8` (default), `utf-8-raw` (no invalid-byte replacement), `utf-16le`, `utf-16be`, `ascii`, `big5`. Other IANA charsets are supported best-effort.

## Minimal config

```yaml
receivers:
  file_log:
    include: [ /var/log/myapp/*.log ]
    start_at: beginning            # default `end` would skip existing content
    operators:
      - type: json_parser          # parse each line as JSON into attributes
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%dT%H:%M:%S%z'
```

# `file_log` receiver: operators

The `operators` list is a **stanza** pipeline — an ordered chain of small, single-purpose
steps that run on each log entry as it leaves the file reader. This is where parsing,
timestamp/severity extraction, and entry combining happen. The full operator reference lives
upstream in [`pkg/stanza/docs/operators`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/stanza/docs/operators); this page is the catalog plus the wiring rules you need to author a pipeline.

## Pipeline rules

- Every operator has a **`type`**. Each can have a unique **`id`**; if you use the same `type`
  twice, you must give them distinct `id`s (otherwise `id` defaults to `type` and they collide).
- By default each operator passes its output to the **next** operator in the list; the last one
  emits from the receiver. Override with **`output: <id>`** to jump to a specific operator
  (used for branching after a `router`).
- For `file_log` the input is **implicit** — the receiver *is* the `file_input`. Do **not** add
  `file_input` or other input operators to `operators`; start with a parser or general-purpose
  operator. Only parsers and general-purpose operators should be used here.
- The raw line is in the entry **body**. Parsers read from `parse_from` (default `body`) and
  write to `parse_to` (default `attributes`).

## Parsers

| `type` | Parses |
|--------|--------|
| `container` | Docker / CRI-O / containerd container log formats — the standard first parser for Kubernetes pod logs (`/var/log/pods/...`). |
| `json_parser` | A JSON object into fields. |
| `json_array_parser` | A JSON array into a list of values. |
| `csv_parser` | CSV using a configured header. |
| `regex_parser` | Named capture groups of a regex into fields. |
| `key_value_parser` | `k=v` pairs into fields. |
| `uri_parser` | A URI string into its components. |
| `syslog_parser` | RFC 3164 / RFC 5424 syslog. |
| `severity_parser` | A field's value into `SeverityNumber`/`SeverityText`. |
| `time_parser` | A field's value into the record timestamp. |
| `trace_parser` | Trace/span IDs and flags into the record's trace context. |
| `scope_name_parser` | A field into the instrumentation scope name. |

### Embedded timestamp / severity / trace parsing

Most parsers can run a follow-up timestamp, severity, and/or trace parse inline instead of
needing a separate operator — the "complex parser" form. This is the idiomatic shape:

```yaml
operators:
  - type: regex_parser
    regex: '^(?P<ts>\S+) (?P<sev>[A-Z]+) (?P<msg>.*)$'
    timestamp:
      parse_from: attributes.ts
      layout: '%Y-%m-%dT%H:%M:%S%z'   # strptime by default; layout_type: gotime|epoch also supported
    severity:
      parse_from: attributes.sev       # maps INFO/WARN/ERROR/... to SeverityNumber
```

## General-purpose operators

| `type` | Does |
|--------|------|
| `add` | Add a field with a static value or expression. |
| `copy` | Copy a field to another location. |
| `move` | Move/rename a field. |
| `remove` | Delete a field. |
| `retain` | Keep only the listed fields, drop the rest. |
| `flatten` | Flatten a nested map one level. |
| `assign_keys` | Assign keys to a list, turning it into a map. |
| `filter` | Drop an entry that matches an [expression](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/docs/types/expression.md) (entry-level drop, distinct from the `filter` processor). |
| `router` | Branch entries to different downstream operators by expression (use with `output`). |
| `recombine` | Combine multiple consecutive entries into one (e.g. stack traces, multi-line app logs). |
| `regex_replace` | Regex find/replace on a field. |
| `unquote` | Unquote a quoted string field. |
| `sanitize_utf8` | Replace invalid UTF-8 sequences. |
| `noop` | Pass through unchanged (placeholder/testing). |

> `recombine` vs `multiline`: the top-level [`multiline`](advanced.md#multiline-entries) block
> splits the *byte stream* into entries before any operator runs (one regex, line start/end).
> The `recombine` operator merges *already-split* entries using an expression — more flexible
> (e.g. "combine until the next line starts with a timestamp"), at more cost. Prefer `multiline`
> for simple cases.

## Inputs (not used here)

`file_input`, `journald_input`, `syslog_input`, `tcp_input`, `udp_input`,
`windows_eventlog_input`, `stdin` exist in the stanza library but belong to their **own
receivers** (`journald`, `tcplog`, `udplog`, `windowseventlog`, …). Don't place them in
`file_log`'s `operators`.

## Worked example — JSON logs with timestamp and severity

```yaml
receivers:
  file_log:
    include: [ /var/log/myservice/*.json ]
    start_at: beginning
    operators:
      - type: json_parser
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%d %H:%M:%S'
        severity:
          parse_from: attributes.level
      - type: move                 # promote the parsed message into the body
        from: attributes.message
        to: body
```

# `file_log` receiver: known quirks

## `start_at` defaults to `end` — existing files look empty

The single most common "the receiver reads nothing" problem. `start_at` defaults to `end`, so for
a **newly discovered** file the receiver begins at the *current* end and only emits lines appended
**after** it starts watching. A file that already exists and is not being actively written to
produces **zero records**.

```yaml
receivers:
  file_log:
    include: [ /var/log/myapp/*.log ]
    start_at: beginning            # read existing content; default `end` skips it
```

Use `beginning` for batch/static files and first-run backfills; `end` is the right default for
high-volume live logs where you don't want to re-read history on every restart (pair with
`storage` so restarts resume mid-file).

## The type was renamed `filelog` → `file_log` in v0.149.0

New configs should use `file_log`. The old `filelog` name still works as a deprecated alias but
logs a startup warning:

```
warn  "filelog" alias is deprecated; use "file_log" instead  {"otelcol.component.id": "filelog", ...}
```

When reviewing an existing config, `filelog:` is **not** broken — it's the legacy spelling.

## Feature-gated options

Some keys do nothing (or error) unless a feature gate is enabled at startup
(`--feature-gates=<gate>`):

| Option | Required gate | State (v0.162.0) |
|--------|---------------|-------|
| `delete_after_read` | `filelog.allowFileDeletion` | Beta (on by default) |
| `header` parsing | `filelog.allowHeaderMetadataParsing` | Beta (on by default) — no flag needed |
| `ordering_criteria.sort_by.sort_type: mtime` | `filelog.mtimeSortType` | Beta (**on by default** since v0.161.0; Alpha off before) |
| require explicit `ordering_criteria.top_n` when sorting | `filelog.requireExplicitTopN` | Alpha (off by default); unset keeps the legacy default `1`, explicit `0` matches all files |
| include/exclude case-insensitive globbing (Windows) | `filelog.windows.caseInsensitive` | Beta (on by default) |
| protobuf checkpoint encoding | `filelog.protobufCheckpointEncoding` | Beta (**on by default** since v0.156.0; Alpha off in v0.148.0–v0.155.0) — ~7× faster decode, ~31% smaller; reads both formats either way |

## `delete_after_read` conflicts with `start_at: end`

`delete_after_read: true` reads a file then deletes it; it must be paired with
`start_at: beginning` (it is invalid with `start_at: end`) and requires the
`filelog.allowFileDeletion` gate to remain enabled.

## Fingerprinting and re-ingestion

Files are identified by the first `fingerprint_size` bytes (default `1000`, minimum `16`) plus
inode. Two consequences:

- **Decreasing `fingerprint_size`** changes identity for files larger than the new size →
  they are treated as new and **re-ingested**.
- Files whose first `fingerprint_size` bytes are **identical** (e.g. a fixed boilerplate header)
  can be mistaken for the same file. Raise `fingerprint_size` past the common prefix, or ensure
  early bytes differ.

## Don't put input operators in `operators`

The receiver *is* the file input. Adding `file_input` (or `tcp_input`, `journald_input`, …) to the
`operators` list is wrong — only parsers and general-purpose operators belong there. Use the
dedicated receiver for other inputs.

## `multiline` needs exactly one pattern

The `multiline` block must specify **exactly one** of `line_start_pattern` /
`line_end_pattern` — setting both, or neither (while present), is a config error.

## Owner/permission attributes are Unix-only

`include_file_owner_name`, `include_file_owner_group_name`, and `include_file_permissions` are not
supported on Windows.

## Stability

`logs` is **Beta** (production-viable; breaking config changes rare). Distributed in `contrib` and
`k8s` — **not** in the `core` or `otlp` distributions, so a minimal/core collector won't have it.

# `file_log` receiver

| | |
|-|-|
| Kind | receiver |
| Type | `file_log` (formerly `filelog`) |
| Signals | logs (Beta) |
| Distributions | contrib, k8s |
| Go module | `github.com/open-telemetry/opentelemetry-collector-contrib/receiver/filelogreceiver` |
| Upstream README | <https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver> |
| Rename | `filelog` → `file_log` in **v0.149.0**; `filelog` kept as a deprecated alias (runtime warns `"filelog" alias is deprecated; use "file_log" instead`). |

## Description

Tails log files from disk and turns each line (or multi-line entry) into an OTLP log record. You give it a list of glob patterns (`include`), and it discovers matching files, follows them as they grow, and tracks how far it has read into each one. File identity is established by a content **fingerprint** (the first `fingerprint_size` bytes) plus inode, so the receiver keeps reading correctly across both common rotation strategies — move/create and copy/truncate — even when the rotated file no longer matches `include`. Read offsets live in memory by default; point `storage` at a `file_storage` extension to make them survive a collector restart.

The raw line lands in the log record `Body` and a few file attributes (`log.file.name` by default) are attached. Everything else — extracting fields, parsing timestamps and severity, combining stack traces into one record — is done by an ordered pipeline of **stanza operators** under `operators:` (`json_parser`, `regex_parser`, `severity_parser`, `recombine`, …). The operator framework, not the receiver's top-level keys, is where most real-world configuration lives.

The single most common surprise: **`start_at` defaults to `end`**, so a file that already exists and is not being actively appended to yields **no records**. Set `start_at: beginning` to read existing content. See [quirks.md](quirks.md).

For ordered discovery, explicit `ordering_criteria.top_n: 0` means "match all." The
`filelog.requireExplicitTopN` migration gate rejects an omitted `top_n`.
Since v0.160.0, `skip_unmodified_files: true` can avoid reopening and fingerprinting tracked files
whose path and modification time have not changed since the previous poll.

## Main use-cases

Use when:
- You collect logs that an application writes to files (container stdout/stderr on a node, app log files, system logs) rather than sending over OTLP.
- You need to parse unstructured or semi-structured text into structured records — JSON, regex-delimited, syslog, key/value — with timestamp and severity extraction.
- You need durable offset tracking so a collector restart resumes where it left off (pair with a `file_storage` extension via `storage`).
- You tail Kubernetes pod logs (`/var/log/pods/...`) — the canonical node-level log-collection path, typically followed by `k8s_attributes` for metadata enrichment.

Avoid when:
- The source already speaks OTLP — use the [`otlp`](../otlp/README.md) receiver.
- You only need journald, Windows Event Log, TCP/UDP syslog, or stdin — use the dedicated receiver (`journald`, `windowseventlog`, `tcplog`/`udplog`, …) rather than wiring those stanza inputs by hand.

## Related components

- [`otlp` receiver](../otlp/README.md) — the OTLP-native ingress; `file_log` is its file-tailing counterpart for logs.
- [`k8s_attributes`](../k8s_attributes/README.md) — enrich tailed pod logs with Kubernetes metadata (namespace, pod, labels).
- [`transform`](../transform/README.md) — OTTL-based post-processing once records are in the pipeline; the stanza `operators` do parsing at ingest, `transform` does mutation downstream.
- [`memory_limiter`](../memory_limiter/README.md) — belongs **first** in the logs pipeline fed by this receiver.
- [`file_storage`](../file_storage/README.md) extension — set `storage:` to it so read offsets survive collector restarts.

## Details

- [Configuration](configuration.md) — every top-level config key (include/exclude, `start_at`, fingerprinting, sizing, rotation, `storage`, `header`, `ordering_criteria`) with defaults and validation, including `top_n: 0` semantics.
- [Operators](operators.md) — the stanza operator pipeline: the available inputs/parsers/general-purpose operators, the `id`/`output` wiring rules, and embedded timestamp/severity parsing.
- [Verification](verification.md) — a file-based recipe (write lines to a mounted file → `file_log` → `debug`), verified on contrib v0.159.0. Notes why `telemetrygen` cannot drive this receiver.
- [Advanced use-cases](advanced.md) — multiline entries, encodings, durable offsets via `storage`, header metadata parsing, log-rotation handling (`on_truncate`), file ordering, and `compression`.
- [Known quirks](quirks.md) — the `start_at: end` default, the rename/alias, feature-gated options, `delete_after_read` constraints, fingerprint re-ingestion, and stability.

# `file_log` receiver: verification

See [Verification harness](../../SKILL.md#verification-harness) for how to run this end-to-end.

> **`telemetrygen` cannot drive this receiver.** `telemetrygen` emits OTLP over the network; the
> `file_log` receiver reads files from disk. There is no `telemetrygen` flag that writes a log
> file. The input here is a **file you write yourself** and mount into the collector container.

The `file_log` receiver ships in the `contrib` and `k8s` distributions. This recipe runs **one**
collector — `file_log` tailing a mounted directory, feeding a `debug` exporter — and proves the
pipeline by writing three pre-formatted lines and reading them back parsed.

Verified on `otel/opentelemetry-collector-contrib:0.159.0` (2026-08-27).

Config (`file_log.yaml`) — note `start_at: beginning`, without which an existing idle file is
skipped (the receiver's top gotcha):

```yaml
receivers:
  file_log:
    include: [ /logs/*.log ]
    start_at: beginning            # default `end` would read nothing from a static file
    operators:
      - type: regex_parser
        regex: '^(?P<ts>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?P<sev>[A-Z]+) (?P<msg>.*)$'
        timestamp:
          parse_from: attributes.ts
          layout: '%Y-%m-%d %H:%M:%S'
        severity:
          parse_from: attributes.sev
exporters:
  debug:
    verbosity: detailed            # basic verbosity prints no per-record lines
service:
  pipelines:
    logs:
      receivers: [file_log]
      exporters: [debug]
```

Create the log file the receiver will tail:

```bash
mkdir -p /tmp/filelog-verify/logs
cat > /tmp/filelog-verify/logs/app.log <<'EOF'
2026-08-27 09:00:01 INFO service started
2026-08-27 09:00:02 WARN cache miss for key=42
2026-08-27 09:00:03 ERROR failed to connect to upstream
EOF
```

Start the collector, mounting both the config and the log directory:

```bash
IMG=otel/opentelemetry-collector-contrib:0.159.0
docker run -d --name fcol \
  -v "$PWD/file_log.yaml:/etc/otelcol-contrib/config.yaml" \
  -v /tmp/filelog-verify/logs:/logs \
  $IMG
```

**What proves it worked:** count the records the `debug` exporter logged, and confirm severity
was parsed:

```bash
docker logs fcol 2>&1 | grep -c 'LogRecord #'
docker logs fcol 2>&1 | grep -E 'SeverityText|-> msg:'
```

Expected output — **3 records**, each with the
raw line in the body, a parsed `msg` attribute, the default `log.file.name=app.log` attribute,
and severity mapped from the text:

```
3
SeverityText: INFO
     -> msg: Str(service started)
SeverityText: WARN
     -> msg: Str(cache miss for key=42)
SeverityText: ERROR
     -> msg: Str(failed to connect to upstream)
```

If the count is **0**, the most likely cause is the `start_at: end` default (the file existed
before the collector started and isn't being appended to) — set `start_at: beginning`, or append
a new line *after* the collector is running.

To watch live tailing instead, leave the collector running and append a line:

```bash
echo "2026-08-27 09:00:04 INFO new line after startup" >> /tmp/filelog-verify/logs/app.log
docker logs fcol 2>&1 | grep -c 'LogRecord #'     # now 4
```

Tear down:

```bash
docker rm -f fcol
rm -rf /tmp/filelog-verify
```

## Note on the deprecated alias

Using `filelog:` (the pre-v0.149.0 name) instead of `file_log:` still works but logs a warning
at startup:

```
warn  builders/builders.go:40  "filelog" alias is deprecated; use "file_log" instead  {"otelcol.component.id": "filelog", ...}
```

# `file_storage`: advanced use-cases

## Persistent exporter queue (survives restart)

By default an exporter's `sending_queue` is in-memory: data still queued at shutdown (or lost in a crash) is gone. Point the queue at `file_storage` and it becomes a **persistent queue** that survives a Collector restart:

```yaml
extensions:
  file_storage:
    directory: /var/lib/otelcol/storage
    create_directory: true
receivers:
  otlp:
    protocols:
      grpc:
exporters:
  otlp:
    endpoint: backend:4317
    tls:
      insecure: true
    sending_queue:
      enabled: true
      storage: file_storage        # <- queue persisted on disk
service:
  extensions: [file_storage]
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp]
```

If the backend is unreachable and the queue fills disk during an outage, the still-undelivered batches are replayed after a restart instead of being dropped. See the [`otlp` exporter](../otlp_exporter/README.md) for the queue's own sizing knobs.

## Multiple named storage instances

You can run several `file_storage` instances (`type/name`) so different consumers get their own directories — handy to isolate a large, spiky exporter queue from small, latency-sensitive receiver offsets:

```yaml
extensions:
  file_storage/queue:
    directory: /var/lib/otelcol/queue
    create_directory: true
  file_storage/offsets:
    directory: /var/lib/otelcol/offsets
    create_directory: true
receivers:
  file_log:
    include: [/var/log/app/*.log]
    storage: file_storage/offsets
exporters:
  otlp:
    endpoint: backend:4317
    sending_queue:
      enabled: true
      storage: file_storage/queue
service:
  extensions: [file_storage/queue, file_storage/offsets]
  pipelines:
    logs:
      receivers: [file_log]
      exporters: [otlp]
```

Each instance still writes one bbolt file per consuming component inside its own `directory`.

## Tuning rebound compaction for persistent-queue spikes

The classic case for online compaction: a persistent exporter queue balloons during a network outage, then drains once connectivity returns. bbolt is mmap-backed, so the freed pages stay **allocated** — the file does not shrink on its own. Rebound compaction reclaims that space, but only after the spike has clearly passed:

```text
allocated MiB
   │            ┌──────────┐   <- spike: queue fills during outage
   │            │          │
100├─ ─ ─ ─ ─ ─ ┘          │   <- rebound_needed_threshold_mib: arms the "needed" flag
   │                        ╲
 10├─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ╲▼ <- rebound_trigger_threshold_mib: drops below -> compact now
   │                          └────────
   └──────────────────────────────────▶ time
```

- `rebound_needed_threshold_mib` (default `100`) — allocation must exceed this for the extension to consider compaction needed at all (avoids compacting trivial usage).
- `rebound_trigger_threshold_mib` (default `10`) — once "needed" is armed, compaction fires when allocation falls back below this, i.e. when the heavy period is over and the rewrite will be cheap.
- `check_interval` (default `5s`) — how often these conditions are evaluated; **must be > 0** when `on_rebound: true`.

Raise the thresholds if your steady-state working set is large (so normal operation never trips compaction); lower them if you want space reclaimed more eagerly after smaller spikes. `compaction.max_transaction_size` (default `65536`) bounds how many items move per compaction transaction.

To hard-cap growth instead of just reclaiming after the fact, set `max_size` (bytes, default `0` = unlimited). It bounds **each** per-consumer bbolt file: once a file is at the cap, a write that needs to grow it is rejected with a storage-full error (writes that fit existing free space still succeed), so a runaway persistent queue can't fill the disk. If you also enable `on_rebound`, both rebound thresholds (×1,048,576) must be ≤ `max_size` or validation fails — see [quirks.md](quirks.md). In practice keep `rebound_needed_threshold_mib × 1,048,576` strictly **below** `max_size`: the "needed" flag only arms once allocation *exceeds* the threshold, and allocation can never exceed `max_size`, so a threshold exactly at the cap validates but can never fire.

## `create_directory` / `directory_permissions` for ephemeral and container environments

In containers or fresh hosts the data directory often does not exist yet. Rather than pre-creating it in an init step, let the extension do it:

```yaml
extensions:
  file_storage:
    directory: /var/lib/otelcol/storage
    create_directory: true
    directory_permissions: "0700"   # only honored when create_directory is true
```

`directory_permissions` defaults to `"0750"` and is applied minus the process umask. It is **ignored** unless `create_directory: true`. Remember the mounted/host directory must be writable by the uid the Collector runs as (the contrib image is non-root) — see [quirks.md](quirks.md).

# `file_storage`: configuration

All keys live under the extension instance (e.g. `extensions: { file_storage: { … } }`), and the extension must be listed under `service.extensions:` to load. Facts below are traced to the `v0.157.0` contrib source (`config.go`, `factory.go`, `default_others.go`).

## Top-level keys

| Key | Type | Default | Meaning |
|-----|------|---------|---------|
| `directory` | string | `/var/lib/otelcol/file_storage` (non-Windows) / `%ProgramData%\Otelcol\FileStorage` (Windows) | Dedicated data directory holding the per-consumer bbolt files. **Must already exist** unless `create_directory: true`. |
| `timeout` | duration | `1s` | Max time to wait for a file lock before failing the operation. Rarely needs changing. |
| `max_size` | int (bytes) | `0` | Max on-disk size of **each** per-consumer bbolt file, in bytes. A write that would grow the file past this is rejected with a storage-full error; writes that fit in already-allocated free space still succeed. `0` means unlimited. **Must be ≥ 0** and not exceed `math.MaxInt`. Added in v0.156.0. |
| `fsync` | bool | `false` | When `true`, force an `fsync` after each DB write (integrity over performance; flips bbolt's `DB.NoSync`). |
| `create_directory` | bool | `false` | When `true`, create the data (and compaction) directory if missing instead of failing validation. |
| `directory_permissions` | string (octal) | `"0750"` (rwxr-x---) | Permissions used when creating directories, minus the process umask. Only relevant when `create_directory: true`. |
| `recreate` | bool | `false` | When `true`, on certain bbolt corruption panics the extension renames the corrupted db to `{filename}.{ISO 8601 timestamp}.backup` and starts a fresh db so the Collector keeps running. State is lost / may be duplicated (contrib issue #35899). |

The platform-specific `directory` defaults come from `default_others.go` (and its Windows counterpart).

## `compaction`

bbolt is mmap-backed: when usage spikes, the file's allocated size grows and does **not** shrink on its own. Compaction rewrites the db to reclaim that freed-but-still-allocated space.

| Key | Type | Default | Meaning |
|-----|------|---------|---------|
| `compaction.on_start` | bool | `false` | Compact once at Collector start. |
| `compaction.on_rebound` | bool | `false` | Online compaction triggered when rebound thresholds are met (see below). |
| `compaction.directory` | string | same as the default data `directory` | Temp directory used as a midstep during compaction. |
| `compaction.max_transaction_size` | int | `65536` | Max number of items in a single compaction transaction; `0` ignores transaction sizes. **Must be ≥ 0.** |
| `compaction.rebound_needed_threshold_mib` | int | `100` | When allocated size exceeds this, the "compaction needed" flag is armed. |
| `compaction.rebound_trigger_threshold_mib` | int | `10` | When the flag is armed and allocated size drops below this, compaction begins and the flag clears. |
| `compaction.check_interval` | duration | `5s` | How often rebound conditions are checked. **Must be > 0 when `on_rebound: true`.** |
| `compaction.cleanup_on_start` | bool | `false` | Remove leftover `tempdb*` files from the compaction directory at start (left behind if a prior process was killed mid-compaction). |

**Rebound (online) compaction** reclaims space after a transient spike drains: the flag arms only once allocated size has exceeded `rebound_needed_threshold_mib`, then compaction fires when usage drops back below `rebound_trigger_threshold_mib`. See [advanced.md](advanced.md) for tuning.

When `max_size` is set (> 0) **and** `on_rebound: true`, both rebound thresholds (converted MiB → bytes, i.e. ×1,048,576) must be ≤ `max_size`, or validation fails — otherwise the rebound machinery could never fire within the cap. See the validation table in [quirks.md](quirks.md).

## Per-consumer files and naming

The extension creates **one bbolt file per consuming component**, named `<kind>_<type>_<name>` inside `directory`:

- A `file_log` receiver with no instance name → `receiver_filelog_` (note the on-disk name uses the deprecated `filelog` type spelling and a trailing `_` for the empty name; see [quirks.md](quirks.md)).
- A named `journald/myservice` receiver → `receiver_journald_myservice`.

Only `A-Za-z0-9._-` are safe in the name; any other character (including `~` itself) is escaped as `~<hex unicode>`. Over-long names are truncated to stay within OS filename limits, and the component→filename mapping is written to the Collector logs.

## Directory permissions and umask

`directory_permissions` (default `"0750"`) is applied **only** when `create_directory: true` creates a directory, and the effective mode is the configured value minus the process umask. The value must be valid octal with no bits outside `0777` — see the validation table in [quirks.md](quirks.md).

## Minimal example

```yaml
extensions:
  file_storage:
    directory: /var/lib/otelcol/storage
    create_directory: true        # create it on start if missing
service:
  extensions: [file_storage]      # must be listed here to load
```

# `file_storage`: known quirks

## The directory must already exist

By default `file_storage` does **not** create its data directory — if `directory` is missing, config validation fails with `directory must exist: … You can enable the create_directory option to automatically create it`. The fix is exactly what the error says: set `create_directory: true` (and, when compaction is enabled, the compaction directory must exist too, or be auto-created the same way).

## Validation errors → fix

All from `config.go` `Validate()`:

| Condition | Error | Fix |
|-----------|-------|-----|
| `directory` (or `compaction.directory` when `on_start`/`on_rebound` is set) does not exist, and `create_directory: false` | `directory must exist: … You can enable the create_directory option to automatically create it` | Create the dir, or set `create_directory: true`. |
| Path exists but is not a directory | `<path> is not a directory` | Point `directory` at an actual directory. |
| `compaction.max_transaction_size` < 0 | `max transaction size for compaction cannot be less than 0` | Use `0` (ignore sizes) or a positive value. |
| `max_size` < 0 | `max size cannot be less than 0` | Use `0` (unlimited) or a positive byte count. |
| `max_size` > `math.MaxInt` | `max size is too large` | Keep `max_size` within `math.MaxInt` bytes. |
| `max_size` > 0, `on_rebound: true`, `rebound_needed_threshold_mib` × 1,048,576 > `max_size` | `compaction rebound needed threshold cannot be greater than max size` | Lower `rebound_needed_threshold_mib` or raise `max_size`. |
| `max_size` > 0, `on_rebound: true`, `rebound_trigger_threshold_mib` × 1,048,576 > `max_size` | `compaction rebound trigger threshold cannot be greater than max size` | Lower `rebound_trigger_threshold_mib` or raise `max_size`. |
| `compaction.on_rebound: true` with `check_interval <= 0` | `compaction check interval must be positive when rebound compaction is set` | Set `compaction.check_interval` to a positive duration (default `5s`). |
| `create_directory: true` with non-octal `directory_permissions` | `directory_permissions value must be a valid octal representation` | Use a valid octal string, e.g. `"0750"`. |
| `directory_permissions` with bits outside `0777` | `directory_permissions contain invalid bits for file access` | Keep permissions within `0777` (file-access bits only). |

## One bbolt file per consumer, with name escaping

The extension writes a separate bbolt file per consuming component, named `<kind>_<type>_<name>` inside `directory`. Only `A-Za-z0-9._-` are safe; any other character (and `~` itself) is escaped as `~<hex unicode>`. Names too long for the OS filename limit are truncated, and the component→filename mapping is logged. So the file set in your storage dir is keyed to your component instances — renaming a receiver/exporter (or its instance name) creates a **new** file and orphans the old state.

## The on-disk name is `receiver_filelog_`, not `receiver_file_log_`

A `file_log` receiver with no instance name produces a bbolt file named **`receiver_filelog_`** — using the deprecated `filelog` type spelling and a trailing `_` for the empty name — even though your config says `file_log`. This is expected; don't go looking for `receiver_file_log_`.

## bbolt corruption and `recreate` can lose or duplicate state

bbolt is generally durable, but on certain corruption panics the extension can either crash the Collector or — with `recreate: true` — rename the corrupted db to `{filename}.{ISO 8601 timestamp}.backup` and start a fresh, empty db so the Collector keeps running (contrib issue #35899). The trade-off: the new db has **no** prior state, so offsets reset (data re-read) or a persistent queue is lost; a receiver may then re-emit already-processed data (duplicates). Treat `recreate` as availability-over-correctness.

## mmap space is not reclaimed until compaction

bbolt is mmap-backed: after a usage spike (e.g. a persistent queue filling during an outage) the file's allocated size grows and does **not** shrink when the data is removed. Disk/mmap footprint stays high until a compaction runs. Enable `compaction.on_start` and/or `compaction.on_rebound` to reclaim it — see [advanced.md](advanced.md).

## `fsync` is a durability-vs-throughput trade-off

`fsync: false` (the default) leaves bbolt's `DB.NoSync` behavior, which is faster but can lose the very latest writes on an unclean shutdown. `fsync: true` forces an `fsync` after every DB write for maximum integrity at a real throughput cost. Only enable it where losing the last few writes is unacceptable.

## Container volume / permission gotchas

The contrib image runs as a **non-root** uid. A mounted host directory (or a Kubernetes volume) must be **writable by that uid**, or the extension can't create/open its bbolt files. In the verification recipe this is why the scratch tree is `chmod -R 777`. In production, set ownership/`fsGroup` appropriately rather than world-writable. `directory_permissions` only affects directories the extension itself creates (with `create_directory: true`); it does not fix a non-writable mount.

## Reading the data for troubleshooting

The bbolt file's contents are readable with the `strings` utility, which is handy for confirming a consumer actually wrote state. This is an implementation detail and may change if the extension moves off bbolt — don't build tooling on it.

## Stability

The extension is **beta**. It is widely used and stable in practice, but the config surface can still change between releases — confirm keys/defaults against the exact contrib version you run.

# `file_storage` extension

| | |
|-|-|
| Kind | extension (storage) |
| Type | `file_storage` |
| Stability | beta |
| Distributions | contrib, k8s |
| Go module | `github.com/open-telemetry/opentelemetry-collector-contrib/extension/storage/filestorage` |
| Upstream README | <https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/storage/filestorage> |

## Description

A **storage extension** that persists other components' state to local disk, backed by **bbolt** (an mmap-based on-disk B+tree key/value store). It gives the Collector durable, restart-surviving state: a receiver's read offsets, an exporter's persistent send queue, or a stateful processor's decision cache all live in files under a single `directory`. The extension writes **one bbolt file per consuming component**, named `<kind>_<type>_<name>` inside that directory.

Unlike a processor or receiver, `file_storage` does **not** sit in a pipeline. It is loaded by listing it under `service.extensions:`, and is then referenced by other components — receivers via their `storage:` key, exporters via `sending_queue.storage:`. Without it, those components hold their state in memory and lose it on every restart.

## Main use-cases

Use when:
- A receiver (e.g. [`file_log`](../file_log/README.md)) must resume reading where it left off after a restart instead of re-reading or skipping data — persist its offsets via `storage:`.
- An exporter needs a **persistent send queue** that survives a restart or crash — set `sending_queue.storage: file_storage` (see the [`otlp` exporter](../otlp_exporter/README.md)).
- A stateful processor needs durable state across restarts — e.g. the [`tail_sampling`](../tail_sampling/README.md) decision cache or [`drain`](../drain/README.md).

Avoid when:
- You are running a stateless pipeline where losing in-memory buffers on restart is acceptable — the extension adds disk I/O and a writable directory requirement for no benefit.
- The Collector has no stable local disk (e.g. a fully ephemeral container with no volume) — state would not actually survive a restart.

## Related components

- [`file_log`](../file_log/README.md) receiver — its `storage:` key points here to persist read offsets/checkpoints across restarts.
- [`otlp` exporter](../otlp_exporter/README.md) — set `sending_queue.storage: file_storage` for a restart-durable persistent queue (otherwise the queue is in-memory).
- [`tail_sampling`](../tail_sampling/README.md) / [`drain`](../drain/README.md) — stateful processors whose decision/aggregation caches can be made durable via this extension.

## Details

- [Configuration](configuration.md) — full top-level and `compaction` config tables, the bbolt per-consumer file naming, and the directory-permissions/umask detail.
- [Verification](verification.md) — offset-persistence-across-restart recipe (no telemetrygen) proving the extension survives a Collector restart.
- [Advanced use-cases](advanced.md) — persistent exporter queues, multiple named storage instances, rebound-compaction tuning, and `create_directory` for ephemeral/container environments.
- [Known quirks](quirks.md) — directory-must-exist gotcha, the validation-error→fix table, per-consumer file naming and the `receiver_filelog_` surprise, bbolt corruption + `recreate`, mmap reclamation, fsync trade-off, and container permission gotchas.

# `file_storage`: verification

See [Verification harness](../../SKILL.md#verification-harness) for how to run this end-to-end.

`file_storage` is an **extension**, so it does not sit in a pipeline and there is no "telemetrygen → component → debug" path to exercise it directly. The behavior worth proving is **state durability across a Collector restart**. This recipe drives the extension through a [`file_log`](../file_log/README.md) receiver and shows that the receiver's read offset survives a restart: after a restart, already-read lines are **not** re-read even with `start_at: beginning`. No `telemetrygen` is involved — the input is a file on disk (like the `file_log` page). Verified end-to-end on `otel/opentelemetry-collector-contrib:0.154.0`.

## Setup

In a scratch dir (`/tmp/fsverify`), create a `storage/` dir and a `logs/` dir. The contrib image runs as a non-root uid, so a mounted host dir must be writable by it — make the scratch tree world-writable for the repro:

```bash
mkdir -p /tmp/fsverify/storage /tmp/fsverify/logs
printf 'line one\nline two\nline three\n' > /tmp/fsverify/logs/app.log
chmod -R 777 /tmp/fsverify
cd /tmp/fsverify
```

Config `fs.yaml` (minimal repro — omits `memory_limiter` etc. on purpose, to isolate the extension):

```yaml
extensions:
  file_storage:
    directory: /storage          # mounted host dir; exists, so create_directory not needed
receivers:
  file_log:
    include: [/logs/app.log]
    start_at: beginning
    storage: file_storage        # <- receiver persists its read offset here
exporters:
  debug:
    verbosity: detailed          # basic verbosity prints no per-record lines
service:
  extensions: [file_storage]     # extensions must be listed here to load
  pipelines:
    logs:
      receivers: [file_log]
      exporters: [debug]
```

## Run 1 — first read

Start the Collector, mounting **both** the storage and logs dirs:

```bash
IMG=otel/opentelemetry-collector-contrib:0.154.0
docker run -d --name fsrun1 \
  -v "$PWD/fs.yaml:/etc/otelcol-contrib/config.yaml" \
  -v "$PWD/storage:/storage" -v "$PWD/logs:/logs" $IMG
sleep 5
docker logs fsrun1 2>&1 | grep -c '^LogRecord #'        # => 3
docker logs fsrun1 2>&1 | grep 'Body: Str(line'         # => line one/two/three
ls /tmp/fsverify/storage                                # => receiver_filelog_  (the bbolt file)
docker rm -f fsrun1
```

**Verified result:** 3 records (`line one`, `line two`, `line three`), and a bbolt file **`receiver_filelog_`** (32768 bytes) appears in the storage dir. Note the on-disk name is `receiver_filelog_` (the deprecated `filelog` type spelling, trailing `_` for the empty instance name) even though the config uses `file_log` — see [quirks.md](quirks.md).

## Run 2 — restart resumes from the persisted offset

Append 2 lines and restart with the **same** storage dir:

```bash
printf 'line four\nline five\n' >> /tmp/fsverify/logs/app.log   # now 5 lines
docker run -d --name fsrun2 \
  -v "$PWD/fs.yaml:/etc/otelcol-contrib/config.yaml" \
  -v "$PWD/storage:/storage" -v "$PWD/logs:/logs" $IMG
sleep 5
docker logs fsrun2 2>&1 | grep -c '^LogRecord #'        # => 2
docker logs fsrun2 2>&1 | grep 'Body: Str(line'         # => line four/line five ONLY
docker rm -f fsrun2
```

**Verified result:** only **2** records (`line four`, `line five`). The persisted offset means the first 3 lines are **not** re-read, even though `start_at: beginning`.

## Control — without `storage`, everything is re-read

Same restart, but with the `storage:` line removed from the receiver (`fs_nostore.yaml` = `fs.yaml` minus that line):

```bash
sed '/storage: file_storage/d' fs.yaml > fs_nostore.yaml   # fs.yaml minus the receiver's storage: line
docker run -d --name fsctl \
  -v "$PWD/fs_nostore.yaml:/etc/otelcol-contrib/config.yaml" \
  -v "$PWD/storage:/storage" -v "$PWD/logs:/logs" $IMG
sleep 5
docker logs fsctl 2>&1 | grep -c '^LogRecord #'         # => 5  (re-reads everything)
docker rm -f fsctl
```

**Verified result:** **5** records — without `storage`, the restart re-reads all 5 lines (no offset memory). This contrast proves the `file_storage` extension is what preserved the offset.

## Tear down

```bash
docker rm -f fsrun1 fsrun2 fsctl 2>/dev/null
rm -rf /tmp/fsverify
```

# `filter`: advanced use-cases

## Combining conditions

Conditions within a list are ORed — any match drops the item. Use `and`/`or`/`not()` and parentheses inside a single condition for AND logic:

```yaml
processors:
  filter:
    error_mode: ignore
    trace_conditions:
      # OR across the list — drop either name
      - 'span.name == "readiness"'
      - 'span.name == "liveness"'
      # AND within one condition — short, successful spans only
      - '(span.end_time - span.start_time) < Duration("10ms") and span.status.code == STATUS_CODE_OK'
```

## Advanced style: explicit context and per-group error_mode

Each entry in a `*_conditions` list can be an object that pins a `context` and/or overrides `error_mode` for its group of `conditions`. Set `context` only when inference can't resolve it — e.g. a bare `IsRootSpan()` with no path prefix:

```yaml
processors:
  filter:
    error_mode: ignore
    trace_conditions:
      - context: span            # required: IsRootSpan() has no prefix to infer from
        error_mode: propagate    # override just this group
        conditions:
          - IsRootSpan()
      - conditions:
          - spanevent.name == "grpc.timeout"
```

Basic (flat strings) and advanced (objects) styles cannot be mixed within one signal.

## error_mode

```yaml
# Default: ignore — a bad condition logs but does not drop unrelated data
processors:
  filter:
    error_mode: ignore
```

```yaml
# Development: catch broken conditions immediately (must set explicitly; ignore is the default)
processors:
  filter:
    error_mode: propagate   # a failing condition drops the whole batch
```

```yaml
# Noisy environments: continue without logging the error at all
processors:
  filter:
    error_mode: silent
```

## Metric and datapoint filtering

Drop whole metrics by name or type (cheapest), or individual datapoints by value/attribute:

```yaml
processors:
  filter:
    error_mode: ignore
    metric_conditions:
      # metric-only converters (HasAttrKeyOnDatapoint / HasAttrOnDatapoint) carry no
      # path prefix, so their context can't be inferred — pin context: metric explicitly.
      - context: metric
        conditions:
          - 'IsMatch(metric.name, "internal\\..*")'
          - 'metric.type == METRIC_DATA_TYPE_HISTOGRAM'
          - 'HasAttrKeyOnDatapoint("internal")'
          - 'HasAttrOnDatapoint("environment", "test")'
      # datapoint-level check, inferred separately (metric context can't see datapoint.value_int)
      - conditions:
          - 'metric.name == "k8s.pod.phase" and datapoint.value_int == 4'
```

Filter at `metric` level when you want to drop the whole series — it is cheaper than evaluating every datapoint. See [Known quirks](quirks.md) for the drop-last-datapoint rule.

## Filtering by resource attribute

Resource attributes are visible in every signal, so the same condition shape works across them — useful for stripping an environment or service wholesale:

```yaml
processors:
  filter:
    error_mode: ignore
    trace_conditions:
      - 'resource.attributes["deployment.environment"] == "dev"'
    metric_conditions:
      - 'resource.attributes["deployment.environment"] == "dev"'
    log_conditions:
      - 'resource.attributes["deployment.environment"] == "dev"'
```

## OTTL function usage

The processor exposes the standard OTTL converters (see the `otel-ottl` skill) plus the two metric-only functions above:

```yaml
processors:
  filter:
    error_mode: ignore
    trace_conditions:
      - 'IsMatch(resource.attributes["k8s.pod.name"], "canary-.*")'
    log_conditions:
      - 'IsMatch(log.body, ".*(password|secret|api[_-]?key).*")'
      - 'log.attributes["http.request.method"] == nil'
```

## Named instances

```yaml
processors:
  filter/health:
    error_mode: ignore
    trace_conditions:
      - 'span.name == "health_check"'
  filter/debug-logs:
    error_mode: ignore
    log_conditions:
      - 'log.severity_number < SEVERITY_NUMBER_INFO'

service:
  pipelines:
    traces:
      processors: [memory_limiter, filter/health]
    logs:
      processors: [memory_limiter, filter/debug-logs]
```

## OTTL context

The `*_conditions` lists infer their context (`resource`/`scope`/`span`/`spanevent`, `metric`/`datapoint`, `log`, `profile`) from the path prefixes used; the filter processor does **not** expose an `exemplar` context. Path and converter inventories live in the `otel-ottl` skill; the field summary per context is in [configuration.md](configuration.md).

# `filter`: configuration

## Typical config

```yaml
processors:
  filter:
    error_mode: ignore
    trace_conditions:
      - span.name == "health_check"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, filter]
      exporters: [otlphttp]
```

## Top-level keys

| Key | Type | Default | Notes |
|-----|------|---------|-------|
| `error_mode` | string | `ignore` | How OTTL condition-evaluation errors are handled: `propagate`, `ignore`, or `silent`. See [Error mode](#error-mode). |
| `trace_conditions` | list | — | Trace conditions (contexts: `resource`, `scope`, `span`, `spanevent`). |
| `metric_conditions` | list | — | Metric conditions (contexts: `resource`, `scope`, `metric`, `datapoint`). |
| `log_conditions` | list | — | Log conditions (contexts: `resource`, `scope`, `log`). |
| `profile_conditions` | list | — | Profile conditions (contexts: `resource`, `scope`, `profile`). **Development** stability. |
| `traces` / `metrics` / `logs` / `profiles` | block | — | **Deprecated** per-signal blocks and legacy `include`/`exclude` matchers. See [Legacy configuration](#legacy-configuration). |

> **Default `error_mode` is permanently `ignore` as of v0.157.0.** The `processor.filter.defaultErrorModeIgnore` gate is Stable since v0.157.0; registered and cannot be disabled (disabling it fails startup). On v0.153.0–v0.156.0 it was Beta and could still be disabled to restore the old `propagate` default.

## The `*_conditions` fields

`trace_conditions`, `metric_conditions`, `log_conditions`, and `profile_conditions` are the current, recommended surface (documented upstream from v0.146.0). Each takes a list of OTTL boolean conditions; an item is **dropped** when any condition evaluates to `true` (conditions are ORed). The expression language — operators, paths, converter functions — is covered in the `otel-ottl` skill; only the filter-specific surface is described here.

### Contexts and path prefixes

Within a `*_conditions` list, paths are **prefixed by context** and the processor **infers** the context from the prefixes used, so you rarely set it by hand.

| Field | Available contexts | Representative paths |
|-------|--------------------|----------------------|
| `trace_conditions` | `resource`, `scope`, `span`, `spanevent` | `span.name`, `span.attributes[...]`, `span.status.code`, `span.kind`, `span.start_time`, `span.end_time`, `spanevent.name`, `resource.attributes[...]`, `scope.name` |
| `metric_conditions` | `resource`, `scope`, `metric`, `datapoint` | `metric.name`, `metric.type`, `metric.unit`, `datapoint.attributes[...]`, `datapoint.value_int`, `datapoint.value_double`, `resource.attributes[...]` |
| `log_conditions` | `resource`, `scope`, `log` | `log.body`, `log.attributes[...]`, `log.severity_number`, `log.severity_text`, `resource.attributes[...]`, `scope.name` |
| `profile_conditions` | `resource`, `scope`, `profile` | `profile.duration_unix_nano`, `resource.attributes[...]` |

The filter processor supports **only** the contexts above — notably **no `exemplar` context** (that path exists in the `transform` processor, which mutates rather than drops).

**Hierarchy.** Conditions run higher-to-lower (`resource` → `scope` → signal-specific). If a higher-level item is dropped, lower-level conditions for it are skipped: a dropped span skips its `spanevent` conditions. If all datapoints of a metric are dropped, the metric is dropped too; if all span events of a span are dropped, the span is left intact. When one condition mixes paths from two contexts, it is evaluated in the **lower** context (e.g. `resource` + `spanevent` → evaluated per span event). See [Known quirks](quirks.md).

**Metric-only converters.** This processor adds two functions that must run in the **`metric`** context: `HasAttrKeyOnDatapoint(key)` (true if any datapoint carries that attribute key) and `HasAttrOnDatapoint(key, value)` (true if any datapoint has that string key/value). Because they carry no path prefix, their context **cannot be inferred** — pin `context: metric` explicitly (advanced style), or validation fails with `unable to infer context from conditions`.

### Basic vs advanced style

**Basic** — a flat list of OTTL strings, ORed, context inferred per condition:

```yaml
filter:
  error_mode: ignore
  trace_conditions:
    - span.attributes["container.name"] == "app_container_1"
    - resource.attributes["host.name"] == "localhost"
    - span.name == "app_3"
```

**Advanced** — objects that pin a `context` and/or override `error_mode` for a group of `conditions`:

```yaml
filter:
  error_mode: ignore
  trace_conditions:
    - context: span            # set explicitly only when inference can't (e.g. IsRootSpan())
      error_mode: propagate    # overrides the top-level error_mode for this group
      conditions:
        - IsRootSpan()
    - conditions:
        - spanevent.name == "grpc.timeout"
```

Set `context` explicitly only when a condition has no path prefix to infer from (e.g. a bare `IsRootSpan()`), or when a combination of paths/functions/enums is not allowed in a single inferred context. Basic and advanced (and the deprecated forms) **cannot be mixed within one signal** — validation rejects mixing configuration styles.

## Error mode

| Mode | Behavior | When |
|------|----------|------|
| `ignore` | Logs the error and moves to the next condition. | **Default and recommended** — a bad condition won't drop unrelated data. |
| `silent` | Continues without logging. | When error logging is too noisy. |
| `propagate` | Returns the error up the pipeline, dropping the whole payload. | Development/testing — surfaces config mistakes immediately. |

## Legacy configuration

Two older forms still work but are **deprecated** and slated for removal; both are documented upstream only for versions before v0.146.0. Prefer `*_conditions` for new configs.

**Deprecated per-signal OTTL blocks** — same conditions, unprefixed paths inside a fixed context. Migrate by moving each into the matching `*_conditions` list with the context prefix:

| Deprecated | Migrate to |
|------------|------------|
| `traces.resource` / `traces.span` / `traces.spanevent` | `trace_conditions` with `resource.` / `span.` / `spanevent.` prefix |
| `metrics.resource` / `metrics.metric` / `metrics.datapoint` | `metric_conditions` with `resource.` / `metric.` / `datapoint.` prefix |
| `logs.resource` / `logs.log_record` | `log_conditions` with `resource.` / `log.` prefix |
| `profiles.resource` / `profiles.profile` | `profile_conditions` with `resource.` / `profile.` prefix |

**Pre-OTTL include/exclude** (metrics and logs only) — name/attribute/severity matchers. Cannot be combined with any OTTL conditions for the same signal.

Metrics — match by name and/or resource attribute:

```yaml
metrics:
  include:
    match_type: strict   # or regexp
    metric_names:
      - "system.cpu.utilization"
    resource_attributes:
      - key: "env"
        value: "production"
  exclude:
    match_type: regexp
    metric_names:
      - "internal\\..*"
```

Logs — match by severity, body, or attribute:

```yaml
logs:
  include:
    match_type: strict   # or regexp
    severity_number:
      min: "INFO"
      match_undefined: true
    bodies:
      - ".*error.*"
    record_attributes:
      - key: "user_id"
        value: ".*"
```

Semantics: `include` alone keeps only matches; `exclude` alone drops matches; both means `include` is applied first, then `exclude`. `match_type` is `strict` (exact) or `regexp`. For metrics, a `regexp` block (`cacheenabled`, `cachemaxnumentries`) tunes the regex cache when `match_type: regexp`.

# `filter`: known quirks

## Data is permanently dropped

`filter` removes data from the pipeline; there is no recovery downstream. Test conditions with `error_mode: propagate` (explicitly — the default is now `ignore`) and `debug` logging before rolling out.

## Dropping a metric's only datapoint removes the metric

Filtering at the `datapoint` level that empties a metric also removes the metric — if all datapoints of a metric match, the whole metric disappears. Likewise, if all telemetry for a resource is dropped, the resource is removed from the payload, and if all span events of a span are dropped the span itself survives (only the events go). Be deliberate about which level you filter at.

## Orphaned telemetry / broken trace completeness

- Dropping a **parent** span orphans its children — they remain in the trace but point at a span that no longer exists, which breaks waterfall views.
- Dropping spans that logs reference (via trace/span ID) breaks log↔trace correlation.
- Dropping arbitrary spans makes a trace incomplete. When you need a whole-trace keep/drop decision, use `tail_sampling`, not `filter`.

## error_mode behavior

- Default is `ignore`: a failing condition is logged and skipped, valid data survives. The `processor.filter.defaultErrorModeIgnore` gate is Stable since v0.157.0; registered and cannot be disabled (disabling it fails startup).
- Setting `error_mode: propagate` explicitly makes a single failing condition (e.g. a type mismatch or a nil dereference) **drop the entire batch**, not just the offending item. Older collectors (pre-v0.153.0) defaulted to `propagate`; on v0.153.0–v0.156.0 the Beta gate could also be disabled to restore that old default.
- `silent` hides evaluation errors entirely; if a filter "isn't working," confirm it isn't set to `silent`.

## Stability caveats

All signals are **Alpha** (profiles are **Development**). Config keys can still change between releases, and profile filtering may break. Check the upstream README for the running version before treating any key as stable.

## Troubleshooting

**Nothing is dropped.**
- Missing/wrong path prefix — in `*_conditions`, paths must be prefixed (`span.name`, `log.body`, `metric.name`, `datapoint.value_int`); the context is inferred from those prefixes.
- Attribute doesn't exist — add a nil check: `log.attributes["x"] != nil and log.attributes["x"] == "y"`.
- `error_mode: silent` is swallowing evaluation errors — switch to `propagate` while debugging.

**Everything is dropped.**
- Overly broad condition (`IsMatch(span.name, ".*")` matches all). Tighten it and re-test with debug logging.

**Validation error: `configuring multiple configuration styles is not supported`.**
- You mixed basic (flat string list) and advanced (`context`/`conditions` objects) forms in one signal — use one style.

**Validation error: `cannot use context inferred ... conditions and the settings ... at the same time`.**
- You mixed `*_conditions` with a deprecated per-signal block or include/exclude for the same signal. Pick one.

**Verifying effectiveness.** The processor emits `processor_filter_spans.filtered`, `processor_filter_datapoints.filtered`, `processor_filter_logs.filtered` (and `processor_filter_profiles.filtered`, development) — watch these to confirm drops.

## Anti-patterns

**Datapoint-level filtering when a whole metric should go.**

```yaml
# Less efficient — inferred as datapoint context, evaluates every datapoint
metric_conditions:
  - 'datapoint.attributes["env"] == "test"'
```

```yaml
# Prefer — resource condition drops the whole resource in one check
metric_conditions:
  - 'resource.attributes["env"] == "test"'
```

**Dropping parent/correlated spans.**

```yaml
# Risky — orphans children, breaks log correlation
trace_conditions:
  - 'span.kind == SPAN_KIND_SERVER'
```

Filter leaf/noise spans, or move whole-trace decisions to `tail_sampling`.

**Complex negative-lookahead regex.**

```yaml
# Rejected at config load — Go's RE2 engine has no lookahead
log_conditions:
  - 'IsMatch(log.body, "^(?!.*(error|warn|fail)).*$")'
```

`IsMatch` compiles with Go's RE2 engine, which does not support lookahead (`(?!`); the pattern above fails validation outright (`invalid or unsupported Perl syntax: `(?!``). Prefer explicit positive conditions (`log.severity_number < SEVERITY_NUMBER_WARN`).

# `filter` processor

| | |
|-|-|
| Kind | processor |
| Type | `filter` |
| Signals | traces (Alpha), metrics (Alpha), logs (Alpha), profiles (Development) |
| Distributions | core, contrib, k8s |
| Go module | `github.com/open-telemetry/opentelemetry-collector-contrib/processor/filterprocessor` |
| Upstream README | <https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/filterprocessor> |

## Description

Drops telemetry — spans, span events, metrics, datapoints, log records (and, in development, profiles) — from a pipeline when it matches an OTTL condition. Conditions are evaluated per item; when any condition evaluates to `true`, that item is dropped. Conditions in a list are ORed together. Filtering is hierarchical: if a higher-level item is dropped (e.g. a span), the lower-level conditions for it (e.g. its span events) are not evaluated.

The current surface is the per-signal `trace_conditions` / `metric_conditions` / `log_conditions` / `profile_conditions` lists, which carry OTTL conditions with context-prefixed paths (`span.name`, `log.body`, …) and infer their context automatically. The top-level `error_mode` controls what happens when a condition fails to evaluate — `ignore` (the permanent default as of v0.157.0) logs and continues, `propagate` fails the batch, `silent` continues without logging. Two older forms are deprecated: the fixed-context per-signal blocks (`traces.span`, `metrics.metric`, `logs.log_record`, …) and a pre-OTTL include/exclude matcher (metrics and logs only) that cannot be combined with OTTL conditions for the same signal.

## Main use-cases

Use it when:
- You want to drop noise outright — health-check spans, debug/trace logs, internal or test-environment metrics.
- You need to reduce backend volume by removing low-value signals at the collector.
- You want to strip telemetry by resource attribute (environment, service, pod) before it leaves the pipeline.

Avoid it when:
- You want to **modify** rather than drop telemetry — use `transform`.
- You need a content-aware keep/drop decision over a **whole trace** (errors, latency) — use `tail_sampling`.
- Dropping would orphan child spans or break trace/log correlation (see [Known quirks](quirks.md)).

## Related components

- `transform` — rewrites telemetry via OTTL instead of dropping it; same expression language.
- `attributes` / `resource` — add or modify the attributes that filter conditions match on (run before `filter`).
- `tail_sampling` — trace-wide, content-aware keep/drop sampling for traces.
- `probabilistic_sampler` — stateless head sampling by trace ID.
- `routing` — sends telemetry to different pipelines by condition rather than dropping it.

## Details

- [Configuration](configuration.md) — `error_mode`, the `*_conditions` lists with their inferred contexts (`resource`/`scope`/`span`/`spanevent`, `metric`/`datapoint`, `log`, `profile`), basic vs advanced style, and the deprecated per-signal blocks and include/exclude matchers.
- [Verification](verification.md) — telemetrygen recipe that drops a subset of logs by severity so the drop is observable.
- [Advanced use-cases](advanced.md) — combining conditions, `error_mode`, metric/datapoint filtering, resource-attribute filters, OTTL functions, named instances.
- [Known quirks](quirks.md) — orphaned telemetry, the drop-last-datapoint rule, error-mode behavior, stability caveats, anti-patterns, troubleshooting.

# `filter`: verification

See [Verification harness](../../SKILL.md#verification-harness) for how to run this end-to-end.

`filter` ships in the `core`, `contrib`, and `k8s` distributions, so a stock contrib collector can run this.

Verified on `otel/opentelemetry-collector-contrib:0.159.0` with
`ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:v0.159.0`
(2026-08-27).

Config (`filter-verify.yaml`) — drop `ERROR`-and-above logs, let everything below pass:

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
processors:
  filter:
    error_mode: ignore
    log_conditions:
      - 'log.severity_number >= SEVERITY_NUMBER_ERROR'
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [filter]
      exporters: [debug]
```

Send two batches — one that the condition matches (dropped) and one that it doesn't (kept) — see the `otel-telemetrygen` skill:

```bash
# Info logs (severity 9) — below ERROR, expect these to survive
telemetrygen logs --otlp-insecure --otlp-endpoint localhost:4317 \
  --logs 10 --severity-number 9 --severity-text Info --body "info log keep me"
# Error logs (severity 17) — at/above ERROR, expect these to be dropped
telemetrygen logs --otlp-insecure --otlp-endpoint localhost:4317 \
  --logs 10 --severity-number 17 --severity-text Error --body "error log drop me"
```

Flags confirmed against the `otel-telemetrygen` skill (`references/flags.md`): `--otlp-insecure`, `--otlp-endpoint`, `--logs`, `--severity-number` (int32, 1–24; 9=Info, 17=Error per the severity-number reference), `--severity-text`, and `--body`. Telemetrygen validates that severity text agrees with the numeric range, so state both explicitly. No `--duration` here — telemetrygen ignores `--logs` when `--duration` is set, so the count would not be 10.

**What proves it worked:** the `debug` exporter prints the 10 Info records ("info log keep me") and **none** of the 10 Error records ("error log drop me"). Fewer records reach `debug` than were sent, and only the non-matching severity survives.

# `interval`: advanced use-cases

## Keep gauges spiky, smooth everything else

```yaml
processors:
  interval:
    interval: 30s
    pass_through:
      gauge: true
      summary: true
```

Use this when your gauges represent things you can't afford to flatten — request latency, queue depth, current connections.

## Multiple intervals for different pipelines

```yaml
processors:
  interval/fast:
    interval: 15s
  interval/slow:
    interval: 5m

service:
  pipelines:
    metrics/realtime:
      receivers: [otlp]
      processors: [memory_limiter, interval/fast]
      exporters: [otlp/dashboard]
    metrics/longterm:
      receivers: [otlp]
      processors: [memory_limiter, interval/slow]
      exporters: [otlp/warehouse]
```

# `interval`: configuration

## Typical config

```yaml
processors:
  interval:

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, interval]
      exporters: [otlphttp]
```

## Configuration reference

| Key | Type | Default | Notes |
|-----|------|---------|-------|
| `interval` | duration | `60s` | Emission cadence. After each emission, internal state is cleared. |
| `pass_through.gauge` | bool | `false` | If `true`, gauges are forwarded unchanged instead of buffered. |
| `pass_through.summary` | bool | `false` | If `true`, summaries are forwarded unchanged instead of buffered. |

Validation: `interval` must be greater than `0`. A zero or negative value fails startup with `invalid interval value` (`ErrInvalidIntervalValue` in source). `pass_through` booleans have no validation.

> **Doc typo, upstream:** the README's configuration block shows `summary: <boo>l`. It's a typo for `<bool>` — the parameter is a plain boolean.

## What it does to each metric type

| Metric type | Default behavior | Pass-through option |
|-------------|------------------|---------------------|
| Cumulative monotonic sum | Aggregated (only latest value per series emitted per interval) | — |
| Cumulative monotonic histogram | Aggregated | — |
| Cumulative monotonic exponential histogram | Aggregated | — |
| Gauge | Aggregated (lossy — see below) | `pass_through.gauge: true` |
| Summary | Aggregated (lossy — see below) | `pass_through.summary: true` |
| Delta monotonic sum, delta histogram, delta exponential histogram | Accumulated: the interval's points are **added** per series and emitted once, with the earliest start and latest timestamp | — |
| Non-monotonic sum (any temporality) | Passed through unchanged, immediately | — (always pass-through) |

> **Doc drift, upstream:** as of v0.162.0 the upstream README still says all delta metrics pass through unchanged; the code (`processor.go`, PR #50922) aggregates them as above.

## What "lossy" means here

- Monotonic cumulative series: you lose **precision** (intermediate values), not totals — the final cumulative value still represents the full count.
- Delta series: per-point granularity is merged into one delta per interval; the total is preserved.
- Gauges and summaries: actual **data loss**. A value that rose and fell back inside the interval is reduced to whatever the last reading happened to be. If the up-and-down matters (latency spikes, queue depth bursts), set `pass_through.gauge: true` / `pass_through.summary: true` to keep them flowing as-is.

## Behavior example

Monotonic sums arriving into the processor:

| Time | Metric | Temporality | Attributes | Value |
|------|--------|-------------|------------|------:|
| 0 | `test_metric` | Cumulative | `labelA: foo` | 4.0 |
| 2 | `test_metric` | Cumulative | `labelA: bar` | 3.1 |
| 4 | `other_metric` | Delta | `fruitType: orange` | 77.4 |
| 6 | `test_metric` | Cumulative | `labelA: foo` | 8.2 |
| 8 | `test_metric` | Cumulative | `labelA: foo` | 12.8 |
| 10 | `test_metric` | Cumulative | `labelA: bar` | 6.4 |

At the next interval boundary, the processor forwards the latest value of each cumulative series and the sum of each delta series (here a single point):

| Time | Metric | Temporality | Attributes | Value |
|------|--------|-------------|------------|------:|
| 4 | `other_metric` | Delta | `fruitType: orange` | 77.4 |
| 8 | `test_metric` | Cumulative | `labelA: foo` | 12.8 |
| 10 | `test_metric` | Cumulative | `labelA: bar` | 6.4 |

Then state is cleared.

# `interval`: known quirks

## State and restart behavior

- Internal state is keyed by metric name + attribute set and is held entirely in memory.
- After each emission the buffer is cleared. **If no new points arrive in the next interval, nothing is emitted for that series in that interval.**
- The state is not persisted — a collector restart drops every buffered point. For monotonic cumulative metrics this is usually fine (the next scrape carries the running total); for buffered gauges/summaries the most recent value is simply lost until the next sample.

## Troubleshooting

**Nothing comes out of the processor.**
- All upstream metrics are non-monotonic sums — those pass through but produce no buffered emission. Confirm a monotonic sum, histogram, gauge, or summary reaches the processor.
- No new samples arrived in the most recent interval — state is cleared at emission, so a silent source means a silent output. Reduce the source's scrape gap or increase `interval`.

**Delta metrics show no volume drop.**
- Before v0.162.0, delta metrics passed through unchanged. Upgrade the collector.

**Gauge spikes disappeared from dashboards.**
- Aggregation flattened them. Set `pass_through.gauge: true`.

**Memory growth.**
- Cardinality issue. The processor stores one entry per unique (metric, attribute set). Filter or drop high-cardinality attributes upstream (`transform` / `attributes` processor) before reaching `interval`.

**Output cadence is irregular after a restart.**
- Expected. The first interval after startup may emit less than usual because state was empty. Subsequent intervals settle once the upstream resends.

## Anti-patterns

**Stacking `interval` with pass-through-only metrics.**

```yaml
# BAD when the source emits only non-monotonic sums — they pass through anyway
processors:
  interval:
service:
  pipelines:
    metrics:
      processors: [interval]
```

If nothing the source emits is aggregated, remove the processor.

**Using it as a backpressure / rate-limit substitute.**

`interval` reduces *point count* per series at emission boundaries; it does not throttle. Use `memory_limiter` (and the exporter's `sending_queue`) for backpressure.

**Aggregating spiky gauges silently.**

The default aggregates gauges and summaries. If you didn't explicitly think about it, flip `pass_through.gauge: true` until you have — losing a latency spike is harder to debug than ingesting one extra point per minute.

# `interval` processor

| | |
|-|-|
| Kind | processor |
| Type | `interval` |
| Signals | metrics |
| Stability | Alpha |
| Distributions | contrib, k8s |
| Go module | `github.com/open-telemetry/opentelemetry-collector-contrib/processor/intervalprocessor` |
| Upstream README | <https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/intervalprocessor> |

## Description

Buffers metrics and emits one point per series once per interval: the latest value for cumulative series, gauges, and summaries, and the **sum** of the interval's points for delta series. Reduces metric point volume from chatty sources.

By default it aggregates monotonic sums, histograms, and exponential histograms (cumulative or delta) plus gauges and summaries; non-monotonic sums pass through unchanged. Gauges and summaries can be kept flowing as-is with `pass_through`.

## Main use-cases

Use it when:
- A receiver scrapes or emits cumulative metrics far more often than your backend needs (e.g., Prometheus receiver at 1s for a backend that ingests at 60s).
- You want to smooth bursty ingest into a fixed cadence downstream.
- Backend cost scales with point count and you have many slow-changing series.

Avoid it when:
- The pipeline carries mostly non-monotonic sums — they bypass the processor entirely and you've added a stateful component for nothing.
- Gauge spikiness matters and you can't set `pass_through.gauge: true` for the relevant signals.
- The collector restarts often — buffered state is in-memory only and is lost on restart (see [State and restart behavior](quirks.md#state-and-restart-behavior)).

## Related components

- `batch` / exporter `sending_queue.batch` — batches by size/time, does not collapse per-series points. Complementary, not a replacement.
- `metricstransform`, `transform` — rewrite metrics; do not aggregate over time.
- `cumulativetodelta` — converts temporality. Often paired with `interval` so cumulative-source metrics reach a delta-only backend as one delta per interval.

## Details

- [Configuration](configuration.md) — config keys, defaults, per-metric-type behavior, and what "lossy" means.
- [Verification](verification.md) — telemetrygen recipe to confirm point volume drops.
- [Advanced use-cases](advanced.md) — keeping gauges spiky, multiple intervals per pipeline.
- [Known quirks](quirks.md) — state/restart behavior, troubleshooting, anti-patterns.

# `interval`: verification

See [Verification harness](../../SKILL.md#verification-harness) for how to run this end-to-end.

`interval` ships in the `contrib` and `k8s` distributions.

Config (`interval-verify.yaml`):

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
processors:
  interval:
    interval: 10s
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [interval]
      exporters: [debug]
```

Generate cumulative-sum metrics frequently (see the `otel-telemetrygen` skill). Cumulative series keep their latest value per interval (delta series are summed instead):

```bash
telemetrygen metrics --otlp-insecure --otlp-endpoint localhost:4317 \
  --metric-type Sum --aggregation-temporality cumulative --rate 5 --duration 25s
```

**What proves it worked:** instead of ~5 points/sec reaching `debug`, the exporter prints the metric roughly once per 10s interval (one latest value per series). Compare against the same run with the processor removed to see the volume drop.

# `k8s_attributes`: advanced use-cases

## RBAC

The processor reads the Kubernetes API, so its ServiceAccount needs permission for exactly the resources you extract. Grant `get`/`list`/`watch` and nothing more.

**Minimum (any deployment)** — `pods` and `namespaces`:

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otel-collector
rules:
  - apiGroups: [""]
    resources: ["pods", "namespaces"]
    verbs: ["get", "watch", "list"]
```

Add resources as you extract more metadata:

| Extracted metadata | Add resource | API group |
|--------------------|--------------|-----------|
| `k8s.deployment.name` (default heuristic) | *nothing extra* — derived from the pod's owner ReplicaSet name | — |
| `k8s.deployment.uid`, or deployment/ReplicaSet labels or annotations | `replicasets`; `deployments` for `from: deployment` | `apps`, `extensions` |
| `k8s.statefulset.*` / `k8s.daemonset.*` | `statefulsets` / `daemonsets` | `apps` |
| `k8s.job.*`, `k8s.cronjob.uid`, or `from: job` / `from: cronjob` extraction | `jobs`; additionally `cronjobs` for `from: cronjob` | `batch` |
| `k8s.cronjob.name` alone | *nothing extra* — derived from the Job name via heuristic | — |
| `k8s.node.*` or node labels/annotations | `nodes` | `` (core) |

Bind it cluster-wide with a `ClusterRoleBinding` when the collector receives telemetry from many namespaces. When the collector only handles one namespace, use a namespaced `Role` + `RoleBinding` with `filter.namespace` set — least privilege, but it **cannot** read nodes, `k8s.cluster.uid`, or namespace labels/annotations.

## Agent/gateway split (passthrough)

Run the processor twice across two tiers. A per-node **agent** in `passthrough: true` mode only stamps the source pod IP (`k8s.pod.ip`) — no API calls, no RBAC — while a central **gateway** does the real enrichment off that IP.

```yaml
# Agent (DaemonSet) — just record the pod IP
processors:
  k8s_attributes:
    passthrough: true
```

```yaml
# Gateway (Deployment) — enrich off the IP the agent stamped
processors:
  k8s_attributes:
    auth_type: serviceAccount
    extract:
      metadata: [k8s.namespace.name, k8s.pod.name, k8s.deployment.name, k8s.node.name]
    pod_association:
      - sources:
          - {from: resource_attribute, name: k8s.pod.ip}
```

This concentrates the API watch and RBAC on the gateway tier instead of every node.

## Filtering for memory (large clusters)

Memory scales with pods watched (~5 KB/pod). The two biggest levers:

```yaml
# DaemonSet: watch only this node's pods — set KUBE_NODE_NAME via the downward API
filter:
  node_from_env_var: KUBE_NODE_NAME
```

```yaml
# Single-namespace collector: watch only that namespace
filter:
  namespace: production
```

`node_from_env_var` pairs with a pod env var sourced from `spec.nodeName`:

```yaml
env:
  - name: KUBE_NODE_NAME
    valueFrom:
      fieldRef:
        fieldPath: spec.nodeName
```

## Labels & annotations

Turn Kubernetes labels/annotations into resource attributes. Use exact `key` matching where possible; `key_regex` matches the **key** and can fan out to many attributes (and costs CPU):

```yaml
extract:
  labels:
    - {tag_name: app.label.component, key: app.kubernetes.io/component, from: pod}
    - {tag_name: team, key: team, from: deployment}   # needs deployments RBAC + watch
  annotations:
    - {tag_name: git.commit, key: git-commit, from: pod}
  otel_annotations: true   # also lift resource.opentelemetry.io/* annotations
```

Extracting `from:` a workload (deployment/replicaset/statefulset/daemonset/job/cronjob) makes the processor watch that resource — extra RBAC and roughly +30% memory. Only do it when you need it.

## Deployment-name lookup

`k8s.deployment.name` is derived by trimming the pod-template-hash off the pod's owner ReplicaSet name — no ReplicaSet watch, cheaper RBAC, less memory. If you configure an explicit `extract.metadata` list, include `k8s.deployment.name` (or `service.name`) for it to be produced; the omitted list already includes `k8s.deployment.name` by default. The former `deployment_name_from_replicaset` switch was removed in v0.160.0.

The heuristic sets a wrong name for pods owned by a **standalone** ReplicaSet (one with no Deployment); in rare cases deployment names of 247–253 chars come back slightly truncated. Enable `k8s.deployment.uid` to start informer-backed lookup instead, at the cost of `replicasets` RBAC and extra memory:

```yaml
extract:
  metadata: [k8s.deployment.name, k8s.deployment.uid]
```

## `wait_for_metadata`

Block startup until the metadata cache has synced, so telemetry arriving immediately on boot is still enriched:

```yaml
processors:
  k8s_attributes:
    wait_for_metadata: true
    wait_for_metadata_timeout: 30s
```

Trade-off: guaranteed enrichment of early data, but a slower start — and if the sync doesn't finish within `wait_for_metadata_timeout` the collector **fails to start**. Leave it `false` (default) when fast, best-effort startup matters more than enriching the first few records.

# `k8s_attributes`: configuration

## Typical config

```yaml
processors:
  k8s_attributes:
    auth_type: serviceAccount
    extract:
      metadata:
        - k8s.namespace.name
        - k8s.pod.name
        - k8s.pod.uid
        - k8s.deployment.name
        - k8s.node.name

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [k8s_attributes]
      exporters: [otlphttp]
```

## Configuration reference

| Key | Type | Default | Meaning |
|-----|------|---------|---------|
| `auth_type` | string | `serviceAccount` | How to reach the API: `serviceAccount` (in-cluster), `kubeConfig` (local dev, reads `~/.kube/config` or `$KUBECONFIG`), or `none`. |
| `context` | string | `""` | Kubeconfig context to use; only with `auth_type: kubeConfig`. |
| `kube_api_qps` | float32 | `5` | Max queries/sec to the API. Raise if you see client-side throttling warnings. |
| `kube_api_burst` | int | `10` | Max request burst to the API. Raise alongside `kube_api_qps` under throttling. |
| `passthrough` | bool | `false` | Agent mode: only annotate the record with the pod IP (`k8s.pod.ip`), don't query the API or extract metadata. The gateway does the enrichment. |
| `extract` | object | (see below) | Which metadata, labels, and annotations to add. |
| `filter` | object | `{}` | Restrict which pods are watched (node/namespace/label/field). The main memory lever. |
| `pod_association` | array | (see below) | Ordered strategies for tying a record to a pod; first match wins. |
| `exclude` | object | excludes `jaeger-agent`, `jaeger-collector` | Pods to skip (by name, regex allowed; an invalid regex fails config validation since v0.162.0). A pod can also opt out with the annotation `opentelemetry.io/k8s-processor.ignore: "true"`. |
| `wait_for_metadata` | bool | `false` | Block startup until the metadata cache has synced. |
| `wait_for_metadata_timeout` | duration | `10s` | Max wait for the initial sync when `wait_for_metadata: true`; on timeout the collector fails to start. |
| `watch_sync_period` | duration | `5m` | Informer cache resync period (v0.152.0). Set `0s` to disable resync — recommended for very large clusters, where periodic resyncs cause CPU spikes and memory churn. |
| `pod_delete_grace_period` | duration | `120s` | How long a deleted pod's metadata is kept in the cache before eviction (v0.154.0), so late-arriving records for that pod can still be enriched. |

## `extract`

```yaml
extract:
  metadata:        # resource attributes derived from pod/namespace/node/workload
    - k8s.namespace.name
    - k8s.pod.name
  labels:          # turn k8s labels into resource attributes
    - tag_name: app.label.component       # attribute name to write
      key: app.kubernetes.io/component    # exact label key (mutually exclusive with key_regex)
      from: pod                           # pod | namespace | node | deployment | replicaset | statefulset | daemonset | job | cronjob
  annotations:     # same shape as labels, for annotations
    - tag_name: git.commit
      key: git-commit
      from: pod
  otel_annotations: true                  # auto-extract resource.opentelemetry.io/* annotations
```

**`labels` / `annotations` fields:**

| Field | Default | Meaning |
|-------|---------|---------|
| `tag_name` | `k8s.<from>.label.<key>` / `k8s.<from>.annotation.<key>` (singular, v1 semconv — default since v0.161.0; plural `labels`/`annotations` when the semconv gates are disabled, see [quirks](quirks.md)) | Output attribute name. With `key_regex`, may use `$1` backreferences. |
| `key` | — | Exact key to match. Mutually exclusive with `key_regex`. |
| `key_regex` | — | Regex matched against the **key** (not the value). Mutually exclusive with `key`. |
| `from` | `pod` | Source object: `pod`, `namespace`, `node`, `deployment`, `replicaset`, `statefulset`, `daemonset`, `job`, `cronjob`. |

`otel_annotations: true` lifts pod annotations prefixed `resource.opentelemetry.io/` straight onto the resource (`resource.opentelemetry.io/foo: bar` → attribute `foo: bar`).

`deployment_name_from_replicaset` was removed in v0.160.0 and is now an unknown-key startup error.
Deployment names use the ReplicaSet-name heuristic unless `k8s.deployment.uid` or deployment /
ReplicaSet label or annotation extraction starts the informer lookup. When configuring an explicit
`extract.metadata` list, include `k8s.deployment.name` (or `service.name`) to emit the derived name;
omitting the list uses the defaults below.

### Default-extracted metadata

Even with no `extract.metadata` list, these are added: `k8s.namespace.name`, `k8s.pod.name`, `k8s.pod.uid`, `k8s.pod.start_time`, `k8s.deployment.name`, `k8s.node.name`, and (when a container identifier is present) `container.image.name` plus the image-tag attribute selected by the semantic-convention gates (`container.image.tags` with the v1 gates enabled — the default since v0.161.0 — or `container.image.tag` on v0; see [quirks](quirks.md)). Set an explicit `metadata` list to narrow this.

### Available metadata fields

| Field | Requires |
|-------|----------|
| `k8s.namespace.name`, `k8s.pod.name`, `k8s.pod.uid`, `k8s.pod.hostname`, `k8s.pod.ip`, `k8s.pod.start_time` | — |
| `k8s.deployment.name`, `k8s.deployment.uid`, `k8s.replicaset.name`, `k8s.replicaset.uid` | pod owned by a ReplicaSet |
| `k8s.statefulset.*`, `k8s.daemonset.*`, `k8s.job.*` | pod owned by that workload |
| `k8s.cronjob.name`, `k8s.cronjob.uid` | pod owned by a Job owned by a CronJob |
| `k8s.node.name`, `k8s.node.uid` | — |
| `k8s.cluster.uid` | reads the `kube-system` namespace UID |
| `k8s.container.name`, `container.id`, `container.image.name`, `container.image.tag` (deprecated, use `container.image.tags`), `container.image.tags`, `container.image.repo_digests` | a container identifier on the record (see [quirks](quirks.md)) |
| `service.namespace`, `service.name`, `service.version`, `service.instance.id` | calculated (see below) |

`service.name` is calculated by semantic-convention priority: `app.kubernetes.io/instance` label → `app.kubernetes.io/name` label → workload name → pod name.

## `filter`

```yaml
filter:
  node: node-1                        # only pods on this node (static name)
  node_from_env_var: KUBE_NODE_NAME   # DaemonSet: only this node's pods (set via downward API)
  namespace: production               # single namespace only
  labels:
    - {key: environment, value: production, op: equals}   # op: equals | not-equals | exists | does-not-exist
  fields:
    - {key: status.phase, value: Running, op: equals}     # op: equals | not-equals
```

`filter.node` / `filter.node_from_env_var` and `filter.namespace` are the two big memory levers — without them a DaemonSet caches every pod in the cluster.

## `pod_association`

Each association must be unique. v0.160.0 rejects duplicate source sets even when their sources are
listed in a different order.

Ordered list; the first association whose **every** source matches wins. Default order: `k8s.pod.ip` resource attr → `ip` resource attr → connection IP → `host.name` (if a valid IP).

```yaml
pod_association:
  - sources: [{from: resource_attribute, name: k8s.pod.uid}]
  - sources:                                       # both must match (AND)
      - {from: resource_attribute, name: k8s.pod.name}
      - {from: resource_attribute, name: k8s.namespace.name}
  - sources: [{from: connection}]                  # IP from the incoming connection
```

| `from` | `name` |
|--------|--------|
| `connection` | not used (uses the incoming connection IP) |
| `resource_attribute` | the attribute key to read |

Max **4** sources per association. Of the container-level attributes, only `container.id` is usable here; `service.version` and `service.instance.id` cannot be (they are computed at runtime).

# `k8s_attributes`: known quirks

## No enrichment usually means an association miss

If telemetry passes through with no `k8s.*` attributes added, the record almost never matched a pod. Walk the [pod_association](configuration.md#pod_association) order: the default starts with the `k8s.pod.ip` resource attribute, then the connection IP. Data that arrives without a pod IP and without a usable connection (e.g. through a gateway) matches nothing. Inspect the incoming record with a `debug` exporter **before** the processor to see what it actually carries, then add a `pod_association` on an attribute the data still has (`k8s.pod.uid`, or `k8s.pod.name` + `k8s.namespace.name`). RBAC gaps produce the same symptom — check with `kubectl auth can-i get pods --as=system:serviceaccount:<ns>:otel-collector`.

## Connection-based association breaks after batching or load balancing

The `from: connection` source reads the IP off the incoming gRPC/HTTP connection. Anything that re-originates the connection — a `batch` processor upstream, a load balancer, a gateway hop — drops that context, and connection association silently stops working. When telemetry crosses a network boundary, switch to a `resource_attribute` source (have a passthrough agent stamp `k8s.pod.ip`, or associate on `k8s.pod.uid`/`k8s.pod.name`).

## Memory scales with pods watched — filter

Without a `filter`, a DaemonSet caches metadata for **every pod in the cluster** (~5 KB each), which OOMKills the collector at scale. Set `filter.node_from_env_var: KUBE_NODE_NAME` (DaemonSet) or `filter.namespace` (single-namespace collector). Extracting labels/annotations `from:` a workload adds another watch (+~30% memory) — only extract what you consume.

## Container attributes need a container identifier

`container.image.name`, `container.image.tag`, `k8s.container.name`, etc. are only added when the record already carries a container identifier — `container.id` (recommended) or `k8s.container.name`. For a multi-container pod without one, the processor can't tell which container the data came from. With `k8s.container.name` but no `k8s.container.restart_count`, it may resolve the wrong container instance; add `k8s.container.restart_count` for an accurate `container.id`.

## The `regex` value-extraction field was removed

The old `extract.labels[].regex` / `extract.annotations[].regex` fields (which parsed a **value** out of a label/annotation) no longer exist — the `k8sattr.fieldExtractConfigRegex.disallow` gate that disallowed them went Stable and was then removed. Extract the full value here, then parse it with `transform` and OTTL's `ExtractPatterns`. This is distinct from `key_regex`, which matches label/annotation **keys** and still works.

## Semantic-convention label/annotation format changed in v0.161.0

Since v0.161.0 the `processor.k8sattributes.EmitV1K8sConventions` and `processor.k8sattributes.DontEmitV0K8sConventions` gates are **Beta (enabled by default)**, so label/annotation attributes use the **singular** v1 semconv form (`k8s.pod.label.<key>`, `k8s.node.annotation.<key>`, `k8s.namespace.label.<key>`, …) and the plural v0 form (`k8s.pod.labels.<key>`) is no longer emitted. The same gates switch `container.image.tag` (string) → `container.image.tags` (slice). Upgrading from ≤ v0.160 silently renames these attributes — repoint dashboards/queries, or dual-emit during migration with `--feature-gates=-processor.k8sattributes.DontEmitV0K8sConventions,processor.k8sattributes.EmitV1K8sConventions`; disable both to restore the old names. Enabling `DontEmitV0K8sConventions` without `EmitV1K8sConventions` is a startup error. The gates affect only the **default** `tag_name`; an explicit `tag_name` is unchanged. (The older `k8sattr.labelsAnnotationsSingular.allow` gate was **removed** in v0.155.0.)

The internal-telemetry gates `processor.k8sattributes.telemetry.enableNewFormatMetrics` and
`processor.k8sattributes.telemetry.disableOldFormatMetrics` are Beta and enabled by default in
v0.160.0; the new metric names are emitted and old-format metrics are disabled by default.

## Don't use it for sidecars, or outside Kubernetes

For a sidecar, inject pod metadata via the Kubernetes downward API as env vars — it's simpler and needs no RBAC. Outside Kubernetes there is no API to watch; use [`resource_detection`](../resource_detection/README.md) for cloud/host attributes instead. Host-network pods also can't be told apart by IP — associate them on `k8s.pod.name`/`k8s.pod.uid`.

## Stability caveats

Traces, metrics, and logs are Stable since v0.161.0 (the Go module is versioned `v1.x`); profiles are Development. Ongoing efficiency work (e.g. `PartialObjectMetadata` informers, and the `otelcol.k8s.pod.association` internal metric — disabled in v0.151.0, then re-enabled in v0.153.0 with a low-cardinality `pod_identifier` attribute) means behavior and internal metrics still shift between releases — confirm against the upstream README for your exact collector version.

# `k8s_attributes` processor

| | |
|-|-|
| Kind | processor |
| Type | `k8s_attributes` |
| Signals | traces (Stable), metrics (Stable), logs (Stable), profiles (Development) — Stable since v0.161.0 (Go module `v1.x`) |
| Distributions | contrib, k8s |
| Go module | `github.com/open-telemetry/opentelemetry-collector-contrib/processor/k8sattributesprocessor` |
| Upstream README | <https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/k8sattributesprocessor> |
| Rename | `k8sattributes` → `k8s_attributes` in v0.146.0; old name kept as a deprecated alias. |

## Description

Enriches telemetry with Kubernetes metadata. The processor watches the Kubernetes API for pod, namespace, node, and workload events and keeps an in-memory cache of their metadata. For each incoming span, metric, or log record it picks a pod by matching one of the configured **pod-association** sources (by default the pod IP, falling back to the connection IP), then adds the configured pod, namespace, node, and workload attributes (`k8s.pod.name`, `k8s.namespace.name`, `k8s.deployment.name`, …) to the record's **resource**.

Association is the crux: enrichment only happens when a record can be tied back to a cached pod. Records arriving with a pod IP (or over a direct connection from the pod) match out of the box; records that have lost that signal — for example after batching or load balancing strips the connection — need an explicit `pod_association` on a resource attribute the data still carries. Memory scales with the number of pods watched, so node/namespace filtering is the main lever for large clusters. The processor needs access to a **Kubernetes API** — in-cluster via a ServiceAccount, or out-of-cluster via `auth_type: kubeConfig` — since there is no cluster to enrich from otherwise; for non-Kubernetes resource attributes, use [`resource_detection`](../resource_detection/README.md).

## Main use-cases

Use it when:
- The collector runs in Kubernetes and you want telemetry correlated with pod/namespace/node/workload metadata.
- You want service discovery from pod labels/annotations (e.g. deriving `service.name`).
- You run an agent/gateway split: a `passthrough` agent stamps the pod IP, a gateway does the full enrichment.

Avoid it when:
- The collector runs **outside** Kubernetes — use [`resource_detection`](../resource_detection/README.md) instead.
- The pod already injects its metadata via the Kubernetes downward API (simpler for sidecars).
- The metadata is already on the telemetry — re-deriving it only adds overhead.

## Related components

- [`resource_detection`](../resource_detection/README.md) — detects environment resource attributes such as cloud, host, and Kubernetes node/cluster metadata. Run it **before** `k8s_attributes`.
- `resource` — sets/edits/removes resource attributes by static value or OTTL. Run it **after** `k8s_attributes` to rename or drop what was added.
- `transform` — OTTL attribute manipulation; the supported replacement for the removed `regex` value-extraction on labels/annotations (see [Known quirks](quirks.md)).

## Details

- [Configuration](configuration.md) — full config table, the `extract` / `filter` / `pod_association` blocks, default-extracted fields, and the available metadata fields.
- [Verification](verification.md) — end-to-end recipe against a real (kind) cluster: telemetrygen → `k8s_attributes` → `debug` showing pod attributes added.
- [Advanced use-cases](advanced.md) — RBAC, agent/gateway passthrough split, namespace/node filtering, workload labels & annotations, deployment-name lookup, `wait_for_metadata`.
- [Known quirks](quirks.md) — association failures, connection-based association vs batching, memory, the removed `regex` field, semconv feature gates, stability caveats.

# `k8s_attributes`: verification

See [Verification harness](../../SKILL.md#verification-harness) for the general approach. Unlike the other component recipes, this one **needs a real Kubernetes cluster** — the processor enriches from the live Kubernetes API, so there is nothing to verify without one. A throwaway [`kind`](https://kind.sigs.k8s.io/) cluster is enough. `k8s_attributes` ships in the `contrib` and `k8s` distributions, so the stock contrib image works.

The idea: run the collector **inside** the cluster (so its ServiceAccount can watch the API), deploy a throwaway workload to enrich against, then send a trace tagged with that workload's pod IP. The default `k8s.pod.ip` pod-association matches it to the cached pod, and the pod's metadata appears on the trace's resource.

## 1. Cluster

```bash
kind create cluster --name k8sattr-verify --wait 90s
```

## 2. RBAC + collector + a workload to associate against

Apply this (ServiceAccount, ClusterRole/Binding, config, collector Deployment):

```yaml
apiVersion: v1
kind: ServiceAccount
metadata: {name: otel-collector, namespace: default}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: {name: otel-collector}
rules:
  - apiGroups: [""]
    resources: ["pods", "namespaces", "nodes"]
    verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata: {name: otel-collector}
subjects: [{kind: ServiceAccount, name: otel-collector, namespace: default}]
roleRef: {kind: ClusterRole, name: otel-collector, apiGroup: rbac.authorization.k8s.io}
---
apiVersion: v1
kind: ConfigMap
metadata: {name: otel-collector-config, namespace: default}
data:
  config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
    processors:
      k8s_attributes:
        auth_type: serviceAccount
        extract:
          metadata: [k8s.namespace.name, k8s.pod.name, k8s.pod.uid, k8s.deployment.name, k8s.node.name]
        pod_association:
          - sources: [{from: resource_attribute, name: k8s.pod.ip}]
    exporters:
      debug:
        verbosity: detailed
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [k8s_attributes]
          exporters: [debug]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: otel-collector, namespace: default}
spec:
  replicas: 1
  selector: {matchLabels: {app: otel-collector}}
  template:
    metadata: {labels: {app: otel-collector}}
    spec:
      serviceAccountName: otel-collector
      containers:
        - name: otel-collector
          image: otel/opentelemetry-collector-contrib:0.160.0
          args: ["--config=/etc/otel/config.yaml"]
          ports: [{containerPort: 4317}]
          volumeMounts: [{name: config, mountPath: /etc/otel}]
      volumes: [{name: config, configMap: {name: otel-collector-config}}]
```

```bash
kubectl apply -f collector.yaml
kubectl create deployment demo --image=nginx --replicas=1   # the workload we'll enrich against
kubectl rollout status deployment/otel-collector
kubectl rollout status deployment/demo
```

## 3. Send a trace tagged with the demo pod's IP

```bash
DEMO_IP=$(kubectl get pod -l app=demo -o jsonpath='{.items[0].status.podIP}')
kubectl port-forward deployment/otel-collector 14317:4317 &   # background

telemetrygen traces --otlp-insecure --otlp-endpoint localhost:14317 \
  --traces 1 --workers 1 \
  --otlp-attributes "k8s.pod.ip=\"$DEMO_IP\""
```

Flags confirmed against the `otel-telemetrygen` skill (`references/flags.md`): `--otlp-insecure`, `--otlp-endpoint`, `--traces` (int, **per worker**; no `--duration`, or the count is ignored), `--workers`, and `--otlp-attributes` (**resource-level** — the right scope, since pod association and the added `k8s.*` attributes all live on the resource; `--telemetry-attributes` would put it at span level where association can't see it). The `k8s.pod.ip` value must be a real pod IP from the cluster so it matches a cached pod.

## What proves it worked

```bash
kubectl logs deployment/otel-collector | grep -A7 'Resource attributes'
```

A successful run sends a trace carrying only `k8s.pod.ip` and shows the demo pod's metadata added to
its resource in this shape:

```
Resource attributes:
     -> k8s.pod.ip: Str(10.244.0.5)
     -> service.name: Str(telemetrygen)
     -> k8s.pod.name: Str(demo-54d7464888-9zhvg)
     -> k8s.namespace.name: Str(default)
     -> k8s.pod.uid: Str(a4c56ba1-2f78-480c-99d0-3160d9c7c42e)
     -> k8s.deployment.name: Str(demo)
     -> k8s.node.name: Str(k8sattr-verify-control-plane)
```

`k8s.deployment.name: demo` is the strongest signal — producing it means the processor resolved the pod purely from the `k8s.pod.ip` we set, then derived the deployment name from the pod's owner ReplicaSet name via the default heuristic. The recipe therefore needs no `replicasets` or `deployments` grants. If you enable `k8s.deployment.uid` or workload label/annotation extraction, grant access to each selected workload source as shown in the [RBAC table](advanced.md#rbac); for example, CronJob extraction requires both `jobs` and `cronjobs`.

## Teardown

```bash
kind delete cluster --name k8sattr-verify
```

# `load_balancing`: advanced use-cases

## Scaling `tail_sampling` (the canonical topology)

`tail_sampling` must see **every span of a trace** to make one keep/drop decision, so it can't be sharded by chance. Put a load-balancing tier in front:

```
agents ──▶ [ LB Collectors: otlp recv → load_balancing(routing_key: traceID) ]
                                  │  consistent hash on trace ID
                                  ▼
           [ sampling Collectors: otlp recv → tail_sampling → otlp/backend ]
```

- The **LB tier** does no sampling — it only fans traces out by trace ID.
- Each **sampling Collector** receives whole traces, so its policies are correct.
- Scale the sampling tier up/down; the resolver picks up the change and the ring rebalances with minimal churn.

The same shape applies to `span_metrics`/`service_graph` downstream, but key on `service` (or `resource`) so all of a service's spans — not each trace — converge on one instance, keeping its RED/series math whole.

## Dynamic discovery

Pick the resolver that matches how backends come and go:

- **`static`** — fixed lab/edge setups. The ring only changes on restart; a dead backend stays in the ring (data to it is lost until you fix the list).
- **`dns`** against a headless Service — backends scale, DNS A-records change, re-resolved every `interval`. Subject to DNS caching delay.
- **`k8s`** EndpointSlice watch — reacts to pod churn faster than DNS and skips DNS caching; needs RBAC (see [configuration.md](configuration.md)). Use `return_hostnames: true` only with a headless Service backing a StatefulSet (stable pod identities).
- **`aws_cloud_map`** — ECS/EC2 fleets registered in Cloud Map; filter by `health_status` and remember the 100-host cap.

## Two-level resiliency model

There are **two** independent queue/retry/timeout layers:

1. **Per-backend** (`protocol.otlp.sending_queue` / `retry_on_failure` / `timeout`) — handles a single backend's transient failures (network blip, brief unavailability). Each sub-exporter has its own.
2. **Load-balancer level** (top-level `sending_queue` / `retry_on_failure` / `timeout`) — when a sub-exporter exhausts its own retries and bounces the data back, the LB level can re-queue and re-dispatch, which on the **next** resolver update may pick a healthy backend. Valuable in elastic environments (K8s scale events) where a backend disappears entirely.

Flow: sub-exporter exhausts → data returns to the LB → LB queue/retry (if enabled) re-dispatches → if all exhausted, data can be dropped. With a `static` resolver and **all** backends down, loss is unavoidable — there is nowhere to re-route.

> Persistent (file-storage-backed) queues are not supported at the per-backend sub-exporter level — the storage config is shared. Put persistence on the LB-level `sending_queue` instead.

## Multiple deterministic front-ends

The hash ring is a pure function of the routing key and the **resolved backend set**. As of v0.157.0, endpoint ordering no longer biases ring construction: two LB Collectors on the same release with the same backend set and `routing_key` compute the **identical** key→backend mapping even if their resolvers return the endpoints in a different order. So you can run several LB front-ends behind a plain round-robin L4 load balancer for HA, and a given trace still always reaches the same sampling backend regardless of which front-end handled it — no shared state or coordinator needed. Keep their configs and versions identical, or the rings can diverge.

## Named instances

Standard `type/name` works, e.g. one tier keyed by trace ID for sampling and another keyed by service for metrics:

```yaml
exporters:
  load_balancing/traces:
    routing_key: traceID
    resolver: { k8s: { service: sampling-collectors.observability } }
    protocol: { otlp: { tls: { insecure: true } } }
  load_balancing/metrics:
    routing_key: service
    resolver: { k8s: { service: spanmetrics-collectors.observability } }
    protocol: { otlp: { tls: { insecure: true } } }
```

# `load_balancing`: configuration

All keys live under the exporter instance (e.g. `exporters: { load_balancing: { … } }`). Facts below are traced to the `v0.157.0` contrib source (`config.go`, `factory.go`, `loadbalancer.go`, the per-signal `*_exporter.go` files, and the resolver files); routing-key and k8s-resolver rows re-checked at `v0.162.0`.

## Top-level keys

| Key | Type | Default | Meaning |
|-----|------|---------|---------|
| `routing_key` | string | empty → see [per-signal table](#routing_key-support-per-signal) | Which value the hash ring is keyed on. |
| `routing_attributes` | list of string | — | Attribute names to build the key from; **only** valid with `routing_key: attributes`. |
| `protocol.otlp` | otlp exporter config | otlp defaults | The per-backend transport (see [protocol.otlp](#protocolotlp)). |
| `resolver` | object | — (required) | Exactly **one** of `static` / `dns` / `k8s` / `aws_cloud_map` (see [Resolvers](#resolvers)). |
| `timeout` | duration | exporter-helper default | Timeout for the load-balancer's **own** queue, before per-backend dispatch. |
| `retry_on_failure` | object | exporter-helper `retry_on_failure` | Retry for the load-balancer level (data bounced back from a sub-exporter). |
| `sending_queue` | object | exporter-helper `sending_queue` | Queue at the load-balancer level, ahead of distribution. |

`timeout`, `retry_on_failure`, and `sending_queue` are the standard exporter-helper knobs — see the `otlp` exporter page for their sub-keys. Here they apply to the **load balancer's own** buffer (the outer level); the **per-backend** queue/retry/timeout are configured separately inside `protocol.otlp` (the inner level). The two levels form the resiliency model described in [advanced.md](advanced.md).

### `routing_key` support per signal

`routing_key` is validated **per signal exporter at startup**, and support differs by signal — this is stricter than the upstream README's prose. The table reflects **v0.162.0**:

| `routing_key` | Traces | Logs | Metrics | Keyed on |
|---------------|:------:|:----:|:-------:|----------|
| `traceID` | ✅ default | ✅ | ❌ | The trace ID. |
| `service` | ✅ | ✅ default | ✅ default | `service.name` resource attribute. |
| `attributes` | ✅ | ✅ | ✅ | The `routing_attributes` values. |
| `resource` | ❌ | ✅ | ✅ | Hash of all resource attributes. |
| `metric` | ❌ | ❌ | ✅ | Metric name. |
| `streamID` | ❌ | ❌ | ✅ | Datapoint stream identity (resource + scope + attributes). |
| `randomness` | ✅ (v0.162.0+) | ❌ | ❌ | OTel randomness: the tracestate `ot=rv` value when present, else the trace ID's low 56 bits. Traces sharing an explicit `rv` (e.g. one session/workflow) go to the same backend. |

- **Traces** accept `traceID` (default, also when empty), `service`, `attributes`, and (since v0.162.0) `randomness`. Any other value fails startup with `unsupported routing_key: <value>`. `randomness` assumes conformant random low bytes in trace IDs and a constant `rv` across a trace; one `rv` group can be arbitrarily large and lands on a single backend — see the upstream README before using it.
- **Logs** accept `service` (default, also when empty), `traceID`, `resource`, `attributes`. For `traceID`, a log without a trace ID gets a random one (so it still lands somewhere). **Version note:** logs routing arrived in **v0.154.0** (contrib PR #46241); at **v0.152.0–v0.153.0** the log exporter ignored `routing_key` and always routed by trace ID without validating the value. See [quirks.md](quirks.md).
- **Metrics** accept `service` (default, also when empty), `resource`, `metric`, `streamID`, `attributes`. Metrics are **Alpha** stability.

### `protocol.otlp`

`protocol.otlp` is a full nested **`otlp` exporter** config — TLS, headers, compression, `sending_queue`, `retry_on_failure`, `timeout`, auth, etc. all apply, per backend.

The one rule: **do not set `protocol.otlp.endpoint`.** The exporter overwrites it with each resolved backend address; the factory seeds it to a placeholder. Setting it yourself has no effect (or is misleading).

```yaml
exporters:
  load_balancing:
    routing_key: traceID
    protocol:
      otlp:
        # endpoint: DO NOT SET — populated per backend
        tls:
          insecure: true
        sending_queue:
          enabled: true          # per-backend queue
        retry_on_failure:
          enabled: true          # per-backend retry
    resolver:
      static:
        hostnames:
          - backend-1:4317
          - backend-2:4317
```

## Resolvers

Configure **exactly one**. Zero resolvers fails startup with `no resolvers specified for the exporter`; more than one fails with `only one resolver should be specified`.

### `static`

| Key | Type | Default | Meaning |
|-----|------|---------|---------|
| `hostnames` | list of string | — | Fixed `host:port` backends. No dynamic updates; the ring only changes on restart. |

### `dns`

Periodically resolves a hostname (e.g. a headless Service) to its A/AAAA records.

| Key | Type | Default | Meaning |
|-----|------|---------|---------|
| `hostname` | string | — (required) | Hostname to resolve. |
| `port` | string | — (empty) | Port appended to each resolved IP. When unset, the load balancer appends `:4317` to any address that has no port. |
| `interval` | duration | `5s` | Re-resolution interval. |
| `timeout` | duration | `1s` | Per-resolution timeout. |

### `k8s`

Watches a Kubernetes Service's `EndpointSlice`s directly (reacts faster than DNS, no DNS caching).

| Key | Type | Default | Meaning |
|-----|------|---------|---------|
| `service` | string | — (required) | `name` or `name.namespace`; namespace defaults to `default` when omitted/undeterminable. |
| `ports` | list of int32 | — (empty) | Ports per backend; multiple ports produce multiple backends per pod. When unset, the load balancer appends `:4317` to each address that has no port. |
| `timeout` | duration | `1m` | List/watch timeout. |
| `return_hostnames` | bool | `false` | Return pod hostnames instead of IPs (requires a headless Service backing a StatefulSet). |

**RBAC:** the Collector's ServiceAccount needs `get`, `list`, `watch` on `discovery.k8s.io/v1` `EndpointSlice`.

Since v0.161.0 endpoints whose `EndpointSlice` `conditions.ready` is explicitly `false` are excluded from the ring; set `publishNotReadyAddresses: true` on the Service to keep not-ready pods routable.

### `aws_cloud_map`

Discovers backends via AWS Cloud Map service discovery.

| Key | Type | Default | Meaning |
|-----|------|---------|---------|
| `namespace` | string | — (required) | Cloud Map namespace. |
| `service_name` | string | — (required) | Cloud Map service. |
| `health_status` | string | `HEALTHY` | Filter: `HEALTHY`, `UNHEALTHY`, `ALL`, `HEALTHY_OR_ELSE_ALL`. |
| `interval` | duration | `30s` | Discovery interval. |
| `timeout` | duration | `5s` | Discovery timeout. |
| `port` | uint16 | — | Optional port override for resolved instances. |
| `owner_account` | string | — | Optional AWS account ID for cross-account service discovery. |

AWS region defaults to `us-east-1` via the SDK default config chain. **Limitation:** returns at most 100 hosts — pagination is not implemented.

## Validation summary

| Condition | Error | When |
|-----------|-------|------|
| `routing_key: attributes` with empty `routing_attributes` | `routing_attributes must be specified when routing_key is "attributes"` | config validation |
| `routing_attributes` set with `routing_key` ≠ `attributes` | `routing_attributes can only be used when routing_key is "attributes"; …` | config validation |
| no resolver configured | `no resolvers specified for the exporter` | startup |
| more than one resolver configured | `only one resolver should be specified` | startup |
| unsupported `routing_key` for the signal | `unsupported routing_key: <value>` | startup (all three signals; traces print the value unquoted, logs/metrics quoted) |

# `load_balancing`: known quirks

## Do not set `protocol.otlp.endpoint`

The exporter overwrites `endpoint` with each resolved backend address — the factory seeds it to a placeholder (`placeholder:4317`). Any value you put there is ignored. Configure everything else (`tls`, `headers`, `compression`, `sending_queue`, `retry_on_failure`, auth) under `protocol.otlp`; leave `endpoint` out.

## Routing-key support is signal-specific (and partly version-specific)

The code is stricter than the upstream README's prose, and logs changed behavior across releases:

- **Logs gained `routing_key` in v0.154.0** (contrib PR #46241), accepting `service` (default), `traceID`, `resource`, `attributes`. At **v0.152.0–v0.153.0** the log exporter **ignored** `routing_key` — it always routed by trace ID and did **not** validate the value, so a `routing_key` set on a logs pipeline was silently inert. If you run an older build, don't assume `service`/`resource` log routing works.
- **Traces** accept only `traceID` (default), `service`, `attributes`, and `randomness` (v0.162.0+). `resource`, `metric`, `streamID` fail startup with `unsupported routing_key: <value>`.
- **Metrics** accept `service` (default), `resource`, `metric`, `streamID`, `attributes`, and are **Alpha** stability (traces/logs are Beta) — config can still change between releases.

Always confirm support against the exact contrib version you run; this surface has been changing.

## Exactly one resolver

`resolver` must contain **one** of `static` / `dns` / `k8s` / `aws_cloud_map`. Zero → `no resolvers specified for the exporter`; two or more → `only one resolver should be specified`. Both are exporter-build errors rather than `Config.Validate()` errors, but `validate` builds the pipeline, so it does surface them (verified on v0.156.0).

## `routing_attributes` is coupled to `routing_key: attributes`

`routing_attributes` is rejected at config validation unless `routing_key` is `attributes`, and `routing_key: attributes` is rejected unless `routing_attributes` is non-empty. They travel together.

## Static resolver + all backends down = unavoidable loss

A `static` resolver never changes its list at runtime. If every listed backend is unreachable, there is nowhere to re-route and data is dropped once the queues drain. Dynamic resolvers (`dns`/`k8s`/`aws_cloud_map`) eventually reflect topology changes, but a mismatch window still exists between a backend dying and the resolver noticing.

## DNS caching delays rebalancing

The `dns` resolver is only as fresh as the DNS responses it gets; intermediate caches/TTLs can delay a backend list update beyond `interval`. The `k8s` resolver (EndpointSlice watch) avoids this and reacts faster to pod churn.

## Hash distribution, not load awareness

The ring balances **keys**, not bytes or backend CPU. It assumes routing-key values are reasonably uniform; a few very hot trace IDs or services can skew load even though key counts are even. The exporter does not look at actual backend load.

## v0.157.0 remaps the ring even when the backend set is unchanged

v0.157.0 increased virtual points per endpoint from 100 to 200, expanded the ring coordinate space from 36,000 to 131,071, and removed endpoint-order bias. Distribution across large backend sets improves, but the same routing key can map to a different backend than it did on v0.156.0. During a rolling upgrade, mixed-version front-end Collectors can therefore disagree on key→backend mapping and split a trace or service across stateful backends. Upgrade the load-balancing front-end tier together (or otherwise drain old instances) before relying on v0.157.0 affinity.

## `return_hostnames` needs a headless StatefulSet

`k8s` `return_hostnames: true` only yields stable, resolvable names when the Service is headless and backs a StatefulSet (stable pod hostnames). With a normal Deployment, use IPs (the default).

## Rebalancing churn

On a backend-set change, roughly `R/N` routes (R = distinct routing-key values, N = backends) move to a different backend. More backends means proportionally less disruption per change — but every move briefly breaks affinity for those keys (e.g. a trace mid-flight can split across two sampling instances). Pairing with `groupbytrace` upstream can tighten per-trace atomicity.

## Single point of failure

One LB Collector in front of the fleet is itself a SPOF. Run several identical LB front-ends (same `resolver`, same `routing_key`) behind an L4 balancer — the deterministic ring means a key still maps to the same backend regardless of which front-end handles it. See [advanced.md](advanced.md).

# `load_balancing` exporter

| | |
|-|-|
| Kind | exporter |
| Type | `load_balancing` |
| Signals | traces (Beta), logs (Beta), metrics (Alpha) |
| Distributions | contrib, k8s |
| Go module | `github.com/open-telemetry/opentelemetry-collector-contrib/exporter/loadbalancingexporter` |
| Upstream README | <https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/loadbalancingexporter> |
| Rename | type renamed from `loadbalancing` to `load_balancing` in **v0.153.0**; the old name is preserved as a deprecated alias. |

## Description

Distributes telemetry across a set of downstream Collector backends so that all records sharing a **routing key** (the trace ID, the `service.name`, etc.) always land on the **same** backend. It wraps a nested `otlp` exporter — one sub-exporter per backend — and picks the target with a **consistent-hash ring** keyed on the routing value. The backend set comes from a `resolver` (`static`, `dns`, `k8s`, or `aws_cloud_map`); when the resolver reports a change, the ring is rebuilt and only ~`R/N` keys (R = routes, N = backends) move, so most traffic keeps its backend.

The point is **affinity, not raw spreading**: stateful downstream processors — `tail_sampling` (needs every span of a trace) and `span_metrics` (needs every span of a service) — only work correctly if a whole trace, or a whole service, reaches one instance. `load_balancing` is the standard way to scale those processors horizontally. Because the ring is deterministic, every front-end Collector running the **same release** with the same resolved backend set computes the **same** key→backend mapping, regardless of endpoint ordering, so you can run several identical front-ends without a coordinator. The routing key is signal-specific (and, for logs, version-specific) — see [Configuration](configuration.md). Before v0.153.0, the type was named `loadbalancing`.

## Main use-cases

Use when:
- You run `tail_sampling` (or `groupbytrace`) on more than one downstream Collector and need every span of a trace to reach the same instance.
- You generate span/service metrics (`span_metrics`, `service_graph`) downstream and need a service's spans pinned to one instance to avoid split/duplicated series.
- You want to fan telemetry across a horizontally-scaled, dynamically-discovered pool of backend Collectors (DNS/headless-service/CloudMap) with stable affinity.

Avoid when:
- You just need plain round-robin/failover to a couple of static endpoints with no affinity requirement — a single `otlp` exporter (optionally with multiple DNS-resolved targets) is simpler.
- You need to route by attribute to **different** pipelines/backends by rule — that is `routing` (a connector), not hash-based distribution.
- Your only goal is throughput batching/retry to one backend — that is the `otlp` exporter's `sending_queue`/`retry_on_failure`.

## Related components

- `tail_sampling` — the primary reason this exporter exists; place `load_balancing` in the tier **in front of** the tail-sampling Collectors so each trace is whole when it arrives.
- `routing` — rule-based routing to named pipelines (a connector); use it for "send tenant X here", not for hash-based affinity.
- `otlp` exporter — the per-backend transport `load_balancing` wraps under `protocol.otlp`; its queue/retry/timeout semantics apply per backend.
- `groupbytrace` / `span_metrics` — downstream processors that depend on the per-trace / per-service affinity this exporter provides.

## Details

- [Configuration](configuration.md) — full config tables for `routing_key` (and its per-signal support), `routing_attributes`, `protocol.otlp`, the four resolver types with every sub-key and default, and the load-balancer-level queue/retry/timeout.
- [Verification](verification.md) — telemetrygen recipe with a `static` resolver fanning four services across two backend Collectors, proving per-service affinity and stable distribution.
- [Advanced use-cases](advanced.md) — the tail_sampling scaling topology, DNS/k8s/aws_cloud_map dynamic discovery, the two-level resiliency model, and running multiple deterministic front-ends.
- [Known quirks](quirks.md) — `protocol.otlp.endpoint` must be unset, logs ignored `routing_key` before v0.154.0, traces reject `resource`/`metric`/`streamID`, resolver count validation, the v0.157.0 ring remap, static-resolver data-loss window, and stability caveats.

# `load_balancing`: verification

See [Verification harness](../../SKILL.md#verification-harness) for how to run this end-to-end.

`load_balancing` ships in the `contrib` and `k8s` distributions. This recipe needs **three** collectors — one front-end running the exporter and **two** backends — so they can talk over a shared network. The idea: route by `service`, send four distinct `service.name`s, and confirm each service's spans always land on **exactly one** backend (affinity), with the services spread across both.

> Use **v0.153.0 or newer** — the type is `loadbalancing` (one word) in v0.152.0 and earlier; it was renamed to `load_balancing` in v0.153.0. Verified on `otel/opentelemetry-collector-contrib:0.159.0` (2026-08-27).

Front-end config (`front.yaml`) — the exporter under test, with a `static` resolver listing the two backends:

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
exporters:
  load_balancing:
    routing_key: service          # each service.name hashes to one backend
    protocol:
      otlp:
        tls:
          insecure: true          # do NOT set protocol.otlp.endpoint — populated per backend
    resolver:
      static:
        hostnames:
          - lb-backend-a:4317
          - lb-backend-b:4317
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [load_balancing]
```

Backend config (`backend.yaml`, used by both backends) — a plain `debug` exporter so each backend prints what it received:

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
exporters:
  debug:
    verbosity: detailed           # basic verbosity prints no per-span lines
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [debug]
```

Start all three on one Docker network (unique names; only the front-end publishes a host port):

```bash
docker network create lbnet
IMG=otel/opentelemetry-collector-contrib:0.159.0
docker run -d --name lb-backend-a --network lbnet -v "$PWD/backend.yaml:/etc/otelcol-contrib/config.yaml" $IMG
docker run -d --name lb-backend-b --network lbnet -v "$PWD/backend.yaml:/etc/otelcol-contrib/config.yaml" $IMG
docker run -d --name lb-front     --network lbnet -p 14317:4317 \
  -v "$PWD/front.yaml:/etc/otelcol-contrib/config.yaml" $IMG
```

Send four services (and `delta` twice, to prove the route is stable) — see the `otel-telemetrygen` skill:

```bash
for svc in alpha beta gamma delta; do
  go run github.com/open-telemetry/opentelemetry-collector-contrib/cmd/telemetrygen@v0.159.0 \
    traces --otlp-endpoint localhost:14317 --otlp-insecure \
    --traces 30 --workers 1 --service "$svc"
done
# repeat delta -> must hit the same backend as the first delta batch
go run github.com/open-telemetry/opentelemetry-collector-contrib/cmd/telemetrygen@v0.159.0 \
  traces --otlp-endpoint localhost:14317 --otlp-insecure \
  --traces 30 --workers 1 --service delta
```

Flags confirmed against the `otel-telemetrygen` skill (`references/flags.md`): `--otlp-endpoint` (string), `--otlp-insecure` (bool), `--traces` (int, **per worker**; ignored if `--duration` is set, so no `--duration` here and `--workers 1` makes the per-service count exact), and `--service` (string, default `telemetrygen`; sets the `service.name` resource attribute — the exact attribute `routing_key: service` hashes on). No flag invents a value.

**What proves it worked:** count which `service.name`s appear in each backend's debug log:

```bash
for c in lb-backend-a lb-backend-b; do
  echo "--- $c ---"
  docker logs "$c" 2>&1 | grep -oE 'service.name: Str\((alpha|beta|gamma|delta)\)' | sort | uniq -c
done
```

Each service must appear in **exactly one** backend — never both:

```
--- lb-backend-a ---
  30 service.name: Str(alpha)
  30 service.name: Str(beta)
--- lb-backend-b ---
  61 service.name: Str(delta)
  30 service.name: Str(gamma)
```

`alpha`/`beta` reached only `lb-backend-a`, `gamma`/`delta` only `lb-backend-b` — clean per-service affinity, with the four services split across both backends. (The counts are per-export-batch occurrences, not span totals — they vary with batching; what matters is which backend each service appears in.) The repeated `delta` batch landed on `lb-backend-b` again, confirming the routing is a deterministic hash of `service.name`, not round-robin. Which service maps to which backend depends on the hash and the backend list, so the exact split varies; what is invariant is that **a given service never appears in more than one backend**.

Tear down:

```bash
docker rm -f lb-front lb-backend-a lb-backend-b
docker network rm lbnet
```

# `log_dedup`: advanced use-cases

## Exclude volatile fields

```yaml
processors:
  log_dedup:
    interval: 30s
    exclude_fields:
      - body.timestamp
      - attributes.request_id
      - attributes.trace_id
      - attributes.span_id
```

Records that differ only in those fields are treated as duplicates.

## Deduplicate only errors

```yaml
processors:
  log_dedup:
    interval: 60s
    conditions:
      - log.severity_number >= SEVERITY_NUMBER_ERROR
```

INFO/WARN/DEBUG logs pass through immediately. Only ERROR+ are buffered.

## Whitelist match keys

```yaml
processors:
  log_dedup:
    interval: 60s
    include_fields:
      - attributes.error_code
      - attributes.service.name
      - body.error_message
```

Groups all errors with the same code from the same service, regardless of every other attribute.

## Per-source named instances

```yaml
processors:
  log_dedup/access:
    interval: 10s
    conditions:
      - 'log.attributes["log.type"] == "access"'
  log_dedup/health:
    interval: 30s
    conditions:
      - 'log.attributes["log.type"] == "health_check"'

service:
  pipelines:
    logs:
      processors: [memory_limiter, log_dedup/access, log_dedup/health]
```

## Multi-tenant buckets with `metadata_keys`

In gateway deployments where one pipeline carries logs for many tenants, aggregating across tenants would merge their counts and lose the routing metadata. `metadata_keys` partitions aggregation per unique combination of client-metadata values (gRPC/HTTP headers), and each emitted log keeps a context carrying its original metadata so a downstream `headers_setter` can re-attach the header.

```yaml
processors:
  log_dedup:
    interval: 30s
    metadata_keys: [x-scope-orgid]
    metadata_cardinality_limit: 1000
```

Always set `metadata_cardinality_limit` when using `metadata_keys` in production: `0` (the default) means the number of tenant buckets is unbounded and memory can grow without limit — the processor logs a startup warning in that case. When the limit is reached, logs with a new metadata combination are rejected with a permanent error rather than silently merged.

Requires the receiver to propagate client metadata (e.g. the OTLP receiver's `include_metadata: true`).

## OTTL context for `conditions`

`conditions` runs in the [OTTL log context](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl/contexts/ottllog). Prefix paths with their context name: `log.body`, `log.attributes["…"]`, `resource.attributes["…"]`, `log.severity_number`, `log.severity_text`, plus all OTTL converters. Un-prefixed paths (`attributes["…"]`, `severity_number`) still work but are **deprecated** since v0.153.0 — the processor rewrites and logs them at startup, and support may be removed. See the `otel-ottl` skill for full path and function inventories.

# `log_dedup`: configuration

## Typical config

```yaml
processors:
  log_dedup:

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [memory_limiter, log_dedup]
      exporters: [otlphttp]
```

## Configuration reference

| Key | Type | Default | Notes |
|-----|------|---------|-------|
| `interval` | duration | `10s` | Aggregation window. Must be `> 0`. Counter resets after each emission. |
| `log_count_attribute` | string | `log_count` | Attribute name for the duplicate count. Must be non-empty. |
| `timezone` | string | `UTC` | IANA timezone for `first_observed_timestamp`/`last_observed_timestamp` (e.g., `America/New_York`). |
| `conditions` | []string | `[]` | OTTL log-context expressions. Empty = deduplicate all logs. Non-empty = only deduplicate logs matching at least one condition; non-matching logs pass through unchanged. Prefix paths with their context name (`log.attributes["x"]`, `resource.attributes["y"]`, `log.severity_number`). |
| `include_fields` | []string | `[]` | Whitelist of fields used to compute log identity. Mutually exclusive with `exclude_fields`. |
| `exclude_fields` | []string | `[]` | Fields removed before hashing **and** stripped from emitted logs. Mutually exclusive with `include_fields`. |
| `metadata_keys` | []string | `[]` | Client-metadata keys (e.g. gRPC/HTTP headers like `x-scope-orgid`) that partition aggregation into separate buckets. Logs with different values for these keys are counted independently, and each emitted log keeps a context carrying its original metadata so downstream extensions (e.g. `headers_setter`) can route it. Case-insensitive; duplicates rejected. Empty = one shared bucket. Added in v0.155.0. |
| `metadata_cardinality_limit` | uint32 | `0` | Caps the number of distinct `metadata_keys` combinations tracked at once. `0` = unbounded (a startup warning is logged when `metadata_keys` is set but this stays `0`). Once the cap is hit, new combinations are rejected with a permanent error. Added in v0.155.0. |

## Validation rules

- `include_fields` and `exclude_fields` cannot both be set.
- Field paths must start with `body.` or `attributes.`.
- The entire `body` cannot be included or excluded — only nested fields when `body` is a map.
- Use `.` as the nesting delimiter. Escape literal dots in keys with `\` (e.g., `attributes.host\.name`).
- Duplicate entries in `include_fields`, `exclude_fields`, or `metadata_keys` (case-insensitive for the latter) are rejected.
- `timezone` must resolve in the IANA database.

## Fallback behavior

If `include_fields` is set but none of the listed fields exist on a record, the processor falls back to comparing all fields (body, attributes, severity).

## What gets matched

Two log records are considered identical when **all** of these match exactly: resource attributes, instrumentation scope (name, version, attributes), log body (content and type), log attributes, and severity (number and text). The processor stores a 64-bit hash per unique combination.

`include_fields` / `exclude_fields` customize which parts of the record feed the hash. When `metadata_keys` is set, matching happens **within** each metadata bucket — two otherwise-identical logs arriving with different metadata values are never merged (see [advanced](advanced.md#multi-tenant-buckets-with-metadata_keys)).

## What gets emitted

At the end of each `interval`, one log per unique hash is emitted with:
- The original body, attributes, severity, resource.
- `log_count` (or the configured name) — duplicate count, int.
- `first_observed_timestamp` — RFC3339 string in the configured timezone.
- `last_observed_timestamp` — RFC3339 string in the configured timezone.
- `Timestamp` and `ObservedTimestamp` set to **emission time** (not original log time).

Counter resets after emission.

# `log_dedup`: known quirks

## Memory model

- One entry per unique hash for the duration of the interval.
- High-cardinality fields (`request_id`, `trace_id`, timestamps) blow up memory — exclude them or use `conditions` to scope what's deduplicated.
- Long intervals accumulate more unique entries; short intervals emit faster with smaller buffers.
- `metadata_keys` multiplies memory: one full bucket of hashes per unique metadata combination. Leaving `metadata_cardinality_limit` at `0` (unbounded) is a memory-growth trap — cap it, and expect a startup warning if you don't.

## Telemetry

The processor emits a histogram `otelcol_dedup_processor_aggregated_logs` with the count of logs aggregated per emission. Useful for tuning. The metric is currently **Development** stability — the name or shape can change between releases.

## Troubleshooting

**No logs are deduplicated.**
- `conditions` may exclude everything — comment them out to confirm.
- Every record may have a unique identifier (`request_id`, body timestamp). Add those to `exclude_fields`.
- Records may arrive slower than `interval` allows duplicates to form — increase `interval`.

**Memory grows without bound.**
- High-cardinality unique fields in the hash input — narrow with `exclude_fields` or `include_fields`.
- Interval too long for the volume — reduce it.
- Scope what's deduplicated with `conditions`.

**Logs arrive in bursts matching the interval.**
- Expected. The processor emits at interval boundaries. Reduce `interval` for lower latency, or move repetitive logs into a dedicated pipeline so other logs aren't held back.

**Different resources/severities aren't being grouped.**
- Resource attributes and severity are always part of the hash — there is no option to ignore them. Use a `resource` or `transform` processor upstream to normalize them first if that's the goal.

## Validation errors

| Error | Fix |
|-------|-----|
| `cannot define both exclude_fields and include_fields` | Choose one. |
| `an excludefield must start with body or attributes` | Prefix every path with `body.` or `attributes.`. |
| `cannot exclude the entire body` | Only nested fields are exclud-able (e.g., `body.timestamp`). |
| `duplicate metadata_key "…"` | Remove the repeated key (comparison is case-insensitive). |
| `too many batcher metadata-value combinations` (runtime, permanent error) | `metadata_cardinality_limit` reached — raise it, or narrow `metadata_keys`. |

## Anti-patterns

**Indiscriminate dedup.**

```yaml
# BAD — critical events get hidden behind log_count
processors:
  log_dedup:
    interval: 60s
```

Scope with `conditions` so audit, security, and one-off error logs pass through individually:

```yaml
processors:
  log_dedup:
    conditions:
      - 'log.attributes["log.type"] == "health_check"'
      - 'log.attributes["log.type"] == "heartbeat"'
```

**`include_fields` that includes unique IDs.**

```yaml
# BAD — request_id makes every record unique; nothing dedupes
processors:
  log_dedup:
    include_fields:
      - attributes.error_code
      - attributes.request_id
```

Include only identifying fields, never per-record identifiers.

**Very long intervals.**

```yaml
# BAD — 30 minutes of in-memory buffering, large bursty emissions
processors:
  log_dedup:
    interval: 30m
```

Prefer shorter intervals plus aggressive `exclude_fields` for cardinality control.

**Forgetting timezone for ops teams.**

```yaml
# Confusing if the team operates in PST
processors:
  log_dedup:
    timezone: UTC  # default
```

Set the timezone your operators read timestamps in.

# `log_dedup` processor

| | |
|-|-|
| Kind | processor |
| Type | `log_dedup` |
| Signals | logs |
| Stability | Alpha (processor); Development (telemetry metric `otelcol_dedup_processor_aggregated_logs`) |
| Distributions | contrib, k8s |
| Go module | `github.com/open-telemetry/opentelemetry-collector-contrib/processor/logdedupprocessor` |
| Upstream README | <https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/logdedupprocessor> |

**Rename in v0.151.0:** the type was renamed `logdedup` → `log_dedup` to match snake_case. The legacy name is a deprecated alias. New configs should use `log_dedup`; existing `logdedup` configs keep working.

## Description

Aggregates identical log records over a time window and emits a single log with a `log_count` attribute. Instead of forwarding every occurrence of a repeated message, the processor counts duplicates seen within an `interval` and emits one representative record per unique combination at the end of the window.

## Main use-cases

Use it when:
- The pipeline carries high-volume repetitive logs (health checks, polling, retry storms, connection errors).
- You want to reduce backend storage cost while preserving frequency information.
- You care about first/last observed timestamps of a recurring event, not every individual occurrence.

Avoid it when:
- Every entry must be preserved (audit, compliance, security logs).
- Sub-second precision per occurrence is required.
- The backend already deduplicates — you'll deduplicate twice.
- Each log carries a unique identifier or timestamp that makes every record unique (use `exclude_fields` to fix this, or skip the processor).

## Related components

- `groupbyattrs` — groups by attribute, does not deduplicate.
- `transform` — can rewrite/strip fields via OTTL, does not aggregate.
- `filter` — drops records, does not aggregate.

## Details

- [Configuration](configuration.md) — config keys, defaults, validation, and how records are matched/emitted.
- [Verification](verification.md) — telemetrygen recipe to confirm dedup works.
- [Advanced use-cases](advanced.md) — scoping with conditions, field whitelists, per-source instances.
- [Known quirks](quirks.md) — memory model, troubleshooting, anti-patterns.

# `log_dedup`: verification

See [Verification harness](../../SKILL.md#verification-harness) for how to run this end-to-end.

`log_dedup` ships in the `contrib` and `k8s` distributions, so a stock contrib collector can run this.

Config (`dedup-verify.yaml`):

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
processors:
  log_dedup:
    interval: 5s
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [log_dedup]
      exporters: [debug]
```

Generate many identical log records (see the `otel-telemetrygen` skill):

```bash
telemetrygen logs --otlp-insecure --otlp-endpoint localhost:4317 \
  --logs 50 --body "health check ok"
```

Don't add `--duration` here: telemetrygen ignores `--logs` when `--duration` is set, so the count would be whatever its default rate happens to emit instead of 50.

**What proves it worked:** the `debug` exporter prints **one** aggregated log per 5s interval carrying a `log_count` attribute of 50 (plus `first_observed_timestamp` / `last_observed_timestamp`), not 50 separate records.

# `memory_limiter`: advanced use-cases

## Pair with `GOMEMLIMIT`

Set the `GOMEMLIMIT` environment variable to roughly **80% of the hard limit**. It tunes Go's garbage collector to target that heap level, so the runtime reclaims memory *before* the limiter has to start refusing — the two work together rather than the limiter being the only brake:

```yaml
# Container env
env:
  - name: GOMEMLIMIT
    value: "3200MiB"   # ~80% of limit_mib

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 4000
    spike_limit_mib: 800
```

`GOMEMLIMIT` is a soft target for the GC pacer; `memory_limiter` is the hard backstop that refuses data. Use both. See <https://pkg.go.dev/runtime#hdr-Environment_Variables>.

## Percentage mode for Kubernetes

When the pod's memory limit varies across environments, use `limit_percentage` so the collector adapts without a config change. It reads the cgroup memory limit (v1/v2) or falls back to `/proc/meminfo`:

```yaml
# Pod spec: resources.limits.memory: 2Gi
processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 15
# Hard limit = 1.5 GiB (75% of 2 GiB), soft limit = 1.2 GiB (60%)
```

`limit_mib` takes precedence if both are set. Percentage mode is most reliable on Linux with cgroups; on macOS/Windows it reads total *system* memory, which is rarely what you want (see [Known quirks](quirks.md)).

## Size `spike_limit_mib` for the traffic shape

`spike_limit_mib` must cover the largest heap increase that can occur in a single `check_interval`. The buffer between soft and hard limit is the recovery headroom:

| Traffic | Config |
|---------|--------|
| Steady | `check_interval: 1s`, `spike_limit_mib: 800` (20% of 4000) |
| Bursty (2× spikes) | `check_interval: 500ms`, `spike_limit_mib: 1200` (30%) — check more often *and* widen the buffer |
| Low-latency, predictable | `check_interval: 1s`, `spike_limit_mib: 400` (10%) — tight control |

If the collector still OOMs, the spike outran the buffer between two checks: lower `check_interval` and/or raise `spike_limit_mib`.

## Multi-pipeline: one config, one checker

A single `memory_limiter` definition referenced from every pipeline shares one background checker (the [singleton](quirks.md), keyed on identical config), so there is no redundant GC:

```yaml
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 4000
    spike_limit_mib: 800

service:
  pipelines:
    traces:  { processors: [memory_limiter, batch] }
    metrics: { processors: [memory_limiter, batch] }
    logs:    { processors: [memory_limiter, batch] }
```

## Tuning the GC intervals

Four keys shape forced GC so it does not pin the CPU when memory hovers near the limit. The `min_gc_interval_*` keys are the **floor** between forced GCs; the `max_gc_interval_*` keys **cap the exponential backoff** the processor applies when a GC reclaims `< 5%` of heap (memory pinned by live references — see [Known quirks](quirks.md)). Keep the soft-limited interval `>=` the hard-limited one on both pairs (the hard limit is more urgent, so it should GC at least as eagerly):

```yaml
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 4000
    min_gc_interval_when_soft_limited: 10s   # default — opportunistic floor
    min_gc_interval_when_hard_limited: 0s    # default — GC every over-hard-limit check
    max_gc_interval_when_soft_limited: 30s   # default — backoff cap
    max_gc_interval_when_hard_limited: 30s   # default — backoff cap
```

Reach for these only if debug logs show `Forcing a GC` firing so often that CPU suffers (raise the `min`), or the backoff messages (`Forced GC did not reclaim enough memory…`) show GC still burning CPU during a downstream outage (lower the `max`, or set it to `0` only if you want to keep GC pinned at the `min`). The defaults are right for most deployments.

# `memory_limiter`: configuration

## Typical config

```yaml
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 4000
    spike_limit_mib: 800

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp]
```

Hard limit 4000 MiB, soft limit 3200 MiB, checked every second. `memory_limiter` is first in the processor list (see [Pipeline placement](#pipeline-placement)).

## Configuration reference

| Key | Type | Default | Meaning |
|-----|------|---------|---------|
| `check_interval` | duration | `0s` (**required**, must be `> 0`) | Time between memory measurements. `1s` is recommended; `500ms` for spiky traffic. |
| `limit_mib` | uint32 | `0` | Hard limit, in MiB of **heap**. Required unless `limit_percentage` is set; **takes precedence** over `limit_percentage` if both are set. |
| `spike_limit_mib` | uint32 | 20% of `limit_mib` | Expected max heap increase between checks. Must be `< limit_mib`. Soft limit = `limit_mib - spike_limit_mib`. |
| `limit_percentage` | uint32 | `0` | Hard limit as a percentage of total memory (`> 0`, `<= 100`). Used **only when `limit_mib` is unset/0**. On Linux reads the cgroup limit, falling back to `/proc/meminfo`; on non-Linux it reads total *system* memory (rarely a container limit). See [Known quirks](quirks.md). |
| `spike_limit_percentage` | uint32 | 20% of `limit_percentage` | Spike as a percentage of total memory. Must be `< limit_percentage` and `<= 100`. Only used with `limit_percentage`. |
| `min_gc_interval_when_soft_limited` | duration | `10s` | Minimum (floor) time between forced GCs while above the soft limit. Must be `>= min_gc_interval_when_hard_limited`. |
| `min_gc_interval_when_hard_limited` | duration | `0s` | Minimum (floor) time between forced GCs while above the hard limit (`0s` = GC on every over-limit check). Should be `<=` the soft-limited interval. |
| `max_gc_interval_when_soft_limited` | duration | `30s` | Caps the exponential backoff between forced GCs on the soft-limit path. When a forced GC reclaims `< 5%` of heap the interval doubles (from the min floor, or 95% of `check_interval`, whichever is larger) up to this cap, then resets once a GC is effective or heap drops. `0` disables backoff (GC stays at the min). Must be `>= min_gc_interval_when_soft_limited` unless `0`; and `>= max_gc_interval_when_hard_limited` when both caps are set. |
| `max_gc_interval_when_hard_limited` | duration | `30s` | Same backoff mechanism for the hard-limit path. Must be `>= min_gc_interval_when_hard_limited` unless `0`. |

One of `limit_mib` or `limit_percentage` is mandatory; `check_interval` is always mandatory.

## Soft limit vs hard limit

The processor derives two thresholds and acts on the current heap allocation:

| State | Threshold | Action |
|-------|-----------|--------|
| Below soft limit | heap `< limit - spike` | Normal operation; data accepted. |
| Above soft limit | heap `>= limit - spike` | Refuse data (`ErrDataRefused`); opportunistic GC throttled between `min_gc_interval_when_soft_limited` and `max_gc_interval_when_soft_limited`. Logs *"Memory usage is above soft limit. Refusing data."* |
| Above hard limit | heap `>= limit` | Refuse data; aggressive GC throttled between `min_gc_interval_when_hard_limited` and `max_gc_interval_when_hard_limited`. Logs *"Memory usage is above hard limit. Forcing a GC."* |

If a forced GC reclaims `< 5%` of heap (e.g. the memory is held by live references in exporter queues during a downstream outage), the processor treats it as ineffective and **exponentially backs off** the GC interval on that path — doubling up to the `max_gc_interval_*` cap — to avoid burning CPU on futile collections during recovery. The interval resets as soon as a GC frees memory or heap drops on its own.

Refusal is a **non-permanent** error, so receivers retry with backoff and clients can rate-limit. Recovery is automatic: once heap falls below the soft limit it logs *"Memory usage back within limits. Resuming normal operation."* The processor also reports its health via `componentstatus` — a **recoverable-error** status while refusing, and `StatusOK` once back within limits — visible through status-aware extensions (e.g. `healthcheckv2`).

## Validation rules

Startup fails (config error) when:

| Rule | Error message |
|------|---------------|
| `check_interval` not `> 0` | `'check_interval' must be greater than zero` |
| Neither limit set | `'limit_mib' or 'limit_percentage' must be greater than zero` |
| `spike_limit_mib >= limit_mib` | `'spike_limit_mib' must be smaller than 'limit_mib'` |
| `spike_limit_percentage >= limit_percentage` (percentage mode) | `'spike_limit_percentage' must be smaller than 'limit_percentage'` |
| `limit_percentage > 100` or `spike_limit_percentage > 100` | `'limit_percentage' and 'spike_limit_percentage' must be greater than zero and less than or equal to hundred` |
| `min_gc_interval_when_soft_limited < min_gc_interval_when_hard_limited` | `'min_gc_interval_when_soft_limited' should be larger than 'min_gc_interval_when_hard_limited'` |
| `max_gc_interval_when_soft_limited` set but `< min_gc_interval_when_soft_limited` | `'max_gc_interval_when_soft_limited' must be greater than or equal to 'min_gc_interval_when_soft_limited' (or 0 to disable)` |
| `max_gc_interval_when_hard_limited` set but `< min_gc_interval_when_hard_limited` | `'max_gc_interval_when_hard_limited' must be greater than or equal to 'min_gc_interval_when_hard_limited' (or 0 to disable)` |
| both max caps set and `max_gc_interval_when_soft_limited < max_gc_interval_when_hard_limited` | `'max_gc_interval_when_soft_limited' should be larger than or equal to 'max_gc_interval_when_hard_limited' (when both are set)` |

## Pipeline placement

`memory_limiter` must be the **first** processor in every pipeline so it refuses data before it enters allocating processors (`batch`, `transform`, `tail_sampling`) and so backpressure reaches receivers early:

```yaml
# Good — refuse before anything buffers
processors: [memory_limiter, tail_sampling, batch]
```

```yaml
# Bad — memory limiting and sampling happen after batching
processors: [batch, memory_limiter, tail_sampling]
```

# `memory_limiter`: known quirks

## It measures heap, not RSS

The limit is compared against Go **heap allocation** (`runtime.MemStats.Alloc`), not process RSS. Total process memory typically runs **50–100 MiB higher** (Go runtime, stacks, CGO, OS page caches, mmap'd files). So `limit_mib: 4000` means a process that sits around ~4050 MiB. Set the container memory limit **10–15% above** `limit_mib`, or the kernel OOM-kills the process before the limiter ever refuses. Seeing RSS exceed `limit_mib` is expected, not a bug.

## `check_interval` and one of the limits are required

There are no usable defaults for the two load-bearing keys. `check_interval` defaults to `0s`, which is a validation error, and you must set `limit_mib` **or** `limit_percentage` (`> 0`). A config with only one of these fails at startup:

```
'check_interval' must be greater than zero
'limit_mib' or 'limit_percentage' must be greater than zero
```

## `limit_mib` wins over `limit_percentage`

If both are set, `limit_mib` is used and `limit_percentage` is ignored — silently. Set only the one you mean.

## `spike_limit` must be smaller than the limit

`spike_limit_mib < limit_mib` (and `spike_limit_percentage < limit_percentage`) is validated. If the spike limit equals or exceeds the limit, the soft limit would be `<= 0` and startup fails with `'spike_limit_mib' must be smaller than 'limit_mib'`.

## GC-interval ordering is enforced

`min_gc_interval_when_soft_limited` must be `>=` `min_gc_interval_when_hard_limited` (the hard limit is more urgent, so it cannot GC *less* often). Inverting them fails startup with `'min_gc_interval_when_soft_limited' should be larger than 'min_gc_interval_when_hard_limited'`. The same ordering holds for the `max_gc_interval_*` caps when both are set, and each `max` (unless `0`) must be `>=` its own `min` — see the [validation rules](configuration.md#validation-rules).

## GC backs off when it stops helping

Since v0.156.0, forced GC no longer fires unconditionally on every over-limit check. When a GC reclaims `< 5%` of heap — typically because the memory is pinned by live references such as exporter sending queues during a downstream outage — the processor treats it as ineffective and **doubles** the interval before the next forced GC, up to `max_gc_interval_when_soft_limited` / `max_gc_interval_when_hard_limited` (both default `30s`). This prevents a CPU-burning GC loop that would starve recovery. The interval resets the moment a GC frees memory or heap drops. Set the relevant `max_gc_interval_*` to `0` to opt out and keep GC pinned at the `min` interval.

## It reports component health

Since v0.156.0 the processor publishes health events via `componentstatus.ReportStatus`: a **recoverable-error** event while it is refusing data and `StatusOK` once heap is back within limits. Status-aware extensions (e.g. `healthcheckv2`) surface this, so a collector under memory pressure can report unhealthy without you scraping logs or refusal metrics.

## `limit_percentage` needs cgroups to be useful

Percentage mode reads the container's memory limit from cgroups v2 (`/sys/fs/cgroup/memory.max`) or v1 (`/sys/fs/cgroup/memory/memory.limit_in_bytes`), and treats an "unlimited" cgroup value (`9223372036854771712`) as absent, falling back to `/proc/meminfo`. On macOS/Windows it reads **total system memory**, not a container limit, so the percentage is rarely meaningful there. Prefer `limit_mib` on non-Linux hosts. If total-memory detection fails entirely the collector refuses to start — switch to `limit_mib`.

## It is a safety net, not a sizing tool

The limiter does not make a small collector handle more load — below capacity it simply **refuses data constantly** (refusal metrics, persistent *"Refusing data"* logs, receivers seeing backpressure). Memory-limiter metric names have changed across Collector releases — v0.156.0 renamed them to a component-specific prefix (`otelcol_processor_memory_limiter_accepted_*` / `otelcol_processor_memory_limiter_refused_*`) — so check the exact Collector version before writing alerts. The fix is to size the collector for the workload or scale out, then keep `memory_limiter` as the backstop. Treat sustained refusal as an under-provisioning signal, not a tuning problem.

## Refusal is non-permanent — receivers must retry

When refusing, the processor returns a **non-permanent** error (gRPC `Unavailable`, *"data refused due to high memory usage"*). Well-behaved receivers (otlp, prometheus, …) retry with backoff and clients rate-limit. A receiver or client that does **not** retry will lose that data. This is by design: the limiter applies backpressure rather than buffering.

## Singleton checker across identical configs

Pipelines referencing the same `memory_limiter` config share one background goroutine and timer (keyed on the config), so GC isn't triggered redundantly per pipeline. Two *differently named* instances with different limits run independent checkers — intentional, but watch that their limits don't sum past the container budget.

## The extension variant

Since v0.142.0 the memory limiter is also available as an **extension** that acts as gRPC/HTTP middleware on a receiver (v0.147.0 added streaming support and fixed a multi-interceptor startup panic):

```yaml
extensions:
  memory_limiter:
    check_interval: 1s
    limit_mib: 4000
receivers:
  otlp:
    protocols:
      grpc:
        middlewares: [{id: memory_limiter}]
service:
  extensions: [memory_limiter]
```

The extension reached **Beta** stability in v0.158.0. It is still **not compiled into the stock core or contrib distributions** (`metadata.yaml` lists no distributions), so it requires a custom build. The processor form remains the bundled pipeline component; choose between them based on whether refusal should happen as receiver middleware or in a pipeline. Since core v0.162.0 the extension emits `otelcol_memorylimiter_refused_requests` (attribute `transport`) for refused network requests.

## Stability caveats

Traces/metrics/logs are Beta; profiles are Alpha. The profiles path can still change between releases — confirm against the upstream README for the exact collector version before relying on it.

# `memory_limiter` processor

| | |
|-|-|
| Kind | processor |
| Type | `memory_limiter` |
| Signals | traces (Beta), metrics (Beta), logs (Beta), profiles (Alpha) |
| Distributions | core, contrib, k8s |
| Go module | `go.opentelemetry.io/collector/processor/memorylimiterprocessor` |
| Upstream README | <https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/memorylimiterprocessor> |

## Description

A safety valve that keeps the Collector from being OOM-killed. A background goroutine reads Go heap allocation (`runtime.MemStats.Alloc`) every `check_interval` and compares it against two thresholds derived from the configured limit: a **soft limit** (`limit_mib - spike_limit_mib`) and a **hard limit** (`limit_mib`). Above the soft limit the processor sets a `mustRefuse` flag and every data path returns a **non-permanent** `ErrDataRefused` to the preceding component — backpressure that well-behaved receivers retry with backoff. Above the hard limit it additionally forces a `runtime.GC()` (with exponential backoff when a GC proves ineffective, capped by the `max_gc_interval_*` keys). It also reports its health via `componentstatus` — recoverable-error while refusing, OK once recovered. When heap drops back below the soft limit it resumes normally; no manual intervention.

It measures **heap**, not process RSS — total process memory runs 50–100 MiB higher, so container limits should sit above `limit_mib`. The processor does not mutate data (`MutatesData: false`); it only accepts or refuses. It is a safety net, not a sizing or performance tool — a chronically undersized collector will simply refuse data constantly. Use it together with the `GOMEMLIMIT` environment variable so Go's GC paces toward the same target.

## Main use-cases

Use it when:
- **Always** — on every production Collector, as the **first** processor in every pipeline.
- You run in a memory-capped environment (Kubernetes pod, Docker container) and must avoid OOM kills under traffic spikes.
- You want early backpressure to receivers so clients rate-limit at the source before buffers fill downstream.

Avoid it when:
- You are reaching for it to fix throughput on an undersized collector — size the collector first, then add this as a safety net (see [Known quirks](quirks.md)).
- A minimal verification/repro pipeline where you are isolating another component on purpose (the skill's verification configs omit it deliberately).

## Related components

- `batch` / exporter `sending_queue.batch` — batching allocates buffers; `memory_limiter` must sit **before** it so refusal happens before data is buffered.
- `tail_sampling`, `transform`, `log_dedup` — stateful/allocating processors; place them after `memory_limiter` so memory pressure refuses data before it reaches them.
- The **`memory_limiter` extension** — same checker exposed as gRPC/HTTP middleware (Beta since v0.158.0, but not included in stock distributions); see [Known quirks](quirks.md).

## Details

- [Configuration](configuration.md) — full config table (`check_interval`, `limit_mib` vs `limit_percentage`, spike limits, min/max GC intervals), validation rules, and the soft/hard-limit mechanism with GC backoff.
- [Verification](verification.md) — telemetrygen recipe that forces refusal with a deliberately tiny limit and shows `ErrDataRefused` in the logs.
- [Advanced use-cases](advanced.md) — `GOMEMLIMIT` pairing, percentage mode for Kubernetes, spike sizing for bursty traffic, multi-pipeline singleton behavior.
- [Known quirks](quirks.md) — heap-vs-RSS, required fields, `limit_percentage` cgroup detection, GC-interval ordering, the singleton checker, the extension variant, stability caveats.

# `memory_limiter`: verification

See [Verification harness](../../SKILL.md#verification-harness) for how to run this end-to-end.

`memory_limiter` ships in the `core`, `contrib`, and `k8s` distributions, so a stock contrib collector can run this.

Because the processor reacts to **real heap pressure**, the trick to verifying it deterministically is to set the limit *below the collector's own idle heap* so it refuses from the first check — no need to actually exhaust memory.

Config (`memlimit-verify.yaml`) — limit far below the idle heap so the soft limit is breached immediately:

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 20
    spike_limit_mib: 5
exporters:
  debug:
    verbosity: detailed
service:
  telemetry:
    logs:
      level: debug
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [debug]
```

Wait ~5s for the first memory check, then send a few traces (see the `otel-telemetrygen` skill):

```bash
telemetrygen traces --otlp-insecure --otlp-endpoint localhost:4317 --traces 10 --workers 1
```

Flags confirmed against the `otel-telemetrygen` skill (`references/flags.md`): `--otlp-insecure`, `--otlp-endpoint`, `--traces` (int, traces **per worker**; ignored if `--duration` is set, so no `--duration` here), and `--workers` (default 1). The exact count does not matter here — every export is refused regardless.

**What proves it worked** (verified run, contrib 0.152.0, idle heap ~17–20 MiB > the 15 MiB soft limit):

1. The collector logs the limit being breached:
   ```
   warn  Memory usage is above hard limit. Forcing a GC.   {"cur_mem_mib": 20}
   info  Memory usage after GC.                            {"cur_mem_mib": 17}
   warn  Memory usage is above soft limit. Refusing data.  {"cur_mem_mib": 17}
   ```
2. telemetrygen's export fails with the non-permanent gRPC error (it retries with backoff, which is exactly the intended backpressure):
   ```
   traces export: ... rpc error: code = Unavailable desc = data refused due to high memory usage
   ```
3. **Zero** spans reach the `debug` exporter (`grep -c '^Span #'` in the collector logs returns `0`) — the data was refused before the pipeline, not dropped silently downstream.

To watch normal operation instead, raise `limit_mib` well above the idle heap (e.g. `limit_mib: 4000`) and the same send passes through with spans printed by `debug`.

# `otlp` receiver: advanced use-cases

## Enabling only one protocol

Once `protocols:` is written, only the sub-blocks present are active. To accept gRPC only (and refuse to even open the HTTP port):

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
```

HTTP-only is the mirror image — list only `http:`. Listing neither fails startup (`must specify at least one protocol when using the OTLP receiver`).

## `include_metadata` for downstream metadata consumers

Per-request gRPC/HTTP headers (and other client metadata) are **not propagated into the downstream pipeline** by default. Downstream features that read them need the receiver to carry them into the pipeline. Server authenticator extensions are different: they read the transport headers directly and do not require `include_metadata`.

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        include_metadata: true     # headers survive into the pipeline
```

`include_metadata: true` is the prerequisite for:
- [`routing`](../routing/README.md) by request metadata.
- `attributes` / `resource` actions that use `from_context: metadata.*`.

Without it, those downstream consumers see no metadata and behave as if the headers were absent. The [`load_balancing`](../load_balancing/README.md) exporter's `routing_key: attributes` reads telemetry resource/scope/record attributes, not transport metadata; copy a header into a telemetry attribute first if it must become a load-balancing key.

## CORS for browser OTLP

Browser RUM agents send OTLP/HTTP cross-origin and require CORS preflight handling:

```yaml
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
        cors:
          allowed_origins:
            - https://app.example.com
          allowed_headers:
            - "*"
          max_age: 7200
```

## mTLS (require client certificates)

Set `client_ca_file` on the protocol's `tls` block to require and verify client certificates:

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        tls:
          cert_file: /certs/server.crt
          key_file: /certs/server.key
          client_ca_file: /certs/client-ca.crt   # mutual TLS
```

## Large-payload tuning

Reject-on-size limits differ per protocol. Raise them when senders batch aggressively:

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        max_recv_msg_size_mib: 16        # gRPC message cap (default 4 MiB)
      http:
        endpoint: 0.0.0.0:4318
        max_request_body_size: 67108864  # 64 MiB (default 20 MiB)
```

## Named instances

The `type/name` pattern lets one Collector run several OTLP listeners — e.g. an internal-only plaintext listener and an external mTLS one:

```yaml
receivers:
  otlp/internal:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
  otlp/external:
    protocols:
      grpc:
        endpoint: 0.0.0.0:5317
        tls:
          cert_file: /certs/server.crt
          key_file: /certs/server.key
          client_ca_file: /certs/client-ca.crt

service:
  pipelines:
    traces/internal:
      receivers: [otlp/internal]
    traces/external:
      receivers: [otlp/external]
```

Give each named instance a distinct `endpoint` — two listeners cannot share a port.

# `otlp` receiver: configuration

All keys live under the receiver instance (`receivers: { otlp: { … } }`). The only top-level key is `protocols:`, which holds two optional sub-blocks. Facts below trace to the core **v1.66.0 / v0.160.0** source (`receiver/otlpreceiver/config.go`, `config/configgrpc/configgrpc.go` `ServerConfig`, `config/confighttp/server.go` `ServerConfig`, and `config/configtls`).

## Top-level

| Key | Type | Default | Meaning |
|-----|------|---------|---------|
| `protocols.grpc` | object | enabled by factory default | OTLP over gRPC (see [grpc](#protocolsgrpc)). |
| `protocols.http` | object | enabled by factory default | OTLP over HTTP — protobuf and JSON (see [http](#protocolshttp)). |

**Factory default** enables **both** protocols: gRPC on `localhost:4317`, HTTP on `localhost:4318`. Once you write a `protocols:` block, only the sub-blocks you include are active; the common idiom

```yaml
receivers:
  otlp:
    protocols:
      grpc: {}     # accept per-protocol defaults
      http: {}
```

enables both at their defaults. Listing only `grpc:` disables HTTP, and vice versa.

**Validation:** if neither `grpc` nor `http` is set, startup fails with `must specify at least one protocol when using the OTLP receiver`.

> The default endpoints bind to `localhost`. In a container or pod you must set `endpoint: 0.0.0.0:4317` / `0.0.0.0:4318` — see [quirks.md](quirks.md).

## `protocols.grpc`

`configgrpc.ServerConfig`. Common keys:

| Key | Type | Default | Meaning |
|-----|------|---------|---------|
| `endpoint` | string | `localhost:4317` | Listen address. Set `0.0.0.0:4317` to accept non-loopback traffic. |
| `transport` | string | `tcp` | Socket type. gRPC servers accept only `tcp` or `unix`. |
| `tls` | object | — (plaintext) | Server TLS (see [TLS](#tls-server)). |
| `max_recv_msg_size_mib` | int | 4 (gRPC default) | Max received message size, in MiB. Raise for large batches. |
| `max_concurrent_streams` | uint32 | 0 (unlimited) | Max concurrent HTTP/2 streams per connection. |
| `read_buffer_size` | int | `524288` (512 KiB) | gRPC read buffer. |
| `write_buffer_size` | int | 0 (gRPC default) | gRPC write buffer. |
| `keepalive` | object | — | Server keepalive (see [keepalive](#grpc-keepalive)). |
| `auth` | object | — | `authenticator:` referencing an auth extension. |
| `include_metadata` | bool | `false` | Propagate per-request gRPC metadata (headers) into the downstream pipeline. Required for metadata routing and `from_context: metadata.*`; server authenticator extensions read transport headers directly. |
| `middlewares` | list | — | gRPC server middleware extensions. |

### grpc keepalive

| Key | Type | Meaning |
|-----|------|---------|
| `server_parameters.max_connection_idle` | duration | Close a connection after this much idle time. |
| `server_parameters.max_connection_age` | duration | Max total connection lifetime. |
| `server_parameters.max_connection_age_grace` | duration | Grace period after `max_connection_age` before forced close. |
| `server_parameters.time` | duration | Ping interval on idle connections. |
| `server_parameters.timeout` | duration | Ping ack timeout. |
| `enforcement_policy.min_time` | duration | Minimum client ping interval the server tolerates. |
| `enforcement_policy.permit_without_stream` | bool | Allow client pings with no active stream. |

## `protocols.http`

`confighttp.ServerConfig` (squashed) plus three OTLP URL-path overrides from `receiver/otlpreceiver/config.go`. Common keys:

| Key | Type | Default | Meaning |
|-----|------|---------|---------|
| `endpoint` | string | `localhost:4318` | Listen address. Set `0.0.0.0:4318` to accept non-loopback traffic. |
| `traces_url_path` | string | `/v1/traces` | Path for trace export. |
| `metrics_url_path` | string | `/v1/metrics` | Path for metric export. |
| `logs_url_path` | string | `/v1/logs` | Path for log export. |
| `tls` | object | — (plaintext) | Server TLS (see [TLS](#tls-server)). |
| `cors` | object | — | Browser CORS (see [cors](#cors)). |
| `auth` | object | — | `authenticator:` referencing an auth extension. |
| `max_request_body_size` | int | `20971520` (20 MiB) | Max accepted request body. Raise for large batches. |
| `include_metadata` | bool | `false` | Propagate per-request HTTP headers into the pipeline (see grpc note above). |
| `response_headers` | map | — | Static headers added to every response. |
| `compression_algorithms` | list | none, gzip, zstd, zlib, snappy, deflate, lz4, x-snappy-framed | Accepted request `Content-Encoding`s. |
| `read_timeout` | duration | 0 (none) | Max time to read the full request. |
| `read_header_timeout` | duration | 0 (none) | Max time to read request headers. |
| `write_timeout` | duration | 0 (none) | Max time to write the response. |
| `keepalive.idle_timeout` | duration | `1m` | Max idle time on a keep-alive connection. |
| `keepalive.enabled` | bool | `true` | Whether to allow HTTP keep-alives. |
| `middlewares` | list | — | HTTP server middleware extensions. |

Core v0.160.0 deprecates the flat HTTP `idle_timeout` and `keep_alives_enabled` keys. Non-no-op
legacy values cannot be configured with the new `keepalive` block: `idle_timeout` conflicts when
nonzero, and `keep_alives_enabled` conflicts when `false`. The no-op legacy values
`idle_timeout: 0` and `keep_alives_enabled: true` may coexist with `keepalive`.

There is **no profiles URL-path field** — the HTTP config exposes only traces/metrics/logs override keys. Profiles (Alpha) are still served over HTTP, but on a fixed, non-configurable path: `/v1development/profiles`.

The HTTP endpoint accepts both **OTLP/protobuf** and **OTLP/JSON** on the same paths; the encoding is selected by `Content-Type`.

### cors

| Key | Type | Meaning |
|-----|------|---------|
| `allowed_origins` | list of string | Origins permitted to make cross-origin OTLP requests (e.g. browser RUM). |
| `allowed_headers` | list of string | Additional request headers allowed cross-origin. |
| `exposed_headers` | list of string | Response headers browsers may expose to the caller (`Access-Control-Expose-Headers`). |
| `max_age` | int | Seconds browsers may cache the CORS preflight. |

## TLS (server)

Both `grpc.tls` and `http.tls` use `configtls.ServerConfig`. Common keys:

| Key | Type | Meaning |
|-----|------|---------|
| `cert_file` | string | Server certificate (enables TLS). |
| `key_file` | string | Server private key. |
| `ca_file` | string | CA bundle used to verify client certificates. |
| `client_ca_file` | string | CA bundle for mutual TLS — require and verify client certs. |

See `config/configtls` for the full set (min/max version, cipher suites, reload interval).

## Validation summary

| Condition | Error | When |
|-----------|-------|------|
| neither `grpc` nor `http` configured | `must specify at least one protocol when using the OTLP receiver` | config validation |
| `max_recv_msg_size_mib` out of range | `invalid max_recv_msg_size_mib value, must be between 1 and …` | startup |

# `otlp` receiver: known quirks

## The default endpoint is `localhost`, not `0.0.0.0`

This is the single most common "the receiver gets no data in Docker/k8s" problem. Each protocol defaults to `localhost:4317` / `localhost:4318` — a deliberate security-hardening default so a Collector started with no explicit endpoint doesn't expose an open OTLP port on every interface. The consequence: inside a container or pod, traffic arrives on a **non-loopback** interface, and a `localhost`-bound listener silently accepts nothing from outside — no error, just no data.

To receive external traffic, set the endpoint explicitly:

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
```

If a containerized receiver "gets no data," check this first.

## Must specify at least one protocol

A `protocols:` block with neither `grpc` nor `http` fails at config validation with `must specify at least one protocol when using the OTLP receiver`. Note the inverse trap: once you write `protocols:`, only the sub-blocks you include are active — listing only `grpc:` **disables** the HTTP listener that the bare factory default would have enabled. Use `grpc: {}` / `http: {}` (empty maps) to keep a protocol on at its defaults.

## gRPC is 4317, HTTP is 4318

The two protocols listen on different ports: gRPC `4317`, HTTP `4318`. Sending OTLP/HTTP to `:4317` (or gRPC to `:4318`) fails. With `telemetrygen`, gRPC is the default and `--otlp-http` switches to the HTTP exporter — point each at the matching port.

## OTLP/JSON works on the HTTP port

The HTTP endpoint accepts both OTLP/protobuf and **OTLP/JSON** on the same paths (`/v1/traces`, `/v1/metrics`, `/v1/logs`); the encoding is chosen by `Content-Type`. There is no separate JSON port or path. gRPC is protobuf only.

## `include_metadata` is off by default

Per-request headers/metadata are not propagated into downstream consumers unless `include_metadata: true`. Metadata [`routing`](../routing/README.md) and `attributes` / `resource` actions using `from_context: metadata.*` silently see no metadata without it. Server authenticator extensions still read transport headers directly, and [`load_balancing`](../load_balancing/README.md) with `routing_key: attributes` hashes telemetry attributes rather than transport metadata.

## No configurable profiles URL path

The HTTP config exposes `traces_url_path`, `metrics_url_path`, and `logs_url_path` overrides only. There is no `profiles_url_path` key — profiles (Alpha) are still served over HTTP, but on a fixed path the receiver hardcodes: `/v1development/profiles`. (Note: the upstream README mentions a `profiles_url_path` and a `/v1/profiles` default; neither exists in the config struct or the handler — the path is fixed.)

## Stability is per signal

Traces, metrics, and logs are **Stable**; **profiles** are **Alpha**. Treat profile ingestion as still-evolving and subject to breaking change, even though the same receiver serves it.

# `otlp` receiver

| | |
|-|-|
| Kind | receiver |
| Type | `otlp` |
| Signals | traces (Stable), metrics (Stable), logs (Stable), profiles (Alpha) |
| Distributions | core, contrib, k8s, otlp |
| Go module | `go.opentelemetry.io/collector/receiver/otlpreceiver` |
| Upstream README | <https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver> |

## Description

Receives telemetry in the **OpenTelemetry Protocol (OTLP)** over gRPC and/or HTTP — the native wire format of the OpenTelemetry SDKs, the Collector's own `otlp` exporter, and most other OTLP-speaking agents. It is the default, canonical ingress for a Collector pipeline and supports traces, metrics, and logs at **Stable** stability (profiles are **Alpha**). The single config block is `protocols:`, holding optional `grpc:` and `http:` sub-blocks; you must enable at least one. The gRPC side listens on `4317`, the HTTP side on `4318` (accepting both OTLP/protobuf and OTLP/JSON on the same endpoint).

The most important operational fact is the **default endpoint**: each protocol binds to `localhost` (`localhost:4317` / `localhost:4318`), not `0.0.0.0`. That is a deliberate security default — inside a container or Kubernetes pod, traffic arrives on a non-loopback interface, so a `localhost`-bound receiver silently accepts nothing from outside. To receive external traffic you must set `endpoint: 0.0.0.0:4317` (and `:4318`). This is the single most common "the receiver gets no data in Docker/k8s" gotcha — see [quirks.md](quirks.md).

## Main use-cases

Use when:
- You need to ingest OTLP from OpenTelemetry SDKs, the OTel Collector `otlp` exporter, or any OTLP-native agent — the default and recommended Collector ingress.
- You want one receiver to handle traces, metrics, and logs over gRPC, HTTP/protobuf, and HTTP/JSON.
- You front a tier that needs per-request metadata **inside the downstream pipeline** (metadata routing, or `from_context: metadata.*` actions) — set `include_metadata: true`. Server authenticator extensions read transport headers directly and do not require it.

Avoid when:
- The source speaks a non-OTLP protocol (Prometheus scrape, Jaeger, Zipkin, Kafka, syslog, filelog, …) — use the matching receiver instead.
- You only need OTLP over HTTP and want to be explicit — still this receiver, just configure only the `http:` sub-block.

## Related components

- [`otlp` exporter](../otlp_exporter/README.md) — the sending counterpart; a Collector-to-Collector hop is this receiver on one end and the `otlp_grpc`/`otlp` exporter on the other.
- [`memory_limiter`](../memory_limiter/README.md) — belongs **first** in every pipeline fed by this receiver, to apply backpressure before buffers grow.
- [`routing`](../routing/README.md) — to route by request metadata, this receiver must run with `include_metadata: true` so the headers survive into the pipeline.
- [`load_balancing`](../load_balancing/README.md) — hashes telemetry resource/scope/record attributes (not transport metadata) when `routing_key: attributes` is selected.

## Details

- [Configuration](configuration.md) — the `protocols.grpc` and `protocols.http` sub-blocks with every key, default, the URL-path overrides, and the "at least one protocol" validation rule.
- [Verification](verification.md) — telemetrygen recipe sending OTLP/gRPC (and OTLP/HTTP) into a single collector with a `debug` exporter, binding `0.0.0.0` so containerized traffic is accepted.
- [Advanced use-cases](advanced.md) — enabling a single protocol, `include_metadata` for downstream metadata consumers, CORS for browser OTLP, mTLS, large-payload tuning, and named instances.
- [Known quirks](quirks.md) — the `localhost` vs `0.0.0.0` default (top item), the "at least one protocol" error, the 4317/4318 ports, OTLP/JSON on the HTTP port, `include_metadata` default, and per-signal stability.

# `otlp` receiver: verification

See [Verification harness](../../SKILL.md#verification-harness) for how to run this end-to-end.

The `otlp` receiver ships in every distribution (`core`, `contrib`, `k8s`, `otlp`). This recipe runs **one** collector — the `otlp` receiver feeding a `debug` exporter — and sends spans into it with `telemetrygen`, first over gRPC (`:4317`) then over HTTP (`:4318`). Verified on `otel/opentelemetry-collector-contrib:0.154.0`.

> **Why `0.0.0.0`, not `localhost`?** The default endpoint binds to `localhost`, which inside a container only accepts loopback traffic from *within* the container. `telemetrygen` runs on the host and reaches the container through a published port, i.e. a non-loopback interface — so the receiver must bind `0.0.0.0` or it silently accepts nothing. This is the receiver's most common gotcha; the recipe sets it explicitly.

Config (`otlp.yaml`) — both protocols bound to `0.0.0.0`, `debug` at detailed verbosity so each span is printed:

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
exporters:
  debug:
    verbosity: detailed           # basic verbosity prints no per-span lines
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [debug]
```

Start the collector, publishing both OTLP ports to the host:

```bash
IMG=otel/opentelemetry-collector-contrib:0.154.0
docker run -d --name otlp-recv -p 4317:4317 -p 4318:4318 \
  -v "$PWD/otlp.yaml:/etc/otelcol-contrib/config.yaml" $IMG
```

Send 5 spans over **gRPC** (port 4317) — see the `otel-telemetrygen` skill:

```bash
telemetrygen traces --otlp-endpoint localhost:4317 --otlp-insecure \
  --traces 5 --workers 1
```

Then send 5 more over **HTTP** (port 4318) with `--otlp-http`:

```bash
telemetrygen traces --otlp-http --otlp-endpoint localhost:4318 --otlp-insecure \
  --traces 5 --workers 1
```

Flags confirmed against the `otel-telemetrygen` skill (`references/flags.md`): `--otlp-endpoint` (string), `--otlp-insecure` (bool, plaintext — matches the `tls`-less receiver), `--otlp-http` (bool, switches to the HTTP exporter and the `:4318` path), and `--traces` (int, **per worker**; ignored when `--duration` is set, so no `--duration` here and `--workers 1` makes the count exact). No flag invents a value.

**What proves it worked:** count the spans the `debug` exporter logged:

```bash
docker logs otlp-recv 2>&1 | grep -c 'Span #'
```

`--traces N` counts **traces, not spans** — `telemetrygen` emits **2 spans per trace** (a parent and a child), so each `--traces 5 --workers 1` run produces **10 span records**, and the two runs (gRPC + HTTP) together yield **20**. Verified on `otel/opentelemetry-collector-contrib:0.154.0`:

```
20
```

If the count is 0, the most likely cause is a `localhost`-bound endpoint (data never reached the receiver) — confirm `endpoint: 0.0.0.0:…` in the config.

Tear down:

```bash
docker rm -f otlp-recv
```

# `otlp_grpc` exporter: advanced use-cases

## Tuning the queue for throughput vs. latency

The queue defaults are `num_consumers: 10` and `queue_size: 1000`. If you opt into queue batching,
its defaults are `batch.min_size: 8192` and `batch.flush_timeout: 200ms`. The levers:

- **Throughput** — raise `num_consumers` (more parallel senders) and `queue_size` (deeper buffer to absorb bursts). Larger `batch.min_size` packs more per request, reducing per-RPC overhead.
- **Latency** — when batching is enabled, lower `batch.flush_timeout` (flush sooner) and `batch.min_size` (smaller batches leave the queue faster). The trade-off is more, smaller RPCs.

```yaml
exporters:
  otlp_grpc:
    endpoint: gateway:4317
    tls:
      insecure: true
    sending_queue:
      num_consumers: 20
      queue_size: 10000
      batch:
        flush_timeout: 5s        # tolerate latency for fewer, fuller batches
        min_size: 16384
```

`batch.min_size` must be ≤ `queue_size` when their sizers match — see [quirks.md](quirks.md).

## Persistent queue via `storage`

By default the queue is in-memory: data buffered at a crash or restart is lost. Point `sending_queue.storage` at a `file_storage` extension to persist the queue to disk so it survives restarts:

```yaml
extensions:
  file_storage:
    directory: /var/lib/otelcol/queue

exporters:
  otlp_grpc:
    endpoint: backend:4317
    sending_queue:
      storage: file_storage      # queue persists across restarts

service:
  extensions: [file_storage]
```

> The [`file_storage`](../file_storage/README.md) extension backs persistent queues. Persistence trades durability for disk I/O. **`wait_for_result` is not supported with a persistent `storage` queue** (see below).

## `wait_for_result` trade-off

`sending_queue.wait_for_result: true` makes the caller block until the export result is known, so the pipeline learns about failures synchronously instead of fire-and-forget. It costs throughput (the producer waits) and is **incompatible with a persistent `storage` queue** — enabling both is a config error. Use it only when you need synchronous delivery confirmation and an in-memory queue.

## mTLS to a secure backend

Drop `insecure` and supply client credentials to authenticate to a TLS backend:

```yaml
exporters:
  otlp_grpc:
    endpoint: backend.example.com:4317
    tls:
      ca_file: /certs/backend-ca.crt
      cert_file: /certs/client.crt        # mTLS
      key_file: /certs/client.key
    headers:
      x-tenant-id: team-a                 # static per-request header
```

## Multiple named exporters

The `type/name` pattern fans a pipeline out to several backends, each with its own queue/retry/TLS:

```yaml
exporters:
  otlp_grpc/primary:
    endpoint: primary:4317
    tls: { insecure: true }
  otlp_grpc/backup:
    endpoint: backup:4317
    tls: { insecure: true }

service:
  pipelines:
    traces:
      exporters: [otlp_grpc/primary, otlp_grpc/backup]
```

## Relationship to `load_balancing`

[`load_balancing`](../load_balancing/README.md) wraps a per-backend `otlp` exporter under `protocol.otlp` — one sub-exporter per resolved backend. Everything on this page (TLS, `headers`, `compression`, `sending_queue`, `retry_on_failure`, `timeout`) applies **per backend** there. The one difference: under `load_balancing` you do **not** set `protocol.otlp.endpoint` — the load balancer populates it from the resolver. Reach for `load_balancing` when you need trace-ID / service affinity across a pool (to scale `tail_sampling`/`span_metrics`); reach for a plain `otlp_grpc` exporter when you just send to one backend (or fan out to a fixed few via named instances).

# `otlp_grpc` exporter: configuration

All keys live under the exporter instance — `exporters: { otlp_grpc: { … } }` or, via the deprecated alias, `exporters: { otlp: { … } }`. Facts below trace to core **v1.66.0 / v0.160.0** source (`exporter/otlpexporter/config.go` + `factory.go`, `config/configgrpc/configgrpc.go` `ClientConfig`, `config/configtls`, `config/configretry/backoff.go`, and `exporter/exporterhelper/internal/queuebatch/config.go` + `queue_sender.go`); `sending_queue.batch.partition` rows re-checked at v1.68.0 / v0.162.0.

## Top-level (gRPC client) keys

| Key | Type | Default | Meaning |
|-----|------|---------|---------|
| `endpoint` | string | — (**required**) | gRPC target (`host:port`, gRPC naming syntax). Accepts `http://`, `https://`, `dns://` prefixes (stripped internally for validation). |
| `compression` | string | `gzip` | `gzip`, `snappy`, `zstd`, or none/empty to disable. Default set by the factory. |
| `tls` | object | — | gRPC client TLS (see [tls](#tls)). |
| `headers` | map[string]string | — | Static headers added to every gRPC request. |
| `timeout` | duration | `5s` | Per-attempt send timeout. |
| `keepalive` | object | — | gRPC client keepalive: `time`, `timeout`, `permit_without_stream`. |
| `read_buffer_size` | int | 0 (gRPC default) | gRPC read buffer. The factory leaves this unset (the exporter reads almost nothing). |
| `write_buffer_size` | int | `524288` (512 KiB) | gRPC write buffer. |
| `wait_for_ready` | bool | `false` | Block RPCs until the connection is ready instead of failing fast. Core v0.157.0 fixed this option being parsed but not applied on gRPC client calls. |
| `balancer_name` | string | `round_robin` | gRPC client-side load-balancing policy across resolved addresses. |
| `authority` | string | — | Overrides the `:authority` pseudo-header. |
| `user_agent` | string | — (build-info default) | Overrides the default gRPC user-agent header. Empty keeps the build-derived default. |
| `auth` | object | — | `authenticator:` referencing an auth extension (e.g. bearer/OAuth2). |
| `middlewares` | list | — | gRPC client middleware extensions. |

**Validation:** an empty `endpoint` fails with `requires a non-empty "endpoint"`.

## `tls`

`configtls.ClientConfig` (squashed). Common keys:

| Key | Type | Default | Meaning |
|-----|------|---------|---------|
| `insecure` | bool | `false` | Use **plaintext** gRPC (no TLS). Set `true` for an internal/test backend with no TLS. |
| `insecure_skip_verify` | bool | `false` | Use TLS but skip server-cert verification. |
| `ca_file` | string | — | CA bundle to verify the server certificate. |
| `cert_file` | string | — | Client certificate (for mTLS). |
| `key_file` | string | — | Client private key (for mTLS). |
| `server_name_override` | string | — | Override the SNI / verified server name. |

`insecure` (plaintext) and `insecure_skip_verify` (TLS without verification) are different: the first disables TLS entirely, the second keeps TLS but trusts any certificate.

## `retry_on_failure`

`configretry.BackOffConfig` — exponential backoff with jitter. After `max_elapsed_time` the data is **dropped**.

| Key | Type | Default | Meaning |
|-----|------|---------|---------|
| `enabled` | bool | `true` | Whether to retry failed sends. |
| `initial_interval` | duration | `5s` | Wait before the first retry. |
| `randomization_factor` | float | `0.5` | Jitter applied to each interval. |
| `multiplier` | float | `1.5` | Growth factor per retry. |
| `max_interval` | duration | `30s` | Cap on the per-retry interval. |
| `max_elapsed_time` | duration | `5m` | Total retry budget; after this, the data is dropped. Set `0` to retry forever. |

## `sending_queue`

The buffer between the pipeline and the gRPC sender. **Enabled by default.** Its optional `batch`
sub-block uses defaults from `NewDefaultQueueConfig`, but batching itself is disabled by default
unless the `pkg.exporterhelper.queueBatchEnabled` feature gate is enabled.

| Key | Type | Default | Meaning |
|-----|------|---------|---------|
| `enabled` | bool | `true` | Whether to enqueue before sending. |
| `num_consumers` | int | `10` | Concurrent senders draining the queue. |
| `queue_size` | int | `1000` | Max items buffered, in `sizer` units. Must be > 0. |
| `sizer` | string | `requests` | Unit for `queue_size`: `requests`, `items`, or `bytes`. |
| `block_on_overflow` | bool | `false` | `false` → overflow returns a retryable error immediately; `true` → enqueue blocks until space frees. |
| `wait_for_result` | bool | `false` | Block the caller until the export result is known. **Not supported with a persistent `storage` queue.** |
| `storage` | component ID | — | A `file_storage` extension ID; turns the queue **persistent** (survives restarts). |
| `batch` | object | disabled | Built-in batching (see [batch](#sending_queuebatch)). Add `batch: {}` to enable it with defaults, or enable the migration gate globally. |

> `storage` references a [`file_storage`](../file_storage/README.md) extension. Persistence survives Collector restarts at the cost of disk I/O; see [advanced.md](advanced.md).

### `sending_queue.batch`

When enabled, flushes at `flush_timeout` or when `min_size` is reached, whichever comes first.
Add `batch: {}` beneath `sending_queue` to opt in with these defaults. The alpha
`pkg.exporterhelper.queueBatchEnabled` feature gate also enables batching by default for exporters
using `NewDefaultQueueConfig`; this is phase 1 of the Collector batching migration. A separate
`batch` processor remains supported for pipeline-level batching.

| Key | Type | Default | Meaning |
|-----|------|---------|---------|
| `flush_timeout` | duration | `200ms` | Max time a partial batch waits before flushing. Must be > 0. |
| `sizer` | string | `items` | Unit for the batch sizes: only `items` or `bytes`. |
| `min_size` | int | `8192` | Flush once the batch reaches this many items (the soft trigger). |
| `max_size` | int | `0` (unlimited) | Hard cap; when > 0, a batch is split to never exceed it. |
| `partition.metadata_keys` | list of string | — | One batcher per distinct combination of these client-metadata values. |
| `partition.cache_size` | int | `10000` | Max active partition batchers (LRU); must be > 0. Exposed as `otelcol_exporter_queue_batch_partition_cache_size` / `_capacity`. Core v0.162.0+. |
| `partition.idle_timeout` | duration | `90s` | How long an empty partition lives before removal; must be > 0. Configurable (and default raised to 90s) in core v0.162.0. |

## Validation summary

| Condition | Error / rule | When |
|-----------|--------------|------|
| empty `endpoint` | `requires a non-empty "endpoint"` | config validation |
| `sending_queue.batch.flush_timeout` ≤ 0 | rejected | config validation |
| `batch.sizer` not `items`/`bytes` | rejected | config validation |
| `batch.min_size` > `queue_size` (matching sizers) | rejected | config validation |
| `batch.max_size` < `min_size` (when `max_size` > 0) | rejected | config validation |
| `queue_size` ≤ 0 | `` `queue_size` must be positive `` | config validation |
| `num_consumers` ≤ 0 | `` `num_consumers` must be positive `` | config validation |

# `otlp_grpc` exporter: known quirks

## The type is now `otlp_grpc` — and there's a separate `otlphttp`

The canonical type was renamed from `otlp` to **`otlp_grpc`** in core **v1.50.0** (≈ contrib v0.144.0). The old name **`otlp` still works** as a deprecated alias (it emits a deprecation warning) and is what most existing configs use, so both `exporters: { otlp: … }` and `exporters: { otlp_grpc: … }` configure this same gRPC exporter. The rename disambiguates it from the **`otlphttp`** exporter, which is a *different* component speaking OTLP over **HTTP**. If your backend has no gRPC endpoint, you want `otlphttp`, not this one — they are not interchangeable.

## Exporter batching vs. the `batch` processor

This exporter's `sending_queue` is enabled by default, but queue batching is **disabled by default**.
Add `sending_queue.batch: {}` to activate the 200ms / 8192-item defaults, or enable the
alpha `pkg.exporterhelper.queueBatchEnabled` migration gate. The separate `batch` processor is still
Beta and supported; use it when batching must happen at pipeline level (for example, once before
fan-out to several exporters), and place it after processors that may drop data. If both layers are
enabled, tune them deliberately because double batching can add latency.

## gzip compression is on by default

The factory sets `compression: gzip`. To send uncompressed gRPC (e.g. a CPU-constrained agent, or a backend that rejects gzip), set `compression: none` (or empty) explicitly. The other accepted values are `snappy` and `zstd`.

## `insecure: true` is needed for plaintext

A backend with no TLS (a sibling Collector on a trusted network, a local test) requires `tls: { insecure: true }`. Without it the exporter attempts a TLS handshake and the connection fails. Note the distinction from `insecure_skip_verify: true`, which keeps TLS but skips certificate verification — different setting, different behavior.

## `wait_for_result` is incompatible with a persistent `storage` queue

`sending_queue.wait_for_result: true` and `sending_queue.storage` (a `file_storage`-backed persistent queue) cannot both be set. Pick synchronous delivery confirmation **or** restart-durable persistence, not both.

## `wait_for_ready` only takes effect on v0.157.0 and newer

Before core v0.157.0, `wait_for_ready: true` was accepted in the gRPC client config but was not added to the RPC call options, so sends still failed fast while the connection was not ready. v0.157.0 applies `grpc.WaitForReady` correctly. Do not rely on this setting for outage behavior on older Collectors.

## `min_size` must be ≤ `queue_size`

When batching is enabled and the sizers match, `sending_queue.batch.min_size` (default 8192) must not exceed `sending_queue.queue_size` (default 1000 in `requests` units — but if you switch `queue_size` to `items`, the 8192 default `min_size` can exceed it and fail validation). If you set `queue_size` in items, raise it above `min_size`, or lower `min_size`. Also: `batch.flush_timeout` must be > 0, and `batch.max_size` (when > 0) must be ≥ `min_size`.

## Data is dropped after `max_elapsed_time`

`retry_on_failure` retries with exponential backoff, but only until `max_elapsed_time` (default **5m**). After that budget is exhausted the data is **dropped** — retries are not infinite by default. Set `max_elapsed_time: 0` to retry forever (at the risk of unbounded queue growth), or pair with a persistent `storage` queue to survive longer outages.

## Stability is per signal

Traces, metrics, and logs are **Stable**; **profiles** are **Alpha**. Treat profile export as experimental and subject to breaking change, even though the same exporter serves it.

# `otlp_grpc` exporter

| | |
|-|-|
| Kind | exporter |
| Type | `otlp_grpc` (deprecated alias `otlp`) |
| Signals | traces (Stable), metrics (Stable), logs (Stable), profiles (Alpha) |
| Distributions | core, contrib, k8s, otlp |
| Go module | `go.opentelemetry.io/collector/exporter/otlpexporter` |
| Upstream README | <https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlpexporter> |
| Rename | type renamed from `otlp` to `otlp_grpc` in core **v1.50.0** (≈ contrib v0.144.0); the old name `otlp` is preserved as a deprecated alias and still works, emitting a deprecation warning. |

## Description

Sends telemetry in **OTLP over gRPC** to a downstream OTLP endpoint — another Collector's `otlp` receiver, or any OTLP/gRPC backend. It is the standard egress for a pipeline and supports traces, metrics, and logs at **Stable** stability (profiles are **Alpha**). The canonical type is now **`otlp_grpc`**, renamed from `otlp` in core v1.50.0 to disambiguate it from the separate `otlphttp` exporter (OTLP over HTTP). The old name **`otlp` still works** as a deprecated alias — and is what the vast majority of existing configs use — so both `exporters: { otlp: … }` and `exporters: { otlp_grpc: … }` configure this component. Only `endpoint` is required; gRPC requests are **gzip-compressed by default**.

The exporter's `sending_queue` and `retry_on_failure` are enabled by default. Queue **batching is
opt-in**: add a `sending_queue.batch` block, or enable the alpha
`pkg.exporterhelper.queueBatchEnabled` migration gate to make the factory's 200ms / 8192-item
batch defaults active. The separate `batch` processor remains Beta and supported for pipeline-level
batching (for example, once before fan-out to several exporters). See
[configuration.md](configuration.md) and [quirks.md](quirks.md).

## Main use-cases

Use when:
- You send telemetry from one Collector to another (a gateway hop, an agent→gateway tier) over OTLP/gRPC.
- You export to an OTLP/gRPC-native backend or vendor endpoint.
- You want built-in queuing and retry, with optional exporter-side batching.

Avoid when:
- The backend only speaks OTLP over **HTTP** — use the `otlphttp` exporter instead.
- You need trace-ID / service affinity across a pool of backends (to scale `tail_sampling`/`span_metrics`) — use [`load_balancing`](../load_balancing/README.md), which wraps this exporter per backend.
- The destination isn't OTLP at all (Prometheus remote-write, Kafka, a vendor's proprietary protocol) — use that exporter.

## Related components

- [`otlp` receiver](../otlp/README.md) — the receiving counterpart; a Collector-to-Collector hop is this exporter on one end and the `otlp` receiver on the other.
- `otlphttp` exporter — the sibling that speaks OTLP over **HTTP** instead of gRPC; pick it when the backend has no gRPC endpoint.
- [`load_balancing`](../load_balancing/README.md) — wraps a per-backend `otlp` exporter under `protocol.otlp`; its queue/retry/TLS semantics are this exporter's.
- [`memory_limiter`](../memory_limiter/README.md) — the back-pressure guard at the **front** of the pipeline this exporter terminates.
- The **`batch` processor** — still supported for pipeline-level batching; distinct from the exporter's opt-in `sending_queue.batch`.

## Details

- [Configuration](configuration.md) — `endpoint`, `compression`, `tls`, `headers`, `timeout`, the `retry_on_failure` block, and the full `sending_queue` (+ `batch`) reference with every default and validation rule.
- [Verification](verification.md) — a two-collector chain (A exports to B) proving spans sent to A arrive in B's `debug` log.
- [Advanced use-cases](advanced.md) — tuning the queue for throughput vs. latency, the persistent queue via `storage`, `wait_for_result`, mTLS to a secure backend, named instances, and the relationship to `load_balancing`.
- [Known quirks](quirks.md) — the `otlp`→`otlp_grpc` rename and `otlphttp` distinction (top item), exporter batching vs. the `batch` processor, default gzip, `insecure: true` for plaintext, the pre-v0.157.0 `wait_for_ready` bug, `wait_for_result` vs. persistent `storage`, `min_size` ≤ `queue_size`, drop-after-`max_elapsed_time`, and per-signal stability.

# `otlp_grpc` exporter: verification

See [Verification harness](../../SKILL.md#verification-harness) for how to run this end-to-end.

The exporter ships in every distribution (`core`, `contrib`, `k8s`, `otlp`). The cleanest proof is a **two-collector chain**: collector **A** receives OTLP and exports it to collector **B**; collector **B** receives it and prints it with `debug`. If spans sent to A appear in B's log, the export hop worked. Verified on `otel/opentelemetry-collector-contrib:0.159.0` (2026-08-27).

> The config uses the deprecated alias `otlp` for the exporter (still the most common form). The canonical `otlp_grpc` name is identical in behavior. This recipe is a **minimal repro** — it omits `memory_limiter` and other production scaffolding on purpose, to isolate the export hop.

Collector **A** config (`a.yaml`) — receives, then exports to B over plaintext gRPC:

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
exporters:
  otlp:                            # the exporter under test (alias of otlp_grpc)
    endpoint: otlp-b:4317
    tls:
      insecure: true               # plaintext gRPC to B
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp]
```

Collector **B** config (`b.yaml`) — receives and prints:

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
exporters:
  debug:
    verbosity: detailed            # basic verbosity prints no per-span lines
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [debug]
```

Start both on one Docker network (distinct names; only A publishes a host port):

```bash
docker network create otlpnet
IMG=otel/opentelemetry-collector-contrib:0.159.0
docker run -d --name otlp-b --network otlpnet \
  -v "$PWD/b.yaml:/etc/otelcol-contrib/config.yaml" $IMG
docker run -d --name otlp-a --network otlpnet -p 4317:4317 \
  -v "$PWD/a.yaml:/etc/otelcol-contrib/config.yaml" $IMG
```

Send 5 spans into **A** — see the `otel-telemetrygen` skill:

```bash
go run github.com/open-telemetry/opentelemetry-collector-contrib/cmd/telemetrygen@v0.159.0 \
  traces --otlp-endpoint localhost:4317 --otlp-insecure --traces 5 --workers 1
```

Flags confirmed against the `otel-telemetrygen` skill (`references/flags.md`): `--otlp-endpoint` (string), `--otlp-insecure` (bool, plaintext to A's TLS-less receiver), `--traces` (int, **per worker**; ignored when `--duration` is set, so no `--duration` and `--workers 1` makes the count exact). No flag invents a value.

**What proves it worked:** the spans must appear in **B's** debug log, having traversed A's `otlp` exporter:

```bash
docker logs otlp-b 2>&1 | grep -c 'Span #'
```

`--traces N` counts **traces, not spans** — `telemetrygen` emits **2 spans per trace** (a parent and a child), so `--traces 5 --workers 1` produces **10 span records**, all of which should reach B:

```
10
```

Collector A also logs, at startup, the exporter's deprecation warning when the `otlp` alias is used:

```
warn  builders/builders.go:40  "otlp" alias is deprecated; use "otlp_grpc" instead  {"otelcol.component.id": "otlp", "otelcol.component.kind": "exporter", ...}
```

A count of 0 in B (but spans visible in A's own log, if you add a `debug` exporter to A) points at the export hop — usually a wrong `endpoint`, or missing `tls.insecure: true` against a plaintext B.

Tear down:

```bash
docker rm -f otlp-a otlp-b
docker network rm otlpnet
```

# `probabilistic_sampler`: advanced use-cases

## Choosing a mode

| Goal | Mode |
|------|------|
| A predictable, reproducible ratio across multiple tiers/collectors | `proportional` — uses W3C 56-bit randomness and ignores prior decisions, so a 1-in-N stays 1-in-N everywhere. |
| Enforce a sampling **floor** without re-cutting already-sampled data | `equalizing` — factors in prior decisions and only lowers probability to the target, so a downstream sampler does not double-sample. |
| Sample **logs by a record attribute**, or stay on the legacy seed-based hash | `hash_seed` — the only mode that supports `attribute_source: record`. |

```yaml
processors:
  probabilistic_sampler:
    mode: proportional
    sampling_percentage: 10
```

## Consistent `hash_seed` across a fleet (hash_seed mode)

In `hash_seed` mode the keep/drop decision depends on `hash_seed`. Every collector in the same sampling tier must use the **same** seed, or two collectors will make different decisions for the same TraceID and the effective rate becomes unpredictable. Pin the seed explicitly rather than relying on the `0` default when more than one collector samples the same stream:

```yaml
processors:
  probabilistic_sampler:
    mode: hash_seed
    hash_seed: 22
    sampling_percentage: 25
```

Use a **different** seed at each tier if you intentionally want independent decisions (e.g. cascading samplers); use the same seed when you want them to agree.

## Sampling logs by attribute

Sample logs on a record attribute instead of the TraceID — useful when logs carry a high-cardinality key (such as a request or session ID) that should drive a consistent decision:

```yaml
processors:
  probabilistic_sampler:
    mode: hash_seed
    sampling_percentage: 20
    attribute_source: record
    from_attribute: request.id
```

`attribute_source: record` requires `from_attribute`; with `attribute_source: traceID` (the default) the log's TraceID is hashed.

## `sampling_priority` overrides (logs)

Set `sampling_priority` to the name of an attribute that overrides the per-record decision. When a record carries that attribute, its value is read as a sampling percentage (`0`–`100`) that replaces `sampling_percentage` for that record — `0` never samples, `>= 100` always samples — handy for always keeping flagged records (set the attribute to `100`):

```yaml
processors:
  probabilistic_sampler:
    sampling_percentage: 5
    sampling_priority: log.priority
```

For traces, the equivalent override is the fixed `sampling.priority` attribute (not configurable), and it is binary rather than a percentage: `0` drops the span, any non-zero value keeps it.

## Combining head + tail sampling

Probabilistic head sampling at the **edge** and `tail_sampling` at a **gateway** compose well: cut bulk volume cheaply and statelessly near the source, then keep the survivors that matter (errors, slow traces) at the gateway. Prefer `proportional`/`equalizing` at the edge so the W3C threshold is carried in `tracestate` and the gateway stays consistent rather than re-cutting blindly.

```yaml
# Edge collector — cheap stateless head sampling
processors:
  probabilistic_sampler:
    mode: proportional
    sampling_percentage: 30
```

```yaml
# Gateway collector — content-aware tail sampling on the survivors
processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
```

## The `fail_closed` trade-off

`fail_closed: true` (default) **drops** items that have sampling-related errors — for example, missing or zero randomness (no usable TraceID / W3C randomness). This is the safe choice: it never lets un-decidable data skew the rate. Set `fail_closed: false` only when you would rather keep such items than lose them, accepting that the effective sampling rate becomes less precise.

# `probabilistic_sampler`: configuration

## Typical config

```yaml
processors:
  probabilistic_sampler:
    sampling_percentage: 25

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [probabilistic_sampler]
      exporters: [otlphttp]
```

## Configuration reference

| Key | Type | Default | Meaning |
|-----|------|---------|---------|
| `sampling_percentage` | float32 | (required) | Percentage sampled. `>= 100` keeps all; `0` keeps none. |
| `mode` | string | `hash_seed` | Sampling algorithm: `proportional`, `equalizing`, or `hash_seed`. See [Modes](#modes). |
| `hash_seed` | uint32 | `0` | Seed for the FNV hash. Only used in `hash_seed` mode. All collectors in a tier must share the same value (see [Known quirks](quirks.md)). |
| `fail_closed` | bool | `true` | Reject items with sampling-related errors (e.g. missing or zero randomness). When `false`, such items pass through. |
| `sampling_precision` | int | `4` | Number of hex digits used to encode the sampling threshold (range `1`–`14`). Higher precision is more exact but capped per mode (see [Known quirks](quirks.md)). |
| `attribute_source` | string | `traceID` | **Logs only.** Randomness source: `traceID` or `record`. |
| `from_attribute` | string | `""` | **Logs only.** Log attribute used for sampling when `attribute_source: record`. |
| `sampling_priority` | string | `""` | **Logs only.** Name of a log-record attribute read as a per-record sampling percentage (0–100) that overrides `sampling_percentage`: `0` never samples, `>= 100` always samples. |

## Modes

| Mode | Randomness | Behavior |
|------|------------|----------|
| `hash_seed` (default) | FNV hash of TraceID (or attribute), 14-bit | Hashes the source value with `hash_seed` and compares against a 14-bit threshold. The only mode that supports sampling logs by a record attribute. |
| `proportional` | W3C 56-bit randomness | Ignores any prior sampling decision and samples a predictable 1-in-N fraction. Best for a consistent ratio across tiers. |
| `equalizing` | W3C 56-bit randomness | Factors in prior sampling decisions and lowers probability to a floor across a pipeline, so a downstream sampler does not re-cut already-sampled data below the target. |

The 56-bit modes use the W3C Trace Context randomness (`tracestate` `ot=rv:`/`th:`), which makes their decisions composable across multiple samplers; `hash_seed` uses its own seed-dependent FNV hash and does not compose. All three modes still record the effective threshold on the outgoing item.

## Traces vs logs

| Aspect | Traces | Logs |
|--------|--------|------|
| Randomness source | Always the TraceID | TraceID (`attribute_source: traceID`) or a record attribute (`attribute_source: record` + `from_attribute`) |
| Per-item priority override | Fixed `sampling.priority` attribute (binary: `0` drops, non-zero keeps) | Configurable via `sampling_priority` (percentage: `0`–`100`) |
| Threshold encoding (downstream) | W3C `tracestate` (`ot=th:...`) | Log attributes (`sampling.threshold` / `sampling.randomness`) |
| Stability | Beta | Alpha |

`attribute_source`, `from_attribute`, and `sampling_priority` are logs-only and have no effect on traces.

# `probabilistic_sampler`: known quirks

## `fail_closed` drops zero/invalid-randomness items

With `fail_closed: true` (the default), any item whose randomness source is missing or zero — for example a span with no usable TraceID, or a log without the configured `from_attribute` — is **dropped**, because the sampler cannot make a valid decision. If you see fewer items than the configured percentage would predict, check for items lacking randomness before assuming the rate is wrong. Setting `fail_closed: false` lets these items through at the cost of a less precise effective rate.

`hash_seed` mode treats an item that **already carries** W3C randomness or threshold (`ot=rv:`/`th:` in `tracestate`) as an error too, since it does not compose with prior consistent sampling. With `fail_closed: true` those items are also dropped — use `proportional`/`equalizing` at any tier that sits downstream of an already-sampling collector or SDK.

## `hash_seed` must match across collectors

In `hash_seed` mode the decision is `FNV(source, hash_seed) vs threshold`. Two collectors with **different** seeds make different keep/drop choices for the same TraceID, so chaining them (or load-balancing across them) yields an unpredictable combined rate. Every collector in one sampling tier must share the same `hash_seed`. The 56-bit `proportional`/`equalizing` modes avoid this by using W3C randomness instead of a seed.

## Precision is capped per mode

`sampling_precision` (1–14 hex digits) controls how finely the threshold is encoded, but each mode has a practical ceiling: roughly 4 for `hash_seed` (it is only 14-bit), about 5 for `equalizing`, and higher for `proportional` though still bounded by float32 rounding of `sampling_percentage`. Asking for more precision than a mode supports does not improve accuracy.

## Threshold persists downstream via tracestate

For traces, the sampling threshold is written into the W3C `tracestate` (`ot=th:...`); for logs it is written into `sampling.threshold` / `sampling.randomness` log attributes. This persistence is intentional — it lets a downstream sampler stay consistent — but it means the threshold travels with the data and a later component can read or act on it. Stripping `tracestate` between tiers breaks `proportional`/`equalizing` consistency.

## Child spans/logs are sampled with probability ≥ parent

To preserve trace completeness, a child item is kept with probability at least that of its parent — the sampler will not drop a child of a kept parent purely by chance. This reduces (but does not fully eliminate) orphaned-child situations; combined with consistent per-TraceID hashing, whole traces tend to be kept or dropped together.

## Counts are statistical, not exact

The kept fraction only **approaches** `sampling_percentage`% as the number of items grows. For small N the observed ratio is noisy and varies run to run; never assert an exact count from a single run.

## Stability caveats

Traces are Beta and logs are Alpha — log-side config keys (`attribute_source`, `from_attribute`, `sampling_priority`) and behavior can still change between releases. Confirm against the upstream README for the exact collector version before relying on log sampling in production.

# `probabilistic_sampler` processor

| | |
|-|-|
| Kind | processor |
| Type | `probabilistic_sampler` |
| Signals | traces (Beta), logs (Alpha) |
| Distributions | core, contrib, k8s |
| Go module | `github.com/open-telemetry/opentelemetry-collector-contrib/processor/probabilisticsamplerprocessor` |
| Upstream README | <https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/probabilisticsamplerprocessor> |

## Description

Head sampling: makes a deterministic keep/drop decision **per item** by hashing the TraceID (or, for logs, a configurable attribute) and comparing it against a threshold derived from `sampling_percentage`. The decision is stateless and content-blind — no buffering, no waiting — so the same TraceID is always kept or always dropped, and all collectors that share the same `hash_seed` agree on the outcome. Because the hash is uniform over trace IDs, sampling roughly `sampling_percentage`% of items.

This is the opposite of **tail sampling** (`tail_sampling`), which buffers the **whole trace** and makes one decision after a wait window using trace-wide signals (errors, latency, span count). Head sampling is cheap and immediate but cannot react to span content; tail sampling is content-aware but stateful, requires buffering, adds latency, and needs all spans of a trace to land on the same instance. In **all three modes** the sampler records the effective threshold on the outgoing item — the trace's `tracestate` (`ot=th:...`) for traces, and log attributes (`sampling.threshold` / `sampling.randomness`) for logs. The W3C-randomness modes (`proportional` / `equalizing`) additionally *read* the incoming threshold and compose with it, so chained samplers stay consistent; the default `hash_seed` mode decides locally with its own FNV hash and does not compose (it ignores — and, with `fail_closed`, drops — items that already carry W3C randomness/threshold).

## Main use-cases

Use it when:
- You want cheap, stateless volume reduction at the edge with no buffering or co-location requirement.
- You want a predictable, reproducible keep/drop decision per TraceID across a fleet of collectors (consistent `hash_seed`).
- You want to sample logs by TraceID or by a record attribute.
- You want to combine head sampling at the edge with `tail_sampling` at a gateway (see [Advanced use-cases](advanced.md)).

Avoid it when:
- You need a content-aware decision over a **whole trace** — keep all errors, keep slow traces — use `tail_sampling`. See its README for the head-vs-tail trade-off.
- You need exact counts; sampling is statistical, so the kept fraction only approaches `sampling_percentage`% over many items.
- Dropping would orphan child spans or break correlation — note the child-≥-parent completeness rule in [Known quirks](quirks.md) mitigates but does not eliminate this.

## Related components

- `tail_sampling` — head's counterpart: trace-wide, content-aware keep/drop after buffering. Stateful and latency-adding, but can act on errors/latency.
- `filter` — drops telemetry by OTTL condition rather than by a probability; use it for deterministic noise removal, not statistical sampling.
- `transform` — mutates telemetry via OTTL instead of dropping it; run it before/after to set the attributes a record-based sampler reads.

## Details

- [Configuration](configuration.md) — full config table, the three sampling modes explained, and the traces-vs-logs differences.
- [Verification](verification.md) — telemetrygen recipe that sends many traces and shows roughly half reach `debug`.
- [Advanced use-cases](advanced.md) — mode selection, consistent `hash_seed` across a fleet, logs sampling by attribute, `sampling_priority` overrides, head+tail combinations, the `fail_closed` trade-off.
- [Known quirks](quirks.md) — `fail_closed` dropping zero/invalid randomness, hash-seed agreement, precision caps, tracestate persistence, child-≥-parent completeness, statistical counts, stability caveats.

# `probabilistic_sampler`: verification

See [Verification harness](../../SKILL.md#verification-harness) for how to run this end-to-end.

`probabilistic_sampler` ships in the `core`, `contrib`, and `k8s` distributions, so a stock contrib collector can run this.

Config (`probabilistic-verify.yaml`) — sample 50% of traces:

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
processors:
  probabilistic_sampler:
    sampling_percentage: 50
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [probabilistic_sampler]
      exporters: [debug]
```

Send a good number of traces — each gets a fresh random TraceID, so roughly half clear the 50% threshold (see the `otel-telemetrygen` skill):

```bash
telemetrygen traces --otlp-insecure --otlp-endpoint localhost:4317 --traces 500 --workers 1
```

Flags confirmed against the `otel-telemetrygen` skill (`references/flags.md`): `--otlp-insecure`, `--otlp-endpoint`, `--traces` (int, traces **per worker**; ignored if `--duration` is set, so no `--duration` here or the count would not be 500), and `--workers` (default 1). With `--workers 1` the total is exactly 500 — the denominator for the ratio below; raising `--workers` multiplies the total. telemetrygen assigns a random TraceID per trace, which is what makes the hash spread evenly.

Use a large `N` (≥ 500) so the ratio is clearly readable — with only a handful of traces the kept fraction is noisy.

**What proves it worked:** noticeably fewer traces reach the `debug` exporter than were sent — close to half. Because the decision is statistical, the exact fraction varies run to run, but at `N = 500` it lands near 50% (a verified run kept 257/500 ≈ 51%). The signal is "clearly fewer than sent, in the neighborhood of 50%", not an exact number.

# `prometheus`: advanced use-cases

## Named instances

Run more than one scrape configuration as independent receiver instances with the `prometheus/<name>` syntax:

```yaml
receivers:
  prometheus/apps:
    config:
      scrape_configs:
        - job_name: 'apps'
          static_configs:
            - targets: ['app-a:9090', 'app-b:9090']
  prometheus/infra:
    config:
      scrape_configs:
        - job_name: 'node'
          static_configs:
            - targets: ['node-exporter:9100']

service:
  pipelines:
    metrics:
      receivers: [prometheus/apps, prometheus/infra]
      exporters: [otlp]
```

## Target Allocator (OpenTelemetry Operator)

Instead of static jobs, fetch a dynamically-assigned, sharded target list from the OpenTelemetry
Operator's Target Allocator (or a compatible endpoint). The block embeds the full `confighttp`
client configuration (so `tls`, `proxy_url`, etc. are available) plus the allocator-specific keys:

```yaml
receivers:
  prometheus:
    target_allocator:
      endpoint: http://my-targetallocator-service
      interval: 30s
      collector_id: collector-1     # literal id — must NOT contain ${...}
```

`endpoint` must be a valid request URI, `interval` must be a positive duration, and `collector_id`
is required and must not contain `${` (see [quirks.md](quirks.md)). Optional `http_sd_config` and
`http_scrape_config` tune the discovery and scrape HTTP clients respectively.

The Target Allocator is also how you avoid duplicate scrapes when running multiple collector
replicas — it shards targets across the collectors rather than every replica scraping everything.

## Native histograms

The receiver auto-converts scraped Prometheus native histograms to OTel exponential histograms.
This is now unconditional — the old `receiver.prometheusreceiver.EnableNativeHistograms` gate
graduated to stable and has been removed, so there is no gate to toggle. You still need **two**
things in the Prometheus scrape config:

1. `scrape_native_histograms: true` (globally under `global`, or per-job)
2. `PrometheusProto` included in `scrape_protocols` (required until Prometheus supports native histograms over text formats)

```yaml
receivers:
  prometheus:
    config:
      global:
        scrape_native_histograms: true
      scrape_configs:
        - job_name: 'native-hist'
          scrape_protocols: [PrometheusProto, OpenMetricsText1.0.0, OpenMetricsText0.0.1, PrometheusText0.0.4]
          static_configs:
            - targets: ['app:9090']
```

If a metric has both classic buckets and native histogram buckets, only the native histogram buckets are used.

## Debug API server (`api_server`)

Host a local Prometheus agent-mode API server to inspect targets, config, and service discovery for
debugging. The `server_config` is the standard Collector `confighttp` server config:

```yaml
receivers:
  prometheus:
    api_server:
      enabled: true
      server_config:
        endpoint: "localhost:9090"   # defaults to 127.0.0.1:9090 if omitted
```

`server_config` is the standard confighttp server config (defaults: endpoint `127.0.0.1:9090`,
`read_timeout: 10m`), and `max_connections` (default `512`) caps simultaneous HTTP connections.

Endpoints mirror the Prometheus agent-mode API:

- `/api/v1/targets`
- `/api/v1/targets/metadata`
- `/api/v1/status/config`
- `/api/v1/scrape_pools`
- `/metrics`

## Relabeling and metric filtering

`relabel_configs` and `metric_relabel_configs` behave exactly as in Prometheus — do most of your
target selection and metric dropping at the source rather than downstream. Note the `$$` escaping
rule for literal `$` in regex/replacement values (see [quirks.md](quirks.md)).

```yaml
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'k8s-pods'
          kubernetes_sd_configs:
            - role: pod
          relabel_configs:
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              regex: "true"
              action: keep
          metric_relabel_configs:
            - source_labels: [__name__]
              regex: "(request_duration_seconds.*|response_duration_seconds.*)"
              action: keep
```

## Splitting scrape configs into files (`scrape_config_files`)

Keep large or shared scrape definitions out of the main config with Prometheus's
`scrape_config_files` (glob paths to files holding `scrape_configs`). This counts as a valid scrape
source for validation (you do not also need inline `scrape_configs`):

```yaml
receivers:
  prometheus:
    config:
      scrape_config_files:
        - /etc/otel/scrape_configs/*.yaml
```

## Feature gates

- `receiver.prometheusreceiver.EnableCreatedTimestampZeroIngestion` (from v0.113.0) — **alpha, off by default**. Injects created-timestamps as 0-valued samples. Off by default due to higher CPU cost at high metric volume.
- `receiver.prometheusreceiver.IgnoreScopeInfoMetric` (from v0.148.0) — **stable since v0.162.0** (beta and on by default v0.156.0–v0.161.0). `otel_scope_info` is no longer used for scope-attribute extraction and is converted like any other metric; scope attributes come from `otel_scope_<name>` labels. The gate can no longer be disabled and is slated for removal in v0.164.0, after which referencing it in `--feature-gates` fails startup.

# `prometheus`: configuration

## Typical config

```yaml
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'my-service'
          scrape_interval: 5s
          static_configs:
            - targets: ['my-service:9090']

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [memory_limiter]
      exporters: [otlp]
```

## Configuration reference

These are the receiver's **own** top-level keys. The `config:` block holds the standard Prometheus
configuration — `global`, `scrape_configs`, `scrape_config_files`, service discovery, relabeling —
documented in the upstream [Prometheus configuration reference][promcfg]; it is not reproduced here.

| Key | Type | Default | Notes |
|-----|------|---------|-------|
| `config` | embedded Prometheus config | — | The Prometheus configuration, same YAML structure as `prometheus.yml`. Holds `global`, `scrape_configs`, `scrape_config_files`, and related sections. Any field you omit takes Prometheus's own default for that setting. |
| `trim_metric_suffixes` | bool | `false` | [**Experimental**] Trims unit and counter-type suffixes from metric names, e.g. `singing_duration_seconds_total` → `singing_duration`. Useful to restore OpenTelemetry-style names. |
| `target_allocator` | object | — | Optional. Client config to fetch dynamically-assigned scrape targets from the OpenTelemetry Operator's Target Allocator. See table below and [advanced.md](advanced.md). |
| `api_server` | object | — | Optional. Hosts a local Prometheus agent-mode API server for debugging. See table below and [advanced.md](advanced.md). |
| `scrape_on_shutdown` | bool | `false` | Perform one final scrape before the receiver closes. This last scrape ignores the configured scrape interval. (v0.155.0) |
| `discovery_reload_on_startup` | bool | `false` | Discover targets immediately on start-up instead of waiting for the configured interval before initializing the scrape pools. (v0.155.0) |
| `initial_scrape_offset` | duration | `0s` | Fixed delay added before the initial scrape of targets, on top of the receiver's internal load-balancing offset. Avoids readiness races and backend rate-limits right after start-up. (v0.155.0) |

[promcfg]: https://prometheus.io/docs/prometheus/latest/configuration/configuration/

### `target_allocator` keys

The block squashes the Collector `confighttp` [client configuration][confighttp] (so `endpoint`,
`tls`, `proxy_url`, etc. are available) and adds:

| Key | Type | Default | Notes |
|-----|------|---------|-------|
| `endpoint` | string | — | **Required.** Target Allocator URL. Must be a valid request URI. |
| `interval` | duration | — | How often to refresh the target list. Must be a **positive** duration. |
| `collector_id` | string | — | **Required.** Identifies this collector instance to the allocator. Must be non-empty. A **literal** `${` is rejected, but `${ENV}` references are env-expanded by the Collector before validation (see [quirks.md](quirks.md)). |
| `http_sd_config` | Prometheus HTTP SD config | — | HTTP service-discovery config used to fetch targets. |
| `http_scrape_config` | Prometheus HTTP client config | — | HTTP client config applied when scraping the discovered targets. |
| `tls`, `proxy_url`, … | (from `confighttp.ClientConfig`) | — | Standard confighttp client options. |

[confighttp]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/confighttp/README.md#client-configuration

### `api_server` keys

| Key | Type | Default | Notes |
|-----|------|---------|-------|
| `enabled` | bool | `false` | Turn the debug API server on. |
| `max_connections` | int | `512` | Maximum number of simultaneous HTTP connections the server accepts. |
| `server_config` | `confighttp.ServerConfig` | endpoint `127.0.0.1:9090`, `read_timeout: 10m` | Standard confighttp [server config][confighttp-server] (`endpoint`, `tls`, CORS, timeouts). The endpoint is populated by default, so enabling the server does not require setting it. |

[confighttp-server]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/confighttp/README.md#server-configuration

## Validation rules

| Rule | Error message |
|------|---------------|
| At least one of `config.scrape_configs`, `config.scrape_config_files`, or `target_allocator` must be set. | `no Prometheus scrape_configs or target_allocator set` |
| Prometheus server-only features are rejected: `remote_write`, `remote_read`, `rule_files`, `alert_config.relabel_configs`, `alert_config.alertmanagers`. | `unsupported features: …` (the offending features, one per line) |
| `target_allocator.endpoint` must parse as a valid request URI. | `TargetAllocator endpoint is not valid: …` |
| `target_allocator.collector_id` must be non-empty and must not contain `${`. | `CollectorID is not a valid ID` |
| `target_allocator.interval` must be a positive duration. | `interval must be a positive duration, got …` |

## Scrape protocols and exposition formats

The receiver supports the protocols/exposition formats accepted by the Prometheus scrape config:

- `PrometheusProto`
- `OpenMetricsText1.0.0`
- `OpenMetricsText0.0.1`
- `PrometheusText1.0.0`
- `PrometheusText0.0.4`

Since Prometheus 3.0, scrapes need a proper `Content-Type` header. For back-compat, the receiver
defaults each scrape config's `fallback_scrape_protocol` to `PrometheusText0.0.4` when it is unset,
so targets that do not return a content type still parse.

## How scraped series map to OTel

- `target_info` is dropped; its attributes populate the OTel **Resource**.
- `otel_scope_name` / `otel_scope_version` labels are dropped and populate the Instrumentation **Scope** name/version.
- Other `otel_scope_<name>` labels populate Scope **attributes**. `otel_scope_info` is not used for scope attributes (since v0.156.0 by default, unconditionally since v0.162.0) and is converted like any other metric.
- The `job` target label (its value comes from `job_name`) maps to `service.name`, and `instance` maps to `service.instance.id` (plus `server.address`/`server.port` when discernible). Kubernetes SD meta-labels map to `k8s.*` resource attributes — see the upstream [resource attribute mapping][resmap].
- Native histograms are converted to OTel exponential histograms (see [advanced.md](advanced.md)).

[resmap]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/prometheusreceiver/resource_attribute_mapping.md

# `prometheus`: known quirks

## No auto-sharding — duplicate scrapes across replicas

The receiver is **stateful** and the Collector **cannot auto-scale scraping** across replicas. If
you run multiple collector replicas with the **same** config, each replica scrapes every target, so
you get **duplicate metrics**. Options:

- Use the **Target Allocator** (`target_allocator`) to shard targets across replicas — see [advanced.md](advanced.md).
- Or **manually shard**: give each replica a different scrape configuration.

This is the upstream README's headline warning; treat the receiver as work-in-progress for
horizontal scaling.

## `$` must be escaped as `$$`

The Collector interprets `$` in config as an environment-variable reference. To use a **literal**
`$` inside the Prometheus config — common in relabel regexes and `replacement` values — escape it as
`$$`. An unescaped `$foo` will be substituted (or blanked) at config load, silently breaking the
relabel rule.

## `collector_id: collector-${HOSTNAME}` — expanded, not rejected

You may see examples using a templated `collector_id` like `collector-${HOSTNAME}` for the Target
Allocator. Contrary to what the "must not contain `${`" rule suggests, the Collector's own
environment-variable substitution expands `${HOSTNAME}` **before** the receiver validates the
config, so `collector-${HOSTNAME}` becomes `collector-<hostname>` and validates fine — this is
actually a working way to get per-pod uniqueness when `HOSTNAME` is set (as it is in Kubernetes
pods). If `HOSTNAME` is unset the value silently collapses to `collector-`.

The `CollectorID is not a valid ID` validation error fires only when a **literal** `${` reaches the
receiver — i.e. when you escape it as `$${...}` (the same `$$` escaping used for literal `$` in
relabel rules) so it survives env-substitution. So the real trap is over-escaping: `$${HOSTNAME}`
is rejected, plain `${HOSTNAME}` is not. An empty `collector_id` is also rejected.

## `use_start_time_metric` / `start_time_metric_regex` were removed

These two keys used to live on this receiver but were **removed in v0.143.0** and are gone from the
current config. To adjust metric start times from `process_start_time_seconds`, use the
**`metric_start_time`** processor (renamed from `metricstarttime` in v0.148.0; the old name is kept
as a deprecated alias). Putting `use_start_time_metric` in the receiver config now is an unknown-key error.

## Prometheus server features are rejected

This is a *scrape-only* receiver, not a Prometheus server. These sections under `config:` are
rejected with `unsupported features: …`:

- `remote_write`
- `remote_read`
- `rule_files`
- `alert_config.relabel_configs`
- `alert_config.alertmanagers`

For Remote Write ingestion use the `prometheus_remote_write` receiver; for rules/alerting run an
actual Prometheus or use OTel processors/connectors.

## `fallback_scrape_protocol` defaulting (Prometheus 3.0)

Since Prometheus 3.0, scrapes without a proper `Content-Type` header fail. The receiver
auto-defaults each scrape config's `fallback_scrape_protocol` to `PrometheusText0.0.4` when unset,
preserving the older lenient behavior. If you scrape native histograms you must still add
`PrometheusProto` to `scrape_protocols` explicitly — the fallback does not cover that.

## Validation-error → fix

| Error | Cause | Fix |
|-------|-------|-----|
| `no Prometheus scrape_configs or target_allocator set` | Neither inline `scrape_configs`, nor `scrape_config_files`, nor `target_allocator` provided. | Add at least one scrape source. |
| `unsupported features: …` | A Prometheus server-only section is present under `config:`. | Remove `remote_write`/`remote_read`/`rule_files`/`alert_config.*`. |
| `TargetAllocator endpoint is not valid: …` | `target_allocator.endpoint` is not a valid request URI. | Use a full URL like `http://my-ta-service`. |
| `CollectorID is not a valid ID` | `collector_id` is empty, or a **literal** `${` reached the receiver (e.g. escaped as `$${`). Plain `${ENV}` is env-expanded before validation and does not trigger this. | Use a literal id (e.g. `collector-1`), or a `${ENV}` that expands to a valid id. |
| `interval must be a positive duration, got …` | `target_allocator.interval` is zero/negative/unset. | Set a positive duration (e.g. `30s`). |
| Empty/blanked relabel replacement | Unescaped `$` was treated as an env var. | Escape literal `$` as `$$`. |

## Stability

Metrics are **Beta**. The receiver embeds `prometheus/prometheus`'s scrape manager, so Prometheus
scrape behavior tracks the pinned upstream version. `trim_metric_suffixes` is flagged
**Experimental** upstream.

# `prometheus` receiver

| | |
|-|-|
| Kind | receiver |
| Type | `prometheus` |
| Signals | metrics (Beta) |
| Distributions | core, contrib, k8s |
| Go module | `github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver` |
| Upstream README | <https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/prometheusreceiver> |

## Description

Scrapes metrics from Prometheus-format HTTP endpoints and converts them to OTLP metrics. It embeds the **scrape manager and parser from `prometheus/prometheus`**, so the configuration under the top-level `config:` key is the same YAML you would put in a `prometheus.yml` — `global`, `scrape_configs`, `scrape_config_files`, service discovery, and relabeling all behave exactly as in Prometheus. The receiver is a *scrape-only* drop-in: it pulls from targets, it does not run a Prometheus server. Anything omitted under `config:` falls back to Prometheus's own defaults for that setting.

Beyond the embedded Prometheus config, the receiver exposes a handful of extra top-level keys: `trim_metric_suffixes` (restore OTel-style metric names), `target_allocator` (fetch dynamically-assigned targets from the OpenTelemetry Operator's Target Allocator), `api_server` (host a local Prometheus agent-mode debug API), and three start-up/shutdown scrape tuning knobs added in v0.155.0 (`scrape_on_shutdown`, `discovery_reload_on_startup`, `initial_scrape_offset`). Scraped `target_info`/`otel_scope_*` series are consumed to populate the OTel Resource and Instrumentation Scope, and native histograms are auto-converted to OTel exponential histograms.

The two facts that catch people out: the receiver is **stateful and does not auto-shard** — running multiple replicas with the same config scrapes every target multiple times (duplicate metrics); use the Target Allocator or manual sharding instead. And the `$` character in the Prometheus config is interpreted by the Collector as an environment variable, so a literal `$` (in a relabel regex or replacement) must be escaped as `$$`. See [quirks.md](quirks.md).

## Main use-cases

Use when:
- You have applications, exporters, or infrastructure that expose metrics in Prometheus format and you want to pull them into an OTLP pipeline.
- You are migrating an existing `prometheus.yml` into the Collector — paste your `scrape_configs` under `config:` largely unchanged.
- You run on Kubernetes with the OpenTelemetry Operator and want dynamically-assigned, sharded scrape targets — use `target_allocator`.
- You need to scrape native histograms and forward them as OTel exponential histograms.

Avoid when:
- The source already speaks OTLP — use the [`otlp`](../otlp/README.md) receiver.
- The source pushes via Prometheus Remote Write — use the `prometheus_remote_write` receiver instead (push, not pull).
- You need Prometheus *server* features (`remote_write`, `remote_read`, `rule_files`, alerting) — this receiver rejects them; it only scrapes.

## Related components

- [`otlp` receiver](../otlp/README.md) — the OTLP-native ingress; `prometheus` is for Prometheus-format scrape targets.
- [`interval` processor](../interval/README.md) — downsample chatty scrapes by emitting only the latest value per series per interval.
- `prometheus_remote_write` receiver and `prometheus` / [`prometheus_remote_write`](../prometheus_remote_write/README.md) exporters — the Prometheus-family counterparts (push vs pull); pair a scrape ingress here with a Prometheus-format egress.
- [`transform`](../transform/README.md) / [`filter`](../filter/README.md) — drop or mutate metrics after scrape, though Prometheus `metric_relabel_configs` does much of this at the source.

## Details

- [Configuration](configuration.md) — every top-level key (`config`, `trim_metric_suffixes`, `target_allocator`, `api_server`, plus the v0.155.0 `scrape_on_shutdown` / `discovery_reload_on_startup` / `initial_scrape_offset` knobs), every validation rule, scrape protocols, and representative `scrape_config` examples.
- [Verification](verification.md) — a self-scrape recipe (the collector scrapes its own `:8888` telemetry endpoint → `debug`), verified on contrib v0.154.0. Notes why `telemetrygen` cannot drive this receiver.
- [Advanced use-cases](advanced.md) — named instances, the Target Allocator, native histograms, the `api_server`, relabeling, and `scrape_config_files`.
- [Known quirks](quirks.md) — the no-auto-shard/duplicate-scrape warning, `$$` escaping, the `collector_id` `${...}` trap, removed `use_start_time_metric`, rejected server features, and stability.

# `prometheus` receiver: verification

See [Verification harness](../../SKILL.md#verification-harness) for how to run this end-to-end.

The `prometheus` receiver is **scrape-based**, so `telemetrygen` does **not** apply — there is no
OTLP endpoint to push into; the receiver pulls from a Prometheus-format HTTP target. The
self-contained way to test it is to have the collector **scrape its own internal telemetry
endpoint**: `service.telemetry.metrics` can expose a Prometheus `/metrics` endpoint, and the
`prometheus` receiver scrapes that, proving the full scrape → parse → OTLP → `debug` path with no
external target. Verified on `otel/opentelemetry-collector-contrib:0.154.0`.

This is a deliberately minimal repro: no `memory_limiter`, single pipeline. Do not copy it verbatim
into production — a real metrics pipeline should front the receiver with `memory_limiter`.

Config (`prometheus-verify.yaml`) — the receiver scrapes the collector's own metrics port `:8888`,
`debug` at detailed verbosity so each metric is printed:

```yaml
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'otel-self'
          scrape_interval: 5s
          static_configs:
            - targets: ['0.0.0.0:8888']
exporters:
  debug:
    verbosity: detailed
service:
  telemetry:
    metrics:
      readers:
        - pull:
            exporter:
              prometheus:
                host: 0.0.0.0
                port: 8888
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [debug]
```

Start the collector, publishing the telemetry port so the self-scrape target resolves:

```bash
IMG=otel/opentelemetry-collector-contrib:0.154.0
docker run -d --name prom-recv -p 8888:8888 \
  -v "$PWD/prometheus-verify.yaml:/etc/otelcol-contrib/config.yaml" $IMG
```

Wait ~12s so at least two 5s scrape intervals elapse (the first scrape fires while the collector is
still warming up and reports fewer series), then read the per-scrape batch summary the `debug`
exporter logs each time the receiver delivers a scrape:

```bash
sleep 12
docker logs prom-recv 2>&1 | grep '"otelcol.component.id": "debug"'
```

**What proves it worked:** each line is one completed scrape → parse → OTLP → `debug` cycle. Once
warmed up, every 5s scrape of the collector's own `/metrics` endpoint delivers a steady batch.
Verified on `otel/opentelemetry-collector-contrib:0.154.0` — the first (warm-up) scrape reported
`metrics: 12`, then each subsequent scrape was stable:

```
... "otelcol.component.id": "debug", ... "resource metrics": 1, "metrics": 17, "data points": 18}
```

Spot-check that a well-known self-metric came through (it appears once per scrape):

```bash
docker logs prom-recv 2>&1 | grep -c '\-> Name: otelcol_process_uptime'
```

returns one hit per elapsed scrape, and the distinct `otelcol_*` series scraped is **11**
(`docker logs prom-recv 2>&1 | grep '\-> Name: otelcol_' | sort -u | wc -l`).

If `debug` logs no batch lines: the most likely cause is that no scrape interval has elapsed yet
(wait longer), or the `:8888` target is unreachable — confirm `service.telemetry.metrics` exposes
the Prometheus reader on the same host/port the scrape config targets.

Tear down:

```bash
docker rm -f prom-recv
```

# `prometheus` exporter: advanced use-cases

## Resource attributes: `target_info` vs `resource_constant_labels`

By default, resource attributes are **not** copied onto each metric's labels — they are exposed on a separate `target_info` series, following Prometheus convention. To use them in PromQL you join on the standard identity labels:

```promql
my_metric * on (job, instance) group_left target_info
```

Two ways to bring the attributes onto the metric labels directly instead:

- **Copy the few you need** with the [`transform`](../transform/README.md) processor (datapoint context), so only the relevant attributes become labels and cardinality stays bounded:

  ```yaml
  processors:
    transform:
      metric_statements:
        - context: datapoint
          statements:
            - set(attributes["namespace"], resource.attributes["k8s.namespace.name"])
  ```

- **Select resource labels** with wildcard `included` / `excluded` patterns:

  ```yaml
  exporters:
    prometheus:
      endpoint: 0.0.0.0:9464
      resource_constant_labels:
        included: ["k8s.*", "deployment.environment.name"]
        excluded: ["k8s.pod.uid"]
  ```

The older `resource_to_telemetry_conversion` block is deprecated in v0.160.0.

## OpenMetrics and exemplars

Exemplars are exported **only** in OpenMetrics format, and **only** for histograms and monotonic sums (counters). Enable OpenMetrics to get them:

```yaml
exporters:
  prometheus:
    endpoint: 0.0.0.0:9464
    enable_open_metrics: true
```

There is no exemplar support for native histograms (below).

## Native histograms

OTel exponential histograms are automatically converted to the Prometheus **native histogram** format. To scrape them, the Prometheus server must scrape using the protobuf format and accept native histograms. (No exemplars are emitted for native histograms.)

## Choosing a `translation_strategy`

`translation_strategy` supersedes the deprecated `add_metric_suffixes` when set (see [configuration.md](configuration.md#translation_strategy)). Pick by what your consumer expects:

```yaml
exporters:
  prometheus:
    endpoint: 0.0.0.0:9464
    translation_strategy: UnderscoreEscapingWithSuffixes   # classic Prometheus compatibility
```

- `UnderscoreEscapingWithSuffixes` / `UnderscoreEscapingWithoutSuffixes` — escape names to classic Prometheus form, with or without type/unit suffixes.
- `NoUTF8EscapingWithSuffixes` — keep UTF-8 names, still append suffixes.
- `NoTranslation` — pass names/labels through unaltered.

Note the content-negotiation caveat: Prometheus may still apply underscore escaping on the wire even when you keep UTF-8. The `exporter.prometheusexporter.DisableAddMetricSuffixes` feature gate (beta, since v0.132.0) forces `translation_strategy` to always be used and the deprecated `add_metric_suffixes` to be ignored.

## The `sending_queue` caveat

This exporter accumulates in memory and is scraped, so the standard `exporterhelper` `sending_queue` has far less meaning here than for a pushing exporter — there is no downstream send to queue against. Don't over-tune it. If you do need the full queue reference, it is documented on the [`otlp_grpc` exporter](../otlp_exporter/configuration.md#sending_queue).

# `prometheus` exporter: configuration

All keys live under the exporter instance — `exporters: { prometheus: { … } }`. Facts below trace to contrib **v0.160.0** source (`exporter/prometheusexporter/config.go` + `factory.go`, with the embedded `confighttp.ServerConfig`).

## Top-level keys

| Key | Type | Default | Meaning |
|-----|------|---------|---------|
| `endpoint` | string | — (**required**) | Address the HTTP server binds to; metrics are exposed at the fixed path `/metrics`. From the embedded `confighttp.ServerConfig`. |
| `namespace` | string | — | If set, prefixes every exported series (e.g. `namespace: testns` → `testns_<metric>`). |
| `const_labels` | map[string]string | — | Key/value labels applied to **every** exported metric. |
| `send_timestamps` | bool | `false` | When `true`, appends the underlying sample timestamp to each exposed line. |
| `metric_expiration` | duration | `5m` | How long a series stays exposed without an update; after this it is dropped from `/metrics`. |
| `resource_constant_labels` | object | — | Select resource attributes to add as labels with `included` / `excluded` wildcard patterns. Added in v0.160.0. The legacy `enabled` and `exclude_service_attributes` fields are rejected in this block. |
| `resource_to_telemetry_conversion` | object | — | **Deprecated** in v0.160.0; use `resource_constant_labels`. Its legacy `enabled` / `exclude_service_attributes` fields remain accepted unless the migration gate is enabled. |
| `enable_open_metrics` | bool | `false` | When `true`, exposes using OpenMetrics format. Exemplars are exported **only** in OpenMetrics format, and **only** for histograms and monotonic sums (counters). |
| `without_scope_info` | bool | `false` | When `true`, omits instrumentation-scope labels (`otel_scope_name`, `otel_scope_version`, `otel_scope_schema_url`, and scope attributes). |
| `add_metric_suffixes` | bool | `true` | **Deprecated** (use `translation_strategy`). When `true`, appends type/unit suffixes (e.g. counters get `_total`). **Ignored** when `translation_strategy` is explicitly set. |
| `translation_strategy` | string (enum) | — | When set, takes precedence over `add_metric_suffixes`; controls how OTLP metric/attribute names are translated to Prometheus names. See [translation_strategy](#translation_strategy). |
| `sending_queue` | object | — | The standard `exporterhelper` queue block. Far less meaningful here than for a pushing exporter (see [advanced.md](advanced.md) and [quirks.md](quirks.md)). |

The exporter also accepts all standard HTTP **server** options from the embedded `confighttp.ServerConfig` (`tls`, `cors`, `auth`, `response_headers`, `compression_algorithms`, `max_request_body_size`, and others — cross-reference the confighttp documentation rather than enumerating exhaustively). Server defaults from `NewDefaultServerConfig()`:

| Server key | Default |
|-----|---------|
| `write_timeout` | `30s` |
| `read_header_timeout` | `1m` |
| `keepalive.idle_timeout` | `1m` |
| `keepalive.enabled` | `true` |

The former flat `idle_timeout` and `keep_alives_enabled` fields are deprecated in core v0.160.0.
Nonzero `idle_timeout` and `keep_alives_enabled: false` cannot be set with the new `keepalive` block;
the no-op values `idle_timeout: 0` and `keep_alives_enabled: true` may coexist with it.

## `translation_strategy`

When set, this enum takes precedence over the deprecated `add_metric_suffixes`. `Validate()` rejects any value other than the four below with `invalid translation_strategy: <v>`.

| Value | Meaning |
|-------|---------|
| `UnderscoreEscapingWithSuffixes` | Fully escape names for classic Prometheus compatibility **and** append type/unit suffixes. |
| `UnderscoreEscapingWithoutSuffixes` | Escape special characters to `_`, but no suffixes. |
| `NoUTF8EscapingWithSuffixes` | Keep UTF-8 (no `_` escaping); still append unit / `_total` suffixes. |
| `NoTranslation` | Bypass all translation; names/labels pass through unaltered. |

> **Content-negotiation caveat** (from the upstream README): Prometheus negotiates the exposition format when scraping, so underscore escaping may still be applied on the wire even when this is set to keep UTF-8.

## `sending_queue`

Optional. The standard `exporterhelper` queue block. Because this is a **pull** exporter that accumulates in memory and is scraped, queue/retry semantics are far less meaningful here than for a pushing exporter. Tune it only if you have a specific reason; for the full queue reference see the [`otlp_grpc` exporter's configuration](../otlp_exporter/configuration.md#sending_queue).

## Validation summary

| Condition | Error / rule | When |
|-----------|--------------|------|
| `translation_strategy` set to anything other than the four valid values | `invalid translation_strategy: <v>` | config validation |
| both resource-conversion blocks configured | rejected; use only `resource_constant_labels` | config validation |
| `resource_constant_labels.enabled` or `.exclude_service_attributes` set | rejected; use `included` / `excluded` | config validation |

An empty `endpoint` is not caught by `Config.Validate()`; `endpoint` has no default, so the missing-address error surfaces one step later, at exporter construction during pipeline build — which the `validate` subcommand triggers, reporting `expecting a non-blank address to run the Prometheus metrics handler`.

## Feature gate

| Gate | Stage | Since | Effect |
|------|-------|-------|--------|
| `exporter.prometheusexporter.DisableAddMetricSuffixes` | `beta` | v0.132.0 | When enabled, the deprecated `add_metric_suffixes` is ignored and `translation_strategy` is always used. (Spec PR 4533.) |
| `exporter.prometheus.DisableResourceToTelemetryConversion` | `alpha` | v0.160.0 | Rejects deprecated `resource_to_telemetry_conversion`; use `resource_constant_labels`. |
| `pkg.translator.prometheus.PermissiveLabelSanitization` | `beta` (since v0.162.0) | v0.55.0 | Shared with `prometheus_remote_write`. Enabled by default: labels with a single leading underscore (`_foo`) are kept instead of being rewritten to `key_foo`. Disable to restore the prefixing. |

# `prometheus` exporter: known quirks

## It's a server (pull), not a pusher

This exporter **hosts** an HTTP server and exposes metrics at `/metrics` for a scraper to pull. It does **not** push metrics anywhere. The common mistake is reaching for it when you actually want to **send** metrics out via Prometheus Remote Write — that is the separate [`prometheus_remote_write`](../prometheus_remote_write/README.md) exporter. They are not interchangeable: this one waits to be scraped; that one initiates pushes.

## Same `prometheus` type string as the receiver

The `prometheus` **receiver** (a scrape ingress) and this `prometheus` **exporter** share the type string `prometheus` but are different component classes. They coexist in one config — `receivers: { prometheus: … }` and `exporters: { prometheus: … }` are unrelated components — so don't assume a reference to "the prometheus component" means either one.

## `endpoint` is required, and the path is fixed at `/metrics`

`endpoint` has **no default**; without it the HTTP server has nothing to bind to. The exposition path is always `/metrics` — it is not configurable. `Config.Validate()` does not catch a missing `endpoint` (it only validates `translation_strategy`), so the failure surfaces when the exporter is **constructed during pipeline build**, not from `Config.Validate()`. Because the `validate` subcommand builds pipelines, it does report this — `failed to build pipelines: failed to create "prometheus" exporter for data type "metrics": expecting a non-blank address to run the Prometheus metrics handler` — rather than waiting for server startup.

## Stale series after `metric_expiration`; counters reset on restart

The exporter holds an in-memory accumulator. A series stays exposed only as long as it keeps getting updates within `metric_expiration` (default **5m**); after that it is dropped from `/metrics`. Because the state is in-memory, all accumulated values — including counters — **reset when the collector restarts**, which a scraping Prometheus will see as a counter reset.

## `add_metric_suffixes` is deprecated — prefer `translation_strategy`

`add_metric_suffixes` (default `true`) appends type/unit suffixes (counters get `_total`), but it is **deprecated** and is **ignored** whenever `translation_strategy` is explicitly set. The `exporter.prometheusexporter.DisableAddMetricSuffixes` feature gate (stage `beta`, since v0.132.0) makes the collector ignore `add_metric_suffixes` entirely and always use `translation_strategy`. Migrate to `translation_strategy` for new configs.

## Exemplars need OpenMetrics, and only on histograms/counters

Exemplars are exported **only** when `enable_open_metrics: true`, and even then **only** for histograms and monotonic sums (counters). Other metric types never carry exemplars, and native histograms get none at all.

## Horizontal scaling: scrape every replica, no aggregation

Each collector replica exposes only the metrics it itself received — there is no cross-replica aggregation. If you run several replicas, the scraping Prometheus must target **every** replica to get the full picture, and you must handle the resulting per-replica series accordingly. There is no built-in sharding or merging on the exposition side.

## Stability is per signal

Metrics are **Beta** — and metrics is the **only** signal this exporter supports (no traces, logs, or profiles).

# `prometheus` exporter

| | |
|-|-|
| Kind | exporter |
| Type | `prometheus` |
| Signals | metrics (Beta) |
| Distributions | core, contrib |
| Go module | `github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter` |
| Upstream README | <https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/prometheusexporter> |

## Description

This is a **pull / expose** exporter — it does **not push anywhere**. It hosts an HTTP server that exposes your pipeline's metrics in Prometheus exposition format at the fixed path `/metrics`, for a Prometheus server (or any scraper) to scrape. This is the single most common point of confusion: if you want to **push** metrics out via Prometheus Remote Write, you want the [`prometheus_remote_write`](../prometheus_remote_write/README.md) exporter, not this one. It embeds `confighttp.ServerConfig`, so it accepts the standard HTTP server options (`tls`, `cors`, `auth`, `max_request_body_size`, and so on); `MutatesData` is `true`.

Internally it keeps an in-memory **accumulator**: it stores the latest aggregated value per series and re-exposes that value on every scrape, dropping a series after `metric_expiration` (default `5m`) elapses with no update. Metric naming is controlled by `add_metric_suffixes` (default `true`, **deprecated**) or the newer `translation_strategy` enum, which takes precedence when set. See the detail files below.

> **Same type string as the `prometheus` receiver.** The `prometheus` *receiver* (a scrape ingress) and this `prometheus` *exporter* share the type string `prometheus` but are different component classes; they coexist in one config without conflict. Don't confuse the two.

## Main use-cases

Use when:
- You want to expose OTLP-pipeline metrics for a Prometheus server (or any scraper) to pull.
- You want a local, human-scrapable `/metrics` endpoint for debugging a metrics pipeline.
- You are bridging an OTLP pipeline into a pull-based Prometheus world.

Avoid when:
- You need to **push** metrics out — use the [`prometheus_remote_write`](../prometheus_remote_write/README.md) exporter.
- The backend speaks OTLP — use the [`otlp_grpc`](../otlp_exporter/README.md) or `otlphttp` exporter.
- You run many horizontally-scaled collector replicas — each exposes only its own subset of the data, so the scraper must target every replica and there is no cross-replica aggregation.

## Related components

- [`prometheus` receiver](../prometheus/README.md) — the scrape-ingress counterpart (same type string, different class); pulls Prometheus-format metrics *into* a pipeline.
- [`prometheus_remote_write`](../prometheus_remote_write/README.md) exporter — the **push** sibling; pick it when you need to send metrics out via Remote Write rather than be scraped.
- [`otlp_grpc`](../otlp_exporter/README.md) / `otlphttp` exporter — OTLP egress when the backend speaks OTLP.
- [`transform`](../transform/README.md) — copy selected resource attributes onto datapoint attributes so they become Prometheus labels for grouping.

## Details

- [Configuration](configuration.md) — `endpoint`, naming, `resource_constant_labels`, deprecated resource conversion, OpenMetrics, the embedded HTTP server options, and validation rules.
- [Verification](verification.md) — drive an OTLP pipeline with `telemetrygen` and `curl` the exporter's own `/metrics`, proving namespace, suffixing, `const_labels`, timestamps, and latest-value accumulation on contrib v0.154.0. It does not exercise `resource_constant_labels`, added in v0.160.0.
- [Advanced use-cases](advanced.md) — `target_info` vs `resource_constant_labels`, OpenMetrics + exemplars, native histograms, `translation_strategy` choices, and the queue caveat.
- [Known quirks](quirks.md) — pull-not-push, the shared `prometheus` type string, required `endpoint`, stale series and counter resets, `add_metric_suffixes` deprecation, exemplar limits, horizontal-scaling behavior, and per-signal stability.

# `prometheus` exporter: verification

See [Verification harness](../../SKILL.md#verification-harness) for how to run this end-to-end.

Unlike most exporters, you observe the result by **scraping the exporter's own `/metrics` endpoint with `curl`** — there is no downstream backend to watch, because this exporter exposes rather than pushes (the `debug` exporter below is included only to confirm what entered the pipeline). The recipe drives an OTLP pipeline with `telemetrygen` and reads the exposed series back. This is a **minimal repro** — it omits `memory_limiter` and other production scaffolding on purpose, to isolate the exporter's behavior. Verified on `otel/opentelemetry-collector-contrib:0.154.0`.

Config (`config.yaml`) — a `metrics` pipeline from the `otlp` receiver into both `prometheus` and `debug`:

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
exporters:
  prometheus:
    endpoint: 0.0.0.0:9464
    namespace: testns
    const_labels:
      verified_by: ollygarden
    send_timestamps: true
  debug:
    verbosity: detailed
service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheus, debug]
```

Run it, publishing both the OTLP receiver and the `/metrics` server to the host:

```bash
docker run -d --name prom-exp \
  -p 14317:4317 -p 19464:9464 \
  -v "$PWD/config.yaml:/etc/otelcol-contrib/config.yaml" \
  otel/opentelemetry-collector-contrib:0.154.0
```

Drive it with `telemetrygen` — see the `otel-telemetrygen` skill. This emits one cumulative Sum metric named `gen` with **3 datapoints, values 0, 1, 2** (confirmed in the `debug` exporter's output as three `Metric #0 Name: gen` lines with `Value: 0/1/2`):

```bash
telemetrygen metrics --otlp-insecure --otlp-endpoint localhost:14317 \
  --metrics 3 --workers 1 --metric-type Sum \
  --telemetry-attributes 'tg_label="v1"'
```

**What proves it worked:** scrape the exporter's own endpoint:

```bash
curl -s localhost:19464/metrics
```

produces exactly:

```text
# HELP testns_gen_total 
# TYPE testns_gen_total counter
testns_gen_total{job="telemetrygen",otel_scope_name="",otel_scope_schema_url="",otel_scope_version="",tg_label="v1",verified_by="ollygarden"} 2 1781369164374
```

Each piece confirms a specific behavior:

- **`namespace: testns`** → the `testns_` series prefix.
- **`_total` suffix and `# TYPE … counter`** → `add_metric_suffixes` defaults to `true`, and this is a counter, so the type/unit suffix is appended.
- **`const_labels`** → the `verified_by="ollygarden"` label on the series.
- **the telemetry attribute** → the `tg_label="v1"` label.
- **scope labels present** (`otel_scope_name`/`otel_scope_version`/`otel_scope_schema_url`, empty here) → `without_scope_info` defaults to `false`.
- **`send_timestamps: true`** → the trailing millisecond timestamp `1781369164374`.
- **latest-value accumulation** → the three datapoints had values 0, 1, 2 at increasing timestamps; the exposed value is **2**. The exporter keeps and exposes the **latest** aggregated value per series (its in-memory accumulator picks the most recent timestamp), not a sum or a stream.

No `target_info` series appeared with `telemetrygen`'s minimal resource — that mechanism is covered in [advanced.md](advanced.md), not claimed here.

Tear down:

```bash
docker rm -f prom-exp
```

# `prometheus_remote_write` exporter: advanced use-cases

## Write-Ahead Log (WAL)

The WAL persists outgoing metrics to disk before sending, so a crash or backend outage doesn't lose in-flight data. It is **off** unless `wal.directory` is set:

```yaml
exporters:
  prometheus_remote_write:
    http:
      endpoint: https://my-cortex:7900/api/v1/push
    wal:
      directory: ./prom_rw     # setting this enables the WAL
      buffer_size: 100         # default 300
      truncate_frequency: 45s  # default 1m
      lag_record_frequency: 15s
```

When the WAL is enabled, `max_batch_size_bytes` is **ignored** — the WAL `buffer_size` / `truncate_frequency` govern how much is read and how often it is truncated.

This WAL is the **exporter's own** on-disk buffer (backed by a `wal` library, stored under `<directory>/prom_remotewrite`). It is **not** the [`file_storage`](../file_storage/README.md) extension and does not reference a `storage` ID — don't confuse the two.

## RW2 / `protobuf_message` and the feature gate

`protobuf_message` selects the wire format:

- `prometheus.WriteRequest` — Remote Write 1.0 (the default).
- `io.prometheus.write.v2.Request` — Remote Write 2.0: more efficient, always includes metadata, and adds created-timestamp and native-histogram support.

RW2 is only honored when the feature gate `exporter.prometheusremotewritexporter.enableSendingRW2` (alpha, since v0.125.0) is enabled — otherwise `Validate()` errors. Per upstream, **PRW 2.0 is "In Development", only partially implemented, and not ready for usage.** Your backend must also support PRW 2.0.

RW2 also unlocks the UTF-8 `translation_strategy` values (`NoUTF8EscapingWithSuffixes`, `NoTranslation`), which are rejected under RW1.

## `resource_constant_labels` vs `target_info`

By default, resource attributes are **not** copied onto each series — they are emitted on a separate `target_info` metric (`target_info.enabled` defaults to `true`). To use them in PromQL, join on the identity labels:

```promql
my_metric * on (job, instance) group_left target_info{k8s_namespace_name="my-namespace"}
```

Two ways to bring attributes onto the metric labels directly:

- **Copy the few you need** with the [`transform`](../transform/README.md) processor (datapoint context), keeping cardinality bounded:

  ```yaml
  processors:
    transform:
      metric_statements:
        - context: datapoint
          statements:
            - set(attributes["namespace"], resource.attributes["k8s.namespace.name"])
  ```

- **Select resource labels** with wildcard `included` / `excluded` patterns:

  ```yaml
  exporters:
    prometheus_remote_write:
      http:
        endpoint: https://my-cortex:7900/api/v1/push
      resource_constant_labels:
        included: ["k8s.*", "deployment.environment.name"]
        excluded: ["k8s.pod.uid"]
  ```

The legacy `resource_to_telemetry_conversion` block is deprecated in v0.160.0.

## `external_labels`

`external_labels` attaches a fixed set of labels to **every** series — useful for tagging a cluster, region, or collector identity. Values may start with the reserved `__` prefix:

```yaml
exporters:
  prometheus_remote_write:
    http:
      endpoint: https://my-cortex:7900/api/v1/push
    external_labels:
      cluster: prod-eu
      collector: gateway-1
```

## Multiple workers: the `EnableMultipleWorkers` gate

The feature gate `exporter.prometheusremotewritexporter.EnableMultipleWorkers` (alpha, since v0.118.0) changes how `num_consumers` and `max_batch_request_parallelism` behave:

- **Gate off (default):** `remote_write_queue.num_consumers` defaults to `5` and drives **request** parallelism. The factory logs a migration warning if you set it to anything other than `5`.
- **Gate on:** `num_consumers` becomes the **queue-worker** count (defaulting to `1`), handling batches from the queue concurrently, while `max_batch_request_parallelism` drives parallelism for splitting a single batch larger than `max_batch_size_bytes`.

> **Out-of-order caveat.** With the gate on and `num_consumers > 1` (or `max_batch_request_parallelism > 1`), the temporal ordering of samples is no longer guaranteed. Vanilla Prometheus rejects out-of-order samples — enable `tsdb.out_of_order_time_window` on the backend, or set `max_batch_request_parallelism: 1`. Other backends (Thanos / Mimir / VictoriaMetrics) have their own settings.

```yaml
exporters:
  prometheus_remote_write:
    http:
      endpoint: https://my-cortex:7900/api/v1/push
    max_batch_request_parallelism: 1   # safe for backends that reject out-of-order samples
```

## Retrying on HTTP 429

By default the exporter does **not** retry on `429 Too Many Requests` — RW backends (Mimir, Cortex, …) return it under rate limiting, and it is treated as a non-retryable error. The feature gate `exporter.prometheusremotewritexporter.RetryOn429` (alpha, since v0.101.0) makes the exporter retry `429` responses using the standard `retry_on_failure` backoff. Enable it if your backend rate-limits with `429` and you want the exporter to back off and retry rather than drop.

## Choosing a `translation_strategy`

`translation_strategy` supersedes the deprecated `add_metric_suffixes` when set (see [configuration.md](configuration.md#translation_strategy)).

```yaml
exporters:
  prometheus_remote_write:
    http:
      endpoint: https://my-cortex:7900/api/v1/push
    translation_strategy: UnderscoreEscapingWithSuffixes   # classic Prometheus compatibility
```

- `UnderscoreEscapingWithSuffixes` / `UnderscoreEscapingWithoutSuffixes` — escape names to classic Prometheus form, with or without type/unit suffixes. Both work under RW1.
- `NoUTF8EscapingWithSuffixes` / `NoTranslation` — keep UTF-8 / pass through unaltered. **Require PRW 2.0** (rejected under RW1).

# `prometheus_remote_write` exporter: configuration

All keys live under the exporter instance — `exporters: { prometheus_remote_write: { … } }` (the deprecated alias `prometheusremotewrite` also works). Facts below trace to contrib **v0.160.0** source (`exporter/prometheusremotewriteexporter/config.go` + `factory.go` + `wal.go`, with `confighttp.ClientConfig`, `exporterhelper.TimeoutConfig`, and `configretry.BackOffConfig`).

## Top-level keys

| Key | Type | Default | Meaning |
|-----|------|---------|---------|
| `http` | object | standard HTTP client defaults | Nested `confighttp.ClientConfig`. Added in v0.159.0; preferred over the deprecated flat HTTP client settings. |
| `http.endpoint` | string | `http://some.url:9411/api/prom/push` (**must override**) | Remote-write URL to push samples to. The factory ships a placeholder — set your real backend URL. |
| `namespace` | string | `""` | Prefix prepended to every exported metric name. |
| `external_labels` | map[string]string | `{}` | Labels attached to **every** series. Values may start with the reserved `__` prefix. |
| `add_metric_suffixes` | bool | `true` | **Deprecated** (use `translation_strategy`). When `false`, no type/unit suffixes are added; the factory logs a deprecation warning. **Ignored** when `translation_strategy` is explicitly set. See note below on the removal-version discrepancy. |
| `translation_strategy` | string (enum) | `""` (unset) | When set, takes precedence over `add_metric_suffixes`. See [translation_strategy](#translation_strategy). When unset, `add_metric_suffixes` governs. |
| `send_metadata` | bool | `false` | When `true`, generate and send Prometheus metadata. **Ignored under PRW 2.0**, which always includes metadata. |
| `disable_scope_info` | bool | `false` | When `true`, omits instrumentation-scope labels (`otel_scope_name`, `otel_scope_version`, and `otel_scope_*` attributes). |
| `max_batch_size_bytes` | int | `3000000` (≈ 2.86 MB) | Batches larger than this are split into multiple requests. **Ignored when the WAL is enabled** (the WAL `buffer_size` / `truncate_frequency` govern instead). |
| `max_batch_request_parallelism` | int (pointer) | unset (documented as `5`) | Parallel requests when splitting a single oversized batch. When unset (the default), request parallelism is governed by `remote_write_queue.num_consumers`; this key only takes over once the `EnableMultipleWorkers` gate is on. Set to `1` if the backend cannot ingest out-of-order samples. See validation below. |
| `protobuf_message` | string | `prometheus.WriteRequest` (RW1) | `prometheus.WriteRequest` = Remote Write 1.0; `io.prometheus.write.v2.Request` = Remote Write 2.0. RW2 requires the `enableSendingRW2` feature gate (see [advanced.md](advanced.md)). |
| `include_metadata_keys` | []string | `[]` | Forward selected incoming client-metadata values as HTTP request headers. Reserved Remote Write protocol headers are rejected. WAL-replayed requests cannot forward this metadata because the WAL stores only serialized request bytes. Added in v0.158.0. |
| `convert_explicit_histograms_to_nhcb` | bool | `false` | Convert explicit-bucket histograms to Native Histograms with Custom Buckets (schema -53), for RW1 or RW2. Added in v0.158.0. |
| `keep_classic_histograms` | bool | `false` | Also emit classic histogram series during NHCB migration; no effect unless conversion is enabled. |
| `remote_write_queue` | object | see below | Outgoing-request queue. This exporter does **not** use `sending_queue`. |
| `resource_constant_labels` | object | see below | Select resource attributes to convert to labels. Added in v0.160.0. |
| `resource_to_telemetry_conversion` | object | see below | **Deprecated** in v0.160.0; use `resource_constant_labels`. |
| `wal` | object | off unless `directory` set | Write-ahead log. See [wal](#wal). |
| `target_info` | object | `enabled: true` | Emit a `target_info` metric per resource for resource-attribute joins. |
| `retry_on_failure` | object | `configretry.BackOffConfig` | Standard retry block; the factory overrides `InitialInterval` to `50ms`. Cross-reference the [exporterhelper retry docs](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md). |
| `timeout` | duration | per exporterhelper | From the embedded `exporterhelper.TimeoutConfig` (squashed). Cross-reference exporterhelper. |

The `http` block accepts the standard HTTP **client** options from `confighttp.ClientConfig`, such
as `endpoint`, `headers`, `tls`, `compression`, and `auth`. Deprecated flat HTTP settings still
work when `http` is absent; when `http` is present it takes precedence. The alpha
`exporter.prometheusremotewritexporter.removeTopLevelHTTPSettings` gate rejects flat HTTP client
settings so migrations can be tested before their eventual removal. (`timeout` remains the
exporterhelper send timeout at top level.) Cross-reference rather than enumerating:

- [HTTP client settings (confighttp)](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/confighttp/README.md) — note only `snappy` compression is accepted (required by the Remote Write protocol). Headers `Content-Encoding`, `Content-Type`, `X-Prometheus-Remote-Write-Version`, and `User-Agent` cannot be overridden via `headers`.
- [TLS / mTLS settings (configtls)](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md) — **TLS is on by default**; set `http.tls.insecure: true` for plaintext.
- [Retry and timeout settings (exporterhelper)](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md) — note the exporter does **not** support `sending_queue`; it provides `remote_write_queue` instead.

## `remote_write_queue`

| Key | Type | Default | Meaning |
|-----|------|---------|---------|
| `enabled` | bool | `true` | When `false`, export requests run synchronously (no queue). |
| `queue_size` | int | `10000` | Maximum number of OTLP metric batches queued at once. Ignored if `enabled: false`. |
| `num_consumers` | int | `5` (or `1` when `EnableMultipleWorkers` gate is on) | Workers used to fan out outgoing requests. See [advanced.md](advanced.md) for how the feature gate changes its meaning. |

## Resource labels

| Key | Type | Default | Meaning |
|-----|------|---------|---------|
| `resource_constant_labels.included` | []string | `[]` | Wildcard patterns selecting resource attributes to convert to labels. |
| `resource_constant_labels.excluded` | []string | `[]` | Wildcard patterns removed from the included set. |

The deprecated `resource_to_telemetry_conversion` block retains `enabled` and
`exclude_service_attributes` for migration. Do not configure both blocks. The alpha
`exporter.prometheusremotewrite.DisableResourceToTelemetryConversion` gate rejects the deprecated
block. `enabled` and `exclude_service_attributes` are not valid under `resource_constant_labels`.

## `wal`

Optional and **off unless `directory` is set**. When enabled, `max_batch_size_bytes` is ignored.

| Key | Type | Default | Meaning |
|-----|------|---------|---------|
| `directory` | string | `""` | Directory to store the WAL in. Setting this enables the WAL. |
| `buffer_size` | int | `300` | Count of elements read from the WAL before truncating. |
| `truncate_frequency` | duration | `1m` | How often the WAL is truncated. |
| `lag_record_frequency` | duration | `15s` | How often the exporter records WAL lag. |

## `target_info`

| Key | Type | Default | Meaning |
|-----|------|---------|---------|
| `enabled` | bool | `true` | When `true`, emits a `target_info` metric per resource metric, carrying resource attributes for PromQL joins. |

## `translation_strategy`

When set, this enum takes precedence over the deprecated `add_metric_suffixes`. `Validate()` rejects any value other than the four below with `invalid translation_strategy: <v>`.

| Value | Meaning |
|-------|---------|
| `UnderscoreEscapingWithSuffixes` | Escape special characters to `_` **and** append type/unit suffixes. |
| `UnderscoreEscapingWithoutSuffixes` | Escape special characters to `_`, but no suffixes. |
| `NoUTF8EscapingWithSuffixes` | Keep UTF-8 (no `_` escaping); still append unit / `_total` suffixes. **Requires PRW 2.0.** |
| `NoTranslation` | Pass names/labels through unaltered. **Requires PRW 2.0.** |

> The upstream README prose calls `UnderscoreEscapingWithSuffixes` the "default", but the **code** default is the empty string, with `add_metric_suffixes: true` governing when `translation_strategy` is unset. Be precise: unset means `add_metric_suffixes` decides.

## Validation summary

| Condition | Error / rule |
|-----------|--------------|
| `max_batch_request_parallelism` set and `< 1` | `max_batch_request_parallelism can't be set to below 1` |
| `remote_write_queue.queue_size < 0` | `remote write queue size can't be negative` |
| `remote_write_queue.enabled` and `queue_size == 0` | `a 0 size queue will drop all the data` |
| `remote_write_queue.num_consumers < 0` | `remote write consumer number can't be negative` |
| `max_batch_size_bytes < 0` | `max_batch_size_bytes must be greater than 0` |
| `max_batch_size_bytes == 0` | not an error — **reset to `3000000`** |
| `http.compression` set and not `snappy` | `compression type must be snappy` (empty is allowed) |
| `protobuf_message` = `io.prometheus.write.v2.Request` without the `enableSendingRW2` gate | `remote write v2 is only supported with the feature gate exporter.prometheusremotewritexporter.enableSendingRW2` |
| `translation_strategy` not one of the four valid values | `invalid translation_strategy: <v>` |
| `translation_strategy` = `NoUTF8EscapingWithSuffixes` or `NoTranslation` under RW1 | `translation strategy <v> requires Prometheus Remote Write 2.0 (UTF-8 support)` |
| `include_metadata_keys` contains `Content-Encoding`, `Content-Type`, `User-Agent`, or `X-Prometheus-Remote-Write-Version` (case-insensitive) | rejected because it collides with a reserved Remote Write header |
| both resource-label blocks configured | `cannot configure both resource_to_telemetry_conversion and resource_constant_labels; resource_to_telemetry_conversion is deprecated` |
| legacy fields under `resource_constant_labels` | `enabled and exclude_service_attributes are not supported under resource_constant_labels; use included and excluded instead` |

> Note: `Validate()` does **not** catch an unset `http.endpoint` — the factory ships a non-empty placeholder default, so the failure surfaces at send time, not config validation.

# `prometheus_remote_write` exporter: known quirks

## It pushes — it does not expose

This exporter **initiates** outbound Remote Write requests to a backend URL. It does **not** host a `/metrics` endpoint to be scraped — that is the separate [`prometheus`](../prometheus_exporter/README.md) exporter. The common mistake is reaching for one when you want the other: "push to a remote URL" is this component; "expose for a scraper" is the other. They are not interchangeable.

## The type was renamed in v0.154.0

The type string changed from `prometheusremotewrite` (no underscores) to `prometheus_remote_write` in **v0.154.0**. The old form still works as a **deprecated alias** and logs a deprecation warning at startup. Use the snake_case `prometheus_remote_write` in new configs.

## Non-cumulative monotonic, histogram, and summary metrics are dropped

Per upstream: **"Non-cumulative monotonic, histogram, and summary OTLP metrics are dropped by this exporter."** This is a Remote Write 1.0 limitation. If your pipeline produces delta sums, OTLP histograms, or summaries and you need them in the backend, convert them upstream (e.g. cumulative-ize deltas) or move to RW2 where applicable — under RW1 they silently disappear.

## TLS is on by default; snappy is the only compression

TLS is **enabled by default** — a plaintext backend needs `http.tls.insecure: true`, otherwise the connection fails. Compression must be `snappy` (the only value `Validate()` accepts; empty is allowed): the Remote Write protocol requires it, so you cannot switch to gzip/zstd.

## `remote_write_queue`, not `sending_queue`

This exporter does **not** use the standard `exporterhelper` `sending_queue`. Queueing is configured under `remote_write_queue` (`enabled`, `queue_size`, `num_consumers`). Configuring `sending_queue` has no effect here.

| Validation error | Fix |
|------------------|-----|
| `a 0 size queue will drop all the data` | Set `remote_write_queue.queue_size` > 0, or disable the queue (`enabled: false`). |
| `remote write queue size can't be negative` | Use a non-negative `queue_size`. |
| `remote write consumer number can't be negative` | Use a non-negative `num_consumers`. |
| `compression type must be snappy` | Remove the `compression` key, or set it to `snappy`. |
| `max_batch_size_bytes must be greater than 0` | Use a non-negative `max_batch_size_bytes` (0 is auto-reset to `3000000`). On v0.156.0 and earlier this error misspelled the field as `max_batch_byte_size`. |
| `max_batch_request_parallelism can't be set to below 1` | Set it to `1` or higher. |
| `remote write v2 is only supported with the feature gate …` | Enable the `enableSendingRW2` gate, or keep `protobuf_message: prometheus.WriteRequest`. |
| `invalid translation_strategy: <v>` | Use one of the four valid enum values. |
| `translation strategy <v> requires Prometheus Remote Write 2.0 (UTF-8 support)` | Switch to an escaping strategy, or enable RW2. |

## `http.endpoint` ships a placeholder, not a real default

The factory default for `http.endpoint` is `http://some.url:9411/api/prom/push` — a non-functional placeholder. `Validate()` does **not** catch it, so if you forget to override it the failure surfaces only when the exporter tries to send. Always set your real backend URL.

## HTTP client settings belong under `http`

Put `endpoint`, `tls`, `headers`, `compression`, and the other `confighttp.ClientConfig` fields under
`http`. Deprecated flat settings still work by default when `http` is absent, and a configured
`http` block takes precedence. Enabling the alpha
`exporter.prometheusremotewritexporter.removeTopLevelHTTPSettings` gate makes flat HTTP settings a
startup error, which is useful for validating a migration.

## `add_metric_suffixes` deprecation — version discrepancy

`add_metric_suffixes` (default `true`) is **deprecated** in favor of `translation_strategy`. The factory logs a deprecation warning when it is `false` (`add_metric_suffixes is deprecated. Please use translation_strategy: UnderscoreEscapingWithoutSuffixes instead.`). The `config.go` comment claims it would be removed in **v0.153.0**, but the key remains present. Treat it as live-but-deprecated; migrate to `translation_strategy` for new configs.

## RW2 is not production-ready

`protobuf_message: io.prometheus.write.v2.Request` (Remote Write 2.0) requires the `exporter.prometheusremotewritexporter.enableSendingRW2` feature gate, and per upstream is **"In Development", only partially implemented, and not ready for usage.** Stay on RW1 (`prometheus.WriteRequest`, the default) for production.

## The WAL is not the `file_storage` extension

The exporter's `wal` block is its **own** on-disk write-ahead log, written under `wal.directory`. It is unrelated to the [`file_storage`](../file_storage/README.md) extension and takes no `storage` ID — don't try to wire it to a storage extension.

## Name and label normalization mutates data

The exporter normalizes OTLP metric names and attributes to Prometheus naming rules (so `MutatesData` is effectively true). Details of the normalization are in the [`pkg/translator/prometheus`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/translator/prometheus) module. Since v0.162.0 the shared `pkg.translator.prometheus.PermissiveLabelSanitization` gate is Beta (on by default), so labels with a single leading underscore (e.g. `_foo`) are no longer rewritten to `key_foo`; disable it with `--feature-gates=-pkg.translator.prometheus.PermissiveLabelSanitization` to restore the old prefixing.

## Stability is per signal

Metrics are **Beta** — and metrics is the **only** signal this exporter supports (no traces, logs, or profiles).

# `prometheus_remote_write` exporter

| | |
|-|-|
| Kind | exporter |
| Type | `prometheus_remote_write` |
| Signals | metrics (Beta) |
| Distributions | core, contrib |
| Go module | `github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter` |
| Upstream README | <https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/prometheusremotewriteexporter> |

> **Renamed in v0.154.0.** The type string was `prometheusremotewrite` (no underscores); it is now `prometheus_remote_write`. The old `prometheusremotewrite` still works as a deprecated alias and logs a deprecation warning — migrate to the snake_case type.

## Description

This is a **push** exporter — it sends OTLP metrics out as Prometheus Remote Write requests to any Remote-Write-compatible backend (Cortex, Mimir, Thanos, Prometheus itself with the receiver enabled, and similar). This is the mirror image of the [`prometheus`](../prometheus_exporter/README.md) exporter, which **hosts** a `/metrics` endpoint to be scraped: this one **initiates** outbound writes. If you are unsure which you want, "push to a remote URL" is this component; "expose for a scraper" is the other.

By default it speaks **Remote Write 1.0** (`protobuf_message: prometheus.WriteRequest`), **requires TLS** (set `http.tls.insecure: true` for plaintext), and uses **snappy** compression (the only value the protocol — and this exporter's `Validate()` — accepts). HTTP client settings belong under `http`; deprecated flat settings remain accepted by default for migration. Outgoing requests are queued through `remote_write_queue` (this exporter does **not** use the standard `sending_queue`). The full key list, defaults, and validation rules are in [configuration.md](configuration.md).

## Main use-cases

Use when:
- Your metrics backend speaks Prometheus Remote Write (Cortex, Mimir, Thanos, VictoriaMetrics, Prometheus with `--web.enable-remote-write-receiver`).
- You want the collector to **push** metrics out rather than be scraped.
- You are bridging an OTLP pipeline into a Prometheus-protocol world without running a separate scraper.

Avoid when:
- You want to be **scraped** — use the [`prometheus`](../prometheus_exporter/README.md) exporter (hosts `/metrics`).
- The backend speaks OTLP — use the [`otlp_grpc`](../otlp_exporter/README.md) or `otlphttp` exporter.
- You rely heavily on non-cumulative monotonic sums, OTLP histograms, or summaries under RW1 — these are **dropped** by this exporter (see [quirks.md](quirks.md)).

## Related components

- [`prometheus` exporter](../prometheus_exporter/README.md) — the **pull / expose** sibling (hosts `/metrics`); same family, opposite direction. This one pushes.
- [`otlp_grpc`](../otlp_exporter/README.md) / `otlphttp` exporter — OTLP egress when the backend speaks OTLP.
- [`prometheus` receiver](../prometheus/README.md) — scrape **ingress** that pulls Prometheus-format metrics into a pipeline.
- [`resource`](../resource/README.md) / [`transform`](../transform/README.md) — shape resource attributes before `resource_constant_labels` or `target_info` export.
- [`file_storage`](../file_storage/README.md) — only loosely related: the WAL here is the exporter's **own** on-disk buffer, **not** the `file_storage` extension (see [quirks.md](quirks.md)).

## Details

- [Configuration](configuration.md) — `http.endpoint`, naming/metadata options, histogram conversion, `remote_write_queue`, resource conversion, WAL, RW version, HTTP/TLS/retry options, and validation rules.
- [Verification](verification.md) — push to a Prometheus container with the remote-write receiver enabled and query the series back, proving `namespace` and `external_labels`. Verified end-to-end on contrib v0.159.0.
- [Advanced use-cases](advanced.md) — the WAL, RW2, `resource_constant_labels` vs `target_info`, external labels, worker/retry feature gates, and translation choices.
- [Known quirks](quirks.md) — push-not-pull, the type rename, `http.endpoint`, nested-vs-flat HTTP migration, TLS-on-by-default, snappy-only, dropped metric types, the `remote_write_queue` (not `sending_queue`) distinction, the `add_metric_suffixes` deprecation discrepancy, RW2 readiness, and per-signal stability.

# `prometheus_remote_write` exporter: verification

See [Verification harness](../../SKILL.md#verification-harness) for how to run this end-to-end.

Unlike the pull-based [`prometheus`](../prometheus_exporter/verification.md) exporter (where you scrape the exporter's own `/metrics`), here you **push** to a real backend and query the series back from it. The recipe runs two containers on a shared Docker network: a Prometheus server with the remote-write receiver enabled, and a collector that pushes into it. This is a **minimal repro** — it omits `memory_limiter` and `batch` on purpose, to isolate the exporter's behavior.

**Verified end-to-end on `otel/opentelemetry-collector-contrib:0.159.0` (2026-08-27).** A
`telemetrygen` v0.159.0 cumulative Sum named `gen` (datapoints 0/1/2) pushed through the nested
`http` configuration via RW1 and landed in Prometheus as
`testns_gen_total{job="telemetrygen", verified_by="ollygarden"} 2`.
This recipe's schema is unchanged through v0.160.0; it does not exercise
`resource_constant_labels`.

Create a network:

```bash
docker network create prw-net
```

Start Prometheus with the remote-write receiver enabled (exposes `/api/v1/write`, accepts RW1). The image ships a default config at `/etc/prometheus/prometheus.yml`:

```bash
docker run -d --name prw-prom --network prw-net -p 19090:9090 \
  prom/prometheus:v3.1.0 \
  --config.file=/etc/prometheus/prometheus.yml \
  --web.enable-remote-write-receiver
```

Collector config (`config.yaml`) — a `metrics` pipeline from the `otlp` receiver into `prometheus_remote_write`, pushing to the Prometheus container by its network alias (`debug` is added only to see what arrives):

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
exporters:
  debug:
    verbosity: detailed
  prometheus_remote_write:
    http:
      endpoint: http://prw-prom:9090/api/v1/write
      tls:
        insecure: true
    namespace: testns
    external_labels:
      verified_by: ollygarden
service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheus_remote_write, debug]
```

> The exporter `http.endpoint` host must match the Prometheus container's network alias (`prw-prom` here). A wrong host doesn't fail fast — the push hangs until the request timeout and the collector logs `Permanent error: context deadline exceeded`, `Dropping data`.

Run the collector on the same network:

```bash
docker run -d --name prw-col --network prw-net \
  -v "$PWD/config.yaml:/etc/otelcol-contrib/config.yaml" \
  otel/opentelemetry-collector-contrib:0.159.0
```

Drive it with `telemetrygen` — see the `otel-telemetrygen` skill. Run it as a container **on the same network** targeting the collector's alias (this avoids a host-port-mapped gRPC stall some Docker Desktop setups hit). This emits one **cumulative Sum** named `gen`; `telemetrygen` already defaults to cumulative temporality, which is required here — the exporter **drops non-cumulative (delta) monotonic sums** (see [quirks](quirks.md)). `--metrics N` is **per worker** and is **ignored when `--duration` is set**, so use `--workers 1` and no `--duration`:

```bash
docker run --rm --network prw-net \
  ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:v0.159.0 \
  metrics --otlp-insecure --otlp-endpoint prw-col:4317 \
  --metrics 3 --workers 1 --metric-type Sum
```

**What proves it worked:** query the series back from Prometheus. The counter gets the `_total` suffix (`add_metric_suffixes` defaults to `true`) and the `testns_` namespace prefix:

```bash
curl -s 'http://localhost:19090/api/v1/query?query=testns_gen_total'
```

```json
{"status":"success","data":{"resultType":"vector","result":[
  {"metric":{"__name__":"testns_gen_total","job":"telemetrygen","verified_by":"ollygarden"},
   "value":[1787819647.001,"2"]}]}}
```

- **`namespace: testns`** → the `testns_` series prefix.
- **`_total` suffix** → `add_metric_suffixes` defaults to `true`, and this is a counter.
- **`external_labels`** → the `verified_by="ollygarden"` label on the series.
- **`job="telemetrygen"`** → `service.name` is mapped to the `job` label.
- **value `2`** → the latest cumulative value of the 0/1/2 sequence.
- **`http.tls.insecure: true`** → plaintext push works against the receiver (TLS is on by default otherwise).

Tear down:

```bash
docker rm -f prw-col prw-prom
docker network rm prw-net
```

# `redaction`: advanced use-cases

## Allowlist mode as a strict schema

Set `allow_all_keys: false` and enumerate `allowed_keys` to define exactly which attribute keys may leave the boundary. Any key not on the list — including newly added, unanticipated attributes — is dropped by default. This is the strongest egress guarantee the processor offers.

```yaml
processors:
  redaction:
    allow_all_keys: false
    allowed_keys:
      - http.method
      - http.route
      - http.status_code
      - service.version
    ignored_keys:
      - safe.attribute        # bypasses all checks
```

## Hashing for correlatable-but-masked values

By default a masked value becomes a run of asterisks. Set `hash_function` when you need to **correlate** redacted values without revealing them — e.g. confirm two spans carry the same user id without storing the id. Use `hmac-sha256`/`hmac-sha512` with `hmac_key` so the hash is not a plain digest an attacker could rainbow-table.

```yaml
processors:
  redaction:
    allow_all_keys: true
    blocked_values:
      - '[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}'   # email
    hash_function: hmac-sha256
    hmac_key: ${env:REDACTION_HMAC_KEY}
```

## URL and DB query sanitization

URL sanitization strips high-cardinality path segments (UUIDs, timestamps, numeric IDs) and query content from URL-valued attributes, preserving routing structure for grouping. DB sanitization rewrites query/command text (and optionally span names) so literals don't leak; it only fires on spans carrying `db.system`/`db.system.name` with `CLIENT` or `SERVER` span kind.

```yaml
processors:
  redaction:
    allow_all_keys: true
    url_sanitizer:
      enabled: true
      attributes: [url.full, http.url]
      sanitize_span_name: true
    db_sanitizer:
      sanitize_span_name: true
      sql:
        enabled: true
        attributes: [db.statement, db.query.text]
      redis:
        enabled: true
        attributes: [db.statement]
```

## `blocked_key_patterns` vs `blocked_values`

- `blocked_key_patterns` matches **key names** (e.g. `.*token.*`); when a key matches, its value is masked but the key is kept.
- `blocked_values` matches **value content** (e.g. a card-number regex) regardless of the key name.
- To remove a key entirely rather than mask its value, leave it off `allowed_keys` (with key filtering enabled) instead of using `blocked_key_patterns`.

## The audit summary

When `summary` is `debug` or `info`, the processor records what it did as attributes on each record (zero-count attributes are omitted); the default (empty) and `silent` emit nothing. See the [audit attributes table](configuration.md#audit-attributes) for the full list of `redaction.*` (and `redaction.body.*`) names and their verbosity levels. Use `summary: info` for counts in production; reserve `summary: debug` (which records key names) for short-lived investigation.

# `redaction`: configuration

## Typical config

```yaml
processors:
  redaction:
    allow_all_keys: false
    allowed_keys:
      - http.method
      - http.route
      - http.status_code
    blocked_values:
      - '4[0-9]{12}(?:[0-9]{3})?'   # Visa-like card numbers
    summary: info

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, redaction]
      exporters: [otlp]
```

## Configuration reference

| Key | Type | Default | Notes |
|-----|------|---------|-------|
| `allow_all_keys` | bool | `false` | When `true`, the `allowed_keys` list is disabled and every key is kept. `blocked_values`/`blocked_key_patterns` still apply. |
| `allowed_keys` | []string | `[]` | Keys to retain. **Fails closed** — an empty list removes *all* attributes. Any key not listed is deleted. |
| `ignored_keys` | []string | `[]` | Keys that bypass all redaction checks (never removed, never masked). |
| `ignored_key_patterns` | []string | `[]` | Regex patterns for keys to bypass all redaction. |
| `blocked_key_patterns` | []string | `[]` | Regex patterns; matching keys have their **values masked** (the key is kept). |
| `blocked_values` | []string | `[]` | Regex patterns matched against attribute **values**; matches are masked or hashed. |
| `allowed_values` | []string | `[]` | Regex patterns for values considered safe. **Takes precedence** over `blocked_values` when both match. |
| `redact_all_types` | bool | `false` | When `true`, non-string values are checked via their `AsString()` form against `blocked_values`. When `false`, only string values are checked. |
| `hash_function` | string | `""` | Replace masked values with a hash instead of asterisks. One of `md5`, `sha1`, `sha3`, `hmac-sha256`, `hmac-sha512`. Empty = asterisk masking. |
| `hmac_key` | string | `""` | Secret key for `hmac-sha256` / `hmac-sha512`. Required and length-validated: ≥ 32 bytes for `hmac-sha256`, ≥ 64 bytes for `hmac-sha512`. Supports env expansion (`${env:REDACTION_HMAC_KEY}`). |
| `summary` | string | `""` | Audit verbosity. Empty (the default) emits **no** audit attributes, same as `silent`. `debug` = key names + counts, `info` = counts only, `silent` = none. |
| `url_sanitizer.enabled` | bool | `false` | Enable URL sanitization. |
| `url_sanitizer.attributes` | []string | `[]` | Attribute keys whose URL values are sanitized. |
| `url_sanitizer.sanitize_span_name` | bool | `true` | Sanitize span names containing `/`. |
| `db_sanitizer.sanitize_span_name` | bool | `true` | Sanitize database query span names. |
| `db_sanitizer.<engine>.enabled` | bool | `false` | Enable per-engine query sanitization. `<engine>` is one of `sql`, `redis`, `valkey`, `memcached`, `mongo`, `opensearch`, `es`. |
| `db_sanitizer.<engine>.attributes` | []string | `[]` | Attributes holding the query/command text for that engine. |

## Processing order and precedence

Attributes are evaluated in a fixed order:

1. **Ignored keys** (`ignored_keys`, `ignored_key_patterns`) — always pass through untouched.
2. **Key allow-listing** — with `allowed_keys` set (and `allow_all_keys: false`), keys not on the list are deleted.
3. **Value matching** — surviving attributes have `blocked_values`/`blocked_key_patterns` applied. If a value matches both `blocked_values` and `allowed_values`, **`allowed_values` wins** and the value is kept.
4. **Mask or hash** — marked values become asterisks, or a hash when `hash_function` is set.

Key allow-listing is fail-closed; value matching is not. Relying only on `blocked_values` (with `allow_all_keys: true`) is a much weaker guarantee — value regexes can miss formats they weren't written for.

## Audit attributes

When `summary` is `debug` or `info`, the processor records what it did as attributes on each record (zero-count attributes are omitted). The default (empty) and `silent` emit nothing:

| Attribute | Verbosity | Description |
|-----------|-----------|-------------|
| `redaction.redacted.keys` | debug | Names of keys removed by allow-listing |
| `redaction.redacted.count` | info | Count of removed keys |
| `redaction.masked.keys` | debug | Names of keys whose values were masked |
| `redaction.masked.count` | info | Count of masked values |
| `redaction.allowed.keys` | debug | Names of keys that passed through |
| `redaction.allowed.count` | info | Count of allowed attributes |
| `redaction.ignored.count` | info | Count of ignored attributes |

For map-formatted log bodies, the same metrics are recorded under a `redaction.body.*` prefix. Key allow-listing and value masking apply to the body too.

# `redaction`: known quirks

## `allowed_keys` fails closed — the footgun

With `allow_all_keys: false` (the default) and an empty `allowed_keys`, **every** attribute is dropped — allow-list semantics, not block-list. If attributes vanish unexpectedly, this is almost always why. Either enumerate the keys you want to keep in `allowed_keys`, or set `allow_all_keys: true` to disable key filtering and rely on value masking alone.

## Evaluation order

Attributes are evaluated in a fixed order:

1. **Ignored keys** (`ignored_keys`, `ignored_key_patterns`) — always pass through untouched.
2. **Key allow-listing** — with `allowed_keys` set (and `allow_all_keys: false`), keys not on the list are deleted.
3. **Value matching** — surviving attributes have `blocked_values`/`blocked_key_patterns` applied. If a value matches both `blocked_values` and `allowed_values`, **`allowed_values` wins** and the value is kept.
4. **Mask or hash** — marked values become asterisks, or a hash when `hash_function` is set.

Key allow-listing is fail-closed; value matching is not. Relying only on `blocked_values` (with `allow_all_keys: true`) is a much weaker guarantee — value regexes can miss formats they weren't written for. See [Configuration](configuration.md#processing-order-and-precedence) for the full precedence detail.

## Masking vs. removal

`blocked_key_patterns` masks the **value** of a matching key but keeps the key itself. To drop a key entirely, leave it off `allowed_keys` with key filtering enabled. `blocked_values` masks values, never removes keys.

## Only string values by default

`blocked_values` matches only string values unless `redact_all_types: true` is set, which checks non-string values via their `AsString()` form. Secrets stored in numeric or boolean values slip through otherwise.

## `summary: debug` writes key names into telemetry

`debug` verbosity records the redacted/masked/allowed **key names** as attributes. If a key name is itself sensitive, keep `summary: info` (counts only) in production and reserve `debug` for short-lived investigation.

## DB sanitizer firing conditions

DB sanitization only fires on spans carrying `db.system`/`db.system.name` with `CLIENT` or `SERVER` span kind. Query text on spans without those attributes — or with a different span kind — is not sanitized. Enable per-engine sanitization explicitly (`db_sanitizer.<engine>.enabled: true`) and point it at the attributes holding the query text.

## Per-signal stability

Traces are Beta; logs and metrics are Alpha. The URL sanitizer and DB sanitizer are newer surface area. Verify masking against representative log/metric data before relying on the processor as your only PII control on those signals.

# `redaction` processor

| | |
|-|-|
| Kind | processor |
| Type | `redaction` |
| Signals | traces, logs, metrics |
| Stability | Beta (traces); Alpha (logs, metrics) |
| Distributions | contrib, k8s |
| Go module | `github.com/open-telemetry/opentelemetry-collector-contrib/processor/redactionprocessor` |
| Upstream README | <https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/redactionprocessor> |

The trace path is Beta (production viable, breaking changes rare). The log and metric paths are Alpha — configuration and behavior may change between releases. The URL sanitizer and DB sanitizer are newer surface area; verify masking against representative data before relying on this processor as your only PII control.

## Description

An allow/block-list redaction processor that enforces a data-governance policy on the attributes flowing through a pipeline. It masks or removes sensitive attribute keys and values across span, log, and metric data points, and can additionally sanitize URLs and database query strings. Two mechanisms compose: allow-listing keys (any key not on `allowed_keys` is deleted, failing closed) and masking values (values matching `blocked_values`, or whose key matches `blocked_key_patterns`, are masked with asterisks or replaced by a hash). On top of attribute redaction it can sanitize URLs (strip high-cardinality path segments and query content) and database queries (SQL, Redis, Valkey, Memcached, MongoDB, OpenSearch, Elasticsearch). It **modifies in place** — it never drops or reroutes telemetry.

## Main use-cases

Use it when:
- Only an explicit allow list of attribute keys is permitted to leave a trust boundary (compliance, multi-tenant egress).
- Known-sensitive values (credit cards, emails, tokens) appear in attribute values and must be masked or hashed before export.
- URLs or DB query strings carry high-cardinality or sensitive content (IDs, tokens, literals) you want sanitized.
- You want an audit trail of what was redacted, masked, or allowed through.

Avoid it when:
- You need general-purpose attribute editing (add/upsert/convert) — use `attributes` or `transform`.
- You want to drop entire spans/logs/metrics — use `filter` (redaction never drops).
- Your redaction logic needs arbitrary conditional transformation — OTTL in `transform` is more expressive (but does not fail closed).

## Related components

- `attributes` — general add/update/delete/hash of attributes (no fail-closed allow list).
- `transform` — OTTL-based conditional transformation, including `replace_pattern` masking (does not fail closed).
- `filter` — drops entire records (redaction never drops).
- `resource` — manipulate resource-level attributes.

## Details

- [Configuration](configuration.md) — open when you need config keys, defaults, processing order/precedence, or the audit-attribute names.
- [Verification](verification.md) — open to confirm masking works with a telemetrygen recipe.
- [Advanced use-cases](advanced.md) — open for allowlist-as-schema, hashing for correlation, URL/DB sanitization, and audit summary usage.
- [Known quirks](quirks.md) — open when attributes vanish unexpectedly, masking behaves differently than expected, or per-signal stability matters.

# `redaction`: verification

See [Verification harness](../../SKILL.md#verification-harness) for how to run this end-to-end.

`redaction` ships in the `contrib` and `k8s` distributions.

Config (`redaction-verify.yaml`):

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
processors:
  redaction:
    allow_all_keys: true
    blocked_values:
      - '4[0-9]{12}(?:[0-9]{3})?'   # Visa-like card numbers
    summary: info
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [redaction]
      exporters: [debug]
```

Generate spans carrying an attribute value that matches the blocked pattern. The span-level attribute flag is `--telemetry-attributes` (resource-level `--otlp-attributes` would attach to the Resource, not the span); the value must be quoted (`key="value"`):

```bash
telemetrygen traces --otlp-insecure --otlp-endpoint localhost:4317 \
  --traces 5 --telemetry-attributes 'cc_number="4111111111111111"'
```

**What proves it worked:** the `debug` exporter shows the matching value masked (replaced with asterisks, or hashed if `hash_function` is set) while non-matching attributes pass through; with `summary: info` the span carries a `redaction.masked.count` audit attribute (and, because nothing is removed under `allow_all_keys: true`, an `redaction.allowed.count`).

# `resource`: advanced use-cases

All actions here operate on **resource** attributes and use the same grammar as the [`attributes`](../attributes/README.md) processor. Actions run top-to-bottom; order matters (see [quirks](quirks.md)).

## Normalize `service.name`

Force a canonical service name regardless of what the SDK set, or backfill one when it's missing:

```yaml
processors:
  resource/canonical-name:
    attributes:
      - key: service.name        # always overwrite
        value: checkout
        action: upsert
  resource/default-name:
    attributes:
      - key: service.name        # only if the SDK left it unset
        value: unknown-service
        action: insert
```

## Set `deployment.environment.name`

Tag all telemetry from a collector instance with its environment (semantic conventions renamed `deployment.environment` → `deployment.environment.name`):

```yaml
processors:
  resource:
    attributes:
      - key: deployment.environment.name
        value: production
        action: upsert
      - key: cloud.region
        value: us-east-1
        action: upsert
```

## Drop high-cardinality / sensitive resource keys

Remove resource attributes that inflate cardinality or leak internal data — by exact key or regex:

```yaml
processors:
  resource:
    attributes:
      - key: process.command_line     # noisy, often unbounded
        action: delete
      - pattern: ^internal\..*        # bulk delete by key regex
        action: delete
```

## Hash identifying resource attributes

Anonymize identifiers while keeping them usable as a stable join key (SHA-256, deterministic):

```yaml
processors:
  resource:
    attributes:
      - key: host.id
        action: hash
      - pattern: ^container\.id$
        action: hash
```

## Rename legacy keys with ordered actions

`extract`/`from_attribute` read; `insert`/`upsert` write; `delete`/`hash` clean up. Read before you delete:

```yaml
processors:
  resource:
    attributes:
      # copy legacy key to the semantic-convention key
      - key: k8s.cluster.name
        from_attribute: k8s-cluster
        action: insert
      # parse a region out of the AZ before anything removes it
      - key: cloud.availability_zone
        pattern: ^(?P<cloud_region>[a-z]+-[a-z]+-\d+)[a-z]$
        action: extract
      # only now drop the legacy key
      - key: k8s-cluster
        action: delete
```

## Named instances

Like every component, `resource` supports the `type/name` form so the same type can be configured more than once and selected per pipeline:

```yaml
processors:
  resource/prod-tags:
    attributes:
      - key: deployment.environment.name
        value: production
        action: upsert
  resource/strip-internal:
    attributes:
      - pattern: ^internal\..*
        action: delete

service:
  pipelines:
    traces/prod:
      processors: [resource/prod-tags]
    traces/egress:
      processors: [resource/strip-internal]
```

## Combine with `resource_detection`

The canonical pattern: `resource_detection` discovers infrastructure attributes from the environment, then `resource` normalizes, renames, and adds static values. Place `resource_detection` **first**:

```yaml
processors:
  resource_detection:
    detectors: [env, system, docker, ec2]
    override: false
  resource:
    attributes:
      - key: deployment.environment.name
        value: production
        action: upsert
      - key: k8s.cluster.name
        from_attribute: k8s-cluster
        action: insert
      - key: k8s-cluster
        action: delete

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resource_detection, resource]
      exporters: [otlp]
```

## Combine with `k8s_attributes`

Let `k8s_attributes` provide dynamic pod/namespace/node metadata from the k8s API, and use `resource` only for static cluster-level values that the API can't supply. Don't hard-code pod-level attributes in `resource` — they're per-pod and dynamic:

```yaml
processors:
  k8s_attributes:
    extract:
      metadata: [k8s.pod.name, k8s.namespace.name, k8s.node.name]
  resource:
    attributes:
      - key: k8s.cluster.name        # static, not pod-specific
        value: prod-us-east
        action: upsert

service:
  pipelines:
    traces:
      processors: [k8s_attributes, resource]
```

## Relationship to `attributes`

`resource` and [`attributes`](../attributes/README.md) share an identical action set (`insert`/`update`/`upsert`/`delete`/`hash`/`extract`/`convert`) and the same `value`/`from_attribute`/`from_context`/`pattern`/`converted_type` fields. The only differences:

- **Scope** — `resource` edits resource attributes; `attributes` edits span/log/datapoint attributes.
- **Matching** — `attributes` has `include`/`exclude` (which can even match on resource attributes); `resource` has none.

If you need to edit resource attributes *only for matching telemetry*, reach for `transform` (with a `where`/OTTL condition), not `resource`. `attributes` can *match* on resource attributes via its `resources:` matcher, but it only edits span/log/datapoint attributes — never the resource — so it can't do conditional resource edits either.

# `resource`: configuration

The processor operates on **resource attributes only** — the attributes attached to the `Resource` that describes the entity producing the telemetry. It does **not** touch span, log-record, or metric-datapoint attributes (use the [`attributes`](../attributes/README.md) processor for those). Its only field is a required ordered list of `attributes` actions; there is **no `include`/`exclude` matching block** (unlike `attributes`), so every resource flowing through the pipeline is processed.

## Root configuration

| Field | Type | Default | Required | Description |
|-------|------|---------|----------|-------------|
| `attributes` | list | none | Yes | Ordered list of actions to perform on the resource's attributes. At least one is required (an empty list is invalid); applied in the order specified. |

There is no default configuration — the processor must be explicitly configured with at least one action.

```yaml
processors:
  resource:
    attributes:
      - key: service.name
        value: my-service
        action: upsert
```

## Action fields

Each entry in `attributes` is one operation. The fields are identical to the `attributes` processor's action fields, but `key`/`from_attribute` refer to **resource** attributes.

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `key` | string | Conditional | Resource attribute key to act on. Required for all actions except `delete`/`hash` when only a `pattern` is given. |
| `action` | string | Yes | One of `insert`, `update`, `upsert`, `delete`, `hash`, `extract`, `convert` (case-insensitive). |
| `value` | any | Conditional | Literal value to set (string/int/double/bool). Required for `insert`/`update`/`upsert` unless `from_attribute`, `from_context`, or `default_value` is used. Mutually exclusive with `from_attribute`/`from_context`. |
| `from_attribute` | string | Conditional | Copy the value from another **resource** attribute. Mutually exclusive with `value` and `from_context`. |
| `from_context` | string | Conditional | Pull the value from request context. Mutually exclusive with `value` and `from_attribute`. See [Context values](#context-values). |
| `default_value` | any | No | Fallback used when the chosen source yields nothing — a missing `from_attribute`/`from_context`, or a `value` that resolved to empty from an unset `${env:…}` expansion (v0.152.0). **Honored only by `insert`/`update`/`upsert`** (silently ignored on other actions). It counts as a value source, so an action with only `default_value` is valid; it prevents the action from being skipped. |
| `pattern` | string | Conditional | Regex (Go syntax). For `delete`/`hash`, matches attribute **keys** to act on. For `extract`, the regex applied to the value (must use named capture groups). |
| `converted_type` | string | Conditional | Target type for `convert`: `int`, `double`, or `string`. Required for `convert`. |

Only **one** of `value`, `from_attribute`, or `from_context` may be set per action; setting more than one fails validation.

### Action types

| Action | Requires | Behavior |
|--------|----------|----------|
| `insert` | `key` + one of `value`/`from_attribute`/`from_context` | Adds the resource attribute only if the key does **not** already exist. No-op if the key exists or the source is missing. |
| `update` | `key` + one of `value`/`from_attribute`/`from_context` | Replaces the value only if the key **already** exists. No-op if the key is absent or the source is missing. |
| `upsert` | `key` + one of `value`/`from_attribute`/`from_context` | Insert if absent, update if present (insert + update). No-op only if the source value is missing. The most common action for static configuration. |
| `delete` | `key` and/or `pattern` | Removes the named key and/or every resource key matching `pattern`. |
| `hash` | `key` and/or `pattern` | Replaces the value(s) of the named key and/or keys matching `pattern` with their SHA-256 hash (lowercase hex string). |
| `extract` | `key` + `pattern` (named groups) | Applies the regex to the **string** value of `key` and creates one resource attribute per named capture group. Source value is left unchanged. Only acts on string values; overwrites existing attributes when group names collide. All groups must be named (`(?P<name>…)`). |
| `convert` | `key` + `converted_type` | Converts the existing value of `key` to `int`/`double`/`string`. No-op if the key is absent; on a failed conversion the original value is kept. |

Example covering several actions, all scoped to the resource:

```yaml
processors:
  resource:
    attributes:
      - key: deployment.environment.name   # always set
        value: production
        action: upsert
      - key: k8s.cluster.name              # rename legacy key
        from_attribute: k8s-cluster
        action: insert
      - key: k8s-cluster                    # drop the old key
        action: delete
      - key: host.id                        # anonymize identifier
        action: hash
      - pattern: ^internal\..*              # bulk delete by key regex
        action: delete
      - key: service.port                   # coerce type
        action: convert
        converted_type: int
```

## No include / exclude matching

Unlike the [`attributes`](../attributes/README.md) processor, `resource` has **no `include`/`exclude` block**. There is no way to scope its actions to a subset of resources by service, attribute, or any other matcher — every resource that passes through the pipeline is processed. To apply resource-attribute edits only to matching telemetry, use `transform` with an OTTL condition. The `attributes` processor can match on resource attributes, but its actions still edit only span/log/datapoint attributes, not the resource.

## Context values

`from_context` pulls a value from request context into a resource attribute. Three source kinds:

- **`client.address`** — the client IP (plain key, no prefix). Only available for network receivers (OTLP, Jaeger, …), not file/push ingestion.
- **`metadata.*`** — gRPC metadata / HTTP headers, e.g. `metadata.x-trace-id`. Requires the receiver to set `include_metadata: true`. Multiple values are joined with `;`.
- **`auth.*`** — data from a server authenticator extension, e.g. `auth.username`, `auth.tenant_id`. Requires an authenticator on the receiver; available keys depend on the authenticator. `auth.` and `metadata.` are separate namespaces — there is no fallback between them.

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        include_metadata: true   # required for metadata.* keys

processors:
  resource/context:
    attributes:
      - key: client.ip
        from_context: client.address
        action: insert
      - key: tenant.id
        from_context: auth.tenant_id
        action: insert
```

# `resource`: known quirks

## Operates only on resource attributes

The single most common confusion: `resource` touches **only resource attributes** — the attributes on the `Resource` that describes the producing entity. It does **not** see span, log-record, or metric-datapoint attributes. A config like this silently does nothing useful, because `http.status_code` is a span attribute, not a resource attribute:

```yaml
# WRONG: http.status_code lives on the span, not the resource
resource:
  attributes:
    - key: http.status_code
      action: delete
```

For span/log/datapoint attributes use the [`attributes`](../attributes/README.md) processor — it has the exact same action grammar. `resource` is the resource-scoped twin of `attributes`.

## No include / exclude matching

`resource` has **no `include`/`exclude` block**. Every resource flowing through the pipeline is processed — there's no way to scope its actions to a subset of resources. If you need conditional application (e.g. only rename a key for one service), use `transform` with an OTTL condition. The `attributes` processor can match on resource attributes, but it cannot edit them; its actions still target span/log/datapoint attributes. Attempting to add an `include:` key to a `resource` config is a configuration error.

## Changes are wholesale across the batch

A `Resource` is shared by every span / log record / metric data point grouped under it in a batch. An action on a resource attribute therefore affects **all** of those records at once — there's no per-record granularity. In the `debug` exporter the attribute appears once, in the **Resource attributes** block, not repeated per span. This is what makes `resource` cheap (one edit per resource, not per record) but also means you can't vary the value by record.

## Action order matters

Actions run top-to-bottom and later actions see the effect of earlier ones. The classic mistake is deleting a key before reading it:

```yaml
# WRONG: source is gone before extract runs
- key: cloud.availability_zone
  action: delete
- key: cloud.availability_zone
  pattern: ^(?P<cloud_region>.+)[a-z]$
  action: extract   # no-op, key already deleted

# RIGHT: read first, clean up last
- key: cloud.availability_zone
  pattern: ^(?P<cloud_region>.+)[a-z]$
  action: extract
- key: cloud.availability_zone
  action: delete
```

Rule of thumb: order as `extract`/read → `insert`/`update`/`upsert`/write → `delete`/`hash`/cleanup.

## `delete` and `hash` semantics

- `delete` and `hash` accept `key` and/or `pattern` (Go regex against attribute **keys**). With both set, the named key plus all pattern matches are affected.
- `hash` replaces the value with its **SHA-256** lowercase-hex digest. It's deterministic (same input → same output) and irreversible; add salt at instrumentation time if you need stronger anonymization. Hashing has a higher per-attribute CPU cost than the simple write actions.
- A `pattern` that matches nothing is a silent no-op — not an error.

## `insert` vs `update` vs `upsert`

- `insert` only writes if the key is **absent** — using it to "change" an existing value silently does nothing.
- `update` only writes if the key **already exists** — using it to "set" a missing value silently does nothing.
- `upsert` always writes (the usual choice for guaranteeing a value).

If an attribute "won't change," check you're not using `insert` on an existing key or `update` on a missing one.

## `extract` only works on string values

`extract` applies its regex to the **string** value of `key`; non-string values are skipped. All capture groups must be **named** (`(?P<name>…)`) — anonymous `()` groups are a config error. Group names become new resource attribute keys, overwriting any existing attribute of the same name.

## `convert` failures are silent

`convert` coerces a value to `int`/`double`/`string`. A failed conversion (e.g. `"abc"` → `int`) keeps the **original** value rather than erroring; the failure is only visible at debug log level. Make sure source values are actually convertible.

## One value source per action

Exactly one of `value`, `from_attribute`, or `from_context` may be set per action. Setting more than one fails validation at startup (`error creating AttrProc due to multiple value sources being set`).

## `from_context` depends on transport

`from_context: metadata.*` requires the receiver to set `include_metadata: true`; `client.address` is only populated for network receivers (OTLP, Jaeger), not file/push ingestion; `auth.*` requires a server authenticator extension. Missing context is a silent no-op — the attribute simply isn't created (unless `default_value` is set).

## Stability

Beta for traces, metrics, and logs; **Development** for profiles. The Beta signals are production-viable; treat profiles support as experimental and subject to breaking changes. Confirm the current stability in the upstream README before relying on it for a given signal.

# `resource` processor

| | |
|-|-|
| Kind | processor |
| Type | `resource` |
| Signals | traces (Beta), metrics (Beta), logs (Beta), profiles (Development) |
| Distributions | core, contrib, k8s |
| Go module | `github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourceprocessor` |
| Upstream README | <https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourceprocessor> |

## Description

Modifies the **resource** attributes of telemetry through an ordered list of `attributes` actions. The resource describes the entity producing the telemetry (a service instance, host, container, …), and its attributes are shared across every span, log record, and metric data point from that resource. Each action names an attribute `key` and one of `insert`, `update`, `upsert`, `delete`, `hash`, `extract`, or `convert`; actions run in the order listed, so later actions see the effect of earlier ones. Values can be literal (`value`), copied from another resource attribute (`from_attribute`), or pulled from request context (`from_context` — client IP, transport metadata, or authenticator data).

It uses the **same action grammar as the [`attributes`](../attributes/README.md) processor**, but scoped to the resource: set `service.name`, add `deployment.environment.name`, rename legacy keys to semantic conventions, or drop noisy resource keys. Unlike `attributes`, it has **no `include`/`exclude` matching** — every resource that flows through the pipeline is processed.

## Main use-cases

Use it when:
- You want to set or standardize resource identity — `service.name`, `service.namespace`, `service.instance.id` — across services (`upsert` static values, `insert` defaults).
- You need to add environment / infrastructure context to all telemetry from a collector (`deployment.environment.name`, `cloud.region`, static tags).
- You want to rename or migrate legacy resource keys to semantic conventions (`insert` from `from_attribute`, then `delete` the old key).
- You need to drop high-cardinality or sensitive resource keys, or `hash` identifying resource attributes (`host.id`, container IDs).

Avoid it when:
- You need to edit **span / log / datapoint** attributes — use [`attributes`](../attributes/README.md) (same action set, telemetry scope).
- You need `include`/`exclude` scoping — `resource` has none; use [`attributes`](../attributes/README.md) (which supports resource-attribute matching) or `transform`.
- You want to auto-discover resource attributes from the environment — use [`resource_detection`](../resource_detection/README.md).
- You want Kubernetes pod/namespace/node metadata — use [`k8s_attributes`](../k8s_attributes/README.md).
- You need conditional logic, math, or cross-field transforms — use `transform` (OTTL).

## Related components

- [`attributes`](../attributes/README.md) — the **same `insert`/`update`/`upsert`/`delete`/`hash`/`extract`/`convert` actions**, but applied to span/log/datapoint attributes instead of the resource.
- [`resource_detection`](../resource_detection/README.md) — auto-detects resource attributes (cloud, host, container, k8s) from the environment; commonly run **before** `resource`, which then normalizes/renames the detected keys.
- [`k8s_attributes`](../k8s_attributes/README.md) — enriches resources with Kubernetes pod/namespace/node metadata from the k8s API; use it for dynamic pod-level enrichment rather than static `resource` values.
- `transform` — OTTL-based mutation that can also edit resource attributes (`context: resource`); a superset of `resource`, at the cost of more verbosity.

## Details

- [Configuration](configuration.md) — the `attributes` action list (every action type and its `key`/`value`/`from_attribute`/`from_context`/`pattern`/`converted_type` fields), scoped to **resource** attributes, and why there is no include/exclude matching.
- [Verification](verification.md) — telemetrygen recipe that sets a resource attribute and confirms it in the **Resource attributes** block of `debug` output.
- [Advanced use-cases](advanced.md) — normalizing `service.name`, setting `deployment.environment.name`, dropping high-cardinality keys, hashing identifiers, ordered actions, named instances, and combining with `resource_detection`/`k8s_attributes`.
- [Known quirks](quirks.md) — operates only on resource attributes (the common `attributes` confusion), no include/exclude, action order, batch-wide effect, `delete`/`hash` semantics, and stability.

# `resource`: verification

See [Verification harness](../../SKILL.md#verification-harness) for how to run this end-to-end.

`resource` ships in the `core`, `contrib`, and `k8s` distributions.

Config (`resource-verify.yaml`) — an `upsert` action that sets (or overwrites) the `deployment.environment.name` **resource** attribute:

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
processors:
  resource:
    attributes:
      - key: deployment.environment.name
        value: prod
        action: upsert
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resource]
      exporters: [debug]
```

Send a known number of traces whose **resource** carries `deployment.environment.name` with a *different* value, so the `upsert` overwrite is observable. telemetrygen sets resource attributes with `--otlp-attributes` (telemetry-level attributes use the separate `--telemetry-attributes` flag). Use `--workers 1` so the `--traces` count (which is **per worker**) is exact:

```bash
telemetrygen traces --otlp-insecure --otlp-endpoint localhost:4317 \
  --workers 1 --traces 5 --otlp-attributes 'deployment.environment.name="staging"'
```

Flags used, all confirmed against the `otel-telemetrygen` skill (`references/flags.md`):

- `--otlp-insecure` — disable transport security (common flag).
- `--otlp-endpoint` — destination, defaults to `localhost:4317` for gRPC (common flag).
- `--workers` — concurrent workers; `1` keeps the count exact (common flag).
- `--traces` — traces **per worker** (trace-specific flag).
- `--otlp-attributes` — **resource**-level attribute as `key="value"` (string values must be quoted; repeatable). This is the resource counterpart of `--telemetry-attributes`, which sets span/log/datapoint attributes that the `resource` processor does **not** touch.

**What proves it worked:** in the `debug` exporter output, the **Resource attributes** block (not the span attributes) shows `deployment.environment.name: Str(prod)` — the value sent (`staging`) has been overwritten by the `upsert` action. The attribute appears once per resource, shared across all spans in the batch.

Dropping `--otlp-attributes` and sending plain traces exercises the **absent-key path** of `upsert`: `deployment.environment.name: Str(prod)` still appears in the Resource attributes because the key wasn't present and got inserted. To test the `insert` **action** itself, swap `action: upsert` → `action: insert` in the config — `insert` writes only when the key is absent and leaves a pre-existing value untouched; `update` writes only when the key already exists.

You can run the same recipe with `telemetrygen logs ... --logs 5` (or `metrics ... --metrics 5`) and a matching pipeline; `--logs`/`--metrics` are likewise per-worker, so keep `--workers 1`. `--otlp-attributes` sets the resource for every signal type.

# `resource_detection` — advanced use-cases

## Detector ordering — first detector to set an attribute wins

Detectors run in list order, and the **first** detector to set a given attribute keeps it; later detectors do not overwrite a value already produced earlier in the same run. This matters when several detectors can emit the same key (e.g. `cloud.platform`, `host.id`). The upstream-recommended AWS order, most-specific first, is:

```yaml
processors:
  resource_detection:
    detectors: [lambda, elastic_beanstalk, eks, ecs, ec2]
```

With `detectors: [eks, ec2]`, `cloud.platform` ends up `aws_eks`; reverse them and you would wrongly get `aws_ec2`. Put the most specific platform detector before the more general one. (`override` is a separate axis — it governs detected-vs-incoming, not detector-vs-detector; see [quirks](quirks.md).)

## Combining `env` with a platform detector

A common pattern: pull deploy-time context from the environment **and** auto-detect the platform.

```yaml
processors:
  resource_detection:
    detectors: [env, ec2]
    timeout: 2s
    override: false
```

`env` reads `OTEL_RESOURCE_ATTRIBUTES` (e.g. `service.name`, `service.version`, `deployment.environment.name` injected by your deploy tooling); `ec2` adds the infra identity. `override: false` keeps anything the SDKs already set.

## Routing an attribute to a specific detector

When two detectors can both produce an attribute, disable it on the one you don't want and enable it on the one you do, rather than relying on order:

```yaml
processors:
  resource_detection:
    detectors: [system, ec2]
    system:
      resource_attributes:
        host.name: { enabled: true }
        host.id:   { enabled: false }
    ec2:
      resource_attributes:
        host.name: { enabled: false }
        host.id:   { enabled: true }
```

Here `host.name` comes from `system` and `host.id` from `ec2`, regardless of list order.

## Chaining with the `resource` processor

`resource_detection` discovers; [`resource`](../resource/README.md) normalizes. Run detection first, then clean up what it found — rename legacy keys, drop high-cardinality ones, or hash identifiers:

```yaml
processors:
  resource_detection:
    detectors: [env, system, ec2]
  resource:
    attributes:
      - key: host.id          # drop a high-cardinality detected key
        action: delete
      - key: deployment.environment.name
        value: production      # force a literal regardless of what env said
        action: upsert

service:
  pipelines:
    traces:
      processors: [resource_detection, resource]
```

## Capturing cloud tags / labels

Several cloud detectors can fold instance tags/labels into resource attributes via a regex allow-list:

```yaml
processors:
  resource_detection:
    detectors: [ec2]
    ec2:
      tags:
        - ^team$
        - ^cost-center$
        - ^env.*$
```

`gcp` and `nova` use `labels:` with the same regex semantics; `azure` uses `tags:` and prefixes matches as `azure.tags.<name>`. EC2 tag retrieval needs the `ec2:DescribeTags` IAM permission (or `tags_from_imds: true` with instance metadata tags enabled).

## Periodic refresh

By default detection runs once at startup. Set `refresh_interval` to re-detect for environments where resource attributes change over the Collector's lifetime (cloud instance tags, k8s labels):

```yaml
processors:
  resource_detection:
    detectors: [ec2]
    refresh_interval: 5m
```

Use this sparingly — see the cardinality and performance caveats in [quirks](quirks.md). The upstream guidance is that a single startup detection suffices for most deployments, and intervals below 1 minute are strongly discouraged.

# `resource_detection` — configuration

## Top-level keys

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| `detectors` | `[]string` | `[env]` | Ordered list of detectors to run. Valid values: `env`, `system`, `docker`, `heroku`, `gcp`, `ec2`, `ecs`, `elastic_beanstalk`, `eks`, `lambda`, `azure`, `aks`, `azureappservice`, `azurecontainerapps`, `consul`, `kubeadm`, `oraclecloud`, `k8s_api`, `k8snode` (deprecated → `k8s_api`), `openshift`, `dynatrace`, `hetzner`, `akamai`, `scaleway`, `upcloud`, `vultr`, `digitalocean`, `nova`, `alibaba_ecs`, `tencent_cvm`, `ibmcloud_vpc`, `ibmcloud_classic`. |
| `override` | `bool` | `true` | Whether detected attributes overwrite resource attributes already present on incoming telemetry. `true` overwrites; `false` keeps existing values and only adds missing ones. |
| `refresh_interval` | `duration` | `0` | If `> 0`, re-runs all detectors on this interval. `0` (default) means detect once at startup and cache. |
| `fail_on_missing_metadata` | `bool` | `false` | For supported network metadata detectors, make an unreachable metadata service a hard error that participates in processor retry instead of producing an empty resource. Added in v0.158.0. |
| `timeout` | `duration` | `5s` | HTTP client timeout for detectors that call a metadata service. Inherited from the embedded `confighttp.ClientConfig`. |
| `retry` | object | enabled (see below) | Global exponential-backoff policy applied to every detector attempt, including periodic refreshes. Added in v0.159.0. |

The component embeds the standard `confighttp.ClientConfig`, so other HTTP client knobs (proxy, TLS, headers) are available for the metadata-service detectors; `timeout` is the one you will usually touch.

> Defaults verified against `factory.go` (`createDefaultConfig`) and `config.go` on contrib v0.160.0: `Detectors: [env]`, `Override: true`, `RefreshInterval: 0`, `FailOnMissingMetadata: false`, client `Timeout: 5s`, and retry enabled with the values below.

## Global retry

| Key | Type | Default | Notes |
|-----|------|---------|-------|
| `retry.enabled` | bool | `true` | `false` makes one detector attempt with no retry. |
| `retry.initial_interval` | duration | `1s` | Delay before the first retry. |
| `retry.randomization_factor` | float | `0.5` | Jitter applied to retry intervals. |
| `retry.multiplier` | float | `2` | Backoff growth factor. |
| `retry.max_interval` | duration | `30s` | Maximum delay between attempts. |
| `retry.max_elapsed_time` | duration | `0` | Overall retry budget. `0` uses `timeout` as the overall session bound; a positive value becomes the session bound and `timeout` limits each attempt. |

With retry enabled, setting both `timeout: 0` and `retry.max_elapsed_time: 0` is rejected because
the detector could block startup indefinitely.

## Per-detector configuration

Each detector has its own config block, keyed by the detector name, squashed into the top level:

```yaml
processors:
  resource_detection:
    detectors: [system, ec2]
    system:
      hostname_sources: ["os"]
    ec2:
      tags:
        - ^team$
```

### Selecting which attributes a detector emits — `resource_attributes`

Most detectors expose a `resource_attributes` map that enables/disables individual attributes. This is how you both **opt in** to attributes that are off by default and **route** an attribute to a specific detector when several can produce it:

```yaml
resource_detection:
  detectors: [system, ec2]
  system:
    resource_attributes:
      host.name:
        enabled: true
      host.id:
        enabled: false   # let ec2 own host.id instead
  ec2:
    resource_attributes:
      host.name:
        enabled: false
      host.id:
        enabled: true
```

The set of attributes and their default-enabled state are per detector — consult the detector's `documentation.md` in the upstream source (e.g. `internal/system/documentation.md`). For the `system` detector, only `host.name` and `os.type` are enabled by default; `host.id`, `host.arch`, `os.description`, the `host.cpu.*` family, `host.ip`/`host.mac`/`host.interface`, and others are opt-in.

## Detector catalog

Every detector reports `cloud.provider`/`cloud.platform` plus a platform-specific set. The most-used ones:

| Detector | Source | Key attributes |
|----------|--------|----------------|
| `env` | `OTEL_RESOURCE_ATTRIBUTES` env var (falls back to deprecated `OTEL_RESOURCE`), `k=v,k=v` format | whatever you put in the variable |
| `system` | host machine | `host.name`, `os.type` (default); `host.id`, `host.arch`, `host.cpu.*`, `os.description`, … (opt-in). `hostname_sources` (`["dns","os"]` default; also `cname`, `lookup`) controls how `host.name` is resolved |
| `docker` | Docker daemon (mount the socket) | `host.name`, `os.type`. Use instead of `system` when the Collector runs as a container; **does not work on macOS** |
| `ec2` | EC2 IMDS | `cloud.*`, `host.id`, `host.name`, `host.type`. Optional `tags` (regex list; needs `ec2:DescribeTags` IAM, or `tags_from_imds: true`). Deprecated per-detector `fail_on_missing_metadata` (use the top-level key), plus `max_attempts`, `max_backoff` |
| `ecs` | ECS Task Metadata Endpoint (V4/V3) | `cloud.*`, `aws.ecs.*` |
| `eks` | EC2 IMDS + k8s/EC2 API fallback | `cloud.*`; `k8s.cluster.name` opt-in (needs `EC2:DescribeInstances`). `node_from_env_var` |
| `lambda` | Lambda runtime env vars | `cloud.*`, `faas.*` |
| `gcp` | GCP metadata server | `cloud.*`, `host.*`, `k8s.cluster.name`, `faas.*` per platform (GCE/GKE/Cloud Run/Functions/App Engine). Optional `labels` (regex list; needs `roles/compute.viewer`). Since v0.161.0, `host.type` on GKE is fetched via the Compute API (needs `compute.instances.get`; disable `host.type` to skip the call) |
| `azure` | Azure IMDS | `cloud.*`, `host.*`. Optional `tags` (regex → `azure.tags.<name>`) |
| `aks` | Azure IMDS | `cloud.*`; `k8s.cluster.name` opt-in |
| `azureappservice` | Azure App Service environment variables | `azure.app_service.instance.id`, `azure.resource_group.name`, `cloud.*`, `deployment.environment.name`, `service.name` |
| `azurefunctions` | Azure Functions environment variables (`FUNCTIONS_WORKER_RUNTIME` / `FUNCTIONS_EXTENSION_VERSION`); v0.162.0+ | `azure.resource_group.name`, `cloud.*` (incl. `cloud.resource_id`), `faas.instance`, `service.name`; `deployment.environment.name` opt-in. Flex Consumption plans lack `azure.resource_group.name` / `cloud.resource_id` |
| `azurecontainerapps` | Azure Container Apps environment variables | `azure.container_app.instance.id`, `cloud.platform`, `cloud.provider`, `service.name` |
| `k8s_api` | k8s API server | node/cluster attrs; requires `node_from_env_var` (default `K8S_NODE_NAME`) and `nodes` RBAC. `auth_type` (`serviceAccount` default / `none` / `kubeConfig`). `k8snode` is the deprecated alias |
| `kubeadm` | k8s API (`kubeadm-config` ConfigMap) | `k8s.cluster.name`, `k8s.cluster.uid`. `auth_type` |
| `openshift` | OpenShift/k8s API | `cloud.*`, `k8s.cluster.name`. `address`, `token`, `tls` |
| `heroku` | Heroku dyno metadata env vars | `service.name`, `service.version`, `service.instance.id`, `heroku.*` |
| `dynatrace` | `dt_host_metadata.properties` file | `dt.entity.host`, `host.name`, `dt.smartscape.host` |
| `consul` | Consul agent | node + exploded `_node_meta` |

Additional metadata-service detectors have detector-specific settings, sometimes including a `labels`/`tags` regex list: `hetzner`, `akamai`, `scaleway`, `upcloud`, `vultr`, `digitalocean`, `nova` (OpenStack), `alibaba_ecs`, `tencent_cvm`, `ibmcloud_vpc` (`protocol: http|https`), `ibmcloud_classic`, `oraclecloud`, `elastic_beanstalk`, `consul`. The per-detector `fail_on_missing_metadata` fields on `upcloud`, `vultr`, `nova`, `alibaba_ecs`, and `tencent_cvm` are deprecated; use the top-level key instead.

For the exact attribute list any detector emits, read its `internal/<detector>/documentation.md` in the upstream source — do not assume.

## Detector migration gates

- `processor.resourcedetection.elasticbeanstalk.EmitV1DeploymentConventions` (Alpha, off by
  default) adds `deployment.environment.name` and `deployment.id` alongside the legacy attributes.
- `processor.resourcedetection.elasticbeanstalk.DontEmitV0DeploymentConventions` (Alpha, off by
  default) removes deprecated `deployment.environment` and `service.instance.id`; it is rejected
  unless the emit-v1 gate is also enabled.
- `processor.resourcedetection.consul.prefixMetaAttributes` (Alpha, off by default) emits Consul
  node metadata as `consul.meta.<key>` instead of unprefixed `<key>`.

# `resource_detection` — known quirks

## Fatal detector errors can stop startup

Fatal errors from a configured detector **propagate and can prevent the Collector from starting** after the configured retry budget is exhausted. Every detector uses the global `retry` block; retries are enabled by default, with the default session bounded by `timeout: 5s`. For supported network metadata detectors, an unreachable service is fatal when `fail_on_missing_metadata` is enabled; otherwise it produces an empty resource. Only configure detectors for the environment you are in.

Since v0.158.0, the top-level `fail_on_missing_metadata` flag (default `false`) controls supported network metadata detectors: enable it when an unreachable metadata service should be a hard error that participates in processor retry rather than returning an empty resource. The older per-detector fields on `ec2`, `upcloud`, `vultr`, `nova`, `alibaba_ecs`, and `tencent_cvm` are deprecated; migrate to the top-level key. The `oraclecloud` detector uses a fast probe: a failed probe returns an empty resource; if the probe succeeds but the follow-up fetch fails, it errors only when `fail_on_missing_metadata: true` and otherwise returns an empty resource.

## `override: true` is the default — and it overwrites SDK-set attributes

By default detected attributes **overwrite** resource attributes already on incoming telemetry. If your SDKs already set `service.name`/`host.name` and the Collector then detects different values for its own host, you can silently replace good per-service identity with the Collector's host identity. For Collectors that receive pre-enriched telemetry (gateway/aggregation tier), prefer `override: false`. The `dynatrace` detector documentation explicitly recommends `override: false` for exactly this reason. (Verified on contrib 0.159.0: with a colliding `service.name`, `override: true` took the detected value and `override: false` kept the incoming one.)

## It describes the Collector's host — not the workload that emitted the telemetry

Every detector reports on the environment **the Collector itself runs in**. The k8s-flavored detectors here (`k8s_api`, `kubeadm`, `openshift`, `eks`/`aks` cluster name) describe the node/cluster the Collector runs on, not the pod that produced each record. For per-record pod/namespace/workload metadata, use [`k8s_attributes`](../k8s_attributes/README.md) instead — the two are complementary, not interchangeable.

## Detector ordering vs `override` are two different things

- **Order** within `detectors: [...]` decides which *detector* wins when two detectors emit the same attribute (first wins).
- **`override`** decides whether *detected* values beat values *already on incoming telemetry*.

They are independent; mixing them up leads to surprising precedence. See [advanced](advanced.md) for the recommended AWS ordering.

## `refresh_interval` multiplies metric cardinality

Changing a resource attribute creates a **new metric time series**. With `refresh_interval` set, every refresh that picks up a changed attribute forks new series — which can sharply raise backend storage cost and query cost. Each refresh also re-runs all detectors, so short intervals add CPU/memory load. Leave it at the default (`0`, detect once) unless attributes genuinely change at runtime; intervals below 1 minute are strongly discouraged.

## Internal telemetry added in v0.157.0

The processor now emits three enabled-by-default, **Development**-stability metrics:

| Metric | Type / unit | Attributes | Meaning |
|--------|-------------|------------|---------|
| `otelcol.resourcedetection.attributes.detected` | asynchronous non-monotonic sum / `{attribute}` | — | Number of attributes in the currently detected resource. |
| `otelcol.resourcedetection.detector.duration` | histogram / `s` | `detector`, `outcome` | Per-detector runtime, including retry backoff. |
| `otelcol.resourcedetection.detector.results` | monotonic sum / `{detection}` | `detector`, `outcome`, `error.type` on failure | Detection outcomes by detector; useful for identifying failing metadata lookups. |

The names and attribute sets may still change because all three metrics are Development stability.

## Docker / container caveats

- Use the **`docker`** detector instead of `system` when the Collector runs as a Docker container, and mount the Docker socket (`/var/run/docker.sock`). Note the official images run as non-root since 0.40.0, so socket access needs group permissions.
- The `docker` detector **does not work on macOS**.
- `host.name` from the `system` detector defaults to the FQDN (`hostname_sources: ["dns", "os"]`); inside a container this is often the container ID. Set `hostname_sources: ["os"]` to use the OS-provided hostname only.

## The `resourcedetection` type is a deprecated alias

The component type was renamed `resourcedetection` → `resource_detection` (with an underscore) in **v0.153.0**. The old `resourcedetection` name still works as a deprecated alias and logs a deprecation warning at startup; new configs should use `resource_detection`. The Go module path and internal package are still `resourcedetectionprocessor`.

## Stability is per-signal

Beta for traces, metrics, and logs; **Development** for profiles. Treat profiles support as experimental.

# `resource_detection` processor

| | |
|-|-|
| Kind | processor |
| Type | `resource_detection` (renamed from `resourcedetection` in v0.153.0; the old name remains as a deprecated alias) |
| Signals | traces (Beta), metrics (Beta), logs (Beta), profiles (Development) |
| Distributions | contrib, k8s |
| Go module | `github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor` |
| Upstream README | <https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor> |

## Description

Detects **resource** attributes from the environment the Collector runs in — host machine, cloud provider metadata services, container runtimes, Kubernetes, and environment variables — and merges them onto every span, metric data point, and log record passing through the pipeline, in a form that follows the [OpenTelemetry resource semantic conventions](https://github.com/open-telemetry/semantic-conventions/tree/main/docs/resource). You enable detection by listing one or more named **detectors** in `detectors`; each detector contributes the attributes it knows how to find (e.g. `system` → `host.name`/`os.type`, `ec2` → `cloud.provider`/`cloud.region`/`host.id`, `gcp` → `cloud.platform`/`k8s.cluster.name`).

A global `retry` block applies to detector failures. Azure Container Apps (`azurecontainerapps`)
Azure App Service (`azureappservice`, v0.160.0+), and Azure Functions (`azurefunctions`, v0.162.0+) are available as detectors. Retries are
enabled by default, with the overall default session bounded by the 5s HTTP timeout.

Detection normally runs **once at startup** and the results are cached. Detectors run in the order listed and, by default (`override: true`), the detected values overwrite any existing resource attributes on incoming telemetry; set `override: false` to keep values already present (the safer choice when telemetry arrives pre-enriched from SDKs or upstream Collectors). Unlike [`resource`](../resource/README.md), it does not take an explicit attribute action list — it *discovers* attributes rather than setting literal values, and is commonly chained **before** `resource` so you can rename or trim what it found.

## Main use-cases

Use it when:
- You want host identity on telemetry from a Collector running on a VM or bare metal — `host.name`, `host.id`, `os.type`, `host.arch` (the `system` detector).
- You run the Collector on a cloud platform and want `cloud.*` / `k8s.cluster.name` / `faas.*` attributes auto-populated (`ec2`, `gcp`, `azure`, `aks`, `eks`, … detectors).
- You want to pull resource attributes from `OTEL_RESOURCE_ATTRIBUTES` into telemetry that lacks them (the `env` detector — the default).
- You need infra context attached at the Collector rather than configured into every SDK.

Avoid it when:
- You want to set **literal/static** resource values (`service.name`, `deployment.environment.name`) — use [`resource`](../resource/README.md); detection discovers from the environment, it does not invent values.
- You need per-pod Kubernetes metadata keyed off each record's source pod — use [`k8s_attributes`](../k8s_attributes/README.md); the `k8s_api`/`kubeadm`/`openshift` detectors here describe the **node/cluster the Collector runs on**, not the workload that emitted the telemetry.
- You need to edit span/log/datapoint attributes — use [`attributes`](../attributes/README.md) or `transform`.

## Related components

- [`resource`](../resource/README.md) — sets/renames resource attributes from an explicit action list. Run `resource_detection` first to discover, then `resource` to normalize or drop what was found.
- [`k8s_attributes`](../k8s_attributes/README.md) — per-record pod/namespace/workload enrichment from the k8s API. `resource_detection`'s k8s detectors describe the host/node the Collector itself runs on; the two are complementary.
- [`attributes`](../attributes/README.md) — edits span/log/datapoint attributes (telemetry scope) rather than the resource.

## Details

- [Configuration](configuration.md) — top-level keys (`detectors`, `override`, `refresh_interval`, `retry`, HTTP client settings), the per-detector `resource_attributes` enable/disable mechanism, and the full detector catalog with the attributes each populates.
- [Verification](verification.md) — telemetrygen recipe using the `env` and `system` detectors, confirming detected attributes in the **Resource attributes** block of `debug` output.
- [Advanced use-cases](advanced.md) — detector ordering, combining detectors, `override: false` for pre-enriched telemetry, chaining with `resource`, cloud tag/label capture, and `refresh_interval`.
- [Known quirks](quirks.md) — a failed detector stops the Collector from starting, the `override` footgun, "describes the Collector's host, not the workload", `refresh_interval` cardinality cost, v0.157.0 internal telemetry, the `resourcedetection` deprecated alias, and Docker/cloud environment caveats.

# `resource_detection` — verification

Confirms that detected attributes appear in the **Resource attributes** block of telemetry. Uses the two detectors that work without a cloud/metadata service: `env` (reads `OTEL_RESOURCE_ATTRIBUTES` from the Collector's environment) and `system` (reads the host machine).

Verified on `otel/opentelemetry-collector-contrib:0.159.0` with telemetrygen v0.159.0
(2026-08-27). This `env` + `system` recipe's schema is unchanged through v0.160.0; it does not
exercise `azureappservice`.

See the `otel-collector` SKILL.md **Verification harness** note: this config is a minimal repro and omits `memory_limiter` and other production scaffolding on purpose.

## Config (`rdcol.yaml`)

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  resource_detection:
    detectors: [env, system]
    override: true
    system:
      hostname_sources: ["os"]

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resource_detection]
      exporters: [debug]
```

## Run

The `env` detector reads the **Collector's** environment, so set `OTEL_RESOURCE_ATTRIBUTES` on the container (not on telemetrygen):

```bash
docker run --rm --name rdcol -p 4317:4317 \
  -e OTEL_RESOURCE_ATTRIBUTES='deployment.environment.name=prod,my.detector.test=hello' \
  -v "$PWD/rdcol.yaml:/etc/otelcol-contrib/config.yaml" \
  otel/opentelemetry-collector-contrib:0.159.0

# in another shell — telemetrygen emits 2 spans per trace
go run github.com/open-telemetry/opentelemetry-collector-contrib/cmd/telemetrygen@v0.159.0 \
  traces --otlp-insecure --otlp-endpoint localhost:4317 --traces 5 --workers 1
```

## Expected output

The `debug` exporter prints a **Resource attributes** block carrying both the `env`-detected and `system`-detected attributes:

```
Resource attributes:
     -> service.name: Str(telemetrygen)
     -> deployment.environment.name: Str(prod)   # from env detector
     -> my.detector.test: Str(hello)             # from env detector
     -> host.name: Str(<container hostname>)      # from system detector
     -> os.type: Str(linux)                       # from system detector
```

10 spans total reach the exporter (5 traces × 2 spans). Tear down with `docker rm -f rdcol`.

## Verified result

The `env` detector populated `deployment.environment.name=prod` and `my.detector.test=hello` from `OTEL_RESOURCE_ATTRIBUTES`; the `system` detector populated `host.name` (the container ID, via `hostname_sources: ["os"]`) and `os.type=linux`. All 10 spans carried the merged resource, and telemetrygen's own `service.name=telemetrygen` was kept (neither detector produces it).

`override` was confirmed separately by setting `OTEL_RESOURCE_ATTRIBUTES=service.name=fromenv` (which collides with telemetrygen's `service.name=telemetrygen`): with `override: true` the resource showed `service.name=fromenv` (detector wins); with `override: false` it showed `service.name=telemetrygen` (incoming value preserved).

# `routing`: advanced use-cases

Examples below use context-qualified paths (`resource.attributes[...]`, `span.…`, `log.…`), the recommended style since v0.156.0 — the `context` field is inferred and omitted. Bare paths with an explicit `context` still work for backward compatibility.

## Named instances for independent dimensions

Under the default `action: move`, each piece of telemetry matches at most one route, so a single `routing` instance can't route on two independent dimensions at once. Use one named instance per dimension and list both as exporters of the input pipeline — each gets an independent copy of the data:

```yaml
connectors:
  routing/tenant:
    table:
      - condition: resource.attributes["tenant"] == "acme"
        pipelines: [logs/acme]
      - condition: resource.attributes["tenant"] == "ecorp"
        pipelines: [logs/ecorp]
  routing/region:
    table:
      - condition: resource.attributes["region"] == "us-east"
        pipelines: [logs/east]
      - condition: resource.attributes["region"] == "us-west"
        pipelines: [logs/west]

service:
  pipelines:
    logs/in:
      receivers: [otlp]
      exporters: [routing/tenant, routing/region]   # both evaluate independently
```

## Fan-out to multiple pipelines in one route

To send the same matched telemetry to several pipelines, list them all under one route — telemetry is copied to each:

```yaml
table:
  - condition: span.end_time_unix_nano - span.start_time_unix_nano > 5000000000   # > 5s
    pipelines: [traces/slow, traces/all]       # slow traces go to BOTH
```

## Multi-tenant routing on request metadata

Route on HTTP headers or gRPC metadata using the `otelcol.client.metadata` (HTTP/client) and `otelcol.grpc.metadata` (gRPC) paths — valid in every signal context. Metadata values are lists, so index with `[0]`:

```yaml
connectors:
  routing:
    default_pipelines: [logs/default]
    table:
      - condition: otelcol.client.metadata["X-Tenant"][0] == "acme"
        pipelines: [logs/acme]
      - condition: otelcol.client.metadata["X-Tenant"][0] == "ecorp"
        pipelines: [logs/ecorp]
```

For gRPC traffic use lowercase keys: `otelcol.grpc.metadata["x-tenant"][0]`. The older `request["X-Tenant"]` / `context: request` form still works but is **deprecated as of v0.156.0** (see [quirks](quirks.md)).

## Severity and environment routing

```yaml
# logs: cheap storage for low severity, expensive for errors
table:
  - condition: log.severity_number < SEVERITY_NUMBER_ERROR
    pipelines: [logs/cheap]
  - condition: log.severity_number >= SEVERITY_NUMBER_ERROR
    pipelines: [logs/important]
```

```yaml
# traces: production to one backend, everything else to the default
connectors:
  routing:
    default_pipelines: [traces/dev]
    table:
      - condition: resource.attributes["deployment.environment"] == "production"
        pipelines: [traces/prod]
```

Order routes from specific to general — the first match wins, so a broad route placed first will shadow a narrower one below it.

## `action: copy` vs the default `move`

`move` (the default) removes matched data from any further evaluation, so it never also reaches later routes or `default_pipelines`. `copy` leaves matched data available downstream — use it for an "archive everything, then route" pattern where the same data must hit both the archive route and a more specific one:

```yaml
table:
  - condition: resource.attributes["archive"] == "true"
    action: copy                       # copied to archive, still evaluated below
    pipelines: [logs/archive]
  - condition: log.severity_number >= SEVERITY_NUMBER_ERROR
    pipelines: [logs/errors]           # default move: errors leave here
```

Because `move` is already the default, you rarely set it explicitly; reach for `copy` only when a match must not be terminal.

## Migrating from `routingprocessor`

The connector replaces the `routingprocessor`, which has since been removed from contrib entirely — the OLD config below no longer loads on current releases. The processor sat inside one pipeline and routed to **exporters** via a `from_attribute` + `value` table; the connector bridges pipelines and routes to **pipelines** via OTTL.

```yaml
# OLD (routingprocessor)
processors:
  routing:
    from_attribute: X-Tenant
    table:
      - value: acme
        exporters: [otlp/acme]
service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [routing]
      exporters: [otlp/acme]
```

```yaml
# NEW (routing connector)
connectors:
  routing:
    default_pipelines: [logs/default]
    table:
      - condition: otelcol.client.metadata["X-Tenant"][0] == "acme"
        pipelines: [logs/acme]
service:
  pipelines:
    logs/in:   {receivers: [otlp], exporters: [routing]}
    logs/acme: {receivers: [routing], exporters: [otlp/acme]}
    logs/default: {receivers: [routing], exporters: [otlp/default]}
```

## Migrating from `match_once`

`match_once` was removed in v0.120.0; each item now matches at most one route (under the default `move`). To reproduce the old `match_once: false` (multi-match) behavior:

- **Parallel routers** — split independent dimensions into separate named instances (see above). Each gets its own copy, so an item can match in both. Simplest when no `default_pipelines` is needed.
- **Enumerate combinations** — for a small fixed set, write one route per combination and list all target pipelines in it (e.g. `condition: resource.attributes["env"] == "prod" and resource.attributes["region"] == "east"` → `pipelines: [logs/prod, logs/east]`). Practical only for a handful of dimensions.

Prefer parallel routers; enumeration grows combinatorially.

# `routing`: configuration

## Typical config

```yaml
connectors:
  routing:
    error_mode: ignore
    default_pipelines: [logs/default]
    table:
      # context inferred from the qualified path — no `context` field needed
      - condition: resource.attributes["tenant"] == "acme"
        pipelines: [logs/acme]
      - condition: log.severity_number >= SEVERITY_NUMBER_ERROR
        pipelines: [logs/errors]

service:
  pipelines:
    logs/in:
      receivers: [otlp]
      exporters: [routing]            # connector as exporter of the input pipeline
    logs/acme:
      receivers: [routing]            # connector as receiver of an output pipeline
      exporters: [otlphttp/acme]
    logs/errors:
      receivers: [routing]
      exporters: [otlphttp/errors]
    logs/default:
      receivers: [routing]
      exporters: [otlphttp/default]
```

## Top-level configuration reference

| Key | Type | Default | Required | Meaning |
|-----|------|---------|----------|---------|
| `table` | array of RoutingTableItem | — | **yes** (≥1 route) | The routing table. Routes are evaluated in order; under the default `action: move` each piece of telemetry matches at most one route. |
| `default_pipelines` | array of pipeline IDs | none | no | Pipelines for telemetry that matches no route. **If unset, unmatched telemetry is dropped.** |
| `error_mode` | string | `ignore` | no | How OTTL evaluation errors are handled: `propagate`, `ignore`, or `silent` (see below). The `connector.routing.defaultErrorModeIgnore` gate is Beta and enabled by default in v0.157.0; disabling it restores `propagate`. |

## RoutingTableItem reference

| Key | Type | Default | Required | Meaning |
|-----|------|---------|----------|---------|
| `context` | string | inferred from path | no | OTTL context the expression runs in: `resource`, `span`, `metric`, `datapoint`, `log`, `otelcol`, or the deprecated `request`. Usually omit it — the context is inferred from context-qualified paths (see below). If set, it takes precedence over inference and must be valid for the pipeline's signal (see table below). |
| `statement` | string | — | one of `statement`/`condition` | OTTL statement: `route() where <expr>` to route unchanged, or `<editor> where <expr>` (e.g. `delete_key(resource.attributes, "k") where …`) to mutate matched data in the same pass. **Mutually exclusive with `condition`.** Not allowed for the deprecated `request` context. |
| `condition` | string | — | one of `statement`/`condition` | OTTL boolean expression (no `route() where`). **Mutually exclusive with `statement`.** Required for the deprecated `request` context. |
| `action` | string | `move` | no | `move` (default) removes matched data from further route evaluation and `default_pipelines`; `copy` leaves it available to later routes and the default. |
| `pipelines` | array of pipeline IDs | — | **yes** (≥1) | Output pipelines this route forwards matching telemetry to. Listing several fans the data out to all of them. |

Exactly one of `statement` / `condition` must be set per route — setting both, or neither, fails validation.

## Context inference (recommended)

Since v0.156.0 the connector infers the `context` from **context-qualified paths** in the condition/statement, so the `context` field is usually unnecessary. Prefix each path with its context and drop the field:

```yaml
table:
  - condition: resource.attributes["env"] == "prod"     # -> resource context
    pipelines: [logs/prod]
  - condition: span.attributes["http.method"] == "GET"  # -> span context
    pipelines: [traces/http]
  - condition: log.severity_text == "ERROR"             # -> log context
    pipelines: [logs/errors]
```

The inferred (or explicit) context must be compatible with the pipeline signal — e.g. a `span` context only works in a traces pipeline. A condition with no OTTL paths (e.g. the literal `"true"`) can't be inferred, so it still needs an explicit `context`. Bare paths with an explicit `context` (`context: resource` + `attributes["env"]`) remain valid for backward compatibility.

## Supported contexts by signal

A route's context (inferred or explicit) must be supported by the signal of the pipeline it routes:

| Signal | Supported contexts |
|--------|--------------------|
| Traces | `resource`, `span`, `otelcol` (+ deprecated `request`) |
| Metrics | `resource`, `metric`, `datapoint`, `otelcol` (+ deprecated `request`) |
| Logs | `resource`, `log`, `otelcol` (+ deprecated `request`) |

`otelcol` (request metadata) is valid in every signal. Coarser contexts are cheaper: `resource` evaluates once per resource bundle, while `span`/`metric`/`log` evaluate per item and `datapoint` per data point. Use the coarsest context that expresses your condition.

## `statement` vs `condition`

Both express the same boolean test in OTTL; they differ only in syntax and which OTTL functions you can call:

```yaml
# condition: a bare boolean expression
- condition: resource.attributes["env"] == "prod"
  pipelines: [logs/prod]

# statement: <editor> where <expr> — the editor (e.g. delete_key) is the statement's
#            function; it runs against matched data in the same pass, then the data is
#            routed. Plain routing without mutation is `route() where <expr>`. You cannot
#            chain an editor onto route() with `and` — that is invalid OTTL syntax.
- statement: delete_key(resource.attributes, "internal.debug") where resource.attributes["env"] == "prod"
  pipelines: [logs/prod]
```

Use `condition` for a plain match (recommended for clarity); reach for `statement` only when you want to mutate the data in the same pass. Beyond the standard converters, the connector exposes only the `delete_key` and `delete_matching_keys` editors. For the full expression language — paths, converters, editors — see the `otel-ottl` skill.

## Request metadata routing (`otelcol.*`)

To route on HTTP headers or gRPC metadata, use the `otelcol.client.metadata` (HTTP/client) and `otelcol.grpc.metadata` (gRPC) paths. Metadata values are lists, so index the first element with `[0]`. These paths are valid in every signal context and, unlike the deprecated `request` context, support the full OTTL grammar (`and`/`or`, statements, etc.):

```yaml
- condition: otelcol.client.metadata["X-Tenant"][0] == "acme"
  pipelines: [logs/acme]
- condition: otelcol.grpc.metadata["x-tenant"][0] == "acme"
  pipelines: [logs/acme]
```

gRPC metadata keys are lowercased, so use lowercase keys for gRPC traffic. The receiver must propagate request metadata (the `otlp` receiver does).

### Deprecated `request` context

The `request` context is **deprecated as of v0.156.0** (a warning is logged when it is used) — prefer `otelcol.client.metadata` / `otelcol.grpc.metadata`. It requires `condition` (a `statement` is rejected) and supports only a single simple comparison — `==` or `!=`, no `and`/`or`:

```yaml
- context: request
  condition: request["X-Tenant"] == "acme"
  pipelines: [logs/acme]
```

## `error_mode`

Controls what happens when an OTTL expression fails to evaluate (missing attribute, type mismatch, etc.):

| Value | Behavior |
|-------|----------|
| `propagate` | The connector returns an error and the **payload is dropped** from the collector. |
| `ignore` | The error is logged and the payload is sent to `default_pipelines`. |
| `silent` | Same as `ignore`, but the error is not logged. |

Prefer `ignore` in production so a transient OTTL error doesn't drop data. It is the default in v0.157.0 because `connector.routing.defaultErrorModeIgnore` is Beta and enabled by default. Set `error_mode` explicitly in reusable configs that also target v0.156.0 or earlier, or deployments that disable the gate.

## Pipeline wiring

`routing` is a connector: it bridges pipelines rather than living inside one. Wire it as the **exporter** of the input pipeline and the **receiver** of each output pipeline (including the default). Pipeline definition order does not matter — only that the connector name appears in both roles:

```yaml
service:
  pipelines:
    logs/in:                # input
      receivers: [otlp]
      exporters: [routing]
    logs/acme:              # output
      receivers: [routing]
      exporters: [otlphttp/acme]
    logs/default:           # default fallback output
      receivers: [routing]
      exporters: [otlphttp/default]
```

Every pipeline ID referenced in `table[].pipelines` and in `default_pipelines` must exist as a pipeline that lists `routing` as a receiver, otherwise the collector fails to start.

# `routing`: known quirks

## Unmatched telemetry is dropped without `default_pipelines`

If a piece of telemetry matches no route and `default_pipelines` is not set, it is **silently dropped** — no error, no log. This is the most common cause of "where did my data go?" with this connector. Always configure `default_pipelines` (even just a catch-all archive or a `debug` pipeline) unless dropping unmatched data is genuinely what you want.

## `error_mode: propagate` drops the payload on OTTL errors

With `error_mode: propagate`, any OTTL evaluation error — a missing attribute, a type mismatch — makes the connector return an error and the **whole payload is dropped** from the collector. In v0.157.0 the default is `ignore`: the `connector.routing.defaultErrorModeIgnore` gate is Beta and enabled by default. `ignore` logs the error and sends the payload to `default_pipelines` instead — so it only actually rescues data when `default_pipelines` is set; without it the errored payload still has nowhere to go. Disabling the gate restores the old `propagate` default. Guard fragile conditions with nil checks (`resource.attributes["k"] != nil and resource.attributes["k"] == "v"`) so a missing key doesn't error in the first place.

## `action` defaults to `move`, so matched data skips `default_pipelines`

The default `action` is `move`: a matched item is removed from further route evaluation and does **not** also reach `default_pipelines` or later routes. Use `action: copy` when you want matched data to stay available for subsequent routes and the default (e.g. an archive-everything route placed first).

## `statement` and `condition` are mutually exclusive

Each route needs **exactly one** of `statement` or `condition`. Setting both, or neither, fails config validation. `statement` is the `route() where <expr>` form — or `<editor> where <expr>` (e.g. `delete_key(...) where …`) when you also want to mutate matched data in the same pass (you can't chain an editor onto `route()` with `and`); `condition` is a bare boolean expression. Pick one per route.

## `request` context is deprecated and has a limited grammar

The `request` context is **deprecated as of v0.156.0** (a warning is logged when it is used) — prefer the `otelcol.client.metadata["key"][0]` (HTTP) / `otelcol.grpc.metadata["key"][0]` (gRPC) paths, which are valid in every signal context and support the full OTTL grammar. The legacy `request` context supports only a single `==` or `!=` comparison — no `and`/`or`, and a `statement` is rejected (it must be `condition`). gRPC metadata keys are lowercased, so use lowercase keys (`request["x-tenant"]` / `otelcol.grpc.metadata["x-tenant"][0]`) for gRPC traffic. The receiver must actually propagate request metadata (the `otlp` receiver does; file-based receivers don't).

## Item-level contexts can split a Resource bundle across pipelines

`span`, `metric`, `datapoint`, and `log` contexts evaluate per item, so a single incoming ResourceSpans/ResourceMetrics/ResourceLogs bundle can be torn apart — some items to one pipeline, others to another (or to the default). This is usually intended, but it means downstream pipelines may receive partial resource bundles. Use `resource` context when you want whole bundles kept together.

## Validation errors → fixes

| Error | Cause | Fix |
|-------|-------|-----|
| `the routing table is empty` | `table` missing or empty | Add at least one route. |
| `no condition or statement provided` | route has neither | Add one of `statement`/`condition`. |
| `both condition and statement provided` | route has both | Remove one. |
| `no pipelines defined` | route missing `pipelines` | Add at least one pipeline ID. |
| `invalid Action string: <value>` (inside a `cannot unmarshal the configuration` error) | `action` set to something other than `move`/`copy` | Use `move` or `copy` (or omit for the `move` default). |
| `"request" context requires a 'condition'` | `request` route used a `statement` | Use `condition` for `request` context (or migrate to `otelcol.*` paths). |
| `invalid context: <name>` | unsupported/typo context | Use `resource`, `span`, `metric`, `datapoint`, `log`, `otelcol`, or the deprecated `request` (and one valid for the signal). |

## `match_once` was removed in v0.120.0

`match_once` (deprecated v0.116.0, removed v0.120.0) is gone — each item now matches **at most one route**. Configs using `match_once: false` no longer load. To fan out, list multiple pipelines in one route; to route on independent dimensions, use parallel named instances (see [advanced](advanced.md#migrating-from-match_once)).

## Stability caveats

All three signal pairs (traces, metrics, logs) are **Alpha**. Config surface and behavior can shift between releases — confirm against the upstream README for your exact collector version. The connector replaces the `routingprocessor`, which has been removed from contrib (v0.156.0 rejects it as an unknown type); existing configs must migrate to the connector form.

# `routing` connector

| | |
|-|-|
| Kind | connector |
| Type | `routing` |
| Signals | traces→traces (Alpha), metrics→metrics (Alpha), logs→logs (Alpha) — same-signal routing, no transformation |
| Distributions | contrib, k8s |
| Go module | `github.com/open-telemetry/opentelemetry-collector-contrib/connector/routingconnector` |
| Upstream README | <https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/routingconnector> |

## Description

Routes telemetry to different pipelines based on OTTL conditions, keeping the **same signal type** (traces→traces, metrics→metrics, logs→logs) — it forwards data unchanged, it does not transform it. As a connector it is wired as the **exporter** of an input pipeline and the **receiver** of one or more output pipelines; the input pipeline hands its data to `routing`, which then dispatches it to the matching output pipelines.

The decision is driven by an ordered `table` of routes, each an OTTL `condition` (or a `route() where …` `statement`) evaluated in a `context`. Since v0.156.0 the context is **inferred** from context-qualified paths (`resource.attributes[...]`, `span.…`, `log.…`, `metric.…`, `datapoint.…`, `otelcol.…`), so the `context` field is usually omitted; the deprecated `request` context is the exception. Routes are evaluated **in order** and, under the default `action: move`, each piece of telemetry matches **at most one route** — to fan out to several pipelines, list them all under that route's `pipelines`, or use `action: copy` to keep matched data available to later routes. Telemetry that matches no route goes to `default_pipelines`, or is **dropped** if none is set. This replaces the `routingprocessor`, which has since been removed from contrib. The OTTL expression language itself lives in the `otel-ottl` skill; this page documents the connector's config surface.

## Main use-cases

Use when:
- You need to send telemetry to different backends/pipelines by attribute (tenant, region, environment, severity, service).
- You want a fan-out: the same data to multiple pipelines based on one condition.
- You need request-level routing from HTTP headers / gRPC metadata (multi-tenant isolation before parsing telemetry).
- You want to split high-priority from low-priority data (e.g. errors to expensive storage, the rest to cheap storage).

Avoid when:
- You only need to **drop** telemetry — use `filter`.
- You need to **modify** telemetry — use `transform`.
- You need to change signal type (e.g. spans→metrics) — that is what transforming connectors like `spanmetrics` are for.

## Related components

- `filter` — drops telemetry via OTTL; use it when the goal is to remove data, not redirect it.
- `transform` — mutates telemetry via OTTL; use it to rewrite data rather than route it.
- `otel-ottl` skill — the `condition`/`statement` expression language, contexts, and functions used in the routing `table`.
- `routingprocessor` — the removed processor this connector replaces (see [Known quirks](quirks.md) for migration).

## Details

- [Configuration](configuration.md) — full config tables for top-level keys and `table` items, context inference, supported contexts per signal, `otelcol.*` request-metadata routing, statement-vs-condition rules, `error_mode` semantics, and the connector pipeline wiring.
- [Verification](verification.md) — telemetrygen recipe that splits telemetry across two pipelines, each writing to its own `file` exporter, proving routing and the default fallback.
- [Advanced use-cases](advanced.md) — named instances, fan-out, multi-tenant `otelcol.*` routing, severity/environment routing, `action: copy` vs the default `move`, and migrating from `routingprocessor` / the removed `match_once`.
- [Known quirks](quirks.md) — dropped unmatched data, `error_mode` data loss, `move`-vs-`copy` defaults, mutual exclusivity, the deprecated `request` context, split Resource bundles, a validation-error→fix table, and stability caveats.

# `routing`: verification

See [Verification harness](../../SKILL.md#verification-harness) for how to run this end-to-end.

`routing` ships in the `contrib` and `k8s` distributions, so a stock contrib collector can run this.

The idea: send logs carrying a resource attribute `tenant`, route the ones where `tenant == "acme"` to one pipeline and let everything else fall through to `default_pipelines`. Each output pipeline writes to its **own** `file` exporter, so the split is visible as two distinct files. Run telemetrygen twice — once with the matching attribute, once without — and check which file each batch lands in.

Config (`routing-verify.yaml`):

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
connectors:
  routing:
    error_mode: ignore
    default_pipelines: [logs/default]
    table:
      - context: resource
        condition: attributes["tenant"] == "acme"
        pipelines: [logs/match]
exporters:
  file/match:
    path: /tmp/routed-match.json
  file/default:
    path: /tmp/routed-default.json
service:
  pipelines:
    logs/in:
      receivers: [otlp]
      exporters: [routing]
    logs/match:
      receivers: [routing]
      exporters: [file/match]
    logs/default:
      receivers: [routing]
      exporters: [file/default]
```

Send a matching batch and a non-matching batch — see the `otel-telemetrygen` skill:

```bash
# matches the route -> /tmp/routed-match.json
telemetrygen logs --otlp-insecure --otlp-endpoint localhost:4317 \
  --logs 3 --workers 1 --otlp-attributes 'tenant="acme"'

# no matching attribute -> falls through to /tmp/routed-default.json
telemetrygen logs --otlp-insecure --otlp-endpoint localhost:4317 \
  --logs 2 --workers 1 --otlp-attributes 'tenant="other"'
```

Flags confirmed against the `otel-telemetrygen` skill (`references/flags.md`): `--otlp-insecure` (bool), `--otlp-endpoint` (string), `--logs` (int, **per worker**; ignored if `--duration` is set, so no `--duration` here, and `--workers 1` makes the count exact), and `--otlp-attributes` (`key="value"`, **resource-level** — the right scope, since the route uses `context: resource` and reads `attributes["tenant"]` off the resource). No flag here invents a value.

**What proves it worked:** `/tmp/routed-match.json` holds the 3 records sent with `tenant="acme"` and `/tmp/routed-default.json` holds the 2 records sent with `tenant="other"`. telemetrygen sends nothing on a `tenant` attribute by default, so the split is unambiguous — records only reach `file/match` when the route condition matched, and only reach `file/default` via the unmatched fallback. The `file` exporter writes one newline-delimited JSON object per export, so count the log records inside each file (the resource-level `tenant` attribute appears once per batch, not once per record):

```bash
# each batch is one JSON line carrying its logRecords; count records, not the resource attr
for f in /tmp/routed-match.json /tmp/routed-default.json; do
  echo "$f: $(grep -o '"timeUnixNano"' "$f" | wc -l | tr -d ' ') records"
done
# => routed-match.json: 3 records, routed-default.json: 2 records
```

> Verified end-to-end on `otel/opentelemetry-collector-contrib:0.152.0`: the `tenant="acme"` batch landed only in `routed-match.json` (3 records) and the `tenant="other"` batch only in `routed-default.json` (2 records). Exact JSON field rendering can shift between collector versions; what matters is the matching records land in `routed-match.json` and the rest in `routed-default.json` — and that removing `default_pipelines` would make the non-matching batch vanish entirely (dropped), the behavior [quirks](quirks.md) warns about.

# `tail_sampling`: advanced use-cases

## `and` — require multiple conditions

Sample only traces that are **both** an error **and** slow:

```yaml
- name: slow-errors
  type: and
  and:
    and_sub_policy:
      - name: has-error
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: is-slow
        type: latency
        latency:
          threshold_ms: 1000
```

`and` accepts all policy types except `composite`, `and`, and `drop`. A `not` sub-policy can be nested inside `and_sub_policy` (and inside `drop_sub_policy`), letting you AND a negated condition without a separate top-level policy.

## `composite` — tiered sampling with rate allocation

Allocate a total spans-per-second budget across policies in priority order:

```yaml
- name: tiered
  type: composite
  composite:
    max_total_spans_per_second: 1000
    policy_order: [errors, slow, sample-rest]
    composite_sub_policy:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow
        type: latency
        latency:
          threshold_ms: 1000
      - name: sample-rest
        type: always_sample
    rate_allocation:
      - policy: errors
        percent: 60
      - policy: slow
        percent: 30
      - policy: sample-rest
        percent: 10
```

Each sub-policy gets `(percent/100) * max_total_spans_per_second`; the first matching policy under budget samples the trace. Put an `always_sample` last to use leftover capacity.

A sub-policy omitted from `rate_allocation` receives the default equal share
(`max_total_spans_per_second / number of sub-policies`) instead of an accidental zero rate. That
default is not the unallocated remainder; list every policy explicitly when exact percentages matter.

## In-process sharding

`num_shards` runs independent event loops inside one processor and assigns each trace by trace-ID
hash. This reduces contention at high ingest rates without splitting a trace:

```yaml
processors:
  tail_sampling:
    num_shards: 8
    num_traces: 100000
    policies: [ ... ]
```

`num_traces`, `expected_new_traces_per_sec`, both decision-cache sizes, and per-second limits in
`rate_limiting`, `bytes_limiting`, and composite policies are divided across shards so their
configured totals remain aggregate limits. Limiter `burst_capacity` is not divided. Values above
256 are rejected, and `num_shards > 1` cannot be combined with `tail_storage`.

## Combining policies and dropping noise

Order policies by intent: force-drop noise first (`drop`), then keep errors, then service-specific rules, then a probabilistic baseline. Any `sample` decision keeps the trace unless a `drop` policy fired.

```yaml
policies:
  - name: drop-health
    type: drop
    drop:
      drop_sub_policy:
        - name: health-routes
          type: string_attribute
          string_attribute:
            key: http.route
            values: ["^/health$", "^/ready$", "^/metrics$"]
            enabled_regex_matching: true
  - name: errors
    type: status_code
    status_code:
      status_codes: [ERROR]
  - name: baseline
    type: probabilistic
    probabilistic:
      sampling_percentage: 1
```

## Scaling horizontally with the `load_balancing` exporter

A single tail-sampling instance only works while every span of a trace lands on it. To run **more than one** tail-sampling collector you need a front layer of collectors using the `load_balancing` exporter with `routing_key: traceID`, so all spans of a given trace are routed to the **same** downstream tail-sampling instance:

```yaml
# Layer 1 — routing collectors (any number of replicas)
exporters:
  load_balancing:
    routing_key: traceID
    protocol:
      otlp:
        tls:
          insecure: true
    resolver:
      dns:
        hostname: tail-sampling-layer
        port: "4317"
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [load_balancing]

# Layer 2 — tail-sampling collectors (behind the DNS name above)
processors:
  tail_sampling:
    decision_wait: 30s
    num_traces: 100000
    policies: [ ... ]
```

For high-volume systems also size `decision_cache` larger than `num_traces` so late-arriving spans inherit the prior decision instead of forming a new (partial) trace.

# `tail_sampling`: configuration

## Typical config

```yaml
processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 50000
    policies:
      # Keep all error traces
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      # Sample 10% of everything else
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 10

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling]
      exporters: [otlphttp]
```

## Configuration reference (top-level)

| Key | Type | Default | Notes |
|-----|------|---------|-------|
| `sampling_strategy` | enum | `trace-complete` | `trace-complete`: evaluate the accumulated trace on the timer path (most flexible, higher memory). `span-ingest`: evaluate each incoming batch at ingest — terminal `sampled`/`dropped` outcomes finalize immediately, non-terminal traces finalize as not-sampled on cleanup (rejects stateful policies). Invalid values fail validation. |
| `decision_wait` | duration | `30s` | Time from first span arrival before the decision is made. Buffers spans for this long. Under `span-ingest` it instead controls pending-cleanup finalization timing. |
| `decision_wait_after_root_received` | duration | `0s` | Decide this long after the root span arrives; `0` disables (only `decision_wait` is used). |
| `num_traces` | int | `50000` | Max traces kept in memory. When full, the oldest are evicted (before decision) unless `block_on_overflow`. |
| `num_shards` | int | `1` | Parallel in-process event loops, with traces assigned by trace-ID hash. Maximum `256`; values >1 cannot be combined with `tail_storage`. Aggregate capacities and per-second policy limits are divided across shards. Added in v0.159.0. |
| `expected_new_traces_per_sec` | int | `0` | Hint for pre-allocating the trace buffer; `0` disables pre-allocation. |
| `sample_on_first_match` | bool | `false` | Stop evaluating and sample as soon as one policy matches. Do not combine with tracestate handling: later policies may report a less strict threshold. |
| `block_on_overflow` | bool | `false` | Block ingest instead of dropping the oldest traces when `num_traces` is reached. |
| `drop_pending_traces_on_shutdown` | bool | `false` | On shutdown, drop pending traces instead of deciding with partial data. |
| `maximum_trace_size_bytes` | int | `0` | Traces larger than this are dropped immediately; `0` disables. |
| `decision_cache.sampled_cache_size` | int | `0` | LRU cache of sampled trace IDs so late spans inherit the decision; `0` disables. |
| `decision_cache.non_sampled_cache_size` | int | `0` | LRU cache of dropped trace IDs; `0` disables. |
| `policies` | list | (required) | Sampling policies. At least one is required. |

`tail_storage` (a component ID) offloads span buffering to a tail-storage extension instead of memory, but is behind the alpha `processor.tailsamplingprocessor.tailstorageextension` feature gate — setting it without the gate fails validation. It cannot be combined with `num_shards > 1`.

The alpha `processor.tailsamplingprocessor.usetracestate` gate (off by default) makes the
`probabilistic`, `rate_limiting`, and `bytes_limiting` policies report effective probability
sampling thresholds and rewrites outgoing `th` values. It falls back to the legacy trace-ID hash
when no probability sampling information is present. Do not combine it with
`sample_on_first_match`. See [Tracestate probability sampling](policies.md#tracestate-probability-sampling).

For the full catalog of policy types and their sub-fields, see [Policy types](policies.md).

# `tail_sampling`: policy types

Each policy has `name` and `type`, plus a config block named after the type.

| `type` | Purpose |
|--------|---------|
| `always_sample` | Sample every trace (catch-all / debugging). No sub-config. |
| `latency` | Sample by total trace duration (earliest start to latest end); the lower bound is exclusive. |
| `numeric_attribute` | Sample when a numeric span/resource attribute is within a range. |
| `probabilistic` | Sample a fixed percentage via trace-ID hash, or via W3C probability sampling information when the opt-in tracestate gate is enabled. |
| `status_code` | Sample if any span has a matching status code. |
| `string_attribute` | Sample by string span/resource attribute value (exact or regex). |
| `rate_limiting` | Sample up to a maximum spans-per-second via a token bucket; `spans_per_second` (required) + `burst_capacity` (optional, default 2× the rate). A trace with more spans than the burst never passes. |
| `bytes_limiting` | Sample up to a maximum bytes-per-second via a token bucket; `bytes_per_second` (required) + `burst_capacity` (optional, default 2× the rate). |
| `span_count` | Sample by number of spans in the trace. |
| `trace_state` | Sample by W3C TraceState key/value. |
| `trace_flags` | Sample if the W3C `sampled` trace flag is set on any span in the trace. No sub-config. |
| `boolean_attribute` | Sample by boolean attribute value. |
| `ottl_condition` | Sample when OTTL conditions on spans/span-events match (`span:` and `spanevent:` lists, plus `error_mode`). Prefer path-qualified context (e.g. `span.attributes[...]`, `resource.attributes[...]`, `spanevent.name`) over bare `attributes[...]` to avoid future breaking changes. |
| `and` | Combine sub-policies with AND — all must match to sample. |
| `composite` | Combine sub-policies with priority order and per-policy rate allocation. |
| `not` | Invert the decision of a single wrapped sub-policy via `not_sub_policy`. |
| `drop` | Force-drop traces where all sub-policies match; takes precedence over sampling. |

Key sub-fields for the common policies:

```yaml
# latency: minimum (and optional maximum) trace duration
- name: slow
  type: latency
  latency:
    threshold_ms: 1000
    upper_threshold_ms: 5000   # optional; 0 = no upper bound

# numeric_attribute: span or resource attribute within a range
- name: http-errors
  type: numeric_attribute
  numeric_attribute:
    key: http.response.status_code
    min_value: 400
    max_value: 599

# string_attribute: exact or regex match
- name: by-route
  type: string_attribute
  string_attribute:
    key: http.route
    values: ["/api/users", "^/api/.*"]
    enabled_regex_matching: false   # treat values as regex when true
    invert_match: false             # sample traces that do NOT match

# status_code: OK / ERROR / UNSET
- name: errors
  type: status_code
  status_code:
    status_codes: [ERROR]

# probabilistic: percentage via trace-ID hash
- name: baseline
  type: probabilistic
  probabilistic:
    sampling_percentage: 10.0

# rate_limiting: cap spans per second (token bucket)
- name: cap
  type: rate_limiting
  rate_limiting:
    spans_per_second: 1000
    burst_capacity: 2000   # optional; default 2× spans_per_second
```

The latency interval is **`threshold_ms < duration <= upper_threshold_ms`**. With no upper bound (`upper_threshold_ms: 0`), only durations strictly greater than `threshold_ms` match. v0.157.0 fixed the no-upper-bound case to use this same exclusive lower bound; a trace exactly equal to `threshold_ms` is not sampled.

## Decision precedence

All policies are evaluated (unless `sample_on_first_match: true`), then a single decision is chosen: a `drop` decision wins over everything; otherwise any `sample` decision keeps the trace; if no policy matches, the trace is **not** sampled.

> **`invert_match` note:** the legacy "inverted decisions" behavior is gone — the `disableinvertdecisions` feature gate was stabilized and then removed, so as of v0.156.0 `invert_match: true` always yields a plain sample/not-sample on the negated condition and no longer vetoes other policies. `invert_match` still exists on the `numeric_attribute`, `string_attribute`, and `boolean_attribute` policies. To explicitly suppress traces, use a `drop` policy; to sample on the opposite of a wrapped policy's decision, use a `not` policy instead.

## Tracestate probability sampling

The alpha `processor.tailsamplingprocessor.usetracestate` gate is off by default. When enabled, the `probabilistic` policy reads OpenTelemetry probability sampling fields (`rv` and `th` in the `ot` section) from W3C `tracestate`: it uses explicit `rv` randomness when present, otherwise trace-ID-derived randomness, and falls back to the legacy FNV trace-ID hash (with `hash_salt`) when no probability sampling information exists.

For sampled traces, the processor rewrites each parseable span's outgoing `th` to the smallest effective threshold across policies that voted to sample. Since v0.160.0, top-level and `and`-nested `rate_limiting` and `bytes_limiting` policies report the threshold actually applied; their token-bucket limits and burst behavior are unchanged. These policies inside `composite` still report always-sample. Existing stricter thresholds are preserved. Spans with unparseable tracestate are skipped during rewriting and counted by the Development metric `otelcol_processor_tail_sampling_count_spans_with_unparseable_tracestate`.

Do not combine tracestate handling with `sample_on_first_match`: stopping early can skip a later
policy with a less strict threshold and make adjusted counts incorrect.

The `and`, `composite`, `not`, and `drop` policy types compose other policies; see [Advanced use-cases](advanced.md) for worked examples.

# `tail_sampling`: known quirks

## All spans of a trace must reach the SAME instance

The decision is made per-trace inside one process. If spans of one trace are spread across multiple tail-sampling collectors, each sees only a fragment and makes its own (wrong/partial) decision. Whenever you run more than one tail-sampling collector you **must** put a `load_balancing` exporter layer in front that routes by `traceID`. A single instance needs no load balancer.

`num_shards` does not change this rule. Sharding is internal to one processor and routes a complete
trace to one event loop; it is not coordination among Collector instances.

## Memory scales with `num_traces` and trace size

`num_traces` is the in-flight trace buffer: every trace awaiting a decision is held in memory. Rough estimate `num_traces * avg_spans_per_trace * ~1KB/span` (e.g. 50,000 traces × 20 spans ≈ 1 GB). Longer `decision_wait` means more traces resident at once. When the buffer fills, the oldest traces are evicted **before** their decision (surfacing as the `sampling_trace_dropped_too_early` metric) unless `block_on_overflow` is set. Size it as `traces_per_sec * decision_wait_seconds * safety_factor`.

## `decision_wait` adds latency and a fixed window

Sampled traces are only exported after `decision_wait` expires, so this delay is added before any trace leaves the processor. Set it long enough for distributed traces to assemble; too short and late spans miss the window. Spans that arrive **after** the decision are not retroactively folded into it — they either inherit the cached decision (if `decision_cache` is configured) or risk forming a separate/dropped partial trace.

## Stateful, single-process decision

Tail sampling is inherently stateful: the keep/drop choice is computed once, in one process, from whatever spans were buffered at decision time. There is no cross-instance coordination and no re-evaluation of a trace once decided. Place context-enriching processors (e.g. `k8s_attributes`) **before** `tail_sampling` in the pipeline, since the processor re-batches spans and downstream context can be lost.

## Tracestate rewriting is opt-in and parse-sensitive

The processor only consumes and rewrites OpenTelemetry probability sampling fields when the alpha `processor.tailsamplingprocessor.usetracestate` gate is enabled. If a span's W3C `tracestate` cannot be parsed, its outgoing `th` is not rewritten; monitor `otelcol_processor_tail_sampling_count_spans_with_unparseable_tracestate` (Development stability) for this case. The probabilistic decision still falls back to the legacy trace-ID hash when the trace carries no probability sampling information. Do not combine the gate with `sample_on_first_match`, which can stop before the least-strict matched threshold is known. Before v0.162.0, late-arriving spans of an already-sampled trace could be rewritten with a different threshold than the original decision; v0.162.0 reuses the original decision's threshold.

# `tail_sampling` processor

| | |
|-|-|
| Kind | processor |
| Type | `tail_sampling` |
| Signals | traces |
| Stability | Beta |
| Distributions | contrib, k8s |
| Go module | `github.com/open-telemetry/opentelemetry-collector-contrib/processor/tailsamplingprocessor` |
| Upstream README | <https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor> |

## Description

Tail sampling buffers **all spans of a trace** (grouped automatically by `trace_id` — no `groupbytrace` needed), waits a configurable `decision_wait` for the trace to complete, then evaluates a set of policies against the **whole trace** to make a single keep/drop decision. Because the decision is made after the trace is (mostly) assembled, it can act on trace-wide signals: presence of an error on any span, total latency, span count, specific attributes, etc.

`num_shards` can split that in-process work across parallel event loops while
keeping each trace on one shard. This improves concurrency inside one Collector; it does not replace
trace-affinity routing when scaling across Collector instances.

This is the opposite of **head sampling** (e.g. `probabilistic_sampler`), which decides at the start of a trace based only on the trace ID — before any span content is known. Head sampling is cheap and stateless; tail sampling is content-aware but stateful, requires buffering, and adds latency.

## Main use-cases

Use it when:
- You want to keep **all error traces** regardless of volume.
- You want to keep **slow** traces (latency-based) while down-sampling fast successful ones.
- You need different sampling rates per service, endpoint, tenant, or attribute.
- You need sophisticated logic — combining conditions (`and`), tiered rate allocation (`composite`), or explicit noise removal (`drop`).

Avoid it when:
- Probabilistic head sampling already meets your needs (cheaper, no buffering, no co-location requirement).
- You need the sampling decision at the edge/SDK before data reaches a collector.
- You **cannot guarantee all spans of a trace reach the same collector instance** (see [Known quirks](quirks.md)).

## Related components

- `probabilistic_sampler` — head sampling; stateless, trace-ID based, decided up front. Cheaper but content-blind.
- [`load_balancing`](../load_balancing/README.md) exporter — routes spans by `traceID` so all spans of a trace reach the same tail-sampling instance; required when scaling tail sampling to more than one collector.
- `groupbytrace` — groups spans by trace ID and waits before forwarding. Not needed in front of `tail_sampling`, which already groups by trace ID internally.

## Details

- [Configuration](configuration.md) — top-level config keys (`decision_wait`, `num_traces`, `num_shards`, `decision_cache`, …), defaults, and the minimal pipeline. Open when wiring up the processor or checking a key/default.
- [Policy types](policies.md) — the full catalog of all ~17 policy types and their sub-fields, latency-boundary semantics, and optional W3C tracestate probability handling. Open when choosing or configuring a sampling policy.
- [Verification](verification.md) — telemetrygen recipe to confirm sampling works. Open when you want to prove the config end-to-end.
- [Advanced use-cases](advanced.md) — `and`/`composite`/`drop` combinations, in-process sharding, and scaling out with `load_balancing`. Open when building multi-condition policies or increasing concurrency.
- [Known quirks](quirks.md) — same-instance/loadbalancing requirement, memory model, `decision_wait` latency, late spans, statefulness, and tracestate parsing. Open when sizing memory or debugging missing/partial traces.

# `tail_sampling`: verification

See [Verification harness](../../SKILL.md#verification-harness) for how to run this end-to-end.

`tail_sampling` ships in the `contrib` and `k8s` distributions.

Verified on `otel/opentelemetry-collector-contrib:0.159.0` with
`ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:v0.159.0`
(2026-08-27). This status-code-policy recipe's schema is unchanged through v0.160.0; it does not
exercise tracestate-aware limiting.

Config (`tailsampling-verify.yaml`):

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
processors:
  tail_sampling:
    decision_wait: 5s
    num_traces: 1000
    policies:
      - name: errors-only
        type: status_code
        status_code:
          status_codes: [ERROR]
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling]
      exporters: [debug]
```

Generate traces, some with error status (see the `otel-telemetrygen` skill):

```bash
# OK traces — expect these to be dropped by the errors-only policy
telemetrygen traces --otlp-insecure --otlp-endpoint localhost:4317 \
  --traces 20 --status-code Ok
# Error traces — expect these to survive
telemetrygen traces --otlp-insecure --otlp-endpoint localhost:4317 --traces 20 --status-code Error
```

The `--status-code` flag is confirmed in the `otel-telemetrygen` skill (accepted values: `Unset`/`0`, `Error`/`1`, `Ok`/`2`). It sets the span status that the `status_code` policy evaluates. telemetrygen's integer mapping (`1`=Error, `2`=Ok) intentionally differs from the OpenTelemetry status-code enum (`1`=Ok, `2`=Error), so pass the strings shown above.

**What proves it worked:** after `decision_wait`, the `debug` exporter shows the 20 error traces and none of the 20 OK traces. telemetrygen emits two spans per generated trace, so the verified debug output contained 40 Error spans and no Ok spans.

# `transform`: advanced use-cases

## Multiple statement groups

Split work into ordered groups when statements need different contexts, conditions, or error modes. Groups run top-to-bottom; within a group, statements run top-to-bottom too.

```yaml
processors:
  transform:
    error_mode: ignore
    metric_statements:
      # Group 1 — metric context
      - statements:
          - convert_sum_to_gauge() where metric.name == "process.count"
          - set(metric.description, "process count gauge")
      # Group 2 — datapoint context (separate group: different context)
      - statements:
          - limit(datapoint.attributes, 50, ["host.name", "service.name"])
          - delete_matching_keys(datapoint.attributes, "(?i).*temp.*")
```

Set `shared_cache: true` on groups that intentionally exchange values through the OTTL cache. This
experimental v0.160.0 option keeps one cache for groups of the same signal; by default each group
gets a fresh cache.

## Scoping statements with conditions

Group-level `conditions` pre-gate a whole group; they are OR'd, and AND'd with each statement's `where`.

```yaml
processors:
  transform:
    error_mode: ignore
    log_statements:
      # Only JSON-looking logs
      - conditions:
          - IsMatch(log.body, "^\\{")
        statements:
          - merge_maps(log.cache, ParseJSON(log.body), "upsert")
          - set(log.attributes["parsed"], true)
      # Only one service
      - conditions:
          - resource.attributes["service.name"] == "payment-service"
        statements:
          - delete_matching_keys(log.attributes, "(?i).*(card|ssn|password).*")
```

## Context selection trade-offs

Pick the **broadest** context that still reaches what you need: editing at `resource` runs once per resource, while `datapoint`/`spanevent` runs per item and is more expensive. Inference already prefers the most specific context that covers your paths — set `context:` explicitly only to override it, and prefer to express a transform at the highest level that the data allows. Path inventories per context live in the `otel-ottl` skill; the context table is in [configuration.md](configuration.md).

## Combining with `filter`

`transform` and `filter` share the OTTL language but do opposite jobs: `filter` drops, `transform` rewrites. A common pipeline normalizes then drops:

```yaml
service:
  pipelines:
    logs:
      processors: [memory_limiter, transform, filter]
```

Run `transform` first if `filter`'s conditions depend on attributes `transform` sets (or vice-versa) — ordering in the pipeline is load-bearing.

## error_mode for resilience

```yaml
# Production: a failing statement logs but doesn't drop the batch
processors:
  transform:
    error_mode: ignore
```

```yaml
# Mixed: most groups tolerant, one group must fail loudly
processors:
  transform:
    error_mode: ignore
    metric_statements:
      - error_mode: propagate          # this group only
        statements:
          - set(metric.unit, "By") where metric.name == "system.memory.usage"
      - statements:                     # inherits ignore
          - set(metric.description, metric.cache["description"])
```

## Named instances

```yaml
processors:
  transform/redact:
    error_mode: ignore
    log_statements:
      - replace_pattern(log.body, "\\b\\d{16}\\b", "****")
  transform/enrich:
    error_mode: ignore
    log_statements:
      - set(log.attributes["env"], "prod")

service:
  pipelines:
    logs/redact:
      processors: [memory_limiter, transform/redact]
    logs/enrich:
      processors: [memory_limiter, transform/enrich]
```

## Common transforms

| Goal | Shape (OTTL — see `otel-ottl` skill for function details) |
|------|------------------------------------------------------------|
| Rename an attribute | `set(resource.attributes["new"], resource.attributes["old"])` then `delete_key(resource.attributes, "old")` — set before delete. |
| Redact a value | `replace_pattern(...)` / `delete_key(...)` / `delete_matching_keys(...)`. |
| Set a default | `set(log.severity_text, "INFO") where log.severity_text == nil or log.severity_text == ""`. |
| Limit / truncate attributes | `keep_keys(...)`, `limit(...)`, `truncate_all(...)`. |
| Convert metric types | `convert_sum_to_gauge()`, `convert_gauge_to_sum(...)` (metric context, with a `where` clause). |
| Reshape histograms | `extract_count_metric(...)`, `extract_sum_metric(...)`, `extract_percentile_metric(...)`, `extract_avg_metric(...)` (v0.162.0+). |
| Aggregate datapoints | `aggregate_on_attributes(...)`, `aggregate_on_attribute_value(...)`. |

These are processor-specific OTTL functions; their full signatures and the general OTTL converter set live in the `otel-ottl` skill.

# `transform`: configuration

## Typical config

```yaml
processors:
  transform:
    error_mode: ignore
    log_statements:
      - set(log.attributes["env"], "prod")

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [memory_limiter, transform]
      exporters: [otlphttp]
```

## Top-level keys

| Key | Type | Default | Notes |
|-----|------|---------|-------|
| `error_mode` | string | `ignore` | How OTTL statement-evaluation errors are handled: `ignore`, `silent`, or `propagate`. See [Error mode](#error-mode). |
| `trace_statements` | list | `[]` | Statements run against trace data. |
| `metric_statements` | list | `[]` | Statements run against metric data. |
| `log_statements` | list | `[]` | Statements run against log data. |
| `profile_statements` | list | `[]` | Statements run against profile data (Development stability). |

> The default `error_mode` is `ignore`. The `processor.transform.defaultErrorModeIgnore` gate was removed in v0.162.0, so naming it in `--feature-gates` fails startup; see [quirks](quirks.md#error_mode-propagate-drops-the-whole-batch-on-error).
>
> A `flatten_data` boolean (default `false`, behind the `transform.flatten.logs` alpha feature gate) gives each log record a distinct copy of its resource and scope before transformation, then regroups afterwards — useful when log-level data drives resource/scope edits. It copies and hashes every record, so enable only when needed.

## Error mode

| Mode | Behavior | When |
|------|----------|------|
| `ignore` | Logs the error and moves to the next statement. | **Default** (since v0.153.0) and **recommended in production** — one bad statement won't drop unrelated data. |
| `silent` | Continues without logging. | When error logging is too noisy. |
| `propagate` | Returns the error up the pipeline, dropping the whole payload. | Development/testing — surfaces config mistakes immediately. Was the default before v0.153.0. |

The top-level `error_mode` can be overridden per statement group (see below).

## Statement groups

Each `*_statements` field accepts entries in two styles. **Do not mix the two styles within one signal's list.**

### Flat (inferred-context) form

The recommended form for most cases: a plain list of OTTL statements. The processor **infers the context** from the path prefixes in each statement (e.g. `log.body` → `log` context; `resource.attributes` alone → `resource` context). You write statements without declaring a context.

```yaml
transform:
  error_mode: ignore
  trace_statements:
    - keep_keys(span.attributes, ["service.name", "http.request.method", "http.route"])
    - set(resource.attributes["deployment.environment.name"], "production")
  log_statements:
    - set(log.severity_text, "FAIL") where log.body == "request failed"
```

### Explicit-group form

For complex cases, group related statements and attach options. Each group is an object with an optional `context`, an optional group-level `error_mode`, optional `conditions`, and a required `statements` list.

```yaml
transform:
  error_mode: propagate           # default for all groups
  metric_statements:
    - context: metric             # optional; inferred if omitted
      error_mode: ignore          # optional; overrides top-level for this group
      conditions:                 # optional; OR'd together
        - metric.type == METRIC_DATA_TYPE_SUM
      statements:                 # required
        - set(metric.description, "Sum metric")
        - convert_sum_to_gauge() where metric.name == "system.processes.count"
```

| Group key | Type | Required | Notes |
|-----------|------|----------|-------|
| `context` | string | No | Explicit OTTL context (see [Context values](#context-values)). Usually inferred — only set it to override inference. |
| `error_mode` | string | No | Overrides the top-level `error_mode` for this group only. |
| `conditions` | list | No | Group-level OTTL booleans, OR'd together. If any is true (or the list is empty), the group's statements run. |
| `shared_cache` | bool | No (`false`) | Experimental. Share one OTTL cache map among statement groups of the same signal instead of clearing it between groups. The cache never persists across batches (v0.162.0 fixed a bug where it could). Added in v0.160.0. |
| `statements` | list | Yes | The OTTL statements to execute, in order. |

## Context values

Context determines which paths the statements can reach. Contexts are hierarchical — a more specific context can also read the broader ones above it. The processor picks the **most specific context** that covers all paths in a group; set `context:` explicitly only when inference can't reconcile the paths.

| Signal | Contexts (broad → specific) | Reaches |
|--------|------------------------------|---------|
| traces | `resource`, `scope`, `span`, `spanevent` | `spanevent` can read `resource`, `scope`, `span`, `spanevent` |
| metrics | `resource`, `scope`, `metric`, `datapoint`, `exemplar` | `exemplar` can read `resource`, `scope`, `metric`, `datapoint`, `exemplar` |
| logs | `resource`, `scope`, `log` | `log` can read `resource`, `scope`, `log` |
| profiles | `resource`, `scope`, `profile` | `profile` can read `resource`, `scope`, `profile` (Development) |

The `exemplar` metric context (reaching a datapoint's exemplars) was added in v0.156.0; on older versions `metric_statements` stops at `datapoint`.

Some path/function combinations can't share a context (e.g. a metric-context-only function alongside a `datapoint` path). When inference fails, split the conflicting statements into separate groups. See [Known quirks](quirks.md).

## `conditions` vs `where`

- `conditions` (group-level) gate the **whole group**; the entries are OR'd. If none match, no statement in the group runs.
- `where` (per statement) gates a **single statement**; it is AND'd with the group's conditions.

```yaml
metric_statements:
  - conditions:
      - metric.type == METRIC_DATA_TYPE_SUM        # group runs only for Sum metrics
    statements:
      - set(metric.description, "count") where metric.name == "http.requests"  # AND name matches
```

## Statement ordering and short-circuiting

Statements run top-to-bottom; later statements observe the effects of earlier ones (so set-then-delete order matters). With `error_mode: propagate`, the first failing statement **short-circuits the rest of the payload** — remaining statements never run and the batch is dropped. With `ignore`/`silent`, a failing statement is skipped and the next one still runs.

The OTTL expression surface — functions, paths, editors, converters, grammar, enums — is covered in the `otel-ottl` skill and is not reproduced here.

# `transform`: known quirks

## Statement order matters

Statements run top-to-bottom and later ones see earlier effects. A classic bug is deleting an attribute before reading it:

```yaml
# BAD — delete runs first, set reads nil
- delete_key(resource.attributes, "k8s.namespace.name")
- set(resource.attributes["namespace"], resource.attributes["k8s.namespace.name"])

# GOOD — set first, then delete
- set(resource.attributes["namespace"], resource.attributes["k8s.namespace.name"])
- delete_key(resource.attributes, "k8s.namespace.name")
```

## Mutating shared resource/scope affects everything under it

A `resource`-context statement runs once per resource and changes are seen by **all** spans/metrics/logs sharing that resource. Setting a resource attribute from per-item data (e.g. a single log's attribute) leaks that value across the whole group. When per-item resource edits are genuinely needed for logs, enable `flatten_data` (behind the `transform.flatten.logs` feature gate) so each record gets its own resource copy — at the cost of copying and hashing every record.

## `error_mode: propagate` drops the whole batch on error

Under `propagate`, the first statement that errors (a type mismatch, a nil dereference like `log.cache["x"]["y"]` when `x` is nil) **drops the entire payload**, not just the offending item, and the remaining statements never run. The default is `ignore`, but any config or group that still sets `propagate` behaves this way. Keep `error_mode: ignore` (or `silent`) in production, and nil-check before accessing nested fields:

```yaml
- set(log.attributes["uid"], log.cache["user"]["id"]) where log.cache["user"] != nil
```

`silent` hides evaluation errors entirely — if a transform "isn't working," confirm it isn't set to `silent`. The default flipped from `propagate` to `ignore` in v0.153.0 and became permanent when `processor.transform.defaultErrorModeIgnore` reached Stable in v0.157.0. Disabling the gate failed startup on v0.157.0–v0.161.0; the gate was removed in v0.162.0, so any remaining `--feature-gates` reference to it now fails startup as an unknown gate. Pre-v0.153.0 collectors still default to `propagate`; on v0.153.0–v0.156.0 the Beta gate could be disabled to restore it.

## Context performance

Statements evaluate per item at their inferred context. A `datapoint`- or `spanevent`-context group runs for every datapoint/event, which is far more work than a `resource`- or `metric`-context group. Express each transform at the broadest context that reaches the data. Mixing incompatible paths/functions in one group makes context inference fail — split them:

```yaml
# FAILS — convert_sum_to_gauge is metric-context-only; limit needs datapoint context
metric_statements:
  - convert_sum_to_gauge() where metric.name == "process.count"
  - limit(datapoint.attributes, 100, ["host.name"])
```

```yaml
# WORKS — one group per context
metric_statements:
  - statements:
      - convert_sum_to_gauge() where metric.name == "process.count"
  - statements:
      - limit(datapoint.attributes, 100, ["host.name"])
```

## OTTL version drift

`transform` is a thin shell around OTTL, which evolves quickly: function names, signatures, path syntax, and enums change between collector releases (e.g. the path-prefix style `log.body` vs older bare `body`; `extract_percentile_metric` added in v0.151.0, `extract_avg_metric` in v0.162.0). `set_semconv_span_name` accepts semantic-convention versions `1.37.0` through `1.43.0` in v0.162 and v0.163 (verified 2026-10-15; owner: the `otel-ottl` skill's `references/functions.md`). Validate statements against the running version — see the `otel-ottl` skill — rather than assuming a snippet from another release still parses.

## Don't change identity carelessly

- Setting `span.trace_id`/`span.span_id` orphans spans and breaks trace/log correlation.
- Renaming a metric or stripping all its datapoint attributes (`set(metric.name, ...)`, `delete_matching_keys(datapoint.attributes, ".*")`) collapses distinct series into one — an identity conflict.
- `copy_metric` without a `where` clause that excludes the copy creates an infinite loop. Always gate it.

## Stability caveats

Traces, metrics, and logs are **Beta**; profiles are **Development**. Beta means breaking changes are rare but config keys can still move; profile support may break. Check the upstream README for the running version before treating any key as stable.

## Troubleshooting

**Statement does nothing.**
- A `where` clause or group `conditions` didn't match.
- Wrong context inferred — set `context:` explicitly, or split the group.
- The field is nil; guard with a nil check.
- Enable `service.telemetry.logs.level: debug` to see the `TransformContext` before/after each statement and whether conditions matched.

**Errors logged but telemetry still flows.** Expected under `error_mode: ignore`; use `silent` to quiet the logs once you've accepted the errors.

**Whole payloads disappear.** `error_mode: propagate` plus an erroring statement. Switch to `ignore` or fix the statement.

**"unexpected token" / "unknown context" on startup.** Incompatible paths/functions in one group (split them), a syntax error, an unknown/renamed function, or mixing the flat and explicit-group styles in one signal's list.

## Anti-patterns

**Using `transform` to filter.** Setting a flag so a later `filter` drops the item is a smell — drop directly with `filter`.

**Hardcoding values that should come from detection.** `set(resource.attributes["deployment.environment.name"], "production")` belongs in resource detection or the SDK, not a static transform, unless you genuinely have no better source.

**`propagate` in production.** Any single error drops the batch; use `ignore`/`silent` and nil-guard nested access.

**Over-transformation.** Long chains of rewrites per item add latency and CPU; do at the source what you can, and transform only what must change downstream.

# `transform` processor

| | |
|-|-|
| Kind | processor |
| Type | `transform` |
| Signals | traces (Beta), metrics (Beta), logs (Beta), profiles (Development) |
| Distributions | contrib, k8s |
| Go module | `github.com/open-telemetry/opentelemetry-collector-contrib/processor/transformprocessor` |
| Upstream README | <https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor> |

## Description

Mutates telemetry — spans, span events, metrics, datapoints, log records (and, in development, profiles) — **in place** by running OTTL statements against it. You supply a per-signal list of statements (`trace_statements`, `metric_statements`, `log_statements`, `profile_statements`); each statement executes sequentially against incoming data, optionally guarded by a `where` clause. Unlike `filter`, which drops items, `transform` rewrites them: set, delete, rename, redact, convert types, parse structured fields, and more.

The top-level `error_mode` controls what happens when a statement fails to evaluate — `ignore` (default) logs and continues to the next statement, `silent` continues without logging, `propagate` drops the whole payload. The `processor.transform.defaultErrorModeIgnore` gate was removed in v0.162.0, so naming it in `--feature-gates` fails startup; see [quirks](quirks.md#error_mode-propagate-drops-the-whole-batch-on-error). Statement groups run in the order written, so later statements see the effects of earlier ones. The OTTL language itself (functions, paths, editors/converters, grammar) lives in the `otel-ottl` skill; this page documents only the processor's config surface.

## Main use-cases

Use it when:
- You need to **modify** telemetry — rename or set attributes, set defaults for missing fields, normalize formats across sources.
- You want to redact sensitive values (passwords, tokens, PII) in place rather than dropping the record.
- You need metric reshaping — convert sum↔gauge, extract count/sum/percentile from histograms, scale or aggregate datapoints.
- You want to parse structured data out of an unstructured log body into attributes.

Avoid it when:
- You only need to **drop** whole items — use `filter` (it complements `transform`).
- You need a whole-trace keep/drop decision — use `tail_sampling`.
- You need cross-batch metric aggregation over time — use `interval` or a metrics connector.

## Related components

- `filter` — drops telemetry via the same OTTL language; pair it with `transform` when some items should be removed and others rewritten.
- `attributes` — simpler key/value attribute edits without OTTL, for straightforward add/update/delete.
- `resource` — resource-attribute edits specifically.
- `metricstransform` — name/label-oriented metric renaming and aggregation outside OTTL.
- `routing` — sends telemetry to different pipelines by condition instead of mutating it.

## Details

- [Configuration](configuration.md) — `error_mode`, the statement-group structure, `context` values per signal, the inferred-context form, group-level `conditions`, and statement ordering/short-circuiting.
- [Verification](verification.md) — telemetrygen recipe that sets an attribute on logs so the change is observable in `debug` output.
- [Advanced use-cases](advanced.md) — multiple statement groups, scoping with conditions, context selection trade-offs, combining with `filter`, error_mode for resilience, named instances, common transforms.
- [Known quirks](quirks.md) — statement ordering, shared-resource mutation, propagate dropping the batch, context performance, OTTL version drift, stability caveats, anti-patterns.

# `transform`: verification

See [Verification harness](../../SKILL.md#verification-harness) for how to run this end-to-end.

`transform` ships in the `contrib` and `k8s` distributions, so a stock contrib collector can run this.

Verified on `otel/opentelemetry-collector-contrib:0.159.0` with
`ghcr.io/open-telemetry/opentelemetry-collector-contrib/telemetrygen:v0.159.0`
(2026-08-27). This recipe's schema is unchanged through v0.160.0; it does not use `shared_cache`.

Config (`transform-verify.yaml`) — add an `env` attribute to every log record that telemetrygen does **not** itself produce:

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
processors:
  transform:
    error_mode: ignore
    log_statements:
      - set(log.attributes["env"], "prod")
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [transform]
      exporters: [debug]
```

Send a batch of logs — see the `otel-telemetrygen` skill:

```bash
telemetrygen logs --otlp-insecure --otlp-endpoint localhost:4317 \
  --logs 5 --body "transform me"
```

Flags confirmed against the `otel-telemetrygen` skill (`references/flags.md`): `--otlp-insecure` (bool), `--otlp-endpoint` (string), `--logs` (int, per-worker count; ignored if `--duration` is set, so no `--duration` here), and `--body` (string). No flag here invents a value.

**What proves it worked:** the `debug` exporter prints all 5 log records, and each carries an `env: Str(prod)` attribute in its `Attributes` block. telemetrygen never emits an `env` log attribute, so its presence in the output is unambiguous proof the statement ran. The log body ("transform me") is unchanged — only the attribute was added.

> The exact `debug` rendering of attributes (`-> env: Str(prod)`) can shift between collector versions; what matters is that the `env`/`prod` pair appears on records that didn't have it on input.