otel-ottl
OpenTelemetry Transformation Language (OTTL) expert for writing and debugging telemetry transformations in the OpenTelemetry Collector. Use when authoring or reviewing `transform`, `filter`, `tail_sampling` processor configs or `routing` connector configs, debugging OTTL syntax o
Install
npx skills add https://github.com/ollygarden/opentelemetry-agent-skills/tree/main/skills/otel-ottl
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install ollygarden-opentelemetry-agent-skills@llmmart
git clone https://github.com/ollygarden/opentelemetry-agent-skills.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole ollygarden/opentelemetry-agent-skills collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
OpenTelemetry Transformation Language (OTTL)
OTTL transforms or selects telemetry inside Collector components. This skill is pinned to collector-contrib v0.162.0. Function, path, default, and feature-gate availability varies by release; when the user's version differs, verify against the matching upstream tag.
Dated OTTL feature-gate status for v0.162.0 and v0.163.0 is in references/functions.md.
Workflow
- Choose the component.
transformrewrites,filterdrops,tail_samplingdecides whether to retain traces, androutingsends telemetry to pipelines. A component controls its available contexts and functions. - Choose the lowest usable context. Lower contexts can read their parents (for example, a span
can read
resource.attributes), but parents cannot read children. Usedatapointfor point attributes instead of traversingmetric.data_points. - Verify every emitted function, then write the statement. Confirm each function's exact
identifier and signature in references/functions.md. If an identifier
is absent, treat it as unsupported instead of deriving or substituting a plausible name. An
editor such as
setordelete_keymutates data and may have awherecondition. Converters such asParseJSONandIsMatchreturn values; they do not mutate. - Set error behavior deliberately.
ignorelogs statement errors and continues;silentcontinues without logging;propagatereturns the error and can cause the component to drop the payload. Transform and filter default toignore; the transform default-error gate was removed in v0.162.0, so naming it in--feature-gatesfails startup (history: theotel-collectorskill's transform quirks), and the filter gate is stable and cannot be disabled. Routing also defaults toignorewhile its beta default-error feature gate is enabled. For routing,ignoresends an errored payload todefault_pipelines; configure that fallback or the payload is dropped. - Verify end to end. Validate the exact Collector version, then send known telemetry and inspect file-exporter output. Use the telemetrygen recipe.
set(span.attributes["env"], "prod") where resource.attributes["env"] == nil
Load only what the task needs
- Contexts — exact paths, hierarchy, enums, and request metadata.
- Functions — editor/converter signatures and release availability.
- Quick reference — component YAML, recipes, escaping, troubleshooting, and safe skeletons.
For a single path or function, read only the relevant section instead of loading the full catalogs.
Safety and correctness gates
- Guard optional or polymorphic input before conversion:
where x != nil,IsString(x), or the appropriate type check. - For JSON-object-only work, guard both the type and shape before calling
ParseJSON, for exampleIsString(log.body) and IsMatch(log.body.string, "(?s)^\\s*\\{.*\\}\\s*$"). The RE2(?s)flag admits pretty-printed objects containing newlines. CheckingIsMapafter parsing does not prevent arrays or scalar JSON from being parsed. - On a version-pinned request, confirm every chosen path and function against that release tag;
do not assume a function listed for this skill's v0.162 anchor exists in an older release.
For v0.156 JSON-object parsing,
ParseJSON,IsString, andIsMatchare available without the v0.157 alpha lambda feature gate. - Request metadata is read-only and may contain credentials. Copy only explicitly allowlisted,
non-sensitive keys. OTLP metadata routing requires
include_metadata: trueon the receiver. HTTP/client header spelling may retain its form (otelcol.client.metadata["X-Tenant"][0]); gRPC metadata keys are lowercase (otelcol.grpc.metadata["x-tenant"][0]). - The routing
requestcontext is deprecated as of v0.156; useotelcol.client.metadataorotelcol.grpc.metadata. - Log-record-specific rewrites of shared resource or scope data require
flatten_data: trueand the alphatransform.flatten.logsgate. This copies and regroups data; do not enable it accidentally. - Since v0.161,
set(target, nil)is no longer a no-op: theottl.set.allowNilgate became beta in v0.161 and stable (cannot be disabled) in v0.162. A nil value empties a map, slice, orpcommon.Valuetarget (an attribute key is created with an empty value) and errors on scalar targets. Use awhere source != nilguard when the destination must remain unchanged for missing input. - Hashing an identifier does not necessarily anonymize it. Apply the organization's data-handling policy before retaining deterministic hashes of personal data.
Frequent syntax traps
- In Collector YAML, write an OTTL replacement backreference
${1}as$${1}. A replacement such as$1REDACTEDis literal and silently fails to substitute the capture. - Go RE2 rejects large counted repetitions such as
(.{1024}).*; useSubstringwith a nil/type guard andLen, ortruncate_allfor a map. - Current span-event paths use
spanevent.*, notspan_event.*. Cache paths are context-qualified, such asspan.cache["parsed"]. - Quote any OTTL statement containing a map literal when YAML includes a space after
:, for example'set(log.attributes["a"], {"foo": "bar"})'; otherwise YAML parses:as a mapping. - Since v0.159, polymorphic
pcommon.Valuepaths compare by their underlying type; maps and slices support only equality and inequality, while primitive values also support ordering. - Use
Decode(value, "base64");Base64Decodewas removed in v0.161. - Regex escapes inside OTTL strings are doubled (
\\d,\\s,\\.).
Upstream sources
Files (opentelemetry-agent-skills)
-
evals
-
evals.json 3.6 KB
{ "skill": "otel-ottl", "runtime": "Codex CLI", "model": "gpt-5.6-terra", "cases": [ { "id": "routing-metadata-fallback", "prompt": "We run OpenTelemetry Collector contrib v0.160. Route OTLP logs by the X-Tenant header to logs/acme, but malformed or missing headers must not lose telemetry. Provide the relevant YAML and explain the failure behavior.", "expectations": [ "Uses executable routing YAML with either otelcol.client.metadata[\"X-Tenant\"][0] for HTTP or otelcol.grpc.metadata[\"x-tenant\"][0] for gRPC.", "Enables include_metadata on the OTLP receiver and notes lowercase keys for gRPC metadata.", "Configures default_pipelines and explains that routing error_mode ignore sends evaluation errors there; without it the payload is dropped.", "Does not use the deprecated routing request context.", "Provides a routing condition matching tenant value acme and assigns the match to pipelines: [logs/acme]." ] }, { "id": "yaml-backreference-redaction", "prompt": "For Collector contrib v0.160, write a transform processor that redacts token query-parameter values in span URL attributes while preserving the parameter name. Include how you would verify it.", "expectations": [ "Uses a span-context transform statement with replace_pattern.", "Uses the Collector-YAML replacement $${1}REDACTED and does not present $1REDACTED as valid.", "Guards a potentially absent or non-string URL input.", "Recommends an end-to-end known-input check such as telemetrygen plus file-exporter output.", "Uses ([?&]token=)[^&]+ or an equivalent capture that preserves the parameter delimiter and name while replacing only its value." ] }, { "id": "v0156-json-log-parse", "prompt": "My Collector is pinned to contrib v0.156. Parse JSON-object log bodies and set parsed fields as log attributes only when the body is actually a JSON object. Do I need an alpha lambda feature gate? Give the transform fragment.", "expectations": [ "Uses log context with ParseJSON and a type-safe IsString guard.", "Restricts parsing to JSON-object-shaped input rather than arbitrary JSON arrays or strings.", "Does not require or use v0.157 alpha lambda converters.", "Treats function availability as release-dependent and keeps the answer compatible with v0.156.", "Writes or merges the parsed JSON object into log.attributes under the object-type guard rather than discarding the parse result." ] }, { "id": "offline-ottl-v0163-upgrade-gates", "prompt": "No web lookups please, just your knowledge and skills. We're upgrading collector-contrib from v0.162.0 to v0.163.0. Our transform processor has set_semconv_span_name(\"1.44.0\") and set(span.attributes[\"rid\"], Concat(Split(span.attributes[\"request.id\"], \"-\"), \"\")). Will both work on v0.163.0, and are there new OTTL feature gates we need to know about?", "expectations": [ "Says set_semconv_span_name accepts only 1.37.0 through 1.43.0 in v0.163.0, so \"1.44.0\" is rejected and \"1.43.0\" should be used.", "Says Concat over Split works on v0.163.0 because pkg.ottl.functions.enableDynamicSliceArguments is beta and enabled by default.", "Names the new alpha pkg.ottl.functions.enableExperimental gate for experimental functions, and keeps lambda converters on ottl.functions.enableLambda.", "Frames the answer as verified on the recorded date and suggests confirming against the v0.163.0 tag.", "Does not perform any web search, web fetch, or network command." ] } ] }
-
-
references
-
contexts.md 11.5 KB
# OTTL Contexts Reference Context paths and enums for collector-contrib **v0.162.0**. Higher-level contexts are reachable from lower ones (a span statement can read `resource.attributes`); the reverse is not true. Always pick the most specific context for the work — using `datapoint` to set metric-point attributes is much cheaper than walking through `metric.data_points` from the metric context. ## Contents - [Context hierarchy](#context-hierarchy) - [Resource](#resource-context), [scope](#scope-instrumentation-scope-context), [span](#span-context-beta), and [span event](#span-event-context-beta) - [Metric](#metric-context-beta), [data point](#datapoint-context), and [exemplar](#exemplar-context-v0156) - [Log](#log-context-beta), [profile](#profile-context-development-v0124), and [profile sample](#profile-sample-context-development-v0132) - [Collector request metadata](#otelcol-context-v0147-always-available-since-v0161) - [Enums](#enums) ## Context hierarchy ``` Resource └── Scope (instrumentation library) ├── Span → Span Event ├── Metric → DataPoint → Exemplar ├── Log └── Profile → Profile Sample OTelCol (Collector request/client metadata; read-only) ``` ## Resource context ```ottl resource.attributes # map resource.attributes["service.name"] resource.dropped_attributes_count resource.schema_url resource.cache # transformation-scope scratchpad resource.cache["key"] ``` ```ottl set(resource.attributes["environment"], "production") where resource.attributes["environment"] == nil set(resource.attributes["service.name"], ToLowerCase(resource.attributes["service.name"])) ``` ## Scope (instrumentation scope) context ```ottl scope.name scope.version scope.attributes scope.attributes["key"] scope.dropped_attributes_count scope.schema_url scope.cache scope.cache["key"] ``` ```ottl where not IsMatch(scope.name, "opentelemetry-.*") set(scope.attributes["quality"], "manual") where scope.name == "manual-instrumentation" ``` > Older configs may use `instrumentation_scope.*`. The current canonical prefix is `scope.*`. ## Span context (Beta) ### Identity & relationships ```ottl span.trace_id # bytes span.trace_id.string # hex string span.span_id # bytes span.span_id.string # hex string span.parent_span_id # bytes span.parent_span_id.string # hex string span.flags # W3C trace flags, uint32 (v0.145+) span.trace_state # W3C trace state span.trace_state["key"] ``` ### Metadata ```ottl span.name span.kind # int64 span.kind.string # "Server" | "Client" | "Internal" | … span.status # status object span.status.code # int64 (use STATUS_CODE_* enums) span.status.message # string ``` ### Timing ```ottl span.start_time_unix_nano # int64 span.end_time_unix_nano # int64 span.start_time # time.Time span.end_time # time.Time ``` ### Data ```ottl span.attributes span.attributes["http.method"] span.events # collection span.links # collection span.dropped_attributes_count span.dropped_events_count span.dropped_links_count span.cache["key"] # transformation cache ``` ### Examples ```ottl # Duration in ms set(span.attributes["duration_ms"], (span.end_time_unix_nano - span.start_time_unix_nano) / 1000000) # Mark errored HTTP spans set(span.status.code, STATUS_CODE_ERROR) where IsInt(span.attributes["http.status_code"]) and Int(span.attributes["http.status_code"]) >= 400 # Identify root spans (preferred form, since v0.142 SpanID accepts hex strings too) set(span.attributes["is_root"], true) where IsRootSpan() # Latency-and-kind sampling decision set(span.attributes["sampled"], true) where span.kind == SPAN_KIND_SERVER and (span.status.code == STATUS_CODE_ERROR or (span.end_time_unix_nano - span.start_time_unix_nano) > 1000000000) ``` ## Span Event context (Beta) ```ottl spanevent.name spanevent.time_unix_nano spanevent.time # time.Time spanevent.attributes spanevent.attributes["key"] spanevent.dropped_attributes_count spanevent.event_index # 0-based position within span (v0.120+) spanevent.cache["key"] # Span fields are reachable: span.name, span.attributes, … ``` ```ottl where spanevent.name == "exception" set(spanevent.attributes["span.name"], span.name) set(spanevent.attributes["error.message"], "REDACTED") where IsMatch(spanevent.attributes["error.message"], ".*password.*") ``` > Older configs may use `span_event.*`. The current path prefix is `spanevent.*`. ## Metric context (Beta) ```ottl metric.name metric.description metric.unit metric.type # int64 (use METRIC_DATA_TYPE_* enums) metric.aggregation_temporality # int64 metric.is_monotonic # bool metric.data_points # collection metric.metadata # metadata carried by the metric ``` ```ottl # Normalize metric names to safe characters replace_pattern(metric.name, "[^a-zA-Z0-9_]", "_") # Normalize legacy units set(metric.unit, "s") where metric.unit == "seconds" set(metric.unit, "B") where metric.unit == "bytes" # Filter by type where metric.type == METRIC_DATA_TYPE_GAUGE ``` ## DataPoint context ```ottl datapoint.start_time_unix_nano datapoint.time_unix_nano datapoint.start_time # time.Time datapoint.time # time.Time datapoint.value_double datapoint.value_int datapoint.attributes datapoint.attributes["key"] datapoint.flags datapoint.exemplars # collection # Metric, scope, resource paths are reachable. ``` ```ottl set(datapoint.attributes["value_range"], "high") where datapoint.value_double > 1000 # Rename an attribute set(datapoint.attributes["host.name"], datapoint.attributes["hostname"]) where datapoint.attributes["hostname"] != nil delete_key(datapoint.attributes, "hostname") ``` ## Exemplar context (v0.156+) Individual metric exemplars. Available only in the `transform` processor's `metric_statements` (the `filter` processor does not expose it). Reachable up through `datapoint.*`, `metric.*`, `scope`, and `resource`. ```ottl exemplar.time_unix_nano exemplar.time # time.Time exemplar.double_value exemplar.int_value exemplar.trace_id / exemplar.trace_id.string exemplar.span_id / exemplar.span_id.string exemplar.filtered_attributes # map exemplar.filtered_attributes["key"] exemplar.cache["key"] # datapoint.*, metric.*, scope, resource paths are reachable. ``` ```ottl # Redact a filtered attribute carried on exemplars delete_matching_keys(exemplar.filtered_attributes, "(?i).*(token|secret).*") ``` ## Log context (Beta) ```ottl log.time_unix_nano log.observed_time_unix_nano log.time # time.Time log.observed_time # time.Time log.severity_number # int64 (use SEVERITY_NUMBER_* enums) log.severity_text log.body # any type log.body.string # body coerced to string log.body["key"] # if body is a map log.body[0] # if body is a list log.attributes log.attributes["key"] log.dropped_attributes_count log.flags log.trace_id / log.trace_id.string log.span_id / log.span_id.string log.event_name # for event-shaped logs ``` As of v0.159, OTTL compares `pcommon.Value` paths such as `log.body` by their underlying type. Primitive values support the normal comparison operators, including numeric comparison between integer and double values. Maps and slices support equality and inequality only; incompatible types compare unequal. ```ottl # Parse JSON body into structured attributes set(log.attributes, ParseJSON(log.body.string)) where IsString(log.body) and IsMatch(log.body.string, "(?s)^\\s*\\{.*\\}\\s*$") # Normalize severity text set(log.severity_text, "ERROR") where log.severity_number >= SEVERITY_NUMBER_ERROR # Trace correlation flag set(log.attributes["has_trace"], true) where log.trace_id.string != "00000000000000000000000000000000" # Body-level redaction set(log.body, "REDACTED") where IsMatch(log.body.string, ".*(password|secret|token).*") ``` ## Profile context (Development, v0.124+) Since v0.162, the profile and profile-sample contexts are documented under upstream `pkg/ottl/contexts/xprofile/`, outside OTTL's stability guarantees; their paths are unchanged. ```ottl profile.profile_id profile.profile_id.string profile.attributes profile.attributes["key"] profile.dropped_attributes_count profile.time_unix_nano profile.time # time.Time profile.duration_unix_nano profile.duration profile.sample_type profile.sample_type.type profile.sample_type.unit profile.sample profile.period_type profile.period_type.type profile.period_type.unit profile.period profile.attribute_indices profile.original_payload_format profile.original_payload # []byte cache["key"] ``` ## Profile Sample context (Development, v0.132+) ```ottl profilesample.values # []int64 profilesample.attributes profilesample.attributes["key"] profilesample.link_index profilesample.timestamps_unix_nano # []int64 profilesample.timestamps # []time.Time profilesample.attribute_indices cache["key"] ``` ## OTelCol context (v0.147+, always available since v0.161) The `otelcol` context exposes Collector-side client and request data that is not part of the telemetry payload. It is read-only. Its `ottl.contexts.enableOTelColContext` gate became stable in v0.161 and was removed in v0.162. In v0.156, the routing connector deprecated its old `request` context in favor of these paths. OTLP receivers must set `include_metadata: true` for request metadata to reach downstream routing. HTTP/client key spelling may retain its form; gRPC keys are lowercase. ```ottl otelcol.client.addr otelcol.client.metadata["X-Tenant-Id"][0] # first HTTP/client metadata value otelcol.client.auth.attributes["subject"] # auth extension attributes otelcol.grpc.metadata["x-tenant-id"][0] # incoming gRPC metadata value ``` Prefer `otelcol.*` paths for routing/filtering conditions. If copying values into telemetry, only copy allowlisted, non-sensitive keys; metadata often includes authorization headers, cookies, or API keys. With routing `error_mode: ignore`, an evaluation error goes to `default_pipelines`; without that fallback it is dropped. ## Enums ### Span kind ```ottl SPAN_KIND_UNSPECIFIED # 0 SPAN_KIND_INTERNAL # 1 SPAN_KIND_SERVER # 2 SPAN_KIND_CLIENT # 3 SPAN_KIND_PRODUCER # 4 SPAN_KIND_CONSUMER # 5 ``` ### Span status ```ottl STATUS_CODE_UNSET # 0 STATUS_CODE_OK # 1 STATUS_CODE_ERROR # 2 ``` ### Log severity ```ottl SEVERITY_NUMBER_UNSPECIFIED # 0 SEVERITY_NUMBER_TRACE # 1-4 SEVERITY_NUMBER_DEBUG # 5-8 SEVERITY_NUMBER_INFO # 9-12 SEVERITY_NUMBER_WARN # 13-16 SEVERITY_NUMBER_ERROR # 17-20 SEVERITY_NUMBER_FATAL # 21-24 ``` ### Metric aggregation temporality ```ottl AGGREGATION_TEMPORALITY_UNSPECIFIED # 0 AGGREGATION_TEMPORALITY_DELTA # 1 AGGREGATION_TEMPORALITY_CUMULATIVE # 2 ``` ### Metric data type ```ottl METRIC_DATA_TYPE_NONE # 0 METRIC_DATA_TYPE_GAUGE # 1 METRIC_DATA_TYPE_SUM # 2 METRIC_DATA_TYPE_HISTOGRAM # 3 METRIC_DATA_TYPE_EXPONENTIAL_HISTOGRAM # 4 METRIC_DATA_TYPE_SUMMARY # 5 ``` -
functions.md 21.9 KB
# OTTL Functions Catalog Editor and converter reference for collector-contrib **v0.162.0**. Editors mutate telemetry; converters return values for use in expressions. See the upstream `pkg/ottl/ottlfuncs/README.md` for the authoritative source; since v0.162 the lambda converters are documented in `pkg/ottl/xottl/ottlfuncs/README.md` and `ProfileID` in `pkg/ottl/contexts/xprofile/README.md`. ## Feature gates (verified 2026-10-15) OTTL feature gates in `pkg/ottl/metadata.yaml`. Offline, cite this date, never call a value currently latest, and confirm it against the evidence before deploying. | Gate | v0.162.0 | v0.163.0 | |---|---|---| | `ottl.functions.enableLambda` (lambda converters) | alpha | alpha | | `ottl.set.allowNil` | stable | stable | | `pkg.ottl.functions.enableDynamicSliceArguments` (path/converter slice args of `Concat`, `keep_keys`) | none; `Concat` ungated since v0.161 | beta (on) | | `pkg.ottl.functions.enableExperimental` (new experimental `xottl` functions) | - | alpha | v0.163.0 also rejects non-list literal slice arguments such as `Concat("a", "-")` at parse time (CHANGELOG). Component gates (`transform`, `filter`, `routing`) live on the `otel-collector` component pages. ## Contents - [Transform-only functions](#transform-processor-only-functions) - [Editors](#editors-data-manipulation) - [String](#string-converters), [type conversion](#type-conversion), and [type checking](#type-checking) - [Lambda converters](#lambda-converters-v0157-alpha), [patterns](#pattern-matching), and [parsing](#data-parsing) - [Collections](#collections), [date/time](#datetime), [encoding](#hashing--encoding), and [OpenTelemetry-specific](#opentelemetry-specific) - [Safe function patterns](#function-patterns) ## Transform-processor-only functions The `transform` processor adds the following functions to the common OTTL catalog. They are not generally available in other OTTL-consuming components. The contexts and signatures below are pinned to the released [v0.162.0 transform processor source](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.162.0/processor/transformprocessor/README.md#supported-functions). | Context | Signature | Behavior and limits | |---------|-----------|---------------------| | `metric` | `convert_sum_to_gauge()` | Sum to Gauge; other metric types are unchanged. The conversion can violate Gauge semantics. | | `metric` | `convert_gauge_to_sum(aggregation_temporality, is_monotonic)` | Gauge to Sum; temporality is `"delta"` or `"cumulative"`; other metric types are unchanged. The caller owns the resulting Sum semantics. | | `metric` | `extract_avg_metric(suffix?)` | Creates a Gauge of sum/count per datapoint from Histogram, ExponentialHistogram, or Summary; default suffix `_avg`; skips datapoints with an absent sum or zero count. Added in v0.162. | | `metric` | `extract_count_metric(is_monotonic, suffix?)` | Creates a Sum from Histogram, ExponentialHistogram, or Summary counts; default suffix `_count`; only creates output when datapoints exist. | | `metric` | `extract_percentile_metric(percentile, suffix?)` | Creates a Gauge from Histogram or ExponentialHistogram buckets; `0 < percentile < 100`; default suffix `_p{percentile}`; the result is an interpolated estimate. | | `metric` | `extract_sum_metric(is_monotonic, suffix?)` | Creates a Sum from Histogram, ExponentialHistogram, or Summary sums; default suffix `_sum`; skips datapoints whose sum is absent. | | `datapoint` | `convert_summary_count_val_to_sum(aggregation_temporality, is_monotonic, suffix?)` | Creates a Sum from a Summary count; temporality is `"delta"` or `"cumulative"`; default suffix `_count`. | | `metric` | `convert_summary_quantile_val_to_gauge(attribute_key?, suffix?)` | Creates one Gauge datapoint per Summary quantile; defaults are attribute `quantile` and suffix `.quantiles`. | | `datapoint` | `convert_summary_sum_val_to_sum(aggregation_temporality, is_monotonic, suffix?)` | Creates a Sum from a Summary sum; temporality is `"delta"` or `"cumulative"`; default suffix `_sum`. | | `metric` | `copy_metric(name?, description?, unit?)` | Appends a full copy, optionally overriding metadata. The copy runs through later metric statements, so guard it with a condition that excludes the copy. | | `metric` | `scale_metric(factor, unit?)` | Scales Gauge, Sum, Histogram, and Summary values; optionally changes the unit. | | `metric` | `aggregate_on_attributes(function, attributes?)` | Aggregates Sum, Gauge, Histogram, or ExponentialHistogram datapoints. Functions are `sum`, `max`, `min`, `mean`, `median`, or `count`; histogram types support only `sum`. Omitted attributes retain all keys, while `[]` drops all keys. | | `metric` | `convert_exponential_histogram_to_histogram(distribution, explicit_bounds)` | Converts ExponentialHistogram to explicit Histogram using `upper`, `midpoint`, `uniform`, or `random`; bounds must be non-empty and strictly increasing. Since v0.162 it converts positive, negative, and zero buckets and preserves the total count. This lossy conversion is not specified by OpenTelemetry. | | `metric` | `aggregate_on_attribute_value(function, attribute, values, new_value)` | Aggregates selected attribute values for Sum, Gauge, Histogram, or ExponentialHistogram datapoints; histogram types support only `sum`. | | `datapoint` | `merge_histogram_buckets(target_value, method?)` | Explicit Histograms only. Default `remove_explicit_bound` removes a matching bound; `limit_buckets` requires a positive integer target and reduces resolution. Other metric types are unchanged. | | `log` | `ParseCEF(target)` | Parses Common Event Format into a map, including an optional syslog prefix and string-valued extensions; malformed or empty input errors. | | `log` | `ParseCLF(target, format?)` | Parses CLF (`"clf"`, default) or NCSA combined (`"combined"`) text into a map; malformed or empty input errors. | | `log` | `ParseELF(target)` | Parses a complete W3C Extended Log Format block into directive metadata, fields, and entries; requires a `#Version` directive and `#Fields` before data. Added in v0.158. | | `log` | `ParseLEEF(target)` | Parses LEEF 1.0/2.0 into a map; malformed or empty input errors; attribute values remain strings. | | `span` | `set_semconv_span_name(semconv_version, original_span_name_attribute?)` | Derives low-cardinality HTTP, RPC, messaging, or database span names. v0.162 and v0.163 accept semantic-convention versions 1.37.0 through 1.43.0 (verified 2026-10-15; [v0.163.0 README](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.163.0/processor/transformprocessor/README.md#set_semconv_span_name)); unrelated spans are unchanged. | For full behavior, examples, and edge cases, follow the tag-pinned [metrics](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.162.0/processor/transformprocessor/README.md#convert_sum_to_gauge), [logs](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.162.0/processor/transformprocessor/README.md#parsecef), and [traces](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.162.0/processor/transformprocessor/README.md#set_semconv_span_name) function sections. The registrations that constrain the contexts are also tag-pinned: [metric/datapoint](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.162.0/processor/transformprocessor/internal/metrics/functions.go), [log](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.162.0/processor/transformprocessor/internal/logs/functions.go), and [span](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.162.0/processor/transformprocessor/internal/traces/functions.go). ## Editors (data manipulation) Editors are lowercase. Every OTTL statement contains exactly one. ### `set` ```ottl set(target, value) set(span.attributes["env"], "production") set(log.body, Concat([log.severity_text, ": ", log.body.string], "")) ``` `set` passes `nil` directly to the target: a map or slice path is cleared, a `pcommon.Value` (for example a missing attribute key) is set to an empty value, and scalar or struct-typed targets return an error. This is the `ottl.set.allowNil` behavior, beta (on by default) in v0.161 and stable in v0.162; through v0.160 a nil value was a no-op. Guard missing sources with `where source != nil` when the target must stay unchanged. ### `clear` (v0.160+) ```ottl clear(target) clear(span.attributes["http.request.header.authorization"]) ``` Sets the target to the empty value for its current type: for example, `""`, `0`, `false`, an empty `pcommon.Map`, `pcommon.Slice`, or `pcommon.Value`, or a zero TraceID/SpanID/time. For Go maps, slices, pointers, byte slices, and a current `nil` value, it passes `nil`; whether the path setter accepts that value is target-specific and may produce an evaluation error. ### `append` ```ottl append(target, value) append(target, values = ["tag1", "tag2"]) append(span.attributes["tags"], "processed") ``` If `target` is a scalar, it is converted to a slice first. ### `delete_key` / `delete_matching_keys` / `delete_index` ```ottl delete_key(span.attributes, "internal.debug") delete_matching_keys(span.attributes, "(?i).*password.*") delete_index(log.attributes["tags"], 0) # v0.145+, one slice index delete_index(log.attributes["tags"], 0, 3) # delete indices 0, 1, 2 ``` ### `keep_keys` / `keep_matching_keys` ```ottl keep_keys(span.attributes, ["http.method", "http.status_code", "http.route"]) keep_matching_keys(span.attributes, "(?i)^http\\..*") ``` ### `flatten` ```ottl flatten(target) flatten(target, prefix, depth, resolveConflicts?) # v0.139+ flatten(log.body) flatten(span.attributes, "nested", 2) flatten(log.attributes, resolveConflicts = true) ``` ### `limit` ```ottl limit(target, count, priority_keys[]) limit(span.attributes, 10, ["http.method", "http.status_code"]) ``` ### `merge_maps` ```ottl merge_maps(target, source, "insert" | "update" | "upsert") merge_maps(span.attributes, ParseJSON(span.attributes["meta"]), "upsert") ``` - `insert` — only set keys not already in target - `update` — only overwrite keys already in target - `upsert` — both ### `truncate_all` ```ottl truncate_all(target, max_length) truncate_all(target, max_length, utf8_safe = false) # v0.148+ truncate_all(target, max_length, truncation_marker = "(truncated)") # v0.157+ truncate_all(span.attributes, 256) ``` UTF-8 safe by default since v0.148 — truncates at character boundaries, so the result may be slightly shorter than `max_length`. Pass `utf8_safe = false` for the previous byte-level behavior. The optional `truncation_marker` is included inside `max_length`, so the complete result never exceeds the limit. A marker longer than the limit is an error. ### `stringify_all` ```ottl stringify_all(target) stringify_all(log.attributes) stringify_all(resource.attributes) ``` Converts every non-string value in a map to its string representation. Numbers and booleans become scalar strings, bytes become base64, maps and slices become JSON strings, and empty values become `""`. Added in collector-contrib v0.155.0. ### Dynamic indexing of converter results Converter results can be indexed with expressions that evaluate to a string or integer, not only literals: ```ottl Split(log.body.string, ",")[log.attributes["index"]] ParseJSON(log.body.string)[log.attributes["field_name"]] ``` This was added in collector-contrib v0.155.0. On older collectors, keep converter-result indexes literal, such as `[0]` or `["field"]`. ### `replace_match` / `replace_all_matches` ```ottl replace_match(target, pattern, replacement, function?, format?) replace_all_matches(target, pattern, replacement, function?, format?) replace_match(span.name, "GET ", "") replace_all_matches(span.attributes, "/user/*/id/*", "/user/{userId}/id/{id}") ``` Glob patterns (not regex). Use `replace_pattern`/`replace_all_patterns` for regex. ### `replace_pattern` / `replace_all_patterns` ```ottl replace_pattern(target, regex, replacement, function?, format?) replace_all_patterns(target, "key" | "value", regex, replacement, function?, format?) # Strip query string replace_pattern(attributes["http.url"], "\\?.*$", "") # Redact a token query parameter (YAML form; in a raw OTTL string use ${1}) replace_pattern(attributes["http.url"], "([?&]token=)[^&]+", "$${1}REDACTED") # Map-wide patterns over keys or values replace_all_patterns(span.attributes, "value", "\\d{16}", "[REDACTED]") replace_all_patterns(span.attributes, "key", "^kube_", "k8s.") ``` **Backreferences in YAML.** OTTL uses `${1}`, `${2}`, … The collector's YAML loader treats `$` as an env-var marker, so a single `$` in YAML produces nothing in the OTTL string. Write `$${1}` in YAML to emit `${1}` in OTTL. Writing `"$1REDACTED"` produces a literal `$1REDACTED` — silent failure. **RE2 repeat-count limit.** Patterns like `^(.{1024}).*` fail to compile with "invalid repeat count". For length-based truncation, use `Substring` + `Len` instead: ```ottl set(attributes["db.statement"], Substring(attributes["db.statement"], 0, 1024)) where attributes["db.statement"] != nil and Len(attributes["db.statement"]) > 1024 ``` --- ## String converters ### `Concat`, `Split`, `Substring` ```ottl Concat(["user", span.attributes["user.id"], "action"], "-") # "user-123-action" when user.id == "123" Split(span.name, "/") # ["", "api", "v1", "users"] Split(span.name, "/")[1] # "api" Substring(span.span_id.string, 0, 8) # first 8 chars Substring(target, start, length, utf8_safe?) # byte offsets; utf8_safe defaults false; added v0.156 Concat(Split(span.attributes["request.id"], "-"), "") # v0.161+: values may be a slice-valued path or converter ``` With `utf8_safe=true`, a start inside a multi-byte character advances to the next UTF-8 boundary and an end inside one backs up to the previous boundary. The result stays valid UTF-8 but can be shorter than `length` bytes. ### Case ```ottl ToLowerCase(s) / ToUpperCase(s) ToCamelCase(s) / ToSnakeCase(s) ConvertCase(s, "lower" | "upper" | "camel" | "snake") ``` ### Trim ```ottl Trim(s, cutset?) TrimPrefix(s, prefix) # v0.124+ TrimSuffix(s, suffix) # v0.124+ ``` ### Prefix/suffix tests ```ottl HasPrefix(s, prefix) # bool HasSuffix(s, suffix) # bool where HasPrefix(span.name, "internal.") ``` ### `Format` ```ottl Format("user=%s req=%d", [span.attributes["user.id"], span.attributes["request.count"]]) ``` --- ## Type conversion ```ottl String(v) # any -> string Int(v) # any -> int64 Double(v) # any -> float64 Bool(v) # v0.143+; loose coercion (see below) ParseInt(s, base) # ParseInt("ff", 16) -> 255 Hex(bytes) # bytes -> hex string ``` **`Bool` coercion is loose, not Pythonic.** Booleans pass through; any non-zero integer or double is `true`; zero is `false`; strings use Go boolean parsing (`1`, `t`, `T`, `TRUE`, `true`, `True`, and false equivalents). Invalid strings and unsupported types error, while `nil` returns `nil`. --- ## Type checking ```ottl IsString(v) / IsInt(v) / IsDouble(v) / IsBool(v) IsList(v) / IsMap(v) IsEmpty(v) # nil, zero value, or empty collection IsInCIDR(ip_string, ["10.0.0.0/8", "192.168.0.0/16"]) # v0.146+ ``` `IsInCIDR` returns `false` for invalid IP strings — useful in conditions: `where IsInCIDR(attributes["client.ip"], ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"])`. --- ## Lambda converters (v0.157+, alpha) These converters require the alpha `ottl.functions.enableLambda` feature gate. For slices, two-argument lambdas receive `(index, value)`; for maps they receive `(key, value)`. Use `_` for an ignored parameter. ```ottl All(source, (k, v) => predicate) # every element; true for empty input Any(source, (k, v) => predicate) # at least one; false for empty input Filter(source, (k, v) => predicate) # filtered slice or map Find(source, (k, v) => predicate, mapper?) # first value, mapped value, or nil MapEach(source, (k, v) => mapped_value) # transform values MapKeys(map, (k, v) => mapped_key) # mapped_key must be a string Reduce(source, seed, (acc, k, v) => next_acc) # fold; empty input returns seed When(() => condition, true_value, false_value) # select one value ``` Map iteration order is not stable, so order-dependent `Reduce` expressions are not deterministic for maps. If `MapKeys` creates duplicate keys, which value is retained is unspecified. --- ## Pattern matching ### `IsMatch` ```ottl IsMatch(target, pattern) # bool where IsMatch(span.name, "^GET /api/.*") where not IsMatch(span.name, ".*health.*") ``` ### `ExtractPatterns` ```ottl ExtractPatterns(target, "(?P<user>\\w+)/(?P<action>\\w+)") # Returns map keyed by the named groups. ``` Named capture groups required — unnamed groups are not surfaced as map keys. ### `ExtractGrokPatterns` (v0.130+) ```ottl ExtractGrokPatterns(log.body, "%{IP:client_ip} %{WORD:method} %{URIPATHPARAM:path}") ExtractGrokPatterns(log.body, "%{URI}", true) # named only ExtractGrokPatterns(log.body, "%{MYPATTERN}", true, ["MYPATTERN=%{IP}:%{INT}"]) # custom defs ``` Built-in pattern library covers HTTP, syslog, paths, IPs, dates, etc. Cheaper than expressing the same thing as a raw regex. --- ## Data parsing ```ottl ParseJSON(s) # JSON string -> map | list ParseCSV(s, headers, delimiter?, headerDelim?, mode?) ParseKeyValue(s, kv_delim?, pair_delim?) # ParseKeyValue("a=1&b=2", "=", "&") ParseXML(s) / ParseSimplifiedXML(s) ParseSeverity(value, mapping) # map values -> SEVERITY_NUMBER_* URL(s) # v0.127+; { url.scheme, url.domain, url.port, url.path, … } UserAgent(s) # v0.134+; { user_agent.name, .version, os.name, os.version, … } ``` `ParseJSON`, `IsString`, and `IsMatch` are available in v0.156 and do not require the v0.157 `ottl.functions.enableLambda` feature gate. ```ottl # Conditional JSON parsing set(log.attributes, ParseJSON(log.body.string)) where IsString(log.body) and IsMatch(log.body.string, "(?s)^\\s*\\{.*\\}\\s*$") # URL parsing set(span.attributes["http.host"], URL(span.attributes["http.url"])["url.domain"]) ``` ### Fallback values ```ottl Coalesce([span.attributes["user.id"], resource.attributes["user.id"], "fallback"]) ``` --- ## XML ```ottl GetXML(target, xpath) # select elements InsertXML(target, xpath, value) RemoveXML(target, xpath) ConvertAttributesToElementsXML(target, xpath?) ConvertTextToElementsXML(target, xpath?, elementName?) ``` --- ## Collections ```ottl Len(target) # works for string, list, map Keys(map) / Values(map) # -> list ContainsValue(list, value) # bool Index(target, value) # v0.126+; -1 if not found Sort(list, "asc" | "desc") # v0.125+ SliceToMap(list, [keyPath]?, [valuePath]?) # v0.128+; turn objects into a map ``` ```ottl SliceToMap(resource.attributes["items"], ["name"]) # key by .name SliceToMap(resource.attributes["items"], ["name"], ["value"]) # explicit value ``` --- ## Date/time ```ottl Now() # current time Time(s, format, location?, locale?) # locale added v0.136 FormatTime(t, format) Duration(s) # "1h30m" -> time.Duration TruncateTime(t, duration) Unix(seconds, nanoseconds?) UnixSeconds(t) / UnixMilli(t) / UnixMicro(t) / UnixNano(t) Year(t) / Month(t) / Day(t) / Hour(t) / Minute(t) / Second(t) / Nanosecond(t) / Weekday(t) Hours(d) / Minutes(d) / Seconds(d) / Milliseconds(d) / Microseconds(d) / Nanoseconds(d) ``` --- ## Hashing & encoding ```ottl SHA256(s) / SHA512(s) SHA1(s) / MD5(s) # cryptographically weak; avoid for security FNV(s) # int64 Murmur3Hash(s) / Murmur3Hash128(s) # v0.129+; hexadecimal string XXH3(s) / XXH128(s) # v0.135+; hexadecimal string Decode(value, encoding) # v0.141+; "base64", "base64-raw", "base64-url", "base64-raw-url", IANA charsets Base64Encode(s, variant?) # v0.147+; default base64 Hex(bytes) ``` --- ## OpenTelemetry-specific ```ottl SpanID(bytes | hex_string) # hex strings accepted v0.142+ TraceID(bytes | hex_string) # same ProfileID(bytes | hex_string) # v0.124+ IsRootSpan() # span context only IsValidLuhn(s) # credit card checksum CommunityID(srcIP, srcPort, dstIP, dstPort, protocol?, seed?) # v0.131+; network flow hash ToKeyValueString(map, delim?, pair_delim?, sort?) ``` --- ## Identifier generation ```ottl UUID() UUIDv7() # v0.138+; time-ordered ``` `UUIDv7` is time-ordered; choose the identifier form according to the consuming system's requirements. --- ## Math ```ottl Log(value) # natural logarithm ``` OTTL's basic arithmetic operators (`+`, `-`, `*`, `/`) cover most needs; `Log` is the rare named math converter. --- ## Function patterns ### Safe type conversion ```ottl set(span.attributes["count_int"], Int(span.attributes["count"])) where IsString(span.attributes["count"]) and IsMatch(span.attributes["count"], "^\\d+$") ``` ### Chained string operations ```ottl set(span.attributes["normalized"], ToLowerCase(Trim(span.attributes["value"], " "))) ``` ### Conditional parsing into cache ```ottl set(log.cache["parsed"], ParseJSON(log.body.string)) where IsString(log.body) and IsMatch(log.body.string, "(?s)^\\s*\\{.*\\}\\s*$") ``` Then read from `log.cache["parsed"]` in subsequent statements without paying the parse cost again. ### Hash-based sampling ```ottl set(span.attributes["sampled"], true) where IsMatch(XXH3(span.trace_id.string), "^(0[0-9a-f]|1[0-9])") # ~10.2% ``` -
quick-reference.md 12.7 KB
# OTTL Quick Reference Recipes, regex patterns, and a troubleshooting table. Pair with `contexts.md` for paths and `functions.md` for full signatures. ## Contents - [Common patterns](#common-patterns) - [Processor configuration](#processor-configuration) - [Regular expressions](#regular-expressions) - [Debugging](#debugging) - [Performance](#performance) - [Troubleshooting and verification](#troubleshooting) ## Common patterns ### Attribute management ```ottl # Set attribute set(span.attributes["environment"], "production") # Set if missing set(span.attributes["version"], "unknown") where span.attributes["version"] == nil # Copy from resource (guard: since v0.161 a nil source creates an empty-valued key) set(span.attributes["service"], resource.attributes["service.name"]) where resource.attributes["service.name"] != nil # Rename set(span.attributes["host.name"], span.attributes["hostname"]) where span.attributes["hostname"] != nil delete_key(span.attributes, "hostname") # Drop sensitive keys delete_matching_keys(span.attributes, "(?i).*(password|secret|token|apikey).*") # Whitelist (fail-closed) keep_keys(span.attributes, ["http.method", "http.status_code", "http.route"]) ``` ### String manipulation ```ottl set(span.attributes["method"], ToUpperCase(String(span.attributes["http.method"]))) set(span.attributes["short_id"], Substring(span.span_id.string, 0, 8)) set(span.attributes["service"], Split(resource.attributes["service.name"], "-")[0]) set(log.body, Concat([log.severity_text, ": ", log.body.string], "")) set(log.body, Trim(log.body.string, " \t\n")) ``` ### Data parsing ```ottl # JSON body -> attributes set(log.attributes, ParseJSON(log.body.string)) where IsString(log.body) and IsMatch(log.body.string, "(?s)^\\s*\\{.*\\}\\s*$") # Query string -> map set(span.attributes["params"], ParseKeyValue(span.attributes["http.query"], "=", "&")) # URL parts (v0.127+) set(span.attributes["http.host"], URL(span.attributes["http.url"])["url.domain"]) # User agent (v0.134+) set(span.attributes["client.os"], UserAgent(span.attributes["http.user_agent"])["os.name"]) # Grok (v0.130+) — much cleaner than the equivalent regex set(log.attributes, ExtractGrokPatterns(log.body.string, "%{IP:client_ip} %{WORD:method} %{URIPATHPARAM:path}")) ``` ### Error / status detection ```ottl # HTTP errors set(span.status.code, STATUS_CODE_ERROR) where IsInt(span.attributes["http.status_code"]) and Int(span.attributes["http.status_code"]) >= 400 # Exception spans set(span.status.code, STATUS_CODE_ERROR) where span.attributes["exception.type"] != nil set(span.status.message, span.attributes["exception.message"]) where IsString(span.attributes["exception.message"]) # Slow spans set(span.attributes["slow"], true) where (span.end_time_unix_nano - span.start_time_unix_nano) > 1000000000 ``` ### Filter conditions ```ottl # Skip health checks where not IsMatch(span.name, ".*(health|ready|alive).*") # Errors only where log.severity_number >= SEVERITY_NUMBER_ERROR # Server spans only where span.kind == SPAN_KIND_SERVER # Drop traffic from internal CIDRs where IsInCIDR(span.attributes["client.address"], ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"]) ``` ### Sampling ```ottl # Hash-based ~10.2%; 26 of 256 possible first-byte values match set(span.attributes["sampled"], true) where IsMatch(XXH3(span.trace_id.string), "^(0[0-9a-f]|1[0-9])") # Always-sample errors set(span.attributes["sampled"], true) where span.status.code == STATUS_CODE_ERROR # Latency-based set(span.attributes["sampled"], true) where (span.end_time_unix_nano - span.start_time_unix_nano) > 1000000000 ``` ### Time operations ```ottl # Duration in ms set(span.attributes["duration_ms"], (span.end_time_unix_nano - span.start_time_unix_nano) / 1000000) # Format timestamp set(log.attributes["formatted_time"], FormatTime(log.time, "%Y-%m-%d %H:%M:%S")) # Stamp processing time set(span.attributes["processed_at"], Now()) # Hour bucket set(log.attributes["hour"], Hour(log.time)) ``` ### Redaction ```ottl # Credit cards replace_all_patterns(log.attributes, "value", "\\b(?:\\d{4}[\\s-]?){3}\\d{4}\\b", "****-****-****-****") # Emails replace_all_patterns(log.attributes, "value", "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}", "[REDACTED_EMAIL]") # IPv4 replace_all_patterns(log.attributes, "value", "\\b\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\b", "[REDACTED_IP]") # Deterministic hash example; policy must permit retaining linkable identifiers set(span.attributes["user.email_hash"], SHA256(span.attributes["user.email"])) delete_key(span.attributes, "user.email") ``` ## Processor configuration ### `transform` ```yaml processors: transform: error_mode: ignore # ignore | silent | propagate trace_statements: - context: span statements: - set(span.attributes["processed"], true) log_statements: - context: log statements: - set(log.attributes["source"], "collector") metric_statements: - context: datapoint statements: - set(datapoint.attributes["env"], "prod") ``` Set `shared_cache: true` only on advanced statement groups that must share context cache state across separate iterations. Sharing occurs only among groups in the same transform processor instance that opt in; the option is experimental and defaults to `false`. ### `filter` ```yaml processors: filter: error_mode: ignore trace_conditions: - 'IsMatch(span.name, "^/health.*")' - 'span.kind == SPAN_KIND_INTERNAL' log_conditions: - 'log.severity_number < SEVERITY_NUMBER_WARN' ``` ### `routing` connector ```yaml connectors: routing: error_mode: ignore default_pipelines: [traces/default] table: - condition: 'resource.attributes["env"] == "prod"' pipelines: [traces/prod] - condition: 'span.status.code == STATUS_CODE_ERROR' pipelines: [traces/errors] ``` With `error_mode: ignore`, an evaluation error is logged and the payload is sent to `default_pipelines`; without a configured fallback it is dropped. Metadata-based routing also requires `include_metadata: true` on the receiving OTLP protocol. Use `otelcol.client.metadata["X-Tenant"][0]` for HTTP/client metadata and lowercase `otelcol.grpc.metadata["x-tenant"][0]` for gRPC metadata. ### `tail_sampling` ```yaml processors: tail_sampling: policies: - name: errors type: ottl_condition ottl_condition: error_mode: ignore span: - 'span.status.code == STATUS_CODE_ERROR' - name: slow type: ottl_condition ottl_condition: error_mode: ignore span: - '(span.end_time_unix_nano - span.start_time_unix_nano) > 1000000000' ``` Set the OTTL policy's `error_mode` explicitly; it is nested under `ottl_condition`, not at the `tail_sampling` processor level. ## Regular expressions ### Common patterns ```ottl "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}" # email "\\b\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\b" # IPv4 "\\b(?:\\d{4}[\\s-]?){3}\\d{4}\\b" # credit card "\\b\\d{3}-\\d{3}-\\d{4}\\b" # US phone "https?://[^\\s]+" # URL "[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}" # UUID "\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}(?:\\.\\d{3})?Z?" # ISO 8601 "^\\{.*\\}$" # JSON object "^\\d+$" # digits only ``` ### Escape rules In OTTL strings, escape regex metacharacters with double backslash. Single-backslash YAML chars get eaten before OTTL sees them. | You want in regex | Write in OTTL string | |-------------------|---------------------| | `\d` | `\\d` | | `\s` | `\\s` | | `\w` | `\\w` | | `\.` | `\\.` | | `\*` | `\\*` | | `\+` | `\\+` | **Backreference escape (YAML).** OTTL `${1}` must be written as `$${1}` in YAML — the loader treats `$` as an env-var marker. `"$1REDACTED"` becomes a literal string at the OTTL level, no replacement. **RE2 repeat-count limit.** `(.{1024}).*` fails to compile. Use `Substring` + `Len` for length truncation: ```ottl set(attributes["x"], Substring(attributes["x"], 0, 1024)) where attributes["x"] != nil and Len(attributes["x"]) > 1024 ``` ## Debugging ### Enable debug logging ```yaml service: telemetry: logs: level: debug ``` ### Add probes ```ottl # Track that a statement ran set(span.attributes["ottl.processed"], true) set(span.attributes["ottl.timestamp"], Now()) # ~0.1% sampled debug snapshot (4 of 4096 three-hex-digit prefixes) set(span.attributes["debug.original_name"], span.name) where IsMatch(XXH3(span.trace_id.string), "^00[0-3]") ``` ### Validate before transforming ```ottl where IsString(span.attributes["count"]) and IsMatch(span.attributes["count"], "^\\d+$") where span.attributes["value"] != nil where Len(String(log.body)) > 0 ``` ## Performance **Do** - Order conditions by selectivity (cheap and most-discriminating first). - Use early filtering to drop work at the smallest scope. - Cache expensive function results in `<context>.cache`. - Pick the most specific context (operate on `datapoint`, not on `metric.data_points`). - Batch attribute pruning with `keep_keys` rather than many `delete_key` statements. **Don't** - Repeat expensive function calls inside the same statement set. - Run complex regex on large bodies. - Process telemetry that a cheaper filter could drop earlier in the pipeline. - Suppress error_mode noise (`silent`) without first verifying behavior. ### Cache pattern ```ottl set(span.cache["url_parts"], Split(span.attributes["http.url"], "/")) set(span.attributes["api_version"], span.cache["url_parts"][2]) ``` ### Early-filter ordering ```ottl # Cheap kind check fails first; regex only runs for SERVER spans. where span.kind == SPAN_KIND_SERVER and IsMatch(span.name, "expensive.*") ``` ## Troubleshooting ### Common errors | Symptom | Cause | Fix | |---------|-------|-----| | `nil value`, or an empty attribute appears | Path resolves to a missing field; since v0.161 `set` passes nil through | Add `where path != nil` (or `IsString(path)` etc.) | | `type mismatch` | Wrong types in arithmetic / function args | Convert with `Int`, `Double`, `String` after type-checking | | `invalid regex` | Unescaped metacharacter | Double-escape: `\\d`, `\\.`, `\\s` | | `division by zero` | `/` with a zero divisor | Add `where divisor != 0` | | Backreferences appear literal | YAML ate the `$` | Use `$${1}` in YAML for OTTL `${1}` | | `attributes` processor doesn't touch resource | Wrong processor | Use `resource` processor or `transform` with `context: resource` | | `cache["x"]` errors in span context | Cache requires a context prefix since v0.120 | Write `span.cache["x"]` | | `Bool("yes")` errors | String inputs use Go boolean parsing; arbitrary non-empty strings are not truthy | Use `IsMatch` with an explicit pattern, or check with `where` | | Unknown function `Base64Decode` | Removed in v0.161 (deprecated since v0.141) | Use `Decode(value, "base64")` | Since v0.160, parser diagnostics identify the input kind, line and column, nearby source, and the expected token when available, for example `statement has invalid syntax at 1:5: (expected ")" Key*)`. Odd-length byte literals instead report `byte literals must have an even number of hexadecimal digits` within the wrapped expression error. ### Checklist before shipping 1. Syntax: parens balanced, strings closed, regex escapes doubled. 2. Types: type-check before conversion (`IsString`, `IsInt`). 3. Existence: nil-guard paths that may be absent. 4. Logic: dry-run with the [telemetrygen verification recipe](../../otel-telemetrygen/SKILL.md#verify-collector-behavior) — generate a known input, capture the file-exporter output, confirm the transformation. The eye test misses too many gotchas. 5. Performance: most-selective `where` clause first. ### Safe transformation skeletons ```ottl # Safe JSON parse set(log.cache["parsed"], ParseJSON(log.body.string)) where IsString(log.body) and IsMatch(log.body.string, "^\\s*[\\{\\[].*[\\}\\]]\\s*$") # Safe numeric conversion set(span.attributes["count_int"], Int(span.attributes["count"])) where IsString(span.attributes["count"]) and IsMatch(span.attributes["count"], "^-?\\d+$") # Safe duration calc set(span.attributes["duration_ns"], span.end_time_unix_nano - span.start_time_unix_nano) where span.end_time_unix_nano > span.start_time_unix_nano ```
-
-
SKILL.md 6.7 KB
--- name: otel-ottl description: OpenTelemetry Transformation Language (OTTL) expert for writing and debugging telemetry transformations in the OpenTelemetry Collector. Use when authoring or reviewing `transform`, `filter`, `tail_sampling` processor configs or `routing` connector configs, debugging OTTL syntax or semantics, transforming traces, metrics, logs, or profiles, or converting data-processing requirements into OTTL statements. --- # OpenTelemetry Transformation Language (OTTL) OTTL transforms or selects telemetry inside Collector components. This skill is pinned to collector-contrib **v0.162.0**. Function, path, default, and feature-gate availability varies by release; when the user's version differs, verify against the matching upstream tag. Dated OTTL feature-gate status for v0.162.0 and v0.163.0 is in [references/functions.md](references/functions.md#feature-gates-verified-2026-10-15). ## Workflow 1. **Choose the component.** `transform` rewrites, `filter` drops, `tail_sampling` decides whether to retain traces, and `routing` sends telemetry to pipelines. A component controls its available contexts and functions. 2. **Choose the lowest usable context.** Lower contexts can read their parents (for example, a span can read `resource.attributes`), but parents cannot read children. Use `datapoint` for point attributes instead of traversing `metric.data_points`. 3. **Verify every emitted function, then write the statement.** Confirm each function's exact identifier and signature in [references/functions.md](references/functions.md). If an identifier is absent, treat it as unsupported instead of deriving or substituting a plausible name. An editor such as `set` or `delete_key` mutates data and may have a `where` condition. Converters such as `ParseJSON` and `IsMatch` return values; they do not mutate. 4. **Set error behavior deliberately.** `ignore` logs statement errors and continues; `silent` continues without logging; `propagate` returns the error and can cause the component to drop the payload. Transform and filter default to `ignore`; the transform default-error gate was removed in v0.162.0, so naming it in `--feature-gates` fails startup (history: the `otel-collector` skill's transform quirks), and the filter gate is stable and cannot be disabled. Routing also defaults to `ignore` while its beta default-error feature gate is enabled. For routing, `ignore` sends an errored payload to `default_pipelines`; configure that fallback or the payload is dropped. 5. **Verify end to end.** Validate the exact Collector version, then send known telemetry and inspect file-exporter output. Use the [telemetrygen recipe](../otel-telemetrygen/SKILL.md#verify-collector-behavior). ```ottl set(span.attributes["env"], "prod") where resource.attributes["env"] == nil ``` ## Load only what the task needs - [Contexts](references/contexts.md) — exact paths, hierarchy, enums, and request metadata. - [Functions](references/functions.md) — editor/converter signatures and release availability. - [Quick reference](references/quick-reference.md) — component YAML, recipes, escaping, troubleshooting, and safe skeletons. For a single path or function, read only the relevant section instead of loading the full catalogs. ## Safety and correctness gates - Guard optional or polymorphic input before conversion: `where x != nil`, `IsString(x)`, or the appropriate type check. - For JSON-object-only work, guard both the type and shape before calling `ParseJSON`, for example `IsString(log.body) and IsMatch(log.body.string, "(?s)^\\s*\\{.*\\}\\s*$")`. The RE2 `(?s)` flag admits pretty-printed objects containing newlines. Checking `IsMap` after parsing does not prevent arrays or scalar JSON from being parsed. - On a version-pinned request, confirm every chosen path and function against that release tag; do not assume a function listed for this skill's v0.162 anchor exists in an older release. For v0.156 JSON-object parsing, `ParseJSON`, `IsString`, and `IsMatch` are available without the v0.157 alpha lambda feature gate. - Request metadata is read-only and may contain credentials. Copy only explicitly allowlisted, non-sensitive keys. OTLP metadata routing requires `include_metadata: true` on the receiver. HTTP/client header spelling may retain its form (`otelcol.client.metadata["X-Tenant"][0]`); gRPC metadata keys are lowercase (`otelcol.grpc.metadata["x-tenant"][0]`). - The routing `request` context is deprecated as of v0.156; use `otelcol.client.metadata` or `otelcol.grpc.metadata`. - Log-record-specific rewrites of shared resource or scope data require `flatten_data: true` and the alpha `transform.flatten.logs` gate. This copies and regroups data; do not enable it accidentally. - Since v0.161, `set(target, nil)` is no longer a no-op: the `ottl.set.allowNil` gate became beta in v0.161 and stable (cannot be disabled) in v0.162. A nil value empties a map, slice, or `pcommon.Value` target (an attribute key is created with an empty value) and errors on scalar targets. Use a `where source != nil` guard when the destination must remain unchanged for missing input. - Hashing an identifier does not necessarily anonymize it. Apply the organization's data-handling policy before retaining deterministic hashes of personal data. ## Frequent syntax traps - In Collector YAML, write an OTTL replacement backreference `${1}` as `$${1}`. A replacement such as `$1REDACTED` is literal and silently fails to substitute the capture. - Go RE2 rejects large counted repetitions such as `(.{1024}).*`; use `Substring` with a nil/type guard and `Len`, or `truncate_all` for a map. - Current span-event paths use `spanevent.*`, not `span_event.*`. Cache paths are context-qualified, such as `span.cache["parsed"]`. - Quote any OTTL statement containing a map literal when YAML includes a space after `:`, for example `'set(log.attributes["a"], {"foo": "bar"})'`; otherwise YAML parses `: ` as a mapping. - Since v0.159, polymorphic `pcommon.Value` paths compare by their underlying type; maps and slices support only equality and inequality, while primitive values also support ordering. - Use `Decode(value, "base64")`; `Base64Decode` was removed in v0.161. - Regex escapes inside OTTL strings are doubled (`\\d`, `\\s`, `\\.`). ## Upstream sources - [OTTL package](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.162.0/pkg/ottl) - [Transform processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.162.0/processor/transformprocessor) - [Filter processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.162.0/processor/filterprocessor) - [Routing connector](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.162.0/connector/routingconnector)
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.