{
  "skill": "otel-profiles",
  "cases": [
    {
      "id": "k3d-pods-missing",
      "prompt": "I'm running the OpenTelemetry eBPF profiler as a DaemonSet on a local k3d cluster, image otel/opentelemetry-collector-ebpf-profiler:0.162.0, hostPID and privileged. First it crashed with 'failed to determine system configs: system analysis request was not handled for pid 1234'. I added pid_namespace_translation: true and now it fails with 'failed to attach scheduler monitor: ... neither debugfs nor tracefs are mounted'. What exactly do I need to change so I get profiles for my pods? Give me the config and manifest changes; don't run anything.",
      "expectations": [
        "Explains that k3d nodes are containers, so hostPID only reaches the node container's PID namespace and pods live in descendant PID namespaces.",
        "States that the boolean pid_namespace_translation on 0.162.0 (ebpf-profiler v0.0.202636) only covers the profiler's own namespace, so pod processes stay invisible even once it starts.",
        "Recommends pid_namespace_translation_mode: auto (or recursive) and states it requires ebpf-profiler v0.0.202640 or newer, which 0.162.0 does not ship (a 0.163.0 nightly or the 0.163.0 release).",
        "Warns that each translation key is rejected as an invalid key by the other profiler version, so the key must match the image.",
        "Fixes tracefs by mounting tracefs inside the node's mount namespace (for example a privileged init container using nsenter) plus a /sys/kernel/tracing hostPath, and does not claim that a hostPath alone is enough on k3d.",
        "Does not claim to have run the profiler, applied manifests, or inspected the user's cluster (read-only upstream lookups are fine)."
      ]
    },
    {
      "id": "eks-daemonset-to-pyroscope",
      "prompt": "We want continuous CPU profiling for every pod in our EKS cluster (Linux nodes, recent kernels) using OpenTelemetry, sent to our self-hosted Grafana Pyroscope at pyroscope.observability.svc:4040. Write the Kubernetes manifests and Collector config, with namespace, pod, and service labels on the profiles. Pin real versions and tell me anything still experimental.",
      "expectations": [
        "Uses the otel/opentelemetry-collector-ebpf-profiler image (otelcol-ebpf-profiler distribution) with a pinned version and the profiling receiver.",
        "Starts the Collector with the service.profilesSupport feature gate.",
        "DaemonSet sets hostPID: true, privileged (or the eBPF capabilities), and mounts /sys/kernel/tracing from the host.",
        "Uses k8s_attributes with pod_association on the container.id resource attribute and RBAC granting get/list/watch on at least pods and namespaces.",
        "Exports with otlp_grpc (or otlp_http) to Pyroscope on port 4040, not a separate 4317 port.",
        "Configures memory_limiter with absolute limit_mib/spike_limit_mib, or explicitly warns that percentage limits may resolve against node memory in this pod.",
        "Does not add the batch processor to the profiles pipeline and does not use components absent from otelcol-ebpf-profiler.",
        "Does not require PID namespace translation or a nightly image for EKS, and labels the profiles signal and receiver as Alpha/Development."
      ]
    },
    {
      "id": "offcpu-and-drop-shims",
      "prompt": "On our bare-metal Linux hosts (kernel 6.1, amd64) we run otelcol-ebpf-profiler 0.162.0 in Docker. I want off-CPU profiles in addition to CPU, and I want to drop profiles coming from containerd-shim processes before they leave the host. Write the config and the docker run command; don't execute anything.",
      "expectations": [
        "States that 0.162.0 does not bundle the offcpu extension and that a build including it is needed (a 0.163.0 nightly or release, or a custom OCB build importing go.opentelemetry.io/ebpf-profiler/probes/offcpu).",
        "Configures off-CPU as extensions.offcpu.threshold within (0, 1], receivers.profiling.probes: [offcpu], and offcpu listed under service.extensions.",
        "States that the filter processor is not part of otelcol-ebpf-profiler and proposes a downstream Collector or a custom OCB build for dropping profiles.",
        "Drops shim processes with filter profile_conditions on a resource attribute such as process.executable.name (not on service.name, which the profiler does not set).",
        "The docker run command uses --privileged (or the eBPF capabilities), --pid=host, a /sys/kernel/tracing bind mount, and the service.profilesSupport feature gate."
      ]
    },
    {
      "id": "pyroscope-k8s-labels-missing",
      "prompt": "Our OTel eBPF profiler adds k8s.namespace.name and k8s.pod.name with k8s_attributes (I see them in the debug exporter), and exports to Pyroscope 2.3.1. But Pyroscope's LabelNames API only returns service_name and a few underscore labels, no k8s_namespace_name. Are the labels being dropped? How do I query CPU by namespace from the API?",
      "expectations": [
        "Explains that Pyroscope 2.x keeps dotted OTel attribute names as-is because label sanitization is disabled by default, so there is no k8s_namespace_name label.",
        "Explains that LabelNames lists only Prometheus-style names, so dotted labels look missing even though they are stored.",
        "Shows a query using the quoted dotted label, for example LabelValues with name k8s.namespace.name or a SelectSeries selector {\"k8s.namespace.name\"=\"...\"}.",
        "Uses the CPU profile type process_cpu:cpu:nanoseconds:cpu:nanoseconds or another correct way to select CPU profiles.",
        "Does not recommend re-enabling sanitization or renaming attributes as the required fix."
      ]
    },
    {
      "id": "grafana-cloud-otlp-http-400",
      "prompt": "We run otelcol-ebpf-profiler 0.162.0 on our Linux hosts and want to send profiles to Grafana Cloud Profiles instead of our self-hosted Pyroscope 2.3. I set up otlp_http with endpoint profiles-prod-NNN.grafana.net over HTTPS and a basic-auth header built from our Profiles user ID and an access policy token with profiles:write, but every export fails and the Collector only logs HTTP 400. What is wrong and what exporter config should I use instead? Also, our existing queries filter on k8s.namespace.name; will they work unchanged in Grafana Cloud? Don't run anything.",
      "expectations": [
        "Recommends exporting with otlp_grpc to the stack's Profiles host on port 443 with TLS (not insecure) as the working path for Grafana Cloud.",
        "Authenticates with HTTP Basic auth (user = the stack's numeric Profiles user/instance ID, password = a Cloud Access Policy token with profiles:write) via exporter headers, and does not depend on the basicauth extension, which otelcol-ebpf-profiler does not bundle (or explicitly notes that it needs a relay Collector or OCB build).",
        "Attributes the HTTP 400 to Grafana Cloud's OTLP/HTTP profiles decoding of resource attributes (the integer process.pid becoming an empty-named label, string resource attributes dropped), not to the endpoint path or credentials.",
        "Notes that removing process.pid makes OTLP/HTTP accept the data but loses the resource attributes (service_name becomes unknown_service), so it is not a real fix.",
        "States that k8s.namespace.name queries keep working in Grafana Cloud because k8s.* attributes stay dotted, and that only service.name, service.namespace, service.version and deployment.environment.name become underscore labels there (unlike self-hosted Pyroscope 2.x, which keeps them dotted).",
        "Does not claim to have run the Collector or contacted Grafana Cloud."
      ]
    },
    {
      "id": "keep-one-namespace",
      "prompt": "We run otel/opentelemetry-collector-ebpf-profiler:0.162.0 as a DaemonSet on Kubernetes with k8s_attributes, and it profiles every process on every node: databases, the kubelet, containerd shims. We only want profiles from pods in our 'payments' namespace to reach the backend. Is there a profiling receiver setting to target only those pods? If not, what is the right way to do it, and does it reduce the profiler's CPU cost on the nodes? Don't run anything.",
      "expectations": [
        "States that the profiling receiver has no setting to select processes, containers, pods, or namespaces (it profiles the whole host), and does not invent a receiver key for it.",
        "Filters with the filter processor using profile_conditions on the resource attribute k8s.namespace.name, placed after k8s_attributes, with a condition that drops everything not equal to 'payments' (filter conditions select what to drop).",
        "States that otelcol-ebpf-profiler 0.162.0 does not bundle the filter processor, and offers both options: a custom OCB build of the distribution adding filterprocessor at a matching version (filtering on the node), or exporting to a downstream Collector that has filter and the service.profilesSupport feature gate.",
        "States that filtering does not reduce the profiler's sampling CPU cost on the node, only the exported or stored volume.",
        "Notes that node processes without a pod (kubelet, containerd shims) carry no k8s.namespace.name and are therefore dropped by the same condition.",
        "Does not claim to have run the Collector or a cluster."
      ]
    }
  ]
}

# Exporting profiles to a backend

Profiles leave the Collector over OTLP: `otlp_grpc`, or `otlp_http`, which posts to `<endpoint>/v1development/profiles` unless `profiles_endpoint` overrides it. To relay through a regular Collector, its `otlp` receiver needs the same feature gate; see the `otel-collector` skill's `components/otlp/` pages.

Backend support moves fast. Status below was checked on 2026-10-07; re-check the linked docs before relying on it.

| Backend | OTLP profiles ingestion | Status and source |
|---|---|---|
| Grafana Pyroscope (OSS) | gRPC and HTTP on the HTTP port `4040` | Pyroscope docs call it suitable for development and testing ([docs](https://grafana.com/docs/pyroscope/latest/configure-client/opentelemetry/ebpf-profiler/)). Routes verified in `v2.3.1` source. |
| Elastic (EDOT Collector) | `profiling` receiver plus the `elasticsearch` exporter | Preview from Elastic Stack 9.2, behind `service.profilesSupport` ([docs](https://www.elastic.co/docs/reference/edot-collector/config/configure-profiles-collection)). |
| devfiler | OTLP | Desktop app for development only, not a production backend ([repo](https://github.com/elastic/devfiler)). |
| Datadog Full-Host Profiler | Not documented as accepting upstream OTLP profiles | Preview; a standalone executable built on the OpenTelemetry eBPF profiler that sends through the Datadog Agent ([docs](https://docs.datadoghq.com/profiler/enabling/full_host/)). |
| Grafana Cloud Profiles | gRPC on the stack's Profiles host, port `443` | Verified 2026-10-08 with the `0.162.0` image. OTLP/HTTP authenticates but rejects the profiler's data; see [Grafana Cloud Profiles](#grafana-cloud-profiles). Not in Grafana Cloud's OTLP gateway docs, which list traces, metrics, and logs only. |

## Pyroscope

```yaml
exporters:
  otlp_grpc:
    endpoint: pyroscope.<namespace>.svc:4040   # no separate 4317 port
    tls:
      insecure: true
```

- The Helm chart (`grafana/pyroscope`) enables an Alloy subchart by default; set `alloy.enabled: false` if profiles should arrive only through OpenTelemetry.
- `service_name` comes from `service.name`. Without it, Pyroscope `v2.3.1` uses `unknown_service:<process.executable.name>`; the Pyroscope docs also show an `ingestion_relabeling_rules` `labelmap` from `process.executable.name` to `service_name`.
- Label sanitization is off by default since Pyroscope 2.0 (`-validation.disable-label-sanitization=true`), so dotted OTel attribute names are stored as-is (`k8s.namespace.name`, not `k8s_namespace_name`). The `LabelNames` API lists only Prometheus-style names, which makes the `k8s.*` labels look missing, but `LabelValues` and selectors with quoted dotted names work.
- CPU samples appear as profile type `process_cpu:cpu:nanoseconds:cpu:nanoseconds`.

Query without a UI (port-forward or reach Pyroscope on `4040`; times in milliseconds):

```bash
end=$(($(date +%s)*1000)); start=$((end-300000))
# Which services arrived
curl -s -X POST http://localhost:4040/querier.v1.QuerierService/LabelValues \
  -H 'content-type: application/json' \
  -d "{\"name\":\"service_name\",\"start\":$start,\"end\":$end}"
# CPU per service in one namespace, using a dotted label name
curl -s -X POST http://localhost:4040/querier.v1.QuerierService/SelectSeries \
  -H 'content-type: application/json' \
  -d "{\"profileTypeID\":\"process_cpu:cpu:nanoseconds:cpu:nanoseconds\",
       \"labelSelector\":\"{\\\"k8s.namespace.name\\\"=\\\"default\\\"}\",
       \"groupBy\":[\"service_name\"],\"start\":$start,\"end\":$end,\"step\":300}"
```

In Grafana, add a `grafana-pyroscope-datasource` pointing at `http://<pyroscope>:4040` and browse flame graphs in Drilldown > Profiles.

## Grafana Cloud Profiles

Use `otlp_grpc`. The stack's Profiles details in the Cloud portal give the host (`profiles-prod-NNN.grafana.net`) and the numeric user (instance) ID; create a Cloud Access Policy token with `profiles:write` there. `otelcol-ebpf-profiler` has no `basicauth` extension, so send a precomputed header:

```yaml
exporters:
  otlp_grpc:
    endpoint: profiles-prod-NNN.grafana.net:443
    headers:
      authorization: "Basic ${env:GRAFANA_PROFILES_BASIC_AUTH}"   # base64 of <user-id>:<token>
```

- Stacks symbolize and resource attributes are kept.
- OTLP/HTTP (`https://profiles-prod-NNN.grafana.net/v1development/profiles`) authenticates, but with Collector `0.162.0` every batch fails with HTTP 400 `invalid labels '{="<pid>", ...}' with error: invalid label name ''`. Grafana Cloud's HTTP decoder turns the integer resource attribute `process.pid` into an empty-named label and drops string resource attributes. Deleting `process.pid` gets a 200, but all resource attributes are lost (`service_name="unknown_service"`), so it is not a fix.
- Only `service.name`, `service.namespace`, `service.version`, and `deployment.environment.name` become underscore labels (`service_namespace`, `deployment_environment_name`, ...); every other attribute (`k8s.*`, `container.id`, `process.*`, `thread.name`) keeps its dotted name and is queried quoted, as in self-hosted Pyroscope. Self-hosted Pyroscope `v2.3.1` keeps all four dotted too. Observed 2026-10-10.
- Without `service.name`, `service_name` is `unknown_service:<process.executable.name>`, as in Pyroscope.

# eBPF profiler on Kubernetes

Run one profiler per node as a DaemonSet. Each pod profiles every process on its node, so Kubernetes metadata comes from matching `container.id` to pods, not from pod IPs.

Component details are in the `otel-collector` skill: `components/k8s_attributes/` (association, `extract`, computed `service.name`) and `components/memory_limiter/` (limit keys and validation).

The manifest carries the host requirements from SKILL.md (`hostPID`, `privileged` or the capabilities, the tracefs hostPath). Kubernetes-specific:

- RBAC: get/list/watch on `pods` and `namespaces` covers the extract list below; other metadata needs more (RBAC table in the `otel-collector` skill's `components/k8s_attributes/advanced.md`).
- `memory_limiter`: absolute `limit_mib`/`spike_limit_mib`, below the container memory limit. In a privileged hostPID pod, `limit_percentage` resolved against node memory in field testing (logged `total_memory_mib` equal to the node).

## Manifest

Minimal and generic. Replace the exporter endpoint and cluster name; see `backends.md` for endpoints. On kind/k3d/Docker-in-Docker nodes, apply the changes in `nested-containers.md`.

```yaml
apiVersion: v1
kind: Namespace
metadata:
  name: profiling
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: otel-ebpf-profiler
  namespace: profiling
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otel-ebpf-profiler
rules:
  - apiGroups: [""]
    resources: [pods, namespaces]
    verbs: [get, list, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: otel-ebpf-profiler
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: otel-ebpf-profiler
subjects:
  - kind: ServiceAccount
    name: otel-ebpf-profiler
    namespace: profiling
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-ebpf-profiler
  namespace: profiling
data:
  config.yaml: |
    receivers:
      profiling: {}
    processors:
      memory_limiter:
        check_interval: 1s
        limit_mib: 800
        spike_limit_mib: 200
      k8s_attributes:
        filter:
          node_from_env_var: K8S_NODE_NAME
        pod_association:
          - sources:
              - from: resource_attribute
                name: container.id
        extract:
          metadata:
            - k8s.namespace.name
            - k8s.pod.name
            - k8s.deployment.name
            - k8s.node.name
            - k8s.container.name
            - service.name
      resource:
        attributes:
          - key: k8s.cluster.name
            value: my-cluster
            action: upsert
    exporters:
      otlp_grpc:
        endpoint: profiles-backend.observability.svc:4317
        tls:
          insecure: true
    service:
      pipelines:
        profiles:
          receivers: [profiling]
          processors: [memory_limiter, k8s_attributes, resource]
          exporters: [otlp_grpc]
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-ebpf-profiler
  namespace: profiling
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: otel-ebpf-profiler
  template:
    metadata:
      labels:
        app.kubernetes.io/name: otel-ebpf-profiler
    spec:
      serviceAccountName: otel-ebpf-profiler
      hostPID: true
      tolerations:
        - operator: Exists
      containers:
        - name: profiler
          image: otel/opentelemetry-collector-ebpf-profiler:0.162.0
          args:
            - --config=/conf/config.yaml
            - --feature-gates=+service.profilesSupport
          env:
            - name: K8S_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          securityContext:
            privileged: true
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              memory: 1Gi
          volumeMounts:
            - name: conf
              mountPath: /conf
            - name: tracefs
              mountPath: /sys/kernel/tracing
      volumes:
        - name: conf
          configMap:
            name: otel-ebpf-profiler
        - name: tracefs
          hostPath:
            path: /sys/kernel/tracing
```

Notes:

- `k8s_attributes` computes `service.name` for pod processes (order in the `otel-collector` skill's `components/k8s_attributes/configuration.md`). Processes with no matching pod (kubelet, container runtime, shims, host daemons) get no pod-derived attributes such as `service.name` or `k8s.namespace.name`; they keep `process.*` and whatever other processors add (here `k8s.cluster.name`). Give them a fallback `service.name` (for example `resource` with `action: insert`, which keeps pod-derived values) if the backend needs one.
- Field measurement at 97 Hz (about five times the default) on a small cluster: roughly 6 to 13 mCPU and 110 to 370 MiB per node. Size requests from your own measurement; upstream states 1% CPU and 250 MB as its testing upper bounds.
- To add the `offcpu` probe or OBI correlation, see `receiver.md`; OBI correlation also needs `/sys/fs/bpf` mounted from the host.

## Keep only some workloads

`filter` runs after `k8s_attributes` and drops what its conditions match, so this keeps only one namespace; node processes, which carry no namespace, are dropped too:

```yaml
processors:
  filter/my-namespace:
    error_mode: ignore
    profile_conditions:
      - resource.attributes["k8s.namespace.name"] != "my-namespace"
service:
  pipelines:
    profiles:
      processors: [memory_limiter, k8s_attributes, resource, filter/my-namespace]
```

`otelcol-ebpf-profiler` does not bundle `filter`, so run it in one of two places:

- **On the node:** an OCB build of the [distribution manifest](https://github.com/open-telemetry/opentelemetry-collector-releases/blob/main/distributions/otelcol-ebpf-profiler/manifest.yaml) plus `github.com/open-telemetry/opentelemetry-collector-contrib/processor/filterprocessor` at the same version (`otel-collector-builder` skill). Dropped profiles never leave the node.
- **Downstream:** export `otlp_grpc` to a Collector that has `filter` and the `service.profilesSupport` gate (a gateway or a dedicated relay). Its OTLP profiles decoding must match the profiler's: when the two run different Collector versions, confirm a round trip through `debug` (`verbosity: detailed`) keeps resource attributes, sample attributes, and stack frames.

`filter` drops whole resources but keeps the shared symbol dictionary, so some symbols of dropped processes still travel; only payload size is affected.

## Verify

```bash
kubectl --context <ctx> -n profiling rollout status ds/otel-ebpf-profiler
kubectl --context <ctx> -n profiling logs ds/otel-ebpf-profiler | grep -E 'Attached sched monitor|Everything is ready|error'
```

Then confirm pod workloads (not only node processes like the kubelet or container runtime) appear in the backend with `k8s.*` attributes. If only node processes appear on a kind/k3d cluster, read `nested-containers.md`.

# Nested-container environments (kind, k3d, Docker-in-Docker)

In kind, k3d, minikube with the docker driver, and Docker-in-Docker, each Kubernetes "node" is itself a container. Two things break that work on real nodes (EKS, GKE, AKS, bare-metal kubelets), where `hostPID` reaches the kernel's root PID namespace and tracefs is usually mounted.

Docker Desktop, Colima, and similar VMs are not nested by themselves: a `docker run --pid=host` there reaches the VM's root PID namespace. They become nested only when you run kind or k3d inside them.

## 1. PID namespaces

`hostPID: true` reaches only the node container's PID namespace; pods sit in descendant namespaces. Without translation the receiver fails at startup. The boolean `pid_namespace_translation: true` (ebpf-profiler `v0.0.202636`, Collector `0.162.0`) starts but sees only the node's own processes (kubelet/k3s, containerd, shims), never pods. Use `pid_namespace_translation_mode: auto` (ebpf-profiler `v0.0.202640`+; images in the SKILL.md matrix). Each pin rejects the other pin's key with `invalid keys`, so match the key to the image. Exact errors and logs for each state: `troubleshooting.md`.

Modes: `none` (default; correct on real nodes, which run in the root PID namespace); `exact` (own namespace only, no BTF needed); `auto` (adds descendants when kernel BTF at `/sys/kernel/btf/vmlinux` exposes the layout, else falls back to exact); `recursive` (requires BTF, fails at startup without it). Tasks outside the profiler's namespace tree are dropped.

## 2. tracefs missing inside the node

The node container gets a fresh sysfs without tracefs, so a hostPath of `/sys/kernel/tracing` (or `/sys/kernel/debug`) is an empty directory. Startup fails with `... neither debugfs nor tracefs are mounted` (full string in `troubleshooting.md`).

Fix: a privileged init container that enters the node's mount namespace and mounts tracefs if absent, plus the existing hostPath mount in the profiler container. It is a no-op where tracefs is already mounted. Add to the DaemonSet pod spec from `kubernetes.md`:

```yaml
      initContainers:
        - name: mount-tracefs
          image: busybox:1.37
          command:
            - sh
            - -c
            - >-
              nsenter -t 1 -m -- env PATH=/bin/aux:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin sh -c
              'grep -q " /sys/kernel/tracing tracefs " /proc/mounts
              || mount -t tracefs nodev /sys/kernel/tracing'
          securityContext:
            privileged: true
```

`nsenter -t 1` works because the pod has `hostPID: true`, so PID 1 is the node's init. The explicit `PATH` matters on k3s nodes, which keep `mount` in `/bin/aux`; without it the init container fails with `mount: not found`.

For plain `docker run` of the profiler inside a Docker-in-Docker host, bind-mount `/sys/kernel/tracing` from a level that has tracefs, or mount it in that inner host first.

## Combined changes for kind/k3d

1. An image that pins ebpf-profiler `v0.0.202640`+ (see the SKILL.md version matrix for release status).
2. `pid_namespace_translation_mode: auto` under `receivers.profiling`.
3. The `mount-tracefs` init container above.

# `profiling` receiver

Keys depend on the ebpf-profiler version the distribution pins, not on the Collector version. For any release other than those in the SKILL.md version matrix, read the pin and the config struct at that tag, then run `validate` with the target image:

```bash
gh api 'repos/open-telemetry/opentelemetry-collector-releases/contents/distributions/otelcol-ebpf-profiler/manifest.yaml?ref=v<collector-version>' --jq .content | base64 -d | grep -A1 ebpf-profiler
# then: https://github.com/open-telemetry/opentelemetry-ebpf-profiler/blob/<pin>/collector/config/config_linux.go
```

## Keys (ebpf-profiler `v0.0.202640`; identical on `v0.0.202636` except the translation key)

Source of truth: [`collector/config/config_linux.go`](https://github.com/open-telemetry/opentelemetry-ebpf-profiler/blob/v0.0.202640/collector/config/config_linux.go) and defaults in [`collector/factory_linux.go`](https://github.com/open-telemetry/opentelemetry-ebpf-profiler/blob/v0.0.202640/collector/factory_linux.go).

| Key | Default | Notes |
|---|---|---|
| `samples_per_second` | `20` | CPU sampling frequency. |
| `reporter_interval` / `reporter_jitter` | `5s` / `0.2` | Export cadence. |
| `interpreters.<name>.disabled` | all enabled | `python`, `perl`, `php`, `hotspot`, `ruby`, `v8`, `dotnet`, `go`, `beam`, `luajit`, `thread_context`. |
| `pid_namespace_translation_mode` | `none` | `none`, `auto`, `exact`, `recursive`. Pins before `v0.0.202640` use boolean `pid_namespace_translation`; each rejects the other's key. Only for [nested containers](nested-containers.md). |
| `probes` | `[]` | Extension IDs, for example `[offcpu]`. |
| `include_env_vars` | `""` | Comma-separated env var names to attach as attributes. |
| `filter_min_process_age` | `0` | Skip short-lived processes. |
| `send_error_frames` / `send_idle_frames` | `false` | Include unwinding-error or idle frames. |
| `error_mode` | `propagate` | `ignore` logs receiver startup errors instead of failing the Collector. |
| `no_kernel_version_check` | `false` | For distro kernels with backported eBPF features. |
| `obi_process_ctx` / `bpf_fs_root` | `false` / `/sys/fs/bpf/` | Share span/trace IDs with OpenTelemetry eBPF Instrumentation (OBI) via a pinned map under `<bpf_fs_root>/otel`. |

Other keys are tuning or debugging knobs; read the source before setting them.

## What it emits

Resource attributes `process.pid`, `process.executable.name`, `process.executable.path`, and `container.id` for containerized processes; sample attributes `thread.name`, `thread.id`, `cpu.logical_number`. It does not set `service.name`: add it with `k8s_attributes`, the `resource` processor, or backend relabeling.

## Off-CPU

Off-CPU needs the `offcpu` extension (`go.opentelemetry.io/ebpf-profiler/probes/offcpu`, present in the module since at least `v0.0.202636`). Not bundled in `0.162.0` (images that bundle it: SKILL.md matrix); otherwise build with OCB importing it:

```yaml
extensions:
  offcpu:
    threshold: 0.1          # capture probability, (0.0, 1.0]
receivers:
  profiling:
    probes: [offcpu]
service:
  extensions: [offcpu]
```

Off-CPU samples arrive as sample type `off_cpu` in `nanoseconds` alongside the CPU samples.

# Troubleshooting the eBPF profiler

Match the exact log or error text. Strings are from Collector `0.162.0` / ebpf-profiler `v0.0.202636` and `v0.0.202640` unless noted.

| Error or symptom | Cause | Fix |
|---|---|---|
| `pipeline "profiles": profiling signal support is at alpha level, gated under the "service.profilesSupport" feature gate` | Gate not enabled. | Add `--feature-gates=+service.profilesSupport` (also on any relay Collector with a profiles pipeline). |
| `'config.Config' has invalid keys: pid_namespace_translation_mode` | Key from ebpf-profiler `v0.0.202640`+ on an older pin (for example `0.162.0`). | Use `pid_namespace_translation: true` there, or move to a `v0.0.202640`+ image. |
| `'config.Config' has invalid keys: pid_namespace_translation` | Boolean key removed in `v0.0.202640`. | Use `pid_namespace_translation_mode`. |
| `failed to create "batch" processor, in pipeline "profiles": telemetry type is not supported` | `batch` has no profiles support. | Remove it from the profiles pipeline. |
| `'processors' unknown type: "filter"` | `otelcol-ebpf-profiler` does not bundle `filter`. | Build the distribution with `filterprocessor`, or filter in a downstream Collector; see "Keep only some workloads" in `kubernetes.md`. |
| `'extensions' unknown type: "offcpu"` | Distribution does not bundle the off-CPU probe (`0.162.0`). | Use `0.163.0-nightly.3d037de` or later, or an OCB build importing `go.opentelemetry.io/ebpf-profiler/probes/offcpu`. |
| `failed to attach scheduler monitor: failed to configure tracepoint on tracer.hookPoint{group:"sched", name:"sched_process_free"}: neither debugfs nor tracefs are mounted` | tracefs absent in the container's `/sys`. | Mount `/sys/kernel/tracing` from the host. If the host path is empty (kind/k3d nodes), mount tracefs on the node first: `nested-containers.md`. |
| `mount: not found` from a tracefs init container | k3s keeps `mount` in `/bin/aux`. | Set `PATH` explicitly inside `nsenter` (see `nested-containers.md`). |
| `failed to determine system configs: system analysis request was not handled for pid N` | Profiler is not in the kernel's root PID namespace (nested node container) and translation is off. | `pid_namespace_translation_mode: auto` (or the boolean on older pins), plus the visibility caveat below. |
| Only node processes (kubelet/k3s, containerd, shims) appear; no pods. Log: `only processes traces within the profiler namespace will be collected` | Boolean translation (`v0.0.202636`) only covers the profiler's own PID namespace. | `pid_namespace_translation_mode: auto` on ebpf-profiler `v0.0.202640`+. |
| `recursive PID namespace translation requires readable kernel BTF ...` | `recursive` mode without kernel BTF. | Use `auto` or `exact`, or a kernel with `/sys/kernel/btf/vmlinux`. |
| `host Agent requires kernel version 5.10 or newer but got X.Y.Z` | Old kernel, or a distro kernel with backported eBPF. | Upgrade the kernel, or set `no_kernel_version_check: true` only if the distro backports the needed features. |
| `failed to load eBPF tracer: failed to read kernel symbols: unable to read kallsyms addresses - check capabilities` (or other eBPF load or permission errors) | Not privileged (observed with `docker run` without `--privileged`). | Run privileged or with `CAP_SYS_ADMIN`, `CAP_BPF`, `CAP_PERFMON`, and the host PID namespace. |
| `memory_limiter` logs `total_memory_mib` equal to node memory | Percentage limits resolved against the node, not the container. | Use `limit_mib` and `spike_limit_mib`. |
| Profiles arrive without `k8s.*` attributes | No `container.id` pod association, RBAC gap, or the process is not in a pod. | `pod_association` on `container.id`; RBAC for pods and namespaces; see `otel-collector` `components/k8s_attributes/quirks.md`. |
| Many `containerd-shim-*` or `unknown_service:*` services in the backend | Host processes with no pod and no `service.name`. | Expected. Fallback `service.name` via `resource` `action: insert` (`kubernetes.md`), or drop downstream with `filter` `profile_conditions` on `process.executable.name` (`filter` is not in `otelcol-ebpf-profiler`). |
| Exporter logs only an HTTP status (for example 400) for every batch | The Collector does not log the response body. | Capture one request body (point `otlp_http` at a local listener) and replay it with `curl -v --data-binary @body.bin` and the same headers (`Content-Type: application/x-protobuf`, `Content-Encoding` if compressed, auth) to read the backend's error. For Grafana Cloud's 400 on OTLP/HTTP, see `backends.md`. |
| `Skip pinning eBPF map to share OTel span/trace IDs` (info) | `obi_process_ctx` is off. | Informational. Enable it only to correlate with OBI. |
| Frames show addresses or `unknown` | Symbols not available to the backend. | Backend-dependent; check the backend's symbolization docs. Go binaries are symbolized on the host. |

For anything else, raise `service.telemetry.logs.level: debug` (the receiver's `verbose_mode` adds more eBPF detail) and check the [ebpf-profiler issues](https://github.com/open-telemetry/opentelemetry-ebpf-profiler/issues).