{
  "schema_version": 1,
  "skill_name": "telemetry",
  "evals": [
    {
      "id": "prometheus-rule-authoring-review",
      "prompt": "Our SRE wants to add two rules to a Prometheus rules file: a recording rule that computes the 5-minute request rate per endpoint and an alert that pages when the API error rate exceeds 5% for ten minutes. What should the rule file look like, what sanity checks should run before it is loaded, and what are the common authoring mistakes to avoid?",
      "expected_output": "A rules file with one group containing two rules: a recording rule named with the level:metric:operation convention (for example job:http_requests:rate5m) with expr sum by (job, endpoint) (rate(http_requests_total[5m])), and an alerting rule (for example ApiHighErrorRate) with the error-rate expression, a for: 10m clause, labels such as severity and team, and annotations with a summary and a runbook link. The response states that each rule must set exactly one of record or alert, expr must be present and parse, durations must be valid Prometheus durations, and label values must be strings. It validates with promtool check rules for full PromQL parsing and the bundled telemetry-check --rules for structural sanity before reload, then verifies with the /api/v1/rules endpoint. Common mistakes called out: putting both record and alert on one rule, writing an expression that is too large to review, misusing relabeling so the labels the rule queries do not exist, and forgetting for on alerts that should require sustained conditions.",
      "assertions": [
        "The response produces a rules file with a recording rule using the level:metric:operation naming convention and an alerting rule with expr, for, labels, and annotations",
        "Exactly one of record or alert per rule, a non-empty expr, and valid Prometheus durations are stated as requirements",
        "Validation with promtool check rules and telemetry-check --rules before reload is prescribed, with verification via the rules API",
        "At least three authoring mistakes are named (both record and alert set, oversized expressions, relabeling/label mismatches, missing for)"
      ]
    },
    {
      "id": "otel-collector-pipeline-design",
      "prompt": "We are standing up an OpenTelemetry Collector that receives OTLP traces and metrics from a few services and sends them to a backend. We want to sample 10% of success traces but keep all error traces, and we are worried about the collector using too much memory. How should the pipelines and processors be designed?",
      "expected_output": "A pipeline design with separate traces and metrics pipelines: traces through otlp receiver, memory_limiter, batch, tail_sampling, then the otlp exporter; metrics through otlp receiver, memory_limiter, batch, then the metrics exporter. The response explains that tail_sampling decides per trace at the batch level, so it belongs on the trace pipeline after batching, with policies that keep spans whose status.code or http.status_code indicates an error and a probabilistic policy for the rest, and that the sampling decision should be recorded as a span attribute. It prescribes one memory_limiter processor before exporters with a limit sized against the container memory budget, batching to amortize exporter cost, and warns not to apply trace samplers to the metrics pipeline. It verifies with the collector health endpoint and receiver/exporter metrics advancing, and notes that a debug exporter is for temporary troubleshooting only.",
      "assertions": [
        "Separate traces and metrics pipelines with receivers, processors, and exporters are specified",
        "tail_sampling is placed on the trace pipeline after batching, with error-keeping and probabilistic policies",
        "memory_limiter runs before exporters and is sized against the container memory budget",
        "The response warns that trace samplers must not be applied to the metrics pipeline and verification uses collector health and per-signal metrics"
      ]
    },
    {
      "id": "loki-label-and-retention-review",
      "prompt": "Our team is about to ship a new service and wants to push its logs to Loki with labels for app, environment, tenant, user_id, and request_id so they can filter per user. They also have not set any retention. What is wrong with this plan and what should the label and retention design be?",
      "expected_output": "A review that app, environment, and tenant are reasonable Loki labels because they are low-cardinality and used as index matchers, but user_id and request_id are high-cardinality per-line fields that explode the inverted index and stream count if indexed. The response puts user_id and request_id in the log line and extracts them with LogQL | json or | regexp when needed, and sets retention deliberately per tenant: retention_enabled true with a retention_period and retention_size, enforced by the compactor, with an owner and a review schedule. It explains the cost model: label matchers run against the inverted index (cheap), line filters and parsing run per line (expensive), and a label whose values change with every log line does not belong in the index. Verification uses loki_ingester_streams to confirm stream cardinality stays bounded and compactor metrics to confirm retention is running.",
      "assertions": [
        "app/environment/tenant are accepted as low-cardinality labels while user_id and request_id are called out as high-cardinality index hazards",
        "High-cardinality fields are moved into the log line and extracted with LogQL json or regexp parsing",
        "Retention is set per tenant with retention_enabled, retention_period, retention_size, compactor enforcement, and an owner",
        "The inverted-index versus per-line-filter cost model is explained and loki_ingester_streams is used as the cardinality signal"
      ]
    },
    {
      "id": "prometheus-retention-and-ha-decision",
      "prompt": "Our single Prometheus instance keeps growing: queries are fine, but the disk fills every few months and someone keeps raising the retention flag to keep more history. Management now wants a second instance so we are 'highly available'. How should we reason about retention and HA before adding machines?",
      "expected_output": "A decision process that separates the questions: retention is a deliberate capacity and compliance choice (how long the data must answer which questions), not a flag to raise reactively; and HA is a redundancy choice (two identically configured instances scraping the same targets with consistent external labels so a query layer can deduplicate), which does not increase storage capacity or history length. The response prescribes setting retention by the question the data answers (alerting windows, trend analysis, audit), bounding it with retention.time and retention.size, watching prometheus_tsdb_head_series and compaction metrics for the actual cost, and routing long-term history to a separate store with its own owner instead of stretching the hot instance. It states that two replicas do not double history, do not share rule state, and need a dedup layer or consistent labeling so alerts do not double-fire, and that rule evaluation consistency across replicas matters more than uptime.",
      "assertions": [
        "Retention is framed as a capacity and compliance decision driven by the questions the data must answer, not a reactive flag",
        "HA is framed as redundant identical instances with consistent external labels and a dedup layer, explicitly not a storage or history increase",
        "Retention flags and TSDB metrics for bounding cost are named, with long-term history routed to a separate store",
        "The double-fire risk and rule-state locality of HA pairs are stated"
      ]
    },
    {
      "id": "trace-span-correlation-setup",
      "prompt": "We run the OpenTelemetry Collector and send OTLP logs, metrics, and traces to the backend. When an alert fires on a metric, the on-call engineer has to search logs by timestamp and guess which request was slow. What should we configure so a metric alert can pivot to the exact trace and its log lines?",
      "expected_output": "A correlation setup: the spanmetrics processor derives RED metrics from spans with trace_id exemplars so PromQL histograms carry the trace ID of slow requests; OTLP log records carry trace_id and span_id and the collector's Loki exporter maps them to structured metadata so LogQL can filter {app=\"x\"} | trace_id=\"...\"; and resource attributes such as service.name and deployment.environment flow through all three signals as the join keys. The response explains that correlation depends on context propagation from the application SDKs, so a missing trace_id in logs usually means propagation is not wired, and prescribes verifying the pivot end-to-end with one query: fire a test request, find its trace ID in the metrics exemplar, and confirm the same ID appears in Loki. It notes that instrumenting application code and propagation are backend-engineering territory while the collector-side join is this skill's scope, and that changing the pipeline requires re-verifying the pivot query.",
      "assertions": [
        "spanmetrics with trace_id exemplars and the Loki exporter mapping trace_id/span_id to structured metadata are specified",
        "Resource attributes are identified as the cross-signal join keys",
        "Dependency on application-side context propagation is stated, with a missing trace_id diagnosed as a propagation problem",
        "Verification is an end-to-end pivot query from metric exemplar to trace to log lines"
      ]
    },
    {
      "id": "stack-ingest-outage-diagnosis",
      "prompt": "A dashboard panel shows no data for the last hour for one service, while other services are fine. The Prometheus scrape targets list shows the job as up, the OTel Collector is healthy, and Loki shows the service's logs. What is the evidence-ordered diagnosis, and what should we check at each layer before changing anything?",
      "expected_output": "An evidence-ordered diagnosis that works from the symptom down: first confirm which layer lost data by checking each component's own signals — the Prometheus targets API for scrape health and the actual metric series (the job can be up while the metric is empty, which points at metric_relabel_configs dropping the series), collector receiver and exporter metrics for delivery, and the rules API for evaluation state; then check config-level causes: relabeling that renamed or dropped labels, a scrape config change that changed the series identity, or a recording rule whose expression no longer matches. The response keeps the diagnosis read-only (telemetry-check --rules and --scrape plus read-only API queries) and treats any config change as a mutation requiring confirmation with a rollback path. It explicitly avoids assuming correlation is causation, e.g. a slow query and missing data are separate evidence, and verifies any fix by re-running the checks and confirming the series appears at the delivery boundary.",
      "assertions": [
        "The diagnosis works layer by layer from the symptom with each component's own signals (targets API, series existence, collector receiver/exporter metrics, rules API)",
        "Config-level causes such as relabeling drops, series-identity changes, and rule-expression mismatches are considered",
        "The diagnosis is kept read-only with the bundled checker and read-only API queries, and changes require confirmation with a rollback path",
        "Correlation is not presented as causation and fixes are verified by re-running the checks"
      ]
    },
    {
      "id": "bounded-promql-logql-investigation",
      "prompt": "Design a read-only PromQL and LogQL investigation for elevated API latency. Include safe selectors, a time range and step, syntax versus semantic validation, aggregation or joins, cost limits, and the evidence to record.",
      "expected_output": "A bounded workflow that defines the service, UTC start/end, expected units, and step before querying; uses a selective PromQL matcher and a Loki stream selector before line parsing; validates syntax with promtool or the target parser separately from semantic checks for metric type, labels, aggregation, and join cardinality; starts with an instant existence query before a short range query; and records HTTP status, query scope, time window, step or limit, cardinality, latency, warnings, and interpretation. It avoids broad selectors, unbounded regex, high-cardinality grouping, and unconstrained joins, with explicit timeout, lookback, series/stream, and concurrency limits.",
      "assertions": [
        "PromQL and LogQL examples use selective selectors and explicitly bounded time and resolution",
        "Syntax parsing is distinguished from semantic validation of names, labels, types, grouping, and join cardinality",
        "The workflow records query status, exact scope and time, limits, cardinality, warnings, and interpretation",
        "Broad selectors, expensive regex/parsing, high-cardinality grouping, and unconstrained joins are rejected"
      ]
    },
    {
      "id": "empty-and-partial-query-diagnosis",
      "prompt": "A valid PromQL request returns an empty vector for a ratio, while a Loki range query returns a partial response with warnings. Explain how to report these results and distinguish wrong labels, retention expiry, stale producers, query errors, and incomplete evidence.",
      "expected_output": "The response says an empty vector is unknown and never numeric zero, so a missing numerator or denominator must not be substituted with zero. It distinguishes parser or non-2xx query errors, absent labels or wrong selectors, expired retention, failed targets or dropped series, stale timestamps or stalled producers, and partial responses caused by limits or shards. It checks the exact UTC window, metric and label existence, target and ingest health, retention, response status and warnings, then narrows the query without mutation; any eventual config change requires confirmation and rollback evidence.",
      "assertions": [
        "No data is explicitly reported as unknown rather than zero, including for ratios",
        "Absent labels, retention expiry, failed targets, stale data, query errors, and partial responses are separate diagnoses",
        "Response status and warnings plus time window and backend health are checked before interpretation",
        "The investigation remains read-only and incomplete evidence is not presented as a complete measurement"
      ]
    },
    {
      "id": "telemetry-routing-and-tracing-scope",
      "prompt": "An engineer asks whether the telemetry skill should also teach Grafana dashboards, SLO paging policy, and Tempo trace queries. Explain the ownership boundaries and what tracing coverage is intentionally deferred.",
      "expected_output": "The response routes SLI/SLO definitions, error budgets, alert thresholds, and paging strategy to platform-engineering; routes Grafana dashboards, panels, data sources, Grafana alert rules, contact points, and notification policies to grafana; and keeps telemetry focused on Prometheus/Loki/Collector backend query evidence. It explicitly says Tempo/tracing query operations are deferred to a future named-tool decision, without inventing Tempo commands or claiming trace-query semantics; application context propagation remains backend-engineering work.",
      "assertions": [
        "Platform-engineering is named for SLI/SLO, strategy, and paging ownership",
        "Grafana is named for dashboards, panels, and Grafana-side alerting/contact policy ownership",
        "Tempo/tracing operations are explicitly deferred without unsupported commands or behavioral claims",
        "The response preserves a bounded telemetry backend scope instead of creating an observability mega-skill"
      ]
    }
  ]
}

# Sample Prometheus recording and alerting rules used by telemetry-check.
# Mirrors the structure promtool check rules accepts: groups of recording and
# alerting rules with labels, annotations, and durations.
groups:
  - name: api-slo
    interval: 1m
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total{job="api"}[5m]))
      - record: job:http_errors:rate5m
        expr: sum by (job) (rate(http_requests_total{job="api",status=~"5.."}[5m]))
      - alert: ApiHighErrorRate
        expr: job:http_errors:rate5m / job:http_requests:rate5m > 0.05
        for: 10m
        labels:
          severity: page
          team: platform
        annotations:
          summary: "API error rate above 5% for 10 minutes"
          runbook: "https://example.com/runbooks/api-high-error-rate"
  - name: node-health
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} is down"

# Sample Prometheus scrape configuration used by telemetry-check tests.
scrape_configs:
  - job_name: node-exporter
    scrape_interval: 30s
    static_configs:
      - targets: ["127.0.0.1:9100", "127.0.0.1:9111"]
        labels:
          env: dev
  - job_name: api
    static_configs:
      - targets:
          - "127.0.0.1:8080"

# Telemetry Operations — Source Index

> **Last Updated:** 2026-08-03

This index tracks the authoritative sources behind the telemetry skill
(Prometheus + OpenTelemetry Collector + Loki as one stack) and the refresh
procedure for keeping it current.

## Canonical sources

| Component | Source |
|---|---|
| Prometheus documentation (current) | https://prometheus.io/docs/introduction/overview/ |
| Prometheus configuration (scrape config, relabeling) | https://prometheus.io/docs/prometheus/latest/configuration/configuration/ |
| Prometheus recording rules | https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/ |
| Prometheus alerting rules | https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/ |
| Prometheus storage and retention | https://prometheus.io/docs/prometheus/latest/storage/ |
| `promtool` rule checking | https://prometheus.io/docs/prometheus/latest/command-line/promtool/ |
| Prometheus rule format reference (rulefmt) | https://github.com/prometheus/prometheus/blob/main/model/rulefmt/rulefmt.go |
| OpenTelemetry Collector | https://opentelemetry.io/docs/collector/ |
| Collector components (receivers/processors/exporters) | https://opentelemetry.io/docs/collector/configuration/ |
| Collector sampling | https://opentelemetry.io/docs/collector/sampling/ |
| OpenTelemetry traces and spans | https://opentelemetry.io/docs/concepts/signals/traces/ |
| Loki documentation (current) | https://grafana.com/docs/loki/latest/ |
| Loki storage and retention | https://grafana.com/docs/loki/latest/operations/storage/retention/ |
| LogQL | https://grafana.com/docs/loki/latest/query/ |
| Loki label design guidance | https://grafana.com/docs/loki/latest/get-started/labels/ |
| Prometheus HTTP API and querying | https://prometheus.io/docs/prometheus/latest/querying/api/ |
| PromQL operators and functions | https://prometheus.io/docs/prometheus/latest/querying/operators/ |
| Loki query HTTP API | https://grafana.com/docs/loki/latest/reference/loki-http-api/ |

## Version observations (as of this refresh)

- Prometheus 3.x defaults to UTF-8 metric-name validation; the legacy metric
  name pattern `[a-zA-Z_:][a-zA-Z0-9_:]*` remains the documented pattern for
  recording rule names and is what `promtool` enforces under legacy validation.
- `promtool check rules` validates rule files structurally and parses every
  expression with the full PromQL parser; `telemetry-check` deliberately
  covers the structural subset so it can run with no Prometheus tooling.
- The OpenTelemetry Collector's `memory_limiter` processor is recommended
  before every exporter; `tail_sampling` and `probabilistic_sampler` are the
  two supported sampler processors for traces.
- Loki retention is enforced by the compactor on a per-tenant basis;
  `retention_period` and `retention_size` are per-tenant limits, and
  `retention_enabled` must be `true` for period-based retention.
- OTLP is the current stable protocol; the Collector can also receive/export
  Prometheus exposition format, and its Loki exporter translates structured
  log records into LogQL-compatible streams.

## Refresh procedure

1. Re-check the sources above for new minor or major releases.
2. Update the version observations that changed (defaults, renamed components,
   new processors, changed retention semantics).
3. Re-run the bundled checker against a rules fixture and a scrape config and
   confirm every check still parses: `telemetry/scripts/telemetry-check
   --rules telemetry/fixtures/prometheus-rules.yml --json`.
4. Re-verify the SKILL.md keyword sweep from the validation contract and the
   routing links to `platform-engineering` and `grafana`.

## Related skill sources

- `platform-engineering` owns observability strategy (SLIs, SLOs, what to
  instrument) and its observability reference treats Prometheus, OTel, and
  Loki as one stack; this skill owns operating that stack.
- `grafana` owns dashboards, panels, and Grafana-side alerting; it queries
  Prometheus (PromQL) and Loki (LogQL) but does not operate the backends.
- `traefik` ships Prometheus and OTel configuration for the edge; its
  observability reference documents the metric names this stack ingests.

# Prometheus Operations

> **Last Updated:** 2026-08-03

Operational patterns for the Prometheus half of the telemetry stack: scrape
configuration, recording and alerting rules, relabeling, retention, and high
availability. Sources: Prometheus documentation (prometheus.io/docs,
accessed 2026-08-03) and the rule format reference in the Prometheus source
(reviewed 2026-08-03).

## Scrape configuration

One scrape job is one scrape group: a `job_name`, a `scrape_interval`, a
`scrape_timeout` strictly below the interval, a `metrics_path`, and a target
source. The default `metrics_path` is `/metrics`; TLS and auth go in
`scheme`, `tls_config`, and `basic_authorization`/`authorization`.

```yaml
scrape_configs:
  - job_name: node
    scrape_interval: 30s
    scrape_timeout: 10s
    metrics_path: /metrics
    static_configs:
      - targets: ["node1:9100", "node2:9100"]
```

Verify the live state, not the file: `/api/v1/status/config` returns the
effective config and `/api/v1/targets?state=active` returns per-target scrape
state. A target that never appears in the target list is usually a relabeling
or discovery problem, not a Prometheus outage.

## Recording and alerting rules

Rules files are `groups`, each with a `name`, an optional `interval`, and a
`rules` list. Every rule has exactly one of `record` or `alert` plus an
`expr`; alerting rules may add `for`, `keep_firing_for`, `labels`, and
`annotations`. Group names must be unique within a file; label and annotation
names must be valid label names; label values must be strings.

```yaml
groups:
  - name: api-slo
    interval: 1m
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total{job="api"}[5m]))
      - alert: ApiHighErrorRate
        expr: job:http_errors:rate5m / job:http_requests:rate5m > 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "API error rate above 5%"
```

Rules that touch the same series belong in one group because groups evaluate
sequentially; cross-group timing is undefined. Validate with
`promtool check rules` (full PromQL parsing) and `telemetry-check --rules`
(structural sanity, no external tools) before every reload, then
`promtool reload` via `POST /-/reload` and confirm with
`/api/v1/rules?type=alert`.

Recording rules are caching, not aggregation religion: name them with the
conventional `level:metric:operation` style, keep them idempotent, and prefer
`sum`/`rate` over `count`-style ratios that need division in every query.
Alerting rules should be small and reviewable; a rule whose expression needs
a comment to explain is a candidate for a recording rule instead.

## Relabeling

`relabel_configs` run at target discovery time (before scraping) and
`metric_relabel_configs` run after scraping (per metric). Use them to:

- enforce label naming and drop forbidden labels (`__meta_*`, `job`);
- add scrape metadata (`__address__`, `__scheme__`, `__metrics_path__`);
- drop high-cardinality labels from `metric_relabel_configs` before the TSDB.

Relabeling is the classic silent-breakage point: a dropped or renamed label
changes series identity without an error. Verify with the target's effective
labels in `/api/v1/targets` and a spot-check of `/metrics` on the endpoint.
`keep`/`drop`/`replace`/`labelmap` are the operators you will actually use;
`regex` capture groups feed `replacement` with `$1`-style references.

## Retention

Local retention is `--storage.tsdb.retention.time` (age) and
`--storage.tsdb.retention.size` (bytes); blocks are ~2h and compaction merges
them. Set retention as a deliberate capacity decision (see
`references/04-stack-integration-and-retention.md`), never leave the defaults
for a long-running server. Watch `prometheus_tsdb_head_series` (cardinality),
`prometheus_tsdb_compactions_total` and `prometheus_tsdb_blocks_loaded` for
compaction health, and `prometheus_tsdb_storage_blocks_bytes` for the
footprint. Retention is enforced lazily by compaction — a server under
compaction pressure can exceed its retention window temporarily.

## High availability (HA)

HA for Prometheus means two identical instances scraping the same targets,
with the same rules, and consistent external labels, so a query layer can
deduplicate or shard. The instances do not share state: each has its own TSDB,
its own `for`-counter state, and its own alert evaluation. Practical rules:

- run both replicas with `--web.external-url` stable and identical rule files;
- give replicas distinct `replica` external labels so dedup can pick one;
- never add alert-specific noise that makes the two replicas fire different
  alert instances — dedup is by label sets;
- consider Thanos or Mimir for query federation and long-term retention, but
  only after the two-replica story is correct.

Rule evaluation correctness across replicas matters more than uptime: a
failover that changes when alerts fire is worse than a brief scrape gap.

# OpenTelemetry Collector Operations

> **Last Updated:** 2026-08-03

Operational patterns for the OpenTelemetry Collector in the telemetry stack:
pipeline design, receivers/processors/exporters, sampling, and trace/span
correlation. Sources: OpenTelemetry Collector documentation
(opentelemetry.io/docs/collector, accessed 2026-08-03).

## Pipeline design

A pipeline is a named, directed acyclic chain per signal type (metrics, logs,
traces): one or more `receivers`, zero or more `processors`, one or more
`exporters`. Signals flow through every processor in order, so pipeline length
is a cost and a debugging surface.

```yaml
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch, tail_sampling]
      exporters: [otlp/backend]
    metrics:
      receivers: [otlp, prometheus]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [loki]
```

Design rules:

- keep pipelines per signal — a pipeline that mixes traces and logs becomes
  un-debuggable and couples sampling decisions;
- run one `memory_limiter` processor before every exporter to bound memory
  (`ballast_size_mib` is deprecated; size `check_interval`/`limit_mib` against
  the container limit);
- `batch` after sampling and before exporting amortizes exporter cost;
- an unused receiver or exporter is dead configuration — remove it.

## Receivers, processors, exporters

| Role | Examples | Notes |
|---|---|---|
| Receiver | `otlp` (default port 4317 gRPC / 4318 HTTP), `prometheus`, `filelog`, `hostmetrics` | Receivers own the ingest surface; auth and TLS live here |
| Processor | `memory_limiter`, `batch`, `tail_sampling`, `probabilistic_sampler`, `resource`, `attributes`, `filter`, `transform`, `spanmetrics` | Order matters: sampling before batch changes semantics; `resource` should run early |
| Exporter | `otlp`, `prometheusremotewrite`, `loki`, `logging`/`debug`, `kafka` | Exporters own delivery and retry; failing exporters backpressure the pipeline |

The `logging`/`debug` exporter is the troubleshooting tool: attach it to a
pipeline temporarily to see what actually leaves the collector, then remove
it. Never ship a debug exporter in production config.

## Sampling

Two trace samplers ship with the collector:

- `probabilistic_sampler` — stateless, per-span hash of the trace ID, cheap,
  no cross-span state. Good for high-volume, low-value traffic.
- `tail_sampling` — buffers spans per trace and decides at the batch level,
  so policies can depend on span attributes (status, error, duration). Stateful
  and memory-hungry; belongs on a trace pipeline after `batch`.

Sample with the alerting and debugging goal in mind: keep every error and slow
path (via `tail_sampling` policy on `status.code`, `http.status_code`,
duration), sample the long tail of success traffic, and record the sampling
decision (`sampler.type`/`sampling.score`) as a span attribute so downstream
queries can scale results. Metrics and logs are not sampled by these processors
— do not apply trace samplers to other signal pipelines.

## Trace/span correlation

Correlation is the payoff of the stack: a trace ID in a log line or an
exemplar lets any query pivot from "this happened" to "here is the full
request". The collector supports this by:

- carrying `trace_id`/`span_id` through OTLP log records — the `loki` exporter
  maps them to `trace_id`/`span_id` structured metadata for LogQL matching;
- the `spanmetrics` processor deriving RED (rate/errors/duration) metrics from
  spans, with `trace_id` exemplars on the histogram so PromQL can jump to a
  trace;
- resource attributes (`service.name`, `deployment.environment`) flowing
  through to every signal so logs, metrics, and traces share the join keys.

Correlation is only as good as propagation: if the application does not
propagate context, the collector cannot invent it. Missing trace context in
logs usually means the SDK side is not wired — that is application-level work
for [backend-engineering](../backend-engineering/SKILL.md); the collector side
of the join is this skill.

# Loki Operations

> **Last Updated:** 2026-08-03

Operational patterns for the Loki half of the telemetry stack: ingest, LogQL,
retention, and label design. Sources: Grafana Loki documentation
(grafana.com/docs/loki, accessed 2026-08-03).

## Ingest

Loki ingests via the push API (`POST /loki/api/v1/push`) from Promtail, the
OpenTelemetry Collector `loki` exporter, Grafana Alloy/Agent, or the SDKs.
Ingest health is distributor-side:

- `loki_distributor_bytes_received_total` and
  `loki_distributor_lines_received_total` must advance per tenant;
- `loki_ingester_streams` shows active streams — a flat line here while
  producers push means a config or network problem;
- the ready endpoint (`/ready`) must return 200; `429 Too Many Requests` from
  the distributor means rate limits — the producer retries, but a sustained
  backlog is a capacity signal.

The OTel Collector `loki` exporter maps log records to streams: the
`loki.tenant` and `loki.format` attributes plus configured labels control
stream cardinality (see Labels below). Promtail is the file-tailer option;
choose one producer per source and document it.

## LogQL

LogQL has two layers: stream selectors and pipeline expressions.

- Selectors choose streams by label: `{app="api", env="prod"}`. Label
  matchers are evaluated against the inverted index — this is the cheap part,
  and it is why label design matters.
- Pipeline expressions filter and transform lines: `|= "error"`,
  `|~ "5[0-9][0-9]"`, `| json`, `| regexp "(?P<field>...)"`, `| line_format`.
  These run per line after selection — the expensive part.

Metric queries wrap the pipeline in `rate`, `count_over_time`, etc.:
`sum by (app) (rate({job="api"} |~ "error"[5m]))`. When a query is slow,
the cause is almost always a too-broad selector (too many streams) or
regexp/JSON parsing on every line; fix the selector or pre-extract fields at
ingest, not by writing a faster regexp.

## Retention

Loki retention is per-tenant, enforced by the compactor:

- `retention_enabled: true` in the limits config enables period-based
  retention; `retention_period` sets the age limit and `retention_size` the
  byte limit per tenant;
- the compactor deletes expired chunks and merges index shards; watch
  `loki_compactor_delete_requests_total` and
  `loki_compactor_compactor_running` to confirm it is actually running;
- retention applies at query and delete time, so expired data can still be
  counted until compaction finishes — size-based limits are the practical
  control for runaway log volume.

Set retention before rollout and treat it as a compliance decision with an
owner (see `references/04-stack-integration-and-retention.md`). A Loki with
default retention and no owner will silently grow until storage is the
incident.

## Labels

Loki labels are an inverted index — every distinct label value adds index
entries and stream overhead. The guidance is deliberately simple:

- keep labels to tenant, app, environment, job, and a handful of
  service-defined dimensions;
- never index high-cardinality fields: request IDs, user IDs, trace IDs,
  IPs, timestamps, or any field whose values change per log line;
- put high-cardinality data in the log line itself and extract it with LogQL
  `| json`/`| regexp`, or as OTel structured metadata, when you need it;
- a good rule of thumb: if a label's values exceed the low hundreds of
  distinct values, it is a line field, not a label.

Cardinality damage is silent and cumulative: `loki_ingester_streams`
climbing while label changes are "small" is the leading signal. Stream
sharding and `chunk_target_size` tune large-stream handling, but they do not
make a bad label design good.

# Stack Integration and Retention

> **Last Updated:** 2026-08-03

How the three components of the telemetry stack fit together, and how to make
retention decisions across them instead of per-component by accident. Sources:
Prometheus storage documentation, OpenTelemetry Collector documentation, and
Loki retention documentation (all accessed 2026-08-03).

## The stack as one unit

The components share a data flow — instrumented services emit metrics, logs,
and traces; the OpenTelemetry Collector (or direct exporters) delivers them;
Prometheus stores metrics and evaluates rules; Loki stores logs; Grafana
queries both and displays the result. The stack is operated as one unit
because a change in any layer changes the behavior of every layer above it:

| Layer | Component | Owns |
|---|---|---|
| Collection | OTel Collector receivers, Prometheus scrape jobs | What data enters the stack |
| Processing | Collector processors (batch, sample, resource) | What the data looks like when stored |
| Storage | Prometheus TSDB, Loki chunks/index | How long data lives and how fast queries are |
| Evaluation | Prometheus rules | What the data means (alerts, recording rules) |
| Presentation | Grafana | What humans see (dashboards, Grafana-side alerting) |

Operational decisions therefore cross component boundaries: a label added in
relabeling is a label PromQL sees; a `trace_id` carried by the collector is a
LogQL matcher; a retention window chosen for Prometheus is a gap in history a
dashboards-as-code change cannot recover.

## Retention as a stack decision

Each component has independent retention, and the combined footprint is what
the team pays for:

| Component | Setting | Applies to | Default behavior |
|---|---|---|---|
| Prometheus | `--storage.tsdb.retention.time` / `.size` | Raw samples (blocks) | 15 days; lazily enforced by compaction |
| OTel Collector | exporter queue + `memory_limiter` | In-flight data | Queue backs up then drops oldest on pressure |
| Loki | `retention_enabled`, `retention_period`, `retention_size` (per tenant) | Indexed log streams | No retention unless enabled; compactor enforces it |

Decide per component by the question the data answers:

- hot metrics for alerting: keep enough history to evaluate every rule window
  plus headroom (a `for: 10m` rule needs at least that much history);
- samples for trends: retention length is a capacity trade-off, not a
  correctness requirement — long histories belong in a separate store
  (Thanos/Mimir) with its own owner;
- logs for debugging and audit: retention is a compliance decision with an
  owner; short retention on logs destroys the only evidence an incident
  post-mortem can use.

The failure mode is defaulting every component and discovering the cost when
storage is the incident. Write the decision down: `retention_period`,
retention flags, and the RPO/RTO framing belong in the deployment config and
its review checklist, not in tribal memory.

## Cross-component consistency checks

- Scrape config and collector receivers must agree on the metrics endpoint:
  a `metric_relabel_configs` drop on Prometheus side does not stop the
  collector from exporting the metric elsewhere.
- Rule expressions and recording rules must not depend on labels the scrape
  config does not produce — validate with `telemetry-check --rules` plus a
  live query, not by reading the config.
- LogQL matchers must match labels the `loki` exporter actually attaches —
  a matcher on `app` fails silently when the collector maps it to
  `service.name` only.
- Trace correlation needs `trace_id`/`span_id` in both the OTLP log records
  and the metrics exemplars; verify one pivot query end-to-end after any
  collector pipeline change.

## Alerting on the stack itself

The stack needs its own health rules (in a file you also validate with
`telemetry-check --rules`):

- `up{job=~"prometheus|otel-collector|loki"}` == 0 for component loss;
- `prometheus_tsdb_head_series` growth vs a budget for cardinality;
- `rate(loki_distributor_lines_received_total[5m])` vs a floor to catch
  silent ingest loss;
- `otelcol_exporter_send_failed_ratio` above a threshold for delivery loss;
- retention drift: `prometheus_tsdb_storage_blocks_bytes` and compactor
  delete counters vs the decided policy.

These rules live in this skill's scope (the Prometheus rules file); the
dashboards and Grafana-side alert routes for them are `grafana` territory.

## Routing

- Observability strategy — what to instrument, SLI/SLO design, error budgets —
  is [platform-engineering](../platform-engineering/SKILL.md).
- Dashboards, panels, Grafana alert rules, contact points, and notification
  policies are [grafana](../grafana/SKILL.md).
- This skill owns the collection/ingest/retention layer and the Prometheus
  rules files those layers consume. When a task crosses into those skills,
  route there instead of duplicating their content.

# PromQL and LogQL Query Workflows

> **Last Updated:** 2026-09-01

Use this reference for an offline query review or a live, read-only query investigation. It covers query shape and evidence; it does not replace `promtool` or Loki's parser.

## 1. Define a bounded question

Write down the signal, service/job, environment, exact UTC start and end, resolution, and expected unit before writing the expression. Prefer a narrow selector with equality matchers:

```promql
sum by (job, route) (rate(http_requests_total{job="api", environment="prod"}[5m]))
```

For logs, select streams first and filter lines second:

```logql
{app="api", environment="prod"} |= "timeout" | json | duration_ms > 1000
```

Do not begin with `{}` or `metric{label=~".*"}`. Avoid unbounded regex, arbitrary joins, and multi-day ranges while exploring. A range query must state its step; choose a step no finer than the evidence needs. Bound concurrency and request timeouts at the client/query gateway.

## 2. Validate in two distinct passes

Syntax and semantics are different claims:

1. **Syntax:** parse the expression with `promtool check rules` (PromQL) or the target Loki/Grafana parser. Check balanced delimiters, operators, durations, and LogQL pipeline syntax. A HTTP 200 only proves the endpoint accepted the request, not that the question is correct.
2. **Semantics:** confirm metric/log names, label keys and value types, counter-versus-gauge intent, compatible label sets for binary operators, and aggregation dimensions. For joins, make the matching labels explicit (`on(...)`/`ignoring(...)`) and use `group_left`/`group_right` only when the cardinality relationship is known. For LogQL, verify parsed fields exist and that aggregation is applied to the intended streams.

Record both results separately. A syntactically valid query can return an empty vector because the label was renamed, the metric was never scraped, or retention has expired.

## 3. Capture bounded evidence

For every query record:

- backend and endpoint, HTTP status, and response `status`/error type;
- exact query text or a redacted query identifier;
- UTC `start`, `end`, and Prometheus `step` (or Loki limit/direction);
- selector and grouping labels, estimated series/stream count, and duration;
- whether the result is instant, range, logs, or a derived metric;
- parser result, semantic checks, warnings, and a link/request ID if available.

Use instant queries to test existence, then a short range query to establish behavior. Keep result samples bounded; do not paste raw logs or credentials.

## 4. Control cost and cardinality

Start with a narrow equality selector and a short range. Expand one dimension at a time, checking series count and latency after each change. Prefer recording rules for repeatedly used expensive PromQL expressions and pre-extracted low-cardinality Loki labels for common filters. Keep request limits explicit: maximum lookback, step floor, series/stream limit, bytes/line limit, timeout, and query concurrency. A gateway or tenant limit is a safety net, not permission to issue a broad query.

Avoid `count by` over unbounded labels, grouping by request/user/trace IDs, regex over all streams, JSON/regexp parsing before a selective matcher, and joins where both sides are high cardinality. A label whose value changes per event belongs in the log body or structured metadata, not Loki's index.

## 5. Diagnose empty and partial results without inventing zeros

An empty result is **unknown**, never numeric zero. Distinguish:

| Observation | Next read-only check | Interpretation |
|---|---|---|
| Query parser error / non-2xx | response error type and expression | Query error; fix syntax or request shape |
| Empty with healthy backend | instant existence query, label/series API, exact time window | Could be absent label, wrong selector, or no events |
| Empty only for old time | retention bounds and backend clock | Data expired or outside retention |
| Empty for one target | scrape/ingest target state and relabel output | Failed target, dropped series, or missing stream |
| Stale/flat value | sample timestamps, scrape freshness, exporter metrics | Stale data, stalled producer, or timestamp issue |
| Partial response/warnings | HTTP status, response warnings, shard/limit metrics | Incomplete evidence; do not aggregate as complete |
| Backend timeout/limit | query duration, series/stream limit, concurrency | Cost or capacity rejection; narrow query |

For ratios, do not substitute zero for a missing numerator or denominator. Report `no data`, preserve `NaN`/absence semantics, and state what evidence is missing. A target can be `up` while a particular metric is absent because relabeling or instrumentation changed.

## Ownership and deferral

Route SLI/SLO definitions, error budgets, alert thresholds, and paging strategy to [platform-engineering](../../platform-engineering/SKILL.md). Route Grafana panels, data-source configuration, Grafana alert rules, contact points, and notification policies to [grafana](../../grafana/SKILL.md). This skill supplies backend query evidence those owners consume.

Tempo/tracing query workflows are intentionally deferred to a future named-tool decision. This skill may preserve trace/span IDs for correlation in logs and metrics, but it does not claim Tempo commands, APIs, or trace-query semantics. Route application propagation to [backend-engineering](../../backend-engineering/SKILL.md) and revisit a dedicated Tempo skill only when a concrete operational surface and non-overlapping trigger are established.

#!/usr/bin/env python3
"""Deterministic tests for the telemetry/scripts/telemetry-check tool.

Runs the script as a subprocess so the tests exercise the real CLI surface
(--help, --json, --rules, --scrape, --targets, exit codes, JSON payloads).
Rules fixtures are written to temp directories at test time; scrape-target
tests probe a real local listening socket for the reachable case and a
just-released port for the unreachable case, so no external network is needed.
Also asserts the read-only contract: the script never opens files in write mode.
"""
import json
import re
import socket
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path

ROOT = Path(__file__).resolve().parent.parent
SCRIPT = ROOT / "scripts" / "telemetry-check"
FIXTURES = ROOT / "fixtures"


def run_script(*args):
    return subprocess.run(
        [sys.executable, str(SCRIPT), *args],
        capture_output=True,
        text=True,
        timeout=30,
    )


def load_json(proc):
    return json.loads(proc.stdout)


def free_port():
    """Bind a socket to an ephemeral port and return (port, socket); caller closes."""
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.bind(("127.0.0.1", 0))
    sock.listen(1)
    return sock.getsockname()[1], sock


class HelpTests(unittest.TestCase):
    def test_help_exits_zero_and_advertises_capabilities(self):
        proc = run_script("--help")
        self.assertEqual(proc.returncode, 0)
        self.assertIn("--json", proc.stdout)
        self.assertIn("rule", proc.stdout.lower())
        self.assertIn("scrape", proc.stdout.lower())
        self.assertIn("read-only", proc.stdout.lower())

    def test_version_flag(self):
        proc = run_script("--version")
        self.assertEqual(proc.returncode, 0)
        self.assertIn("telemetry-check", proc.stdout)

    def test_no_args_is_usage_error(self):
        proc = run_script()
        self.assertEqual(proc.returncode, 2)


class RulesFileTests(unittest.TestCase):
    def test_valid_fixture_parses_to_json_and_passes(self):
        proc = run_script("--rules", str(FIXTURES / "prometheus-rules.yml"), "--json")
        self.assertEqual(proc.returncode, 0, proc.stderr)
        payload = load_json(proc)
        self.assertTrue(payload["ok"])
        self.assertEqual(payload["checks"][0]["status"], "ok")
        self.assertEqual(payload["checks"][0]["groups"], 2)
        self.assertEqual(payload["checks"][0]["rules"], 4)
        self.assertEqual(payload["checks"][0]["errors"], [])

    def test_malformed_yaml_is_rejected(self):
        bad = "groups:\n  - name: bad\n   rules:\n     - record: x\n"
        with tempfile.TemporaryDirectory() as tmp:
            path = Path(tmp) / "bad.yml"
            path.write_text(bad, encoding="utf-8")
            proc = run_script("--rules", str(path), "--json")
        self.assertNotEqual(proc.returncode, 0)
        payload = load_json(proc)
        self.assertFalse(payload["ok"])
        self.assertIn("invalid YAML", payload["checks"][0]["error"])

    def test_duplicate_group_names_detected(self):
        rules = (
            "groups:\n"
            "  - name: g1\n"
            "    rules:\n"
            "      - record: a_total\n"
            "        expr: sum(foo)\n"
            "  - name: g1\n"
            "    rules:\n"
            "      - alert: B\n"
            "        expr: up == 0\n"
        )
        with tempfile.TemporaryDirectory() as tmp:
            path = Path(tmp) / "dup.yml"
            path.write_text(rules, encoding="utf-8")
            proc = run_script("--rules", str(path), "--json")
        self.assertEqual(proc.returncode, 1)
        errors = load_json(proc)["checks"][0]["errors"]
        self.assertTrue(any("repeated" in error for error in errors))

    def test_rule_errors_detected(self):
        rules = (
            "groups:\n"
            "  - name: g1\n"
            "    rules:\n"
            "      - record: bad{name}\n"
            "        alert: AlsoAlert\n"
            "        for: 5x\n"
            "        labels:\n"
            "          severity: 5\n"
            "      - record: ok_name\n"
        )
        with tempfile.TemporaryDirectory() as tmp:
            path = Path(tmp) / "errors.yml"
            path.write_text(rules, encoding="utf-8")
            proc = run_script("--rules", str(path), "--json")
        self.assertEqual(proc.returncode, 1)
        errors = load_json(proc)["checks"][0]["errors"]
        joined = "\n".join(errors)
        self.assertIn("only one of 'record' and 'alert'", joined)
        self.assertIn("braces present in recording rule name", joined)
        self.assertIn("invalid duration", joined)
        self.assertIn("not a YAML string", joined)
        self.assertIn("field 'expr' must be set", joined)

    def test_unbalanced_expression_is_detected(self):
        rules = (
            "groups:\n"
            "  - name: g1\n"
            "    rules:\n"
            "      - record: a_total\n"
            "        expr: sum(foo\n"
        )
        with tempfile.TemporaryDirectory() as tmp:
            path = Path(tmp) / "unbalanced.yml"
            path.write_text(rules, encoding="utf-8")
            proc = run_script("--rules", str(path), "--json")
        self.assertEqual(proc.returncode, 1)
        errors = load_json(proc)["checks"][0]["errors"]
        self.assertTrue(any("unbalanced" in error for error in errors))

    def test_quoted_numeric_label_value_is_accepted(self):
        rules = (
            "groups:\n"
            "  - name: g1\n"
            "    rules:\n"
            "      - alert: HighCPU\n"
            "        expr: cpu_usage > 0.9\n"
            "        labels:\n"
            "          severity: \"5\"\n"
            "          team: platform\n"
        )
        with tempfile.TemporaryDirectory() as tmp:
            path = Path(tmp) / "quoted.yml"
            path.write_text(rules, encoding="utf-8")
            proc = run_script("--rules", str(path), "--json")
        self.assertEqual(proc.returncode, 0, proc.stdout)
        self.assertEqual(load_json(proc)["checks"][0]["errors"], [])

    def test_warning_only_findings_are_not_fatal(self):
        rules = (
            "groups:\n"
            "  - name: g1\n"
            "    rules:\n"
            "      - record: my rule\n"
            "        expr: sum(foo)\n"
            "        extra_field: 1\n"
        )
        with tempfile.TemporaryDirectory() as tmp:
            path = Path(tmp) / "warn.yml"
            path.write_text(rules, encoding="utf-8")
            proc = run_script("--rules", str(path), "--json")
        self.assertEqual(proc.returncode, 0, proc.stdout)
        check = load_json(proc)["checks"][0]
        self.assertEqual(check["status"], "ok")
        self.assertTrue(any("metric-name pattern" in w for w in check["warnings"]))
        self.assertTrue(any("unknown rule field" in w for w in check["warnings"]))

    def test_block_scalar_expression_parses(self):
        rules = (
            "groups:\n"
            "  - name: g1\n"
            "    rules:\n"
            "      - alert: SlowQueries\n"
            "        expr: |\n"
            "          histogram_quantile(0.99,\n"
            "            sum by (le) (rate(query_duration_seconds_bucket[5m])))\n"
            "        for: 15m\n"
        )
        with tempfile.TemporaryDirectory() as tmp:
            path = Path(tmp) / "block.yml"
            path.write_text(rules, encoding="utf-8")
            proc = run_script("--rules", str(path), "--json")
        self.assertEqual(proc.returncode, 0, proc.stdout)
        self.assertEqual(load_json(proc)["checks"][0]["errors"], [])

    def test_missing_rules_file_is_fatal(self):
        proc = run_script("--rules", "/nonexistent/rules.yml", "--json")
        self.assertEqual(proc.returncode, 1)
        payload = load_json(proc)
        self.assertEqual(payload["checks"][0]["status"], "error")


class ScrapeTargetTests(unittest.TestCase):
    def test_scrape_config_probes_reachable_and_unreachable(self):
        reachable_port, listener = free_port()
        try:
            with tempfile.TemporaryDirectory() as tmp:
                unreachable_port, probe = free_port()
                probe.close()
                config = (
                    "scrape_configs:\n"
                    "  - job_name: local\n"
                    "    static_configs:\n"
                    "      - targets:\n"
                    "          - '127.0.0.1:%d'\n"
                    "          - '127.0.0.1:%d'\n" % (reachable_port, unreachable_port)
                )
                path = Path(tmp) / "scrape.yml"
                path.write_text(config, encoding="utf-8")
                proc = run_script(
                    "--scrape", str(path), "--json", "--timeout", "1"
                )
        finally:
            listener.close()
        self.assertEqual(proc.returncode, 1)
        check = load_json(proc)["checks"][0]
        self.assertEqual(check["status"], "issues")
        by_target = {entry["target"]: entry for entry in check["targets"]}
        self.assertTrue(by_target["127.0.0.1:%d" % reachable_port]["reachable"])
        self.assertFalse(by_target["127.0.0.1:%d" % unreachable_port]["reachable"])

    def test_targets_plain_list_file(self):
        with tempfile.TemporaryDirectory() as tmp:
            path = Path(tmp) / "targets.txt"
            path.write_text("127.0.0.1:1\n127.0.0.1:2\n", encoding="utf-8")
            proc = run_script("--targets", str(path), "--json", "--timeout", "1")
        self.assertEqual(proc.returncode, 1)
        check = load_json(proc)["checks"][0]
        self.assertEqual(len(check["targets"]), 2)
        self.assertTrue(all(not entry["reachable"] for entry in check["targets"]))

    def test_targets_yaml_list_file(self):
        with tempfile.TemporaryDirectory() as tmp:
            path = Path(tmp) / "targets.yml"
            path.write_text("- 127.0.0.1:1\n- 127.0.0.1:2\n", encoding="utf-8")
            proc = run_script("--targets", str(path), "--json", "--timeout", "1")
        self.assertEqual(proc.returncode, 1)
        check = load_json(proc)["checks"][0]
        self.assertEqual(len(check["targets"]), 2)


class ReadOnlyContractTests(unittest.TestCase):
    def test_script_never_opens_files_for_writing(self):
        source = SCRIPT.read_text(encoding="utf-8")
        pattern = re.compile(r"open\([^)]*['\"][w]['\"]")
        self.assertIsNone(pattern.search(source))
        self.assertIn("read-only", source.lower())


if __name__ == "__main__":
    unittest.main()