Claude Skill

promql

PromQL query writing: selectors, operators, vector matching, functions, aggregations, histograms, subqueries, recording rules, alerting rules, and query optimization. Invoke whenever task involves any interaction with PromQL — writing, debugging, optimizing, reviewing, or explain

LLM Mart · 0 points · 0 views 0 listing impressions 0 install-command copies
Download xobotyi-cc-foundry-plugins_grafana_skills_promql-278c16c.zip · 42 KB

Install

skills CLI npx skills add https://github.com/xobotyi/cc-foundry/tree/master/plugins/grafana/skills/promql
Claude Code claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install xobotyi-cc-foundry@llmmart
Git git clone https://github.com/xobotyi/cc-foundry.git

The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole xobotyi/cc-foundry collection as a plugin from our marketplace. Git is the plain clone.

Skill manifest

PromQL

PromQL is a nested functional language for selecting and aggregating Prometheus time series. Every query is an expression tree: selectors at the leaves, functions and operators at the nodes, a single root value (scalar, instant vector, range vector, or string) at the top. Type discipline makes queries correct; cardinality awareness makes them fast.

Authoritative reference for writing PromQL queries — dashboards, alerts, recording rules, exploration. The companion backend/prometheus skill covers instrumentation (metric types, naming, labels, exporters). When both apply, use backend/prometheus for emitting metrics and this skill for querying them.

References

  • Selectors and data types — [${CLAUDE_SKILL_DIR}/references/selectors-and-types.md] Expression types, instant and range vector selectors, label matchers (=, !=, =~, !~), time durations, offset and @ modifiers, staleness, instant vs range queries, string and float literals
  • Operators — [${CLAUDE_SKILL_DIR}/references/operators.md] Arithmetic, comparison, logical/set, vector matching (on, ignoring, group_left, group_right), aggregation operators, by/without, fill modifiers, operator precedence
  • Functions — [${CLAUDE_SKILL_DIR}/references/functions.md] Full function catalog: counter family, gauge functions, classic and native histograms, *_over_time, existence checks, label manipulation, math, time, type conversion, sorting, trigonometric
  • Native histograms — [${CLAUDE_SKILL_DIR}/references/native-histograms.md] Native vs classic detection, histogram_quantile across both forms, native-only functions (histogram_avg, histogram_count, histogram_sum, histogram_fraction, histogram_stddev, histogram_stdvar), aggregation without le, NHCB semantics, trim operators, migration patterns, function compatibility matrix, gotchas, annotations
  • Subqueries — [${CLAUDE_SKILL_DIR}/references/subqueries.md] Syntax, resolution and alignment, nested subqueries, when to use subqueries vs recording rules, pitfalls
  • Optimization and pitfalls — [${CLAUDE_SKILL_DIR}/references/optimization.md] Cardinality awareness, common pitfalls (rate-of-gauge, aggregate-then-rate, averaging ratios, averaging summary quantiles), counter resets, staleness, rate window sizing, native histograms, diagnostics
  • Recording and alerting rules — [${CLAUDE_SKILL_DIR}/references/recording-alerting-rules.md] Rule file structure, level:metric:operations naming, ratio aggregation patterns, alert syntax, for/keep_firing_for, templating, alerting best practices, anti-patterns

Read the relevant reference before proceeding.


Data Types

Every expression has one type. Functions and operators require specific input types — type mismatch is the most common cause of "query parses but returns nothing".

  • instant vector — set of series, one sample per series at one timestamp. Default result of metric selectors, aggregations, most functions.
  • range vector — set of series, multiple samples per series across a time window. Produced only by range vector selectors (metric[5m]) and subqueries. Cannot be graphed directly — must pass through a function returning an instant vector.
  • scalar — single numeric value, no labels. Used as numeric arguments and in arithmetic.
  • string — function arguments only; cannot be a query result.

Function type rules:

  • rate, irate, increase, delta, idelta, deriv, predict_linear, *_over_time — range vector → instant vector
  • histogram_quantile, histogram_fraction, histogram_* — instant vector (histogram or bucket series) → instant vector
  • Aggregation operators (sum, avg, max, topk, ...) — instant vector → instant vector
  • scalar() — instant vector → scalar; vector() — scalar → instant vector

Selectors

Instant Vector Selectors

http_requests_total                                # all series with this metric name
http_requests_total{job="api"}                     # filtered by label
http_requests_total{job="api", method="GET"}       # multiple matchers — AND
http_requests_total{status=~"5.."}                 # regex match (RE2, fully anchored)
http_requests_total{method!="OPTIONS"}             # negative match
{__name__=~"http_.*"}                              # regex on metric name via __name__

Label matchers: = exact, != not equal, =~ regex match, !~ negative regex.

Regex is fully anchored — =~"foo" matches ^foo$, not arbitrary substrings. Use =~".*foo.*" for substrings.

Required selectors: at least one matcher must not match the empty string. {job=~".*"} is illegal; {job=~".+"} is valid.

Range Vector Selectors

Append [duration] to a selector:

http_requests_total[5m]                            # last 5 minutes of samples
http_requests_total{job="api"}[1h]

Duration units: ms, s, m, h, d, w, y. Combine longest-first: 1h30m, 12h34m56s. Floats with units are not allowed (1.5h is invalid — use 1h30m).

Modifiers

  • offset <duration> — time-shift into the past (or future with negative). Must immediately follow the selector.

    http_requests_total offset 5m
    rate(http_requests_total[5m] offset 1w)            # rate one week ago
    
  • @ <unix-timestamp> — pin evaluation to an absolute time. start() and end() resolve to the range query's bounds.

    http_requests_total @ 1609746000
    rate(http_requests_total[5m] @ end())
    

Staleness: instant vector selectors return the most recent sample within the lookback window (default 5 minutes). Series with no sample in the window disappear from the result.

Full selector reference, lookback delta tuning, instant-vs-range-query semantics: [${CLAUDE_SKILL_DIR}/references/selectors-and-types.md].

Operators

Arithmetic and Comparison

+, -, *, /, %, ^ — IEEE 754 arithmetic. The metric name is dropped from any vector arithmetic result.

==, !=, >, <, >=, <= — by default filter: drop series where the comparison is false. Add bool after the operator to return 0 or 1 instead.

http_requests_total > 100                          # only series whose latest value exceeds 100
http_requests_total > bool 100                     # all series, value is 1 if > 100 else 0

Logical/Set Operators

Operate on label-set membership, ignoring sample values:

  • vector1 and vector2 — intersection
  • vector1 or vector2 — union
  • vector1 unless vector2 — complement
up{job="prom"} or up{job="node"}                       # union
metric and on(device) (other_metric == 0)              # constrained intersection

Vector Matching

The biggest source of "no results returned". Default rule: two elements match if they have exactly the same label set (after the metric name drops). When operands have different label sets, match keywords are required:

  • on(labels) — match only on these labels; ignore all others
  • ignoring(labels) — match on all labels except these

Group modifiers for many-to-one (arithmetic, comparison, trigonometric only — not set operators):

  • group_left(extra_labels) — many on the left, one on the right; copy extra_labels from right to result
  • group_right(extra_labels) — symmetric

Canonical patterns:

# Error ratio: errors carry an extra `code` label; ignore it to match
errors:rate5m{code="500"} / ignoring(code) requests:rate5m

# Many-to-one: same denominator for every code
errors:rate5m / ignoring(code) group_left requests:rate5m

# Join info-style metric: bring `version` label into result
node_filesystem_avail_bytes * on(instance, job) group_left(version) node_exporter_build_info

Aggregation

Instant vector → instant vector with fewer elements.

  • sum, avg, min, max — reduce across dimensions
  • count — number of series; group — value 1 per group
  • topk(k, v) / bottomk(k, v) — k largest/smallest; preserves labels
  • stddev, stdvar — population statistics
  • quantile(φ, v) — quantile across series
  • count_values(label, v) — distribution of values

by vs without:

  • by (labels) keeps only the listed labels
  • without (labels) keeps everything except the listed labels

Prefer without for normal aggregations — preserves job, instance, and friends. Use by for a known output dimension set (typically recording rules).

sum without (instance) (rate(http_requests_total[5m]))    # collapses instance, keeps job/method/status
sum by (job) (rate(http_requests_total[5m]))              # collapses everything except job

Operator Precedence (highest to lowest)

  1. ^
  2. *, /, %, atan2
  3. +, -
  4. ==, !=, <=, <, >=, >
  5. and, unless
  6. or

Left-associative except ^ (right-associative). Use parentheses when mixing comparison and logical operators.

Full operator catalog, fill modifiers, detailed vector matching examples: [${CLAUDE_SKILL_DIR}/references/operators.md].

Key Functions by Category

Counters — rate, irate, increase

The three most important PromQL functions. All take a range vector, return an instant vector, handle counter resets automatically.

  • rate(v[5m]) — per-second average rate over the range, extrapolated. Use for alerts and slow counters.
  • irate(v[1m]) — instant rate from the last two samples. Graphing volatile counters only; never alerts (flaps).
  • increase(v[1h]) — total increase over the range; syntactic sugar for rate(v) * range_seconds. Use for human-readable totals.

Composition rule — rate() first, then aggregate:

sum(rate(http_requests_total[5m]))             # CORRECT
rate(sum(http_requests_total)[5m])             # WRONG — counter resets disappear into the sum

Applies to all counter functions (rate, irate, increase), gauge functions (delta, deriv, predict_linear), and *_over_time functions.

Window sizing: range must contain ≥2 samples. Rule of thumb: 4× the scrape interval, minimum 1m. With 15s scrapes the standard default is [5m].

Gauges — delta, idelta, deriv, predict_linear

For gauges only. Never apply rate() to a gauge.

delta(cpu_temp_celsius[2h])                                  # absolute change over 2h
deriv(node_memory_MemAvailable_bytes[5m])                    # smoothed per-second change
predict_linear(node_filesystem_avail_bytes[1h], 4*3600) < 0  # will fs fill in 4h?

Histograms

Classic histograms expose _bucket{le="..."}, _sum, _count series:

histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# Aggregating across instances — `le` MUST be preserved
histogram_quantile(0.95, sum by (job, le) (rate(http_request_duration_seconds_bucket[5m])))
histogram_quantile(0.95, sum without (instance) (rate(http_request_duration_seconds_bucket[5m])))

# Average from sum/count
rate(http_request_duration_seconds_sum[5m]) / rate(http_request_duration_seconds_count[5m])

Native histograms are single composite series — quantile aggregation does not require le:

histogram_quantile(0.9, sum by (job) (rate(http_request_duration_seconds[5m])))
histogram_fraction(0, 0.2, rate(http_request_duration_seconds[5m]))     # fraction ≤ 200ms
histogram_avg(rate(http_request_duration_seconds[5m]))
histogram_count(rate(http_request_duration_seconds[5m]))                # rate of observations
histogram_sum(rate(http_request_duration_seconds[5m]))

If native histograms are available, prefer them — fewer series, easier aggregation, lower error.

Full native histogram semantics — detection, function compatibility matrix, NHCB behavior, trim operators, migration, gotchas: [${CLAUDE_SKILL_DIR}/references/native-histograms.md].

Aggregation Over Time — *_over_time

Range vector → one summary value per series.

  • avg_over_time(range-vector) — average across the range
  • min_over_time / max_over_time — extremes
  • sum_over_time / count_over_time — totals
  • last_over_time / present_over_time — most recent / existence
  • quantile_over_time(φ, range-vector) — quantile across time
  • stddev_over_time / stdvar_over_time
avg_over_time(node_memory_MemAvailable_bytes[1h])
max_over_time(rate(http_requests_total[5m])[1h:])         # subquery needed — rate is instant vector

Existence and Counter Resets

absent(up{job="critical"})                         # 1 if no series, empty if any
absent_over_time(up{job="critical"}[5m])           # missing for the entire window
resets(http_requests_total[1h])                    # how many counter resets in the range
changes(up[1h])                                    # how many times the value changed

Label Manipulation

label_replace(up, "host", "$1", "instance", "(.+):(\\d+)")        # extract host from instance="host:port"
label_join(rate(x[5m]), "endpoint", " ", "method", "path")        # concatenate labels
info(rate(http_server_duration_count[5m]))                        # auto-join with target_info (experimental)

Full function catalog: [${CLAUDE_SKILL_DIR}/references/functions.md].

Subqueries

Embed a range query inside an instant query. Syntax:

<instant-vector-expression>[<range>:[<resolution>]]

Use when a range-vector function needs the output of another function or aggregation:

max_over_time(rate(http_requests_total[5m])[1h:])             # peak 5m-rate over the last hour
quantile_over_time(0.99, rate(errors[1m])[1d:])               # 99th percentile of per-minute error rates over a day
deriv(rate(http_requests_total[5m])[30m:])                    # is the rate accelerating?

Performance: subqueries re-run the inner expression at every resolution step. A [1d:1m] subquery runs the inner 1,440 times. For expensive inner expressions or repeated use, convert to a recording rule.

Avoid subqueries inside recording rules. Split into two recording rules.

Full syntax, alignment rules, subquery vs recording rule guidance: [${CLAUDE_SKILL_DIR}/references/subqueries.md].

Common Query Idioms

# Request rate by job
sum by (job) (rate(http_requests_total[5m]))

# Error ratio — aggregate numerator and denominator separately
  sum(rate(http_requests_total{status=~"5.."}[5m]))
/
  sum(rate(http_requests_total[5m]))

# P99 latency by handler (classic histogram)
histogram_quantile(0.99, sum by (handler, le) (rate(http_request_duration_seconds_bucket[5m])))

# Top 5 CPU consumers
topk(5, sum by (app) (rate(process_cpu_seconds_total[5m])))

# Disk-full prediction
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 24*3600) < 0

# Alert on missing scrape target
absent(up{job="critical-service"})

# Week-over-week request rate comparison
  rate(http_requests_total[5m])
-
  rate(http_requests_total[5m] offset 1w)

# Batch job staleness
time() - my_batch_last_success_timestamp_seconds > 2 * 3600

Pitfalls — Quick Reference

Positive rules when writing or reviewing queries:

  • Apply rate() to counters only. Use delta(), idelta(), or deriv() for gauges. rate() on a gauge silently produces nonsense.
  • rate() first, then aggregate. sum(rate(x[5m])), never rate(sum(x)[5m]). Counter resets are per-series.
  • Aggregate ratios correctly. Sum numerator and denominator separately, then divide. Never avg(ratio) or avg(avg).
  • Don't average summary quantiles. avg(metric{quantile="0.95"}) is statistically invalid. Use histograms for cross-instance aggregation.
  • Preserve le when aggregating classic histogram buckets. sum by (job, le) or sum without (instance).
  • Range ≥ 4× scrape interval, minimum 1m. Avoid rate(x[1m]) with 30s+ scrape intervals.
  • rate() (not irate()) for alerts. irate() is for graphing volatile counters.
  • without over by for most aggregations — preserves useful labels like job automatically.
  • offset must immediately follow the selector. sum(x offset 5m) is correct; sum(x) offset 5m is invalid.
  • Filter early. Push label matchers into the innermost selector.

Full pitfall catalog with diagnostics: [${CLAUDE_SKILL_DIR}/references/optimization.md].

Recording and Alerting Rules

Recording Rule Naming — level:metric:operations

instance:http_requests:rate5m                   # per-instance rate (no aggregation)
job:http_requests:rate5m                        # rolled up to per-job
job:http_errors_per_requests:ratio_rate5m       # ratio, per job
job:request_latency_seconds:mean5m              # average latency from Summary
  • level — labels remaining in the output (instance, job, cluster, instance_path, etc.)
  • metric — metric name; strip _total after rate()
  • operations — operations applied, newest first. Omit _sum when other ops present; mean replaces rate for sum/count averages; ratio for divisions named with _per_

Recording rule examples and ratio aggregation patterns: [${CLAUDE_SKILL_DIR}/references/recording-alerting-rules.md].

Alerting Rule Essentials

- alert: HighErrorRatio                   # CamelCase community convention
  expr: |
      sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
    /
      sum by (job) (rate(http_requests_total[5m]))
    > 0.05
  for: 10m                                # must be true for the entire window before firing
  keep_firing_for: 5m                     # keep firing this long after expression becomes false
  labels:
    severity: page
  annotations:
    summary: "High error rate on {{ $labels.job }}"
    description: "Error ratio is {{ $value | humanizePercentage }} for {{ $labels.job }}"
    runbook_url: "https://wiki.example.com/runbooks/{{ $labels.job }}"

Alerting principles:

  • Alert on symptoms, not causes. User-visible latency, error rates — not internal-only metrics.
  • Page on one stack layer. If user latency is fine, don't page on a slow sub-component.
  • Every page must be actionable. Nothing to do → delete the alert.
  • Match for: to the rate window. rate(x[5m]) with for: 30s is contradictory.

Full alerting best practices, template variables, anti-patterns: [${CLAUDE_SKILL_DIR}/references/recording-alerting-rules.md].

Application

Writing PromQL queries:

  • Apply conventions silently — don't narrate each rule.
  • Wrap counters in rate() or increase() before any further operation.
  • Aggregate before joining; filter at the innermost selector.
  • Prefer without over by when removing a noise dimension.
  • For histograms, prefer native when available; otherwise preserve le in aggregations.

Writing recording rules:

  • Follow level:metric:operations naming.
  • Build a hierarchy: per-instance rule first, then aggregate up. Strip _total after rate().
  • Aggregate ratios by computing numerator and denominator separately.
  • No subqueries inside recording rules — split into two rules.

Writing alerting rules:

  • CamelCase alert names; severity label for routing.
  • Alert on symptoms at the top of the stack.
  • Set for: to filter blips, sized to match the underlying rate window.
  • Use templating for context-rich notifications ({{ $labels.X }}, {{ $value }}).

Reviewing PromQL:

  • Cite the violation and show the fix inline.
  • Check first: rate() on a counter? Aggregation inside rate()? le preserved?

Integration

backend/prometheus governs instrumentation (metric types, naming, labels, exporters, cardinality). This skill governs query writing. When both apply:

  • "What metric should I emit" — backend/prometheus
  • "How do I query this" — this skill
  • Naming a recording rule output — use level:metric:operations here; that becomes the metric name backend/prometheus rules then apply to

grafana/metricsql (forthcoming) covers VictoriaMetrics' PromQL superset. MetricsQL extends PromQL with extra functions and looser type rules — write standard PromQL when targeting either Prometheus or VictoriaMetrics; reach for MetricsQL-only features only when the deployment is VictoriaMetrics and the standard expression is awkward.

Files (cc-foundry)
  • .dev
    • reference-inventory.json 2.7 KB
      Loading…
  • references
    • functions.md 15.2 KB
      Loading…
    • native-histograms.md 21 KB
      Loading…
    • operators.md 10.8 KB
      Loading…
    • optimization.md 11.4 KB
      Loading…
    • recording-alerting-rules.md 14.2 KB
      Loading…
    • selectors-and-types.md 8.8 KB
      Loading…
    • subqueries.md 6.5 KB
      Loading…
  • SKILL.md 20.6 KB
    ---
    name: promql
    description: >-
      PromQL query writing: selectors, operators, vector matching, functions, aggregations, histograms, subqueries,
      recording rules, alerting rules, and query optimization. Invoke whenever task involves any interaction with PromQL —
      writing, debugging, optimizing, reviewing, or explaining Prometheus queries for dashboards, alerts, recording rules,
      or ad-hoc exploration.
    ---
    
    # PromQL
    
    PromQL is a nested functional language for selecting and aggregating Prometheus time series. Every query is an
    expression tree: selectors at the leaves, functions and operators at the nodes, a single root value (scalar, instant
    vector, range vector, or string) at the top. Type discipline makes queries correct; cardinality awareness makes them
    fast.
    
    Authoritative reference for **writing PromQL queries** — dashboards, alerts, recording rules, exploration. The companion
    `backend/prometheus` skill covers **instrumentation** (metric types, naming, labels, exporters). When both apply, use
    `backend/prometheus` for emitting metrics and this skill for querying them.
    
    ## References
    
    - **Selectors and data types** — [`${CLAUDE_SKILL_DIR}/references/selectors-and-types.md`] Expression types, instant and
      range vector selectors, label matchers (`=`, `!=`, `=~`, `!~`), time durations, offset and `@` modifiers, staleness,
      instant vs range queries, string and float literals
    - **Operators** — [`${CLAUDE_SKILL_DIR}/references/operators.md`] Arithmetic, comparison, logical/set, vector matching
      (`on`, `ignoring`, `group_left`, `group_right`), aggregation operators, `by`/`without`, fill modifiers, operator
      precedence
    - **Functions** — [`${CLAUDE_SKILL_DIR}/references/functions.md`] Full function catalog: counter family, gauge
      functions, classic and native histograms, `*_over_time`, existence checks, label manipulation, math, time, type
      conversion, sorting, trigonometric
    - **Native histograms** — [`${CLAUDE_SKILL_DIR}/references/native-histograms.md`] Native vs classic detection,
      `histogram_quantile` across both forms, native-only functions (`histogram_avg`, `histogram_count`, `histogram_sum`,
      `histogram_fraction`, `histogram_stddev`, `histogram_stdvar`), aggregation without `le`, NHCB semantics, trim
      operators, migration patterns, function compatibility matrix, gotchas, annotations
    - **Subqueries** — [`${CLAUDE_SKILL_DIR}/references/subqueries.md`] Syntax, resolution and alignment, nested subqueries,
      when to use subqueries vs recording rules, pitfalls
    - **Optimization and pitfalls** — [`${CLAUDE_SKILL_DIR}/references/optimization.md`] Cardinality awareness, common
      pitfalls (rate-of-gauge, aggregate-then-rate, averaging ratios, averaging summary quantiles), counter resets,
      staleness, rate window sizing, native histograms, diagnostics
    - **Recording and alerting rules** — [`${CLAUDE_SKILL_DIR}/references/recording-alerting-rules.md`] Rule file structure,
      `level:metric:operations` naming, ratio aggregation patterns, alert syntax, `for`/`keep_firing_for`, templating,
      alerting best practices, anti-patterns
    
    Read the relevant reference before proceeding.
    
    ---
    
    ## Data Types
    
    Every expression has one type. Functions and operators require specific input types — type mismatch is the most common
    cause of "query parses but returns nothing".
    
    - **instant vector** — set of series, one sample per series at one timestamp. Default result of metric selectors,
      aggregations, most functions.
    - **range vector** — set of series, multiple samples per series across a time window. Produced **only** by range vector
      selectors (`metric[5m]`) and subqueries. Cannot be graphed directly — must pass through a function returning an
      instant vector.
    - **scalar** — single numeric value, no labels. Used as numeric arguments and in arithmetic.
    - **string** — function arguments only; cannot be a query result.
    
    **Function type rules:**
    
    - `rate`, `irate`, `increase`, `delta`, `idelta`, `deriv`, `predict_linear`, `*_over_time` — range vector → instant
      vector
    - `histogram_quantile`, `histogram_fraction`, `histogram_*` — instant vector (histogram or bucket series) → instant
      vector
    - Aggregation operators (`sum`, `avg`, `max`, `topk`, ...) — instant vector → instant vector
    - `scalar()` — instant vector → scalar; `vector()` — scalar → instant vector
    
    ## Selectors
    
    ### Instant Vector Selectors
    
    ```promql
    http_requests_total                                # all series with this metric name
    http_requests_total{job="api"}                     # filtered by label
    http_requests_total{job="api", method="GET"}       # multiple matchers — AND
    http_requests_total{status=~"5.."}                 # regex match (RE2, fully anchored)
    http_requests_total{method!="OPTIONS"}             # negative match
    {__name__=~"http_.*"}                              # regex on metric name via __name__
    ```
    
    **Label matchers:** `=` exact, `!=` not equal, `=~` regex match, `!~` negative regex.
    
    **Regex is fully anchored** — `=~"foo"` matches `^foo$`, not arbitrary substrings. Use `=~".*foo.*"` for substrings.
    
    **Required selectors:** at least one matcher must not match the empty string. `{job=~".*"}` is illegal; `{job=~".+"}` is
    valid.
    
    ### Range Vector Selectors
    
    Append `[duration]` to a selector:
    
    ```promql
    http_requests_total[5m]                            # last 5 minutes of samples
    http_requests_total{job="api"}[1h]
    ```
    
    Duration units: `ms`, `s`, `m`, `h`, `d`, `w`, `y`. Combine longest-first: `1h30m`, `12h34m56s`. Floats with units are
    not allowed (`1.5h` is invalid — use `1h30m`).
    
    ### Modifiers
    
    - **`offset <duration>`** — time-shift into the past (or future with negative). Must immediately follow the selector.
    
      ```promql
      http_requests_total offset 5m
      rate(http_requests_total[5m] offset 1w)            # rate one week ago
      ```
    
    - **`@ <unix-timestamp>`** — pin evaluation to an absolute time. `start()` and `end()` resolve to the range query's
      bounds.
    
      ```promql
      http_requests_total @ 1609746000
      rate(http_requests_total[5m] @ end())
      ```
    
    **Staleness:** instant vector selectors return the most recent sample within the lookback window (default 5 minutes).
    Series with no sample in the window disappear from the result.
    
    Full selector reference, lookback delta tuning, instant-vs-range-query semantics:
    [`${CLAUDE_SKILL_DIR}/references/selectors-and-types.md`].
    
    ## Operators
    
    ### Arithmetic and Comparison
    
    `+`, `-`, `*`, `/`, `%`, `^` — IEEE 754 arithmetic. The metric name is dropped from any vector arithmetic result.
    
    `==`, `!=`, `>`, `<`, `>=`, `<=` — by default **filter**: drop series where the comparison is false. Add `bool` after
    the operator to return `0` or `1` instead.
    
    ```promql
    http_requests_total > 100                          # only series whose latest value exceeds 100
    http_requests_total > bool 100                     # all series, value is 1 if > 100 else 0
    ```
    
    ### Logical/Set Operators
    
    Operate on label-set membership, ignoring sample values:
    
    - `vector1 and vector2` — intersection
    - `vector1 or vector2` — union
    - `vector1 unless vector2` — complement
    
    ```promql
    up{job="prom"} or up{job="node"}                       # union
    metric and on(device) (other_metric == 0)              # constrained intersection
    ```
    
    ### Vector Matching
    
    The biggest source of "no results returned". Default rule: **two elements match if they have exactly the same label
    set** (after the metric name drops). When operands have different label sets, match keywords are required:
    
    - **`on(labels)`** — match only on these labels; ignore all others
    - **`ignoring(labels)`** — match on all labels except these
    
    **Group modifiers for many-to-one** (arithmetic, comparison, trigonometric only — not set operators):
    
    - **`group_left(extra_labels)`** — many on the left, one on the right; copy extra_labels from right to result
    - **`group_right(extra_labels)`** — symmetric
    
    **Canonical patterns:**
    
    ```promql
    # Error ratio: errors carry an extra `code` label; ignore it to match
    errors:rate5m{code="500"} / ignoring(code) requests:rate5m
    
    # Many-to-one: same denominator for every code
    errors:rate5m / ignoring(code) group_left requests:rate5m
    
    # Join info-style metric: bring `version` label into result
    node_filesystem_avail_bytes * on(instance, job) group_left(version) node_exporter_build_info
    ```
    
    ### Aggregation
    
    Instant vector → instant vector with fewer elements.
    
    - **`sum`**, **`avg`**, **`min`**, **`max`** — reduce across dimensions
    - **`count`** — number of series; **`group`** — value 1 per group
    - **`topk(k, v)`** / **`bottomk(k, v)`** — k largest/smallest; preserves labels
    - **`stddev`**, **`stdvar`** — population statistics
    - **`quantile(φ, v)`** — quantile across series
    - **`count_values(label, v)`** — distribution of values
    
    **`by` vs `without`:**
    
    - `by (labels)` keeps **only** the listed labels
    - `without (labels)` keeps **everything except** the listed labels
    
    **Prefer `without`** for normal aggregations — preserves `job`, `instance`, and friends. Use `by` for a known output
    dimension set (typically recording rules).
    
    ```promql
    sum without (instance) (rate(http_requests_total[5m]))    # collapses instance, keeps job/method/status
    sum by (job) (rate(http_requests_total[5m]))              # collapses everything except job
    ```
    
    ### Operator Precedence (highest to lowest)
    
    1. `^`
    2. `*`, `/`, `%`, `atan2`
    3. `+`, `-`
    4. `==`, `!=`, `<=`, `<`, `>=`, `>`
    5. `and`, `unless`
    6. `or`
    
    Left-associative except `^` (right-associative). Use parentheses when mixing comparison and logical operators.
    
    Full operator catalog, fill modifiers, detailed vector matching examples:
    [`${CLAUDE_SKILL_DIR}/references/operators.md`].
    
    ## Key Functions by Category
    
    ### Counters — `rate`, `irate`, `increase`
    
    The three most important PromQL functions. All take a range vector, return an instant vector, handle counter resets
    automatically.
    
    - **`rate(v[5m])`** — per-second average rate over the range, extrapolated. Use for alerts and slow counters.
    - **`irate(v[1m])`** — instant rate from the last two samples. Graphing volatile counters only; never alerts (flaps).
    - **`increase(v[1h])`** — total increase over the range; syntactic sugar for `rate(v) * range_seconds`. Use for
      human-readable totals.
    
    **Composition rule — `rate()` first, then aggregate:**
    
    ```promql
    sum(rate(http_requests_total[5m]))             # CORRECT
    rate(sum(http_requests_total)[5m])             # WRONG — counter resets disappear into the sum
    ```
    
    Applies to all counter functions (`rate`, `irate`, `increase`), gauge functions (`delta`, `deriv`, `predict_linear`),
    and `*_over_time` functions.
    
    **Window sizing:** range must contain ≥2 samples. Rule of thumb: 4× the scrape interval, minimum 1m. With 15s scrapes
    the standard default is `[5m]`.
    
    ### Gauges — `delta`, `idelta`, `deriv`, `predict_linear`
    
    For gauges only. Never apply `rate()` to a gauge.
    
    ```promql
    delta(cpu_temp_celsius[2h])                                  # absolute change over 2h
    deriv(node_memory_MemAvailable_bytes[5m])                    # smoothed per-second change
    predict_linear(node_filesystem_avail_bytes[1h], 4*3600) < 0  # will fs fill in 4h?
    ```
    
    ### Histograms
    
    **Classic histograms** expose `_bucket{le="..."}`, `_sum`, `_count` series:
    
    ```promql
    histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))
    
    # Aggregating across instances — `le` MUST be preserved
    histogram_quantile(0.95, sum by (job, le) (rate(http_request_duration_seconds_bucket[5m])))
    histogram_quantile(0.95, sum without (instance) (rate(http_request_duration_seconds_bucket[5m])))
    
    # Average from sum/count
    rate(http_request_duration_seconds_sum[5m]) / rate(http_request_duration_seconds_count[5m])
    ```
    
    **Native histograms** are single composite series — quantile aggregation does not require `le`:
    
    ```promql
    histogram_quantile(0.9, sum by (job) (rate(http_request_duration_seconds[5m])))
    histogram_fraction(0, 0.2, rate(http_request_duration_seconds[5m]))     # fraction ≤ 200ms
    histogram_avg(rate(http_request_duration_seconds[5m]))
    histogram_count(rate(http_request_duration_seconds[5m]))                # rate of observations
    histogram_sum(rate(http_request_duration_seconds[5m]))
    ```
    
    If native histograms are available, prefer them — fewer series, easier aggregation, lower error.
    
    Full native histogram semantics — detection, function compatibility matrix, NHCB behavior, trim operators, migration,
    gotchas: [`${CLAUDE_SKILL_DIR}/references/native-histograms.md`].
    
    ### Aggregation Over Time — `*_over_time`
    
    Range vector → one summary value per series.
    
    - **`avg_over_time(range-vector)`** — average across the range
    - **`min_over_time`** / **`max_over_time`** — extremes
    - **`sum_over_time`** / **`count_over_time`** — totals
    - **`last_over_time`** / **`present_over_time`** — most recent / existence
    - **`quantile_over_time(φ, range-vector)`** — quantile across time
    - **`stddev_over_time`** / **`stdvar_over_time`**
    
    ```promql
    avg_over_time(node_memory_MemAvailable_bytes[1h])
    max_over_time(rate(http_requests_total[5m])[1h:])         # subquery needed — rate is instant vector
    ```
    
    ### Existence and Counter Resets
    
    ```promql
    absent(up{job="critical"})                         # 1 if no series, empty if any
    absent_over_time(up{job="critical"}[5m])           # missing for the entire window
    resets(http_requests_total[1h])                    # how many counter resets in the range
    changes(up[1h])                                    # how many times the value changed
    ```
    
    ### Label Manipulation
    
    ```promql
    label_replace(up, "host", "$1", "instance", "(.+):(\\d+)")        # extract host from instance="host:port"
    label_join(rate(x[5m]), "endpoint", " ", "method", "path")        # concatenate labels
    info(rate(http_server_duration_count[5m]))                        # auto-join with target_info (experimental)
    ```
    
    Full function catalog: [`${CLAUDE_SKILL_DIR}/references/functions.md`].
    
    ## Subqueries
    
    Embed a range query inside an instant query. Syntax:
    
    ```promql
    <instant-vector-expression>[<range>:[<resolution>]]
    ```
    
    Use when a range-vector function needs the output of another function or aggregation:
    
    ```promql
    max_over_time(rate(http_requests_total[5m])[1h:])             # peak 5m-rate over the last hour
    quantile_over_time(0.99, rate(errors[1m])[1d:])               # 99th percentile of per-minute error rates over a day
    deriv(rate(http_requests_total[5m])[30m:])                    # is the rate accelerating?
    ```
    
    **Performance:** subqueries re-run the inner expression at every resolution step. A `[1d:1m]` subquery runs the inner
    1,440 times. For expensive inner expressions or repeated use, convert to a recording rule.
    
    **Avoid subqueries inside recording rules.** Split into two recording rules.
    
    Full syntax, alignment rules, subquery vs recording rule guidance: [`${CLAUDE_SKILL_DIR}/references/subqueries.md`].
    
    ## Common Query Idioms
    
    ```promql
    # Request rate by job
    sum by (job) (rate(http_requests_total[5m]))
    
    # Error ratio — aggregate numerator and denominator separately
      sum(rate(http_requests_total{status=~"5.."}[5m]))
    /
      sum(rate(http_requests_total[5m]))
    
    # P99 latency by handler (classic histogram)
    histogram_quantile(0.99, sum by (handler, le) (rate(http_request_duration_seconds_bucket[5m])))
    
    # Top 5 CPU consumers
    topk(5, sum by (app) (rate(process_cpu_seconds_total[5m])))
    
    # Disk-full prediction
    predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 24*3600) < 0
    
    # Alert on missing scrape target
    absent(up{job="critical-service"})
    
    # Week-over-week request rate comparison
      rate(http_requests_total[5m])
    -
      rate(http_requests_total[5m] offset 1w)
    
    # Batch job staleness
    time() - my_batch_last_success_timestamp_seconds > 2 * 3600
    ```
    
    ## Pitfalls — Quick Reference
    
    Positive rules when writing or reviewing queries:
    
    - **Apply `rate()` to counters only.** Use `delta()`, `idelta()`, or `deriv()` for gauges. `rate()` on a gauge silently
      produces nonsense.
    - **`rate()` first, then aggregate.** `sum(rate(x[5m]))`, never `rate(sum(x)[5m])`. Counter resets are per-series.
    - **Aggregate ratios correctly.** Sum numerator and denominator separately, then divide. Never `avg(ratio)` or
      `avg(avg)`.
    - **Don't average summary quantiles.** `avg(metric{quantile="0.95"})` is statistically invalid. Use histograms for
      cross-instance aggregation.
    - **Preserve `le` when aggregating classic histogram buckets.** `sum by (job, le)` or `sum without (instance)`.
    - **Range ≥ 4× scrape interval, minimum 1m.** Avoid `rate(x[1m])` with 30s+ scrape intervals.
    - **`rate()` (not `irate()`) for alerts.** `irate()` is for graphing volatile counters.
    - **`without` over `by`** for most aggregations — preserves useful labels like `job` automatically.
    - **`offset` must immediately follow the selector.** `sum(x offset 5m)` is correct; `sum(x) offset 5m` is invalid.
    - **Filter early.** Push label matchers into the innermost selector.
    
    Full pitfall catalog with diagnostics: [`${CLAUDE_SKILL_DIR}/references/optimization.md`].
    
    ## Recording and Alerting Rules
    
    ### Recording Rule Naming — `level:metric:operations`
    
    ```text
    instance:http_requests:rate5m                   # per-instance rate (no aggregation)
    job:http_requests:rate5m                        # rolled up to per-job
    job:http_errors_per_requests:ratio_rate5m       # ratio, per job
    job:request_latency_seconds:mean5m              # average latency from Summary
    ```
    
    - **`level`** — labels remaining in the output (`instance`, `job`, `cluster`, `instance_path`, etc.)
    - **`metric`** — metric name; strip `_total` after `rate()`
    - **`operations`** — operations applied, newest first. Omit `_sum` when other ops present; `mean` replaces `rate` for
      sum/count averages; `ratio` for divisions named with `_per_`
    
    Recording rule examples and ratio aggregation patterns: [`${CLAUDE_SKILL_DIR}/references/recording-alerting-rules.md`].
    
    ### Alerting Rule Essentials
    
    ```yaml
    - alert: HighErrorRatio                   # CamelCase community convention
      expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
        /
          sum by (job) (rate(http_requests_total[5m]))
        > 0.05
      for: 10m                                # must be true for the entire window before firing
      keep_firing_for: 5m                     # keep firing this long after expression becomes false
      labels:
        severity: page
      annotations:
        summary: "High error rate on {{ $labels.job }}"
        description: "Error ratio is {{ $value | humanizePercentage }} for {{ $labels.job }}"
        runbook_url: "https://wiki.example.com/runbooks/{{ $labels.job }}"
    ```
    
    **Alerting principles:**
    
    - **Alert on symptoms, not causes.** User-visible latency, error rates — not internal-only metrics.
    - **Page on one stack layer.** If user latency is fine, don't page on a slow sub-component.
    - **Every page must be actionable.** Nothing to do → delete the alert.
    - **Match `for:` to the rate window.** `rate(x[5m])` with `for: 30s` is contradictory.
    
    Full alerting best practices, template variables, anti-patterns:
    [`${CLAUDE_SKILL_DIR}/references/recording-alerting-rules.md`].
    
    ## Application
    
    **Writing PromQL queries:**
    
    - Apply conventions silently — don't narrate each rule.
    - Wrap counters in `rate()` or `increase()` before any further operation.
    - Aggregate before joining; filter at the innermost selector.
    - Prefer `without` over `by` when removing a noise dimension.
    - For histograms, prefer native when available; otherwise preserve `le` in aggregations.
    
    **Writing recording rules:**
    
    - Follow `level:metric:operations` naming.
    - Build a hierarchy: per-instance rule first, then aggregate up. Strip `_total` after `rate()`.
    - Aggregate ratios by computing numerator and denominator separately.
    - No subqueries inside recording rules — split into two rules.
    
    **Writing alerting rules:**
    
    - CamelCase alert names; `severity` label for routing.
    - Alert on symptoms at the top of the stack.
    - Set `for:` to filter blips, sized to match the underlying rate window.
    - Use templating for context-rich notifications (`{{ $labels.X }}`, `{{ $value }}`).
    
    **Reviewing PromQL:**
    
    - Cite the violation and show the fix inline.
    - Check first: `rate()` on a counter? Aggregation inside `rate()`? `le` preserved?
    
    ## Integration
    
    `backend/prometheus` governs instrumentation (metric types, naming, labels, exporters, cardinality). This skill governs
    query writing. When both apply:
    
    - "What metric should I emit" — `backend/prometheus`
    - "How do I query this" — this skill
    - Naming a recording rule output — use `level:metric:operations` here; that becomes the metric name `backend/prometheus`
      rules then apply to
    
    `grafana/metricsql` (forthcoming) covers VictoriaMetrics' PromQL superset. MetricsQL extends PromQL with extra functions
    and looser type rules — write standard PromQL when targeting either Prometheus or VictoriaMetrics; reach for
    MetricsQL-only features only when the deployment is VictoriaMetrics and the standard expression is awkward.
    

Comments (0)

Sign in to join the conversation.

No comments yet.

Reviews (0)

No reviews yet.