promql
PromQL query writing: selectors, operators, vector matching, functions, aggregations, histograms, subqueries, recording rules, alerting rules, and query optimization. Invoke whenever task involves any interaction with PromQL — writing, debugging, optimizing, reviewing, or explain
Install
npx skills add https://github.com/xobotyi/cc-foundry/tree/master/plugins/grafana/skills/promql
claude plugin marketplace add https://llmmart.ai/marketplace.json && claude plugin install xobotyi-cc-foundry@llmmart
git clone https://github.com/xobotyi/cc-foundry.git
The skills CLI installs just this skill, for any of its supported agents. Claude Code installs the whole xobotyi/cc-foundry collection as a plugin from our marketplace. Git is the plain clone.
Skill manifest
PromQL
PromQL is a nested functional language for selecting and aggregating Prometheus time series. Every query is an expression tree: selectors at the leaves, functions and operators at the nodes, a single root value (scalar, instant vector, range vector, or string) at the top. Type discipline makes queries correct; cardinality awareness makes them fast.
Authoritative reference for writing PromQL queries — dashboards, alerts, recording rules, exploration. The companion
backend/prometheus skill covers instrumentation (metric types, naming, labels, exporters). When both apply, use
backend/prometheus for emitting metrics and this skill for querying them.
References
- Selectors and data types — [
${CLAUDE_SKILL_DIR}/references/selectors-and-types.md] Expression types, instant and range vector selectors, label matchers (=,!=,=~,!~), time durations, offset and@modifiers, staleness, instant vs range queries, string and float literals - Operators — [
${CLAUDE_SKILL_DIR}/references/operators.md] Arithmetic, comparison, logical/set, vector matching (on,ignoring,group_left,group_right), aggregation operators,by/without, fill modifiers, operator precedence - Functions — [
${CLAUDE_SKILL_DIR}/references/functions.md] Full function catalog: counter family, gauge functions, classic and native histograms,*_over_time, existence checks, label manipulation, math, time, type conversion, sorting, trigonometric - Native histograms — [
${CLAUDE_SKILL_DIR}/references/native-histograms.md] Native vs classic detection,histogram_quantileacross both forms, native-only functions (histogram_avg,histogram_count,histogram_sum,histogram_fraction,histogram_stddev,histogram_stdvar), aggregation withoutle, NHCB semantics, trim operators, migration patterns, function compatibility matrix, gotchas, annotations - Subqueries — [
${CLAUDE_SKILL_DIR}/references/subqueries.md] Syntax, resolution and alignment, nested subqueries, when to use subqueries vs recording rules, pitfalls - Optimization and pitfalls — [
${CLAUDE_SKILL_DIR}/references/optimization.md] Cardinality awareness, common pitfalls (rate-of-gauge, aggregate-then-rate, averaging ratios, averaging summary quantiles), counter resets, staleness, rate window sizing, native histograms, diagnostics - Recording and alerting rules — [
${CLAUDE_SKILL_DIR}/references/recording-alerting-rules.md] Rule file structure,level:metric:operationsnaming, ratio aggregation patterns, alert syntax,for/keep_firing_for, templating, alerting best practices, anti-patterns
Read the relevant reference before proceeding.
Data Types
Every expression has one type. Functions and operators require specific input types — type mismatch is the most common cause of "query parses but returns nothing".
- instant vector — set of series, one sample per series at one timestamp. Default result of metric selectors, aggregations, most functions.
- range vector — set of series, multiple samples per series across a time window. Produced only by range vector
selectors (
metric[5m]) and subqueries. Cannot be graphed directly — must pass through a function returning an instant vector. - scalar — single numeric value, no labels. Used as numeric arguments and in arithmetic.
- string — function arguments only; cannot be a query result.
Function type rules:
rate,irate,increase,delta,idelta,deriv,predict_linear,*_over_time— range vector → instant vectorhistogram_quantile,histogram_fraction,histogram_*— instant vector (histogram or bucket series) → instant vector- Aggregation operators (
sum,avg,max,topk, ...) — instant vector → instant vector scalar()— instant vector → scalar;vector()— scalar → instant vector
Selectors
Instant Vector Selectors
http_requests_total # all series with this metric name
http_requests_total{job="api"} # filtered by label
http_requests_total{job="api", method="GET"} # multiple matchers — AND
http_requests_total{status=~"5.."} # regex match (RE2, fully anchored)
http_requests_total{method!="OPTIONS"} # negative match
{__name__=~"http_.*"} # regex on metric name via __name__
Label matchers: = exact, != not equal, =~ regex match, !~ negative regex.
Regex is fully anchored — =~"foo" matches ^foo$, not arbitrary substrings. Use =~".*foo.*" for substrings.
Required selectors: at least one matcher must not match the empty string. {job=~".*"} is illegal; {job=~".+"} is
valid.
Range Vector Selectors
Append [duration] to a selector:
http_requests_total[5m] # last 5 minutes of samples
http_requests_total{job="api"}[1h]
Duration units: ms, s, m, h, d, w, y. Combine longest-first: 1h30m, 12h34m56s. Floats with units are
not allowed (1.5h is invalid — use 1h30m).
Modifiers
offset <duration>— time-shift into the past (or future with negative). Must immediately follow the selector.http_requests_total offset 5m rate(http_requests_total[5m] offset 1w) # rate one week ago@ <unix-timestamp>— pin evaluation to an absolute time.start()andend()resolve to the range query's bounds.http_requests_total @ 1609746000 rate(http_requests_total[5m] @ end())
Staleness: instant vector selectors return the most recent sample within the lookback window (default 5 minutes). Series with no sample in the window disappear from the result.
Full selector reference, lookback delta tuning, instant-vs-range-query semantics:
[${CLAUDE_SKILL_DIR}/references/selectors-and-types.md].
Operators
Arithmetic and Comparison
+, -, *, /, %, ^ — IEEE 754 arithmetic. The metric name is dropped from any vector arithmetic result.
==, !=, >, <, >=, <= — by default filter: drop series where the comparison is false. Add bool after
the operator to return 0 or 1 instead.
http_requests_total > 100 # only series whose latest value exceeds 100
http_requests_total > bool 100 # all series, value is 1 if > 100 else 0
Logical/Set Operators
Operate on label-set membership, ignoring sample values:
vector1 and vector2— intersectionvector1 or vector2— unionvector1 unless vector2— complement
up{job="prom"} or up{job="node"} # union
metric and on(device) (other_metric == 0) # constrained intersection
Vector Matching
The biggest source of "no results returned". Default rule: two elements match if they have exactly the same label set (after the metric name drops). When operands have different label sets, match keywords are required:
on(labels)— match only on these labels; ignore all othersignoring(labels)— match on all labels except these
Group modifiers for many-to-one (arithmetic, comparison, trigonometric only — not set operators):
group_left(extra_labels)— many on the left, one on the right; copy extra_labels from right to resultgroup_right(extra_labels)— symmetric
Canonical patterns:
# Error ratio: errors carry an extra `code` label; ignore it to match
errors:rate5m{code="500"} / ignoring(code) requests:rate5m
# Many-to-one: same denominator for every code
errors:rate5m / ignoring(code) group_left requests:rate5m
# Join info-style metric: bring `version` label into result
node_filesystem_avail_bytes * on(instance, job) group_left(version) node_exporter_build_info
Aggregation
Instant vector → instant vector with fewer elements.
sum,avg,min,max— reduce across dimensionscount— number of series;group— value 1 per grouptopk(k, v)/bottomk(k, v)— k largest/smallest; preserves labelsstddev,stdvar— population statisticsquantile(φ, v)— quantile across seriescount_values(label, v)— distribution of values
by vs without:
by (labels)keeps only the listed labelswithout (labels)keeps everything except the listed labels
Prefer without for normal aggregations — preserves job, instance, and friends. Use by for a known output
dimension set (typically recording rules).
sum without (instance) (rate(http_requests_total[5m])) # collapses instance, keeps job/method/status
sum by (job) (rate(http_requests_total[5m])) # collapses everything except job
Operator Precedence (highest to lowest)
^*,/,%,atan2+,-==,!=,<=,<,>=,>and,unlessor
Left-associative except ^ (right-associative). Use parentheses when mixing comparison and logical operators.
Full operator catalog, fill modifiers, detailed vector matching examples:
[${CLAUDE_SKILL_DIR}/references/operators.md].
Key Functions by Category
Counters — rate, irate, increase
The three most important PromQL functions. All take a range vector, return an instant vector, handle counter resets automatically.
rate(v[5m])— per-second average rate over the range, extrapolated. Use for alerts and slow counters.irate(v[1m])— instant rate from the last two samples. Graphing volatile counters only; never alerts (flaps).increase(v[1h])— total increase over the range; syntactic sugar forrate(v) * range_seconds. Use for human-readable totals.
Composition rule — rate() first, then aggregate:
sum(rate(http_requests_total[5m])) # CORRECT
rate(sum(http_requests_total)[5m]) # WRONG — counter resets disappear into the sum
Applies to all counter functions (rate, irate, increase), gauge functions (delta, deriv, predict_linear),
and *_over_time functions.
Window sizing: range must contain ≥2 samples. Rule of thumb: 4× the scrape interval, minimum 1m. With 15s scrapes
the standard default is [5m].
Gauges — delta, idelta, deriv, predict_linear
For gauges only. Never apply rate() to a gauge.
delta(cpu_temp_celsius[2h]) # absolute change over 2h
deriv(node_memory_MemAvailable_bytes[5m]) # smoothed per-second change
predict_linear(node_filesystem_avail_bytes[1h], 4*3600) < 0 # will fs fill in 4h?
Histograms
Classic histograms expose _bucket{le="..."}, _sum, _count series:
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))
# Aggregating across instances — `le` MUST be preserved
histogram_quantile(0.95, sum by (job, le) (rate(http_request_duration_seconds_bucket[5m])))
histogram_quantile(0.95, sum without (instance) (rate(http_request_duration_seconds_bucket[5m])))
# Average from sum/count
rate(http_request_duration_seconds_sum[5m]) / rate(http_request_duration_seconds_count[5m])
Native histograms are single composite series — quantile aggregation does not require le:
histogram_quantile(0.9, sum by (job) (rate(http_request_duration_seconds[5m])))
histogram_fraction(0, 0.2, rate(http_request_duration_seconds[5m])) # fraction ≤ 200ms
histogram_avg(rate(http_request_duration_seconds[5m]))
histogram_count(rate(http_request_duration_seconds[5m])) # rate of observations
histogram_sum(rate(http_request_duration_seconds[5m]))
If native histograms are available, prefer them — fewer series, easier aggregation, lower error.
Full native histogram semantics — detection, function compatibility matrix, NHCB behavior, trim operators, migration,
gotchas: [${CLAUDE_SKILL_DIR}/references/native-histograms.md].
Aggregation Over Time — *_over_time
Range vector → one summary value per series.
avg_over_time(range-vector)— average across the rangemin_over_time/max_over_time— extremessum_over_time/count_over_time— totalslast_over_time/present_over_time— most recent / existencequantile_over_time(φ, range-vector)— quantile across timestddev_over_time/stdvar_over_time
avg_over_time(node_memory_MemAvailable_bytes[1h])
max_over_time(rate(http_requests_total[5m])[1h:]) # subquery needed — rate is instant vector
Existence and Counter Resets
absent(up{job="critical"}) # 1 if no series, empty if any
absent_over_time(up{job="critical"}[5m]) # missing for the entire window
resets(http_requests_total[1h]) # how many counter resets in the range
changes(up[1h]) # how many times the value changed
Label Manipulation
label_replace(up, "host", "$1", "instance", "(.+):(\\d+)") # extract host from instance="host:port"
label_join(rate(x[5m]), "endpoint", " ", "method", "path") # concatenate labels
info(rate(http_server_duration_count[5m])) # auto-join with target_info (experimental)
Full function catalog: [${CLAUDE_SKILL_DIR}/references/functions.md].
Subqueries
Embed a range query inside an instant query. Syntax:
<instant-vector-expression>[<range>:[<resolution>]]
Use when a range-vector function needs the output of another function or aggregation:
max_over_time(rate(http_requests_total[5m])[1h:]) # peak 5m-rate over the last hour
quantile_over_time(0.99, rate(errors[1m])[1d:]) # 99th percentile of per-minute error rates over a day
deriv(rate(http_requests_total[5m])[30m:]) # is the rate accelerating?
Performance: subqueries re-run the inner expression at every resolution step. A [1d:1m] subquery runs the inner
1,440 times. For expensive inner expressions or repeated use, convert to a recording rule.
Avoid subqueries inside recording rules. Split into two recording rules.
Full syntax, alignment rules, subquery vs recording rule guidance: [${CLAUDE_SKILL_DIR}/references/subqueries.md].
Common Query Idioms
# Request rate by job
sum by (job) (rate(http_requests_total[5m]))
# Error ratio — aggregate numerator and denominator separately
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
# P99 latency by handler (classic histogram)
histogram_quantile(0.99, sum by (handler, le) (rate(http_request_duration_seconds_bucket[5m])))
# Top 5 CPU consumers
topk(5, sum by (app) (rate(process_cpu_seconds_total[5m])))
# Disk-full prediction
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 24*3600) < 0
# Alert on missing scrape target
absent(up{job="critical-service"})
# Week-over-week request rate comparison
rate(http_requests_total[5m])
-
rate(http_requests_total[5m] offset 1w)
# Batch job staleness
time() - my_batch_last_success_timestamp_seconds > 2 * 3600
Pitfalls — Quick Reference
Positive rules when writing or reviewing queries:
- Apply
rate()to counters only. Usedelta(),idelta(), orderiv()for gauges.rate()on a gauge silently produces nonsense. rate()first, then aggregate.sum(rate(x[5m])), neverrate(sum(x)[5m]). Counter resets are per-series.- Aggregate ratios correctly. Sum numerator and denominator separately, then divide. Never
avg(ratio)oravg(avg). - Don't average summary quantiles.
avg(metric{quantile="0.95"})is statistically invalid. Use histograms for cross-instance aggregation. - Preserve
lewhen aggregating classic histogram buckets.sum by (job, le)orsum without (instance). - Range ≥ 4× scrape interval, minimum 1m. Avoid
rate(x[1m])with 30s+ scrape intervals. rate()(notirate()) for alerts.irate()is for graphing volatile counters.withoutoverbyfor most aggregations — preserves useful labels likejobautomatically.offsetmust immediately follow the selector.sum(x offset 5m)is correct;sum(x) offset 5mis invalid.- Filter early. Push label matchers into the innermost selector.
Full pitfall catalog with diagnostics: [${CLAUDE_SKILL_DIR}/references/optimization.md].
Recording and Alerting Rules
Recording Rule Naming — level:metric:operations
instance:http_requests:rate5m # per-instance rate (no aggregation)
job:http_requests:rate5m # rolled up to per-job
job:http_errors_per_requests:ratio_rate5m # ratio, per job
job:request_latency_seconds:mean5m # average latency from Summary
level— labels remaining in the output (instance,job,cluster,instance_path, etc.)metric— metric name; strip_totalafterrate()operations— operations applied, newest first. Omit_sumwhen other ops present;meanreplacesratefor sum/count averages;ratiofor divisions named with_per_
Recording rule examples and ratio aggregation patterns: [${CLAUDE_SKILL_DIR}/references/recording-alerting-rules.md].
Alerting Rule Essentials
- alert: HighErrorRatio # CamelCase community convention
expr: |
sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
/
sum by (job) (rate(http_requests_total[5m]))
> 0.05
for: 10m # must be true for the entire window before firing
keep_firing_for: 5m # keep firing this long after expression becomes false
labels:
severity: page
annotations:
summary: "High error rate on {{ $labels.job }}"
description: "Error ratio is {{ $value | humanizePercentage }} for {{ $labels.job }}"
runbook_url: "https://wiki.example.com/runbooks/{{ $labels.job }}"
Alerting principles:
- Alert on symptoms, not causes. User-visible latency, error rates — not internal-only metrics.
- Page on one stack layer. If user latency is fine, don't page on a slow sub-component.
- Every page must be actionable. Nothing to do → delete the alert.
- Match
for:to the rate window.rate(x[5m])withfor: 30sis contradictory.
Full alerting best practices, template variables, anti-patterns:
[${CLAUDE_SKILL_DIR}/references/recording-alerting-rules.md].
Application
Writing PromQL queries:
- Apply conventions silently — don't narrate each rule.
- Wrap counters in
rate()orincrease()before any further operation. - Aggregate before joining; filter at the innermost selector.
- Prefer
withoutoverbywhen removing a noise dimension. - For histograms, prefer native when available; otherwise preserve
lein aggregations.
Writing recording rules:
- Follow
level:metric:operationsnaming. - Build a hierarchy: per-instance rule first, then aggregate up. Strip
_totalafterrate(). - Aggregate ratios by computing numerator and denominator separately.
- No subqueries inside recording rules — split into two rules.
Writing alerting rules:
- CamelCase alert names;
severitylabel for routing. - Alert on symptoms at the top of the stack.
- Set
for:to filter blips, sized to match the underlying rate window. - Use templating for context-rich notifications (
{{ $labels.X }},{{ $value }}).
Reviewing PromQL:
- Cite the violation and show the fix inline.
- Check first:
rate()on a counter? Aggregation insiderate()?lepreserved?
Integration
backend/prometheus governs instrumentation (metric types, naming, labels, exporters, cardinality). This skill governs
query writing. When both apply:
- "What metric should I emit" —
backend/prometheus - "How do I query this" — this skill
- Naming a recording rule output — use
level:metric:operationshere; that becomes the metric namebackend/prometheusrules then apply to
grafana/metricsql (forthcoming) covers VictoriaMetrics' PromQL superset. MetricsQL extends PromQL with extra functions
and looser type rules — write standard PromQL when targeting either Prometheus or VictoriaMetrics; reach for
MetricsQL-only features only when the deployment is VictoriaMetrics and the standard expression is awkward.
Files (cc-foundry)
-
.dev
-
reference-inventory.json 2.7 KB
Loading…
-
-
references
-
functions.md 15.2 KB
Loading… -
native-histograms.md 21 KB
Loading… -
operators.md 10.8 KB
Loading… -
optimization.md 11.4 KB
Loading… -
recording-alerting-rules.md 14.2 KB
Loading… -
selectors-and-types.md 8.8 KB
Loading… -
subqueries.md 6.5 KB
Loading…
-
-
SKILL.md 20.6 KB
--- name: promql description: >- PromQL query writing: selectors, operators, vector matching, functions, aggregations, histograms, subqueries, recording rules, alerting rules, and query optimization. Invoke whenever task involves any interaction with PromQL — writing, debugging, optimizing, reviewing, or explaining Prometheus queries for dashboards, alerts, recording rules, or ad-hoc exploration. --- # PromQL PromQL is a nested functional language for selecting and aggregating Prometheus time series. Every query is an expression tree: selectors at the leaves, functions and operators at the nodes, a single root value (scalar, instant vector, range vector, or string) at the top. Type discipline makes queries correct; cardinality awareness makes them fast. Authoritative reference for **writing PromQL queries** — dashboards, alerts, recording rules, exploration. The companion `backend/prometheus` skill covers **instrumentation** (metric types, naming, labels, exporters). When both apply, use `backend/prometheus` for emitting metrics and this skill for querying them. ## References - **Selectors and data types** — [`${CLAUDE_SKILL_DIR}/references/selectors-and-types.md`] Expression types, instant and range vector selectors, label matchers (`=`, `!=`, `=~`, `!~`), time durations, offset and `@` modifiers, staleness, instant vs range queries, string and float literals - **Operators** — [`${CLAUDE_SKILL_DIR}/references/operators.md`] Arithmetic, comparison, logical/set, vector matching (`on`, `ignoring`, `group_left`, `group_right`), aggregation operators, `by`/`without`, fill modifiers, operator precedence - **Functions** — [`${CLAUDE_SKILL_DIR}/references/functions.md`] Full function catalog: counter family, gauge functions, classic and native histograms, `*_over_time`, existence checks, label manipulation, math, time, type conversion, sorting, trigonometric - **Native histograms** — [`${CLAUDE_SKILL_DIR}/references/native-histograms.md`] Native vs classic detection, `histogram_quantile` across both forms, native-only functions (`histogram_avg`, `histogram_count`, `histogram_sum`, `histogram_fraction`, `histogram_stddev`, `histogram_stdvar`), aggregation without `le`, NHCB semantics, trim operators, migration patterns, function compatibility matrix, gotchas, annotations - **Subqueries** — [`${CLAUDE_SKILL_DIR}/references/subqueries.md`] Syntax, resolution and alignment, nested subqueries, when to use subqueries vs recording rules, pitfalls - **Optimization and pitfalls** — [`${CLAUDE_SKILL_DIR}/references/optimization.md`] Cardinality awareness, common pitfalls (rate-of-gauge, aggregate-then-rate, averaging ratios, averaging summary quantiles), counter resets, staleness, rate window sizing, native histograms, diagnostics - **Recording and alerting rules** — [`${CLAUDE_SKILL_DIR}/references/recording-alerting-rules.md`] Rule file structure, `level:metric:operations` naming, ratio aggregation patterns, alert syntax, `for`/`keep_firing_for`, templating, alerting best practices, anti-patterns Read the relevant reference before proceeding. --- ## Data Types Every expression has one type. Functions and operators require specific input types — type mismatch is the most common cause of "query parses but returns nothing". - **instant vector** — set of series, one sample per series at one timestamp. Default result of metric selectors, aggregations, most functions. - **range vector** — set of series, multiple samples per series across a time window. Produced **only** by range vector selectors (`metric[5m]`) and subqueries. Cannot be graphed directly — must pass through a function returning an instant vector. - **scalar** — single numeric value, no labels. Used as numeric arguments and in arithmetic. - **string** — function arguments only; cannot be a query result. **Function type rules:** - `rate`, `irate`, `increase`, `delta`, `idelta`, `deriv`, `predict_linear`, `*_over_time` — range vector → instant vector - `histogram_quantile`, `histogram_fraction`, `histogram_*` — instant vector (histogram or bucket series) → instant vector - Aggregation operators (`sum`, `avg`, `max`, `topk`, ...) — instant vector → instant vector - `scalar()` — instant vector → scalar; `vector()` — scalar → instant vector ## Selectors ### Instant Vector Selectors ```promql http_requests_total # all series with this metric name http_requests_total{job="api"} # filtered by label http_requests_total{job="api", method="GET"} # multiple matchers — AND http_requests_total{status=~"5.."} # regex match (RE2, fully anchored) http_requests_total{method!="OPTIONS"} # negative match {__name__=~"http_.*"} # regex on metric name via __name__ ``` **Label matchers:** `=` exact, `!=` not equal, `=~` regex match, `!~` negative regex. **Regex is fully anchored** — `=~"foo"` matches `^foo$`, not arbitrary substrings. Use `=~".*foo.*"` for substrings. **Required selectors:** at least one matcher must not match the empty string. `{job=~".*"}` is illegal; `{job=~".+"}` is valid. ### Range Vector Selectors Append `[duration]` to a selector: ```promql http_requests_total[5m] # last 5 minutes of samples http_requests_total{job="api"}[1h] ``` Duration units: `ms`, `s`, `m`, `h`, `d`, `w`, `y`. Combine longest-first: `1h30m`, `12h34m56s`. Floats with units are not allowed (`1.5h` is invalid — use `1h30m`). ### Modifiers - **`offset <duration>`** — time-shift into the past (or future with negative). Must immediately follow the selector. ```promql http_requests_total offset 5m rate(http_requests_total[5m] offset 1w) # rate one week ago ``` - **`@ <unix-timestamp>`** — pin evaluation to an absolute time. `start()` and `end()` resolve to the range query's bounds. ```promql http_requests_total @ 1609746000 rate(http_requests_total[5m] @ end()) ``` **Staleness:** instant vector selectors return the most recent sample within the lookback window (default 5 minutes). Series with no sample in the window disappear from the result. Full selector reference, lookback delta tuning, instant-vs-range-query semantics: [`${CLAUDE_SKILL_DIR}/references/selectors-and-types.md`]. ## Operators ### Arithmetic and Comparison `+`, `-`, `*`, `/`, `%`, `^` — IEEE 754 arithmetic. The metric name is dropped from any vector arithmetic result. `==`, `!=`, `>`, `<`, `>=`, `<=` — by default **filter**: drop series where the comparison is false. Add `bool` after the operator to return `0` or `1` instead. ```promql http_requests_total > 100 # only series whose latest value exceeds 100 http_requests_total > bool 100 # all series, value is 1 if > 100 else 0 ``` ### Logical/Set Operators Operate on label-set membership, ignoring sample values: - `vector1 and vector2` — intersection - `vector1 or vector2` — union - `vector1 unless vector2` — complement ```promql up{job="prom"} or up{job="node"} # union metric and on(device) (other_metric == 0) # constrained intersection ``` ### Vector Matching The biggest source of "no results returned". Default rule: **two elements match if they have exactly the same label set** (after the metric name drops). When operands have different label sets, match keywords are required: - **`on(labels)`** — match only on these labels; ignore all others - **`ignoring(labels)`** — match on all labels except these **Group modifiers for many-to-one** (arithmetic, comparison, trigonometric only — not set operators): - **`group_left(extra_labels)`** — many on the left, one on the right; copy extra_labels from right to result - **`group_right(extra_labels)`** — symmetric **Canonical patterns:** ```promql # Error ratio: errors carry an extra `code` label; ignore it to match errors:rate5m{code="500"} / ignoring(code) requests:rate5m # Many-to-one: same denominator for every code errors:rate5m / ignoring(code) group_left requests:rate5m # Join info-style metric: bring `version` label into result node_filesystem_avail_bytes * on(instance, job) group_left(version) node_exporter_build_info ``` ### Aggregation Instant vector → instant vector with fewer elements. - **`sum`**, **`avg`**, **`min`**, **`max`** — reduce across dimensions - **`count`** — number of series; **`group`** — value 1 per group - **`topk(k, v)`** / **`bottomk(k, v)`** — k largest/smallest; preserves labels - **`stddev`**, **`stdvar`** — population statistics - **`quantile(φ, v)`** — quantile across series - **`count_values(label, v)`** — distribution of values **`by` vs `without`:** - `by (labels)` keeps **only** the listed labels - `without (labels)` keeps **everything except** the listed labels **Prefer `without`** for normal aggregations — preserves `job`, `instance`, and friends. Use `by` for a known output dimension set (typically recording rules). ```promql sum without (instance) (rate(http_requests_total[5m])) # collapses instance, keeps job/method/status sum by (job) (rate(http_requests_total[5m])) # collapses everything except job ``` ### Operator Precedence (highest to lowest) 1. `^` 2. `*`, `/`, `%`, `atan2` 3. `+`, `-` 4. `==`, `!=`, `<=`, `<`, `>=`, `>` 5. `and`, `unless` 6. `or` Left-associative except `^` (right-associative). Use parentheses when mixing comparison and logical operators. Full operator catalog, fill modifiers, detailed vector matching examples: [`${CLAUDE_SKILL_DIR}/references/operators.md`]. ## Key Functions by Category ### Counters — `rate`, `irate`, `increase` The three most important PromQL functions. All take a range vector, return an instant vector, handle counter resets automatically. - **`rate(v[5m])`** — per-second average rate over the range, extrapolated. Use for alerts and slow counters. - **`irate(v[1m])`** — instant rate from the last two samples. Graphing volatile counters only; never alerts (flaps). - **`increase(v[1h])`** — total increase over the range; syntactic sugar for `rate(v) * range_seconds`. Use for human-readable totals. **Composition rule — `rate()` first, then aggregate:** ```promql sum(rate(http_requests_total[5m])) # CORRECT rate(sum(http_requests_total)[5m]) # WRONG — counter resets disappear into the sum ``` Applies to all counter functions (`rate`, `irate`, `increase`), gauge functions (`delta`, `deriv`, `predict_linear`), and `*_over_time` functions. **Window sizing:** range must contain ≥2 samples. Rule of thumb: 4× the scrape interval, minimum 1m. With 15s scrapes the standard default is `[5m]`. ### Gauges — `delta`, `idelta`, `deriv`, `predict_linear` For gauges only. Never apply `rate()` to a gauge. ```promql delta(cpu_temp_celsius[2h]) # absolute change over 2h deriv(node_memory_MemAvailable_bytes[5m]) # smoothed per-second change predict_linear(node_filesystem_avail_bytes[1h], 4*3600) < 0 # will fs fill in 4h? ``` ### Histograms **Classic histograms** expose `_bucket{le="..."}`, `_sum`, `_count` series: ```promql histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) # Aggregating across instances — `le` MUST be preserved histogram_quantile(0.95, sum by (job, le) (rate(http_request_duration_seconds_bucket[5m]))) histogram_quantile(0.95, sum without (instance) (rate(http_request_duration_seconds_bucket[5m]))) # Average from sum/count rate(http_request_duration_seconds_sum[5m]) / rate(http_request_duration_seconds_count[5m]) ``` **Native histograms** are single composite series — quantile aggregation does not require `le`: ```promql histogram_quantile(0.9, sum by (job) (rate(http_request_duration_seconds[5m]))) histogram_fraction(0, 0.2, rate(http_request_duration_seconds[5m])) # fraction ≤ 200ms histogram_avg(rate(http_request_duration_seconds[5m])) histogram_count(rate(http_request_duration_seconds[5m])) # rate of observations histogram_sum(rate(http_request_duration_seconds[5m])) ``` If native histograms are available, prefer them — fewer series, easier aggregation, lower error. Full native histogram semantics — detection, function compatibility matrix, NHCB behavior, trim operators, migration, gotchas: [`${CLAUDE_SKILL_DIR}/references/native-histograms.md`]. ### Aggregation Over Time — `*_over_time` Range vector → one summary value per series. - **`avg_over_time(range-vector)`** — average across the range - **`min_over_time`** / **`max_over_time`** — extremes - **`sum_over_time`** / **`count_over_time`** — totals - **`last_over_time`** / **`present_over_time`** — most recent / existence - **`quantile_over_time(φ, range-vector)`** — quantile across time - **`stddev_over_time`** / **`stdvar_over_time`** ```promql avg_over_time(node_memory_MemAvailable_bytes[1h]) max_over_time(rate(http_requests_total[5m])[1h:]) # subquery needed — rate is instant vector ``` ### Existence and Counter Resets ```promql absent(up{job="critical"}) # 1 if no series, empty if any absent_over_time(up{job="critical"}[5m]) # missing for the entire window resets(http_requests_total[1h]) # how many counter resets in the range changes(up[1h]) # how many times the value changed ``` ### Label Manipulation ```promql label_replace(up, "host", "$1", "instance", "(.+):(\\d+)") # extract host from instance="host:port" label_join(rate(x[5m]), "endpoint", " ", "method", "path") # concatenate labels info(rate(http_server_duration_count[5m])) # auto-join with target_info (experimental) ``` Full function catalog: [`${CLAUDE_SKILL_DIR}/references/functions.md`]. ## Subqueries Embed a range query inside an instant query. Syntax: ```promql <instant-vector-expression>[<range>:[<resolution>]] ``` Use when a range-vector function needs the output of another function or aggregation: ```promql max_over_time(rate(http_requests_total[5m])[1h:]) # peak 5m-rate over the last hour quantile_over_time(0.99, rate(errors[1m])[1d:]) # 99th percentile of per-minute error rates over a day deriv(rate(http_requests_total[5m])[30m:]) # is the rate accelerating? ``` **Performance:** subqueries re-run the inner expression at every resolution step. A `[1d:1m]` subquery runs the inner 1,440 times. For expensive inner expressions or repeated use, convert to a recording rule. **Avoid subqueries inside recording rules.** Split into two recording rules. Full syntax, alignment rules, subquery vs recording rule guidance: [`${CLAUDE_SKILL_DIR}/references/subqueries.md`]. ## Common Query Idioms ```promql # Request rate by job sum by (job) (rate(http_requests_total[5m])) # Error ratio — aggregate numerator and denominator separately sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) # P99 latency by handler (classic histogram) histogram_quantile(0.99, sum by (handler, le) (rate(http_request_duration_seconds_bucket[5m]))) # Top 5 CPU consumers topk(5, sum by (app) (rate(process_cpu_seconds_total[5m]))) # Disk-full prediction predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 24*3600) < 0 # Alert on missing scrape target absent(up{job="critical-service"}) # Week-over-week request rate comparison rate(http_requests_total[5m]) - rate(http_requests_total[5m] offset 1w) # Batch job staleness time() - my_batch_last_success_timestamp_seconds > 2 * 3600 ``` ## Pitfalls — Quick Reference Positive rules when writing or reviewing queries: - **Apply `rate()` to counters only.** Use `delta()`, `idelta()`, or `deriv()` for gauges. `rate()` on a gauge silently produces nonsense. - **`rate()` first, then aggregate.** `sum(rate(x[5m]))`, never `rate(sum(x)[5m])`. Counter resets are per-series. - **Aggregate ratios correctly.** Sum numerator and denominator separately, then divide. Never `avg(ratio)` or `avg(avg)`. - **Don't average summary quantiles.** `avg(metric{quantile="0.95"})` is statistically invalid. Use histograms for cross-instance aggregation. - **Preserve `le` when aggregating classic histogram buckets.** `sum by (job, le)` or `sum without (instance)`. - **Range ≥ 4× scrape interval, minimum 1m.** Avoid `rate(x[1m])` with 30s+ scrape intervals. - **`rate()` (not `irate()`) for alerts.** `irate()` is for graphing volatile counters. - **`without` over `by`** for most aggregations — preserves useful labels like `job` automatically. - **`offset` must immediately follow the selector.** `sum(x offset 5m)` is correct; `sum(x) offset 5m` is invalid. - **Filter early.** Push label matchers into the innermost selector. Full pitfall catalog with diagnostics: [`${CLAUDE_SKILL_DIR}/references/optimization.md`]. ## Recording and Alerting Rules ### Recording Rule Naming — `level:metric:operations` ```text instance:http_requests:rate5m # per-instance rate (no aggregation) job:http_requests:rate5m # rolled up to per-job job:http_errors_per_requests:ratio_rate5m # ratio, per job job:request_latency_seconds:mean5m # average latency from Summary ``` - **`level`** — labels remaining in the output (`instance`, `job`, `cluster`, `instance_path`, etc.) - **`metric`** — metric name; strip `_total` after `rate()` - **`operations`** — operations applied, newest first. Omit `_sum` when other ops present; `mean` replaces `rate` for sum/count averages; `ratio` for divisions named with `_per_` Recording rule examples and ratio aggregation patterns: [`${CLAUDE_SKILL_DIR}/references/recording-alerting-rules.md`]. ### Alerting Rule Essentials ```yaml - alert: HighErrorRatio # CamelCase community convention expr: | sum by (job) (rate(http_requests_total{status=~"5.."}[5m])) / sum by (job) (rate(http_requests_total[5m])) > 0.05 for: 10m # must be true for the entire window before firing keep_firing_for: 5m # keep firing this long after expression becomes false labels: severity: page annotations: summary: "High error rate on {{ $labels.job }}" description: "Error ratio is {{ $value | humanizePercentage }} for {{ $labels.job }}" runbook_url: "https://wiki.example.com/runbooks/{{ $labels.job }}" ``` **Alerting principles:** - **Alert on symptoms, not causes.** User-visible latency, error rates — not internal-only metrics. - **Page on one stack layer.** If user latency is fine, don't page on a slow sub-component. - **Every page must be actionable.** Nothing to do → delete the alert. - **Match `for:` to the rate window.** `rate(x[5m])` with `for: 30s` is contradictory. Full alerting best practices, template variables, anti-patterns: [`${CLAUDE_SKILL_DIR}/references/recording-alerting-rules.md`]. ## Application **Writing PromQL queries:** - Apply conventions silently — don't narrate each rule. - Wrap counters in `rate()` or `increase()` before any further operation. - Aggregate before joining; filter at the innermost selector. - Prefer `without` over `by` when removing a noise dimension. - For histograms, prefer native when available; otherwise preserve `le` in aggregations. **Writing recording rules:** - Follow `level:metric:operations` naming. - Build a hierarchy: per-instance rule first, then aggregate up. Strip `_total` after `rate()`. - Aggregate ratios by computing numerator and denominator separately. - No subqueries inside recording rules — split into two rules. **Writing alerting rules:** - CamelCase alert names; `severity` label for routing. - Alert on symptoms at the top of the stack. - Set `for:` to filter blips, sized to match the underlying rate window. - Use templating for context-rich notifications (`{{ $labels.X }}`, `{{ $value }}`). **Reviewing PromQL:** - Cite the violation and show the fix inline. - Check first: `rate()` on a counter? Aggregation inside `rate()`? `le` preserved? ## Integration `backend/prometheus` governs instrumentation (metric types, naming, labels, exporters, cardinality). This skill governs query writing. When both apply: - "What metric should I emit" — `backend/prometheus` - "How do I query this" — this skill - Naming a recording rule output — use `level:metric:operations` here; that becomes the metric name `backend/prometheus` rules then apply to `grafana/metricsql` (forthcoming) covers VictoriaMetrics' PromQL superset. MetricsQL extends PromQL with extra functions and looser type rules — write standard PromQL when targeting either Prometheus or VictoriaMetrics; reach for MetricsQL-only features only when the deployment is VictoriaMetrics and the standard expression is awkward.
Comments (0)
Sign in to join the conversation.
Reviews (0)
No reviews yet.
No comments yet.