PromQL cheat sheet

up == 0

Every scrape target that is currently failing. Check this first when a dashboard goes blank.

up{job="api"} == 0

up

1 if the last scrape succeeded, 0 if it failed. Generated by Prometheus itself.

== 0

A filter: keeps only series whose value is 0.

Gotcha up == 0 shows failing targets, not missing ones. A target that vanished from service discovery needs absent().

metric{label="value"}

Select series by label with exact, negative and regex matchers.

http_requests_total{job="api", status=~"5..", method!="OPTIONS"}

= !=

Exact match / not equal.

=~ !~

Regex match / not match; fully anchored.

{__name__=~"node_.*"}

Select by metric name pattern.

metric[5m]

A range vector: every sample from the last five minutes, the input rate() and friends need.

rate(http_requests_total[5m])

[5m]

Look-back window. Use at least 4x the scrape interval.

offset 1w

Shift the query back in time, for week-over-week comparison.

[1h:1m]

Subquery: evaluate an expression every minute over an hour.

rate(counter[5m])

Per-second average increase of a counter over the window, handling counter resets.

sum by (job) (rate(http_requests_total[5m]))

rate

Smooth per-second rate; right for graphs and alerts.

irate

Uses only the last two samples; spiky, for fast-moving graphs.

increase

Total increase over the window (rate x seconds).

Gotcha Apply rate before sum, never after. sum(...) of raw counters loses the reset handling and produces nonsense.

sum by (<labels>) (<expr>)

Aggregate across series and keep only the labels you name.

sum by (route) (rate(http_requests_total{job="api"}[5m]))

by (a, b)

Keep these labels, drop the rest.

without (a)

Drop these labels, keep the rest.

avg / max / min / count

Other aggregation operators.

topk(<k>, <expr>)

The k largest series right now, such as the busiest endpoints or the hungriest pods.

topk(5, sum by (pod) (rate(container_cpu_usage_seconds_total[5m])))

bottomk

The k smallest.

Gotcha In a graph, topk is evaluated per step, so different series can appear at different times.

sum(rate(errors[5m])) / sum(rate(total[5m]))

Error ratio for a service: the core SLI behind most availability alerts.

sum(rate(http_requests_total{job="api",status=~"5.."}[5m])) / sum(rate(http_requests_total{job="api"}[5m]))

numerator

Failed requests only.

denominator

Every request.

Gotcha When a service gets no traffic the ratio is 0/0 and returns nothing. Alerts silently stop firing; pair them with a traffic or absent() check.

histogram_quantile(0.99, sum by (le) (rate(<metric>_bucket[5m])))

Estimate a latency percentile from histogram buckets.

histogram_quantile(0.99, sum by (le, route) (rate(http_request_duration_seconds_bucket{job="api"}[5m])))

le

The bucket upper bound label; always keep it in the by clause.

0.99

The quantile: 0.5 median, 0.99 p99.

_sum / _count

Average latency: rate(_sum) / rate(_count).

Gotcha The result can be no more precise than the bucket boundaries. If p99 sits exactly on a bucket edge, the real value is somewhere inside that bucket.

absent(<selector>)

Returns 1 when no series match: alert on metrics that disappeared entirely.

absent(up{job="api"})

absent_over_time(x[10m])

No samples at all for ten minutes.

predict_linear(<gauge>[<window>], <seconds>)

Extrapolate a gauge: alert before the disk is full rather than when it is full.

predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 4 * 3600) < 0

[6h]

History used for the linear fit.

4 * 3600

Predict four hours ahead.

delta / deriv / changes

Functions for gauges and for counting events such as restarts.

changes(kube_pod_container_status_restarts_total[1h]) > 3

delta(g[1h])

Difference between first and last sample of a gauge.

deriv(g[10m])

Per-second slope by linear regression.

changes(x[1h])

How many times the value changed.

avg_over_time / max_over_time

Aggregate each series across time.

CPU utilisation per host

Fraction of CPU time not idle, from node_exporter.

1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))

mode="idle"

Idle seconds; 1 minus idle is busy.

mode="iowait"

Waiting on storage.

mode="steal"

Stolen by the hypervisor, a noisy-neighbour signal.

Memory available ratio

Share of memory still available to applications.

node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes

MemAvailable

Includes reclaimable cache, unlike MemFree.

Filesystem usage

Used fraction per filesystem, ignoring pseudo filesystems.

1 - node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"}

avail vs free

avail excludes root-reserved blocks, matching what apps can use.

Container memory vs limit

Working set as a fraction of the memory limit. At 1 the kernel's cgroup OOM killer ends the container (exit code 137, reason OOMKilled); the kubelet only evicts pods under node memory pressure.

max by (pod, container) (container_memory_working_set_bytes{container!=""}) / max by (pod, container) (kube_pod_container_resource_limits{resource="memory"})

working_set

Usage minus reclaimable cache: what kubelet eviction ranks by, and the closest signal to an OOM kill.

container!=""

Drop the pod-level aggregate series.

topk(10, count by (__name__) ({__name__=~".+"}))

Find the metrics with the most series, the usual cause of a slow or memory-hungry Prometheus.

count by (__name__)

Series per metric name.

/tsdb-status

The UI page with the same information and more.

Gotcha The query itself touches every series. Run it on a quiet moment, and never put user IDs or request IDs in labels.

Alerting rule

An alert with a pending period, severity for routing and a runbook link for the person who gets paged.

groups:
  - name: api
    rules:
      - alert: ApiHighErrorRate
        expr: |
          sum(rate(http_requests_total{job="api",status=~"5.."}[5m]))
            / sum(rate(http_requests_total{job="api"}[5m])) > 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "API 5xx ratio {{ $value | humanizePercentage }}"
          runbook_url: https://runbooks.example.com/api-errors

for: 10m

Must stay true this long before firing; filters blips.

labels.severity

Alertmanager routes on it: page versus ticket.

annotations

Human text; {{ $value }} and {{ $labels.x }} are templated.

runbook_url

The first thing the on-call engineer should click.

Gotcha Page on symptoms users feel (errors, latency), not on causes such as CPU. Causes belong on dashboards and tickets.

Multi-window burn-rate alert

Page when a 99.9% SLO's error budget is burning 14.4x too fast over both a long and a short window.

- alert: ApiErrorBudgetFastBurn
  expr: |
    job:slo_errors_per_request:ratio_rate1h{job="api"} > (14.4 * 0.001)
    and
    job:slo_errors_per_request:ratio_rate5m{job="api"} > (14.4 * 0.001)
  labels:
    severity: page

14.4x over 1h

Spends 2% of a 30-day budget in one hour.

6x over 6h

The usual second page condition (5% of budget).

short window (5m)

Makes the alert reset quickly once the burn stops.

0.001

The error budget of a 99.9% target.

Recording rule

Precompute an expensive expression so dashboards and alerts stay fast.

groups:
  - name: api-recording
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

level:metric:operations

Naming convention: aggregation level, metric, operations applied.

promtool check rules <file>

Validate rule files, and unit-test alerts against synthetic series, before they reach production.

promtool check rules alerts/*.yml && promtool test rules tests/alerts_test.yml

check config

Validate prometheus.yml.

test rules

Run alert unit tests.

amtool silence add <matchers>

Silence alerts during planned maintenance, with an expiry and an author comment.

amtool silence add alertname=ApiHighErrorRate instance=web-01 --duration=2h --comment="kernel patching CHG-1042"

--duration

Always expire silences.

silence query

List active silences.

silence expire ID

End one early.

Gotcha A silence without an expiry outlives the maintenance and hides the next real incident.

curl <prometheus>/api/v1/query

Run PromQL from a terminal or script through the HTTP API.

curl -s 'http://prometheus:9090/api/v1/query' --data-urlencode 'query=up{job="api"}' | jq '.data.result'

/api/v1/query

Instant query; add time= for a past moment.

/api/v1/query_range

start, end and step for a time series.

--data-urlencode

Encode braces, quotes and spaces safely.