Prometheus Recording Rules

Use this page to create Prometheus recording rules that precompute PromQL expressions into new time series for dashboards, alerts, and repeated queries.

Recording rules are evaluated like alerting rules, but they write a new metric instead of sending a notification. The new metric can then be reused in Grafana dashboards, Prometheus alerts, or Thanos queries without repeating the original expression every time.

When should I use recording rules?

Use recording rules when a PromQL expression is correct but expensive, repeated often, or reused in more than one place.

Common examples:

  • A Grafana dashboard panel or variable uses the same expensive query repeatedly.
  • Several alerts use the same aggregation with different thresholds.
  • A query needs a stable, shorter metric name for dashboards or alerts.
  • A high-cardinality source metric should be aggregated before broad dashboard use.

Do not use a recording rule to hide an incorrect query. First confirm the original PromQL returns the data you expect in Prometheus or Thanos.

Create your first recording rule

Start with a PromQL expression that already works. This example records the 99th percentile duration for billing jobs, then reuses the recorded metric in two alert rules.

cat > alert.rules.yml <<'EOF'
groups:
  - name: billing-jobs-health
    rules:
      - record: job_type:billing_jobs_duration_seconds:99p5m
        expr: histogram_quantile(0.99, sum(rate(jobs_duration_seconds_bucket{service="billing-processing"}[5m])) by (job_type, le))

      - alert: BillingProcessingVerySlow
        expr: job_type:billing_jobs_duration_seconds:99p5m > 30
        for: 5m
        labels:
          severity: critical

      - alert: BillingProcessingSlow
        expr: job_type:billing_jobs_duration_seconds:99p5m > 15
        for: 5m
        labels:
          severity: warning
EOF

Upload the rule file through the Sauron API Server:

curl --fail --show-error -i -u sauron:mypassword \
  -T alert.rules.yml \
  'https://api.stage.apatil.developers.oracledx.com/prometheus/rules/alert.rules.yml'

Multiple Prometheus rule files are allowed. Use a file name that makes the rule ownership or purpose clear.

Prove it works before production

Before using a recording rule in production dashboards or alerts:

  1. Query the original PromQL expression in Prometheus or Thanos.
  2. Upload the rule file successfully through the Sauron API Server.
  3. Open https://prometheus.stage.apatil.developers.oracledx.com/rules and confirm the rule group appears.
  4. Query the new recorded metric in Prometheus.
  5. Query the same recorded metric in Thanos.
  6. Use the recorded metric in Grafana dashboards or alert rules only after it appears.

Recording rules are evaluated on the rule evaluation schedule, so the new metric may not appear immediately after upload.

Recording rule reference

The alert.rules.yml file can contain recording rules and alerting rules. Recording rules use record and expr; alerting rules use alert and expr.

The Prometheus Authors provide formal documentation for recording rules.

Basic shape
groups:
  - name: example-recording-rules
    rules:
      - record: job:http_requests:rate5m
        expr: sum(rate(http_requests_total[5m])) by (job)
Reuse a recorded metric in an alert
groups:
  - name: example-alerts
    rules:
      - alert: HighRequestRate
        expr: job:http_requests:rate5m > 100
        for: 10m
        labels:
          severity: warning
Common mistakes
  • The rule file is missing the top-level groups: key.
  • YAML indentation is invalid.
  • The original PromQL query does not return data for the selected time range.
  • Aggregation removes labels that a dashboard or alert still expects.
  • The recorded metric name in the query does not exactly match the record name.
  • A histogram quantile expression omits the le label in the bucket aggregation.
  • The rule was uploaded successfully, but the evaluation schedule has not produced the first sample yet.
Self-check before escalation

Before contacting Sauron support, collect this information:

  • Sauron URL.
  • Rule file name, such as alert.rules.yml.
  • Recorded metric name.
  • Original PromQL expression.
  • Upload command used, with password or token removed.
  • Exact API status code and response message.
  • Whether the rule appears in https://prometheus.stage.apatil.developers.oracledx.com/rules.
  • Whether the recorded metric appears in Prometheus.
  • Whether the recorded metric appears in Thanos.
  • Grafana dashboard or alert rule that uses the recorded metric.
  • Time range tested.
  • Recent metric, scrape, dashboard, or rule changes.
  • Impact, including whether this affects dashboards, alerts, or direct queries.

Do not share passwords, tokens, certificates, private keys, or full credentials in support channels.