Prometheus Recording Rules

Use a recording rule to save the result of a metrics query as a new metric. Dashboards and alerts can reuse that result. Start with Sending Metrics if your service metric is not in Prometheus yet.

When should I use recording rules?

Use recording rules when a PromQL expression is correct but expensive, repeated often, or reused in more than one place.

Common examples:

  • A Grafana dashboard panel or variable uses the same expensive query repeatedly.
  • Several alerts use the same aggregation with different thresholds.
  • A query needs a stable, shorter metric name for dashboards or alerts.
  • A high-cardinality source metric should be aggregated before broad dashboard use.

Do not use a recording rule to hide an incorrect query. First confirm the original PromQL returns the data you expect in Prometheus or Thanos.

Create your first recording rule

Use a test Sauron and an account allowed to upload and delete rules. Ask your Sauron owner for access and review of the shared configuration.

First run a query for a metric your service already sends to Prometheus. Ask your service owner for the metric name if needed. This example uses http_requests_total; replace it with your real request counter and test the sum(rate(...)) expression in Prometheus before uploading.

cat > service-requests.rules <<'EOF'
groups:
  - name: service-requests
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))
EOF

Upload the rule file to a test Sauron through the Sauron API Server. Choose a new filename so this example does not replace an existing rule file. Help fills in this Sauron's configured API username; paste the command and enter its password when curl prompts:

curl --fail --show-error -i -u 'sauron' \
  -T service-requests.rules \
  'https://api.stage.apatil.developers.oracledx.com/prometheus/rules/service-requests.rules'

Multiple Prometheus rule files are allowed. Use a file name that makes the rule ownership or purpose clear.

Prove it works before production

Before using a recording rule in production dashboards or alerts:

  1. Query the original PromQL expression in Prometheus or Thanos.
  2. Upload the rule file successfully through the Sauron API Server.
  3. Open Prometheus Rules and confirm the rule group appears.
  4. Query the new recorded metric in Prometheus.
  5. If Thanos is enabled for this Sauron, query the same recorded metric there.
  6. Use the recorded metric in Grafana dashboards or alert rules only after it appears.

Recording rules are evaluated on the rule evaluation schedule, so the new metric may not appear immediately after upload. When finished, remove only your temporary file with DELETE /prometheus/rules/service-requests.rules in the Sauron API. Check that the rule group disappears; previously recorded samples can remain until retention expires.

Recording rule reference

One rule file can contain recording rules and alert rules. A recording rule uses record and expr; an alert rule uses alert and expr.

The Prometheus Authors provide formal documentation for recording rules.

Basic shape

The first recording rule shows the required groups, rules, record, and expr fields.

Reuse a recorded metric in an alert

groups:
  - name: example-alerts
    rules:
      - alert: HighRequestRate
        expr: job:http_requests:rate5m > 100
        for: 10m
        labels:
          severity: warning

Common mistakes

  • The rule file is missing the top-level groups: key.
  • YAML indentation is invalid.
  • The original PromQL query does not return data for the selected time range.
  • Aggregation removes labels that a dashboard or alert still expects.
  • The recorded metric name in the query does not exactly match the record name.
  • A histogram quantile expression omits the le label in the bucket aggregation.
  • The rule was uploaded successfully, but the evaluation schedule has not produced the first sample yet.

Tips and checks

  • Tips: Use recording rules for calculations reused by dashboards or alerts. Choose a name that describes the result.
  • Check: Confirm the rule appears in Prometheus Rules without an evaluation error, then query its recorded metric. If data is missing, run the original expression and check its time range.