Federated Metric Views with Thanos

Use this advanced guide when a query or dashboard needs metrics from several Saurons. Thanos is the metrics search layer; an additional store connects it to another approved source. For your first service metric, start with Sending Metrics and Grafana Dashboards.

When should I use this?

Use regional Thanos for a regional check and a global view to query several regions together. The global view requests data from connected sources when you query; adding a store does not copy their metrics or change their retention.

Retention with downsampling

Downsampling keeps a less detailed view of older metrics. Instead of showing measurements every few seconds, it lets you query five-minute or one-hour summaries. These show long-term trends, but not the exact timing of short spikes.

With Thanos object storage and downsampling enabled, you can request metric retention up to one year, subject to approval. The Compactor combines stored blocks, creates these summaries, and applies retention settings.

Metric detail Sauron defaults when downsampling is enabled
Original measurements 30 days
Five-minute summaries 60 days
One-hour summaries 90 days

These periods overlap; they do not add together. Confirm the source Sauron's configured periods and required detail with its owner before requesting a change. Connecting global Thanos does not extend a regional source's retention. See official Thanos retention guidance.

If an older query returns no data, check retention and allow a maximum source resolution of 5 minutes or 1 hour, or automatic if available. A query restricted to original measurements cannot use those summaries. See Thanos query resolution options.

Design your global view

This illustrative topology uses support-style names. It is not a verified customer deployment. Replace the names with your approved Saurons, and test the connection in a test Sauron before a production change.

A global Sauron Thanos view queries additional stores phx, iad, and fra connected to three regional Thanos sources
Example query topology. Select to enlarge; close with × or Esc.

The destination is prod.rgbu-common-global. Fill in this matrix with its Sauron owner before adding stores. The known query must already work in its regional source; record its labels and time range too. The gRPC endpoint is the address Thanos uses for this metric connection.

Regional source (example) Store name Approved gRPC endpoint Owner Known metric check
prod.rgbu-common-phx phx Owner supplies host:port Phoenix Sauron owner Record query, labels, time range
prod.rgbu-common-iad iad Owner supplies host:port Ashburn Sauron owner Record query, labels, time range
prod.rgbu-common-fra fra Owner supplies host:port Frankfurt Sauron owner Record query, labels, time range
  • Keep a clear global-to-regional hierarchy. Document each source's data and avoid reaching the same sources through overlapping paths.
  • Prefer one named entry per region for easy inspection and removal. A store name does not add a region label to metrics.
  • Preserve region, environment, and source labels; confirm their actual names before writing a combined query. Ask the Sauron owner to confirm HA replica labels and deduplication, keeping distinct regions and sources separate.
  • Check query warnings and expected regions before trusting a total. Partial responses can return reachable sources while another source fails.

Further reading: Thanos global views, HA, and partial responses.

Before you add a Thanos Store

The destination is where you add the store and run the combined query; the source is the regional Thanos endpoint exposing data. Ask both Sauron owners to confirm API access, approved connection credentials, and the network path. Source allowlists must permit the destination's actual outbound addresses.

Use the source's thanos-grpc hostname and externally reachable port, not its browser thanos URL. The port depends on the ingress setup; confirm it rather than assuming 443 or 10901. Loading an HTTP page does not prove gRPC works. The urls field takes host:port, without https://.

Choose an unused store name: 2–12 lowercase letters, digits, or hyphens, starting with a letter and ending with a letter or digit. Reusing a name updates its existing entry.

Sauron stores the connection certificate/key in a Kubernetes Secret and uses TLS for the remote gRPC client. Have its owner confirm the approved trust setup. Thanos TLS client options provide background; follow the Sauron API procedure below for this task. Keep certificate/key values out of support messages, tickets, and screenshots.

Add an additional store

Start with Phoenix (phx) only. Open the approved global (destination) API from Console Endpoints and check existing names:

GET /v1/thanos/additionalstores

On the approved regional (source) API, an authorized user retrieves its configured certificate and private key:

GET /v1/thanos/servercerts

Use the returned base64 values exactly, including any trailing =, in the destination's API form below. If you lack access, ask the source owner for approved connection material through your team's secure process.

PUT /v1/thanos/additionalstores

{
  "cert": "replace-with-approved-base64-cert",
  "key": "replace-with-approved-base64-key",
  "name": "phx",
  "urls": ["thanos-grpc.<source-sauron-domain>:<approved-port>"]
}

Replace the domain and port with the matrix entry. Check: HTTP 202 means the change was accepted; it does not prove the source is queryable. Inspect the stored name and endpoint with:

GET /v1/thanos/additionalstores/phx

This response also contains the certificate and private key. Inspect it privately. Before updating an existing entry, save its current configuration securely and have the destination owner review the change.

Prove it works before production

  1. In global destination Console Endpoints, find the new Thanos Store view for phx. Refresh until it opens; registration is not instant.
  2. Open that view's Stores tab. Check source reachability and the reported labels/time range.
  3. Open the destination's main Thanos view. Repeat the known regional query with the same time range and source labels. For a source running the Linux host test, start with node_cpu_seconds_total{job="sauron-first-metric-test"} and inspect its returned labels to identify the source.
  4. Check query warnings. Success: Expected regional series appear in the global result, with the source still identifiable.

Then repeat for iad and fra using each source's endpoint and approved connection material. Recheck every intended region before relying on a combined dashboard or alert. Use a source filter and recent time range when investigating a slow or missing result.

Inspect or remove a store

List names with GET /v1/thanos/additionalstores; inspect one with GET /v1/thanos/additionalstores/phx. The named response contains secrets.

When a test is complete or a source is retired, ask the destination owner which dashboards or rules depend on it before deleting its entry:

DELETE /v1/thanos/additionalstores/phx

Check: The request is accepted with 202; refresh the name list until phx is absent. Deleting an entry does not delete regional metrics. Verify the remaining global view still covers the intended sources.

Add multiple stores that share one certificate and key

One entry can contain several endpoints, but it has one certificate and key. Use this shape only after the owners confirm every endpoint accepts the same approved connection material:

{
  "cert": "replace-with-approved-base64-cert",
  "key": "replace-with-approved-base64-key",
  "name": "dev-regions",
  "urls": [
    "thanos-grpc.<source-sauron-domain-1>:<approved-port-1>",
    "thanos-grpc.<source-sauron-domain-2>:<approved-port-2>"
  ]
}

Use the same add and verification procedure, checking each endpoint before trusting the combined result.

Share certificate values for additional stores

GET /v1/thanos/servercerts returns the source's configured certificate and private key in base64. For an owner-approved certificate change, its API also supports:

PUT /v1/thanos/servercerts

{
  "cert": "replace-with-base64-cert",
  "key": "replace-with-base64-key"
}

This changes the source's certificate/key pair; it is not required to add a store. The source owner must coordinate with connected destinations and verify their connections afterward.

Tips and checks

  • Tips: Keep your regional source list current and retain source-identifying labels. Follow Design your global view before combining regional totals.
  • Check: Query each source separately, then the global view. Read query warnings and use the checks below for missing data.
What you see Next check
Store missing Inspect the destination's stored name/endpoint; check registration with its owner.
Connection timeout Confirm hostname/port and destination-to-source network allowlists.
HTTP or TLS error on gRPC Confirm the StoreAPI endpoint, port, and approved certificate/key; ask the Sauron owner to inspect ingress and TLS.
Store reachable, data missing Run the query in the source; compare time range, labels, and global warnings.
Only some regions appear Check each regional store/query; treat the combined result as incomplete until verified.