Create Grafana Dashboards

Use this page to create Grafana dashboards from Sauron metrics and verify the PromQL query before relying on a dashboard for production visibility.

Component Role in this page
Grafana Creates, saves, and displays dashboards and panels.
Prometheus or Thanos Verifies PromQL queries before you add them to Grafana panels.
Sauron metrics datasource Supplies Sauron metrics to Grafana panels.
Create your first Grafana dashboard

Use this flow when you are new to Grafana and want to confirm that Sauron has metrics you can graph.

  1. Open https://grafana.stage.apatil.developers.oracledx.com.
  2. Sign in with the access method configured for your Sauron.
  3. Create a new dashboard and add a panel.
  4. Select the default Sauron metrics datasource.
  5. Start with a simple PromQL query for a metric you know should exist. If you do not know a metric name yet, first use Prometheus or Thanos to explore available metric names.
  6. Set a recent time range, such as the last 1 hour.
  7. Run the query and confirm the panel returns data.
  8. Save the dashboard with a name your team can recognize.

If the panel shows no data, check the time range, datasource, query, dashboard variables, and whether the same query returns data in Prometheus or Thanos.

Before you build the dashboard, verify the exact query directly:

  1. Open Prometheus: https://prometheus.stage.apatil.developers.oracledx.com/graph?g0.expr=up&g0.tab=1.
  2. Open Thanos: https://thanos.stage.apatil.developers.oracledx.com/graph?g0.expr=up&g0.tab=1.
  3. Use the same PromQL query, labels, and time range when you create the Grafana panel.
Prove it works before production

Before treating a dashboard as production-ready:

  1. Run the PromQL query directly in Prometheus or Thanos.
  2. Use the same time range in Grafana.
  3. Confirm the panel datasource is the intended Sauron metrics datasource.
  4. Confirm dashboard variables select labels that exist for the queried series.
  5. Save the dashboard only after the panel returns the expected data.
Manually Creating Grafana Dashboards

See the Grafana UI documentation.

Creating Grafana Dashboards via Grafana HTTP API

See the Grafana HTTP API documentation.

For example, creating a new dashboard:

GRAFANA_SERVICE_ACCOUNT_TOKEN='replace-with-your-grafana-service-account-token'

curl --fail --show-error -i -XPOST \
  -H "Authorization: Bearer ${GRAFANA_SERVICE_ACCOUNT_TOKEN}" \
  -H 'Content-Type: application/json' \
  --data-binary @./dashboard.json \
  'https://grafana.stage.apatil.developers.oracledx.com/api/dashboards/db'

where dashboard.json contains the dashboard definition. The best way to create your initial dashboard JSON is to define it manually within the UI, then export it. This is a little bit funky, but what you export from the UI must be placed within a "dashboard" element in your dashboard.json, like this:

{
  "dashboard": {
    "title": "replace-with-dashboard-json-exported-from-the-ui"
  },
  "overwrite": true
}
Built-in Data sources

Your Sauron Grafana comes with a predefined data source called "Prometheus" that points to your Sauron's Prometheus backend(s).

Thanos provides an HA and stable view of the metrics coming from multiple Prometheus backends, which can have slightly different samples. If you are not sure which datasource your dashboard should use, compare the same query in Grafana, Prometheus, and Thanos for the same time range before changing the dashboard.

Built-in My Sauron Dashboards

Your Sauron's Grafana comes preconfigured with a set of built-in dashboards, under a folder named "My Sauron". These dashboards provide you with a view of the overall health of your Sauron's components, like OpenSearch, OpenSearch Dashboards, and Prometheus.

Self-check before escalation

Most questions for this topic can be resolved with the steps above. If dashboard creation or query results do not behave as expected, check and collect:

  • Sauron Grafana URL.
  • Component or endpoint.
  • Dashboard name and panel name.
  • Datasource selected in the panel.
  • PromQL query.
  • Time range.
  • Authentication method used.
  • Whether the same query works in Prometheus or Thanos.
  • Exact error/status or screenshot.
  • Start time.
  • Recent dashboard, datasource, query, variable, or access changes.
  • Checks completed.
  • Production impact, if any.