Sauron Self-Alerting

Self-alerts report possible problems with Sauron itself, such as storage pressure. They are different from alerts about your service. For a service alert, start with Defining Alerts.

First check whether self-alerting is enabled for your instance. The Sauron API can then list or change its alerts and recipients. Ask your Sauron owner which recipients and settings are approved before changing them.

What can I change?

Task	API area
Enable or disable all self-alerting	/v1/selfalerting/actions
Check self-alerting status	/v1/selfalerting/status
Set custom self-alert email recipients	/v1/selfalerting/toEmails
List available self-alerts	/v1/selfalerting/alerts
Enable or disable one self-alert	/v1/selfalerting/alerts/{alertName}/actions
View or update one self-alert's configuration	/v1/selfalerting/alerts/{alertName}
Control whether future self-alerts are enabled by default	/v1/selfalerting

Do not share passwords, tokens, SMTP credentials, webhook credentials, private keys, or full Alertmanager receiver secrets in support channels.

The commands below show this Sauron's configured API username. Curl prompts for its password. You need an account allowed to change self-alerting for the write commands.

Enable or disable self-alerting

Check the current self-alerting status:

curl --fail --show-error -u 'sauron' \
  'https://api.stage.apatil.developers.oracledx.com/v1/selfalerting/status'

Enable self-alerting:

curl --fail --show-error -i -XPUT -u 'sauron' \
  'https://api.stage.apatil.developers.oracledx.com/v1/selfalerting/actions?action=enable'

Disable self-alerting:

curl --fail --show-error -i -XPUT -u 'sauron' \
  'https://api.stage.apatil.developers.oracledx.com/v1/selfalerting/actions?action=disable'

Set custom email recipients

This API replaces the custom recipient list; it does not append to it. Include every recipient you want to keep. If toEmails is not configured, the contact email is used. Replace the example addresses with approved recipients.

Set one email recipient:

curl --fail --show-error -i -XPUT -u 'sauron' \
  'https://api.stage.apatil.developers.oracledx.com/v1/selfalerting/toEmails?emails=abc@example.com'

Set multiple email recipients:

curl --fail --show-error -i -XPUT -u 'sauron' \
  'https://api.stage.apatil.developers.oracledx.com/v1/selfalerting/toEmails?emails=abc@example.com,xyz@example.com'

Manage individual self-alerts

List available self-alerts:

curl --fail --show-error -u 'sauron' \
  'https://api.stage.apatil.developers.oracledx.com/v1/selfalerting/alerts'

Example response:

[
  {
    "description": "Alert on the Prometheus head series dropping greatly",
    "name": "prometheusseriesdropping"
  }
]

Disable one self-alert:

curl --fail --show-error -i -XPUT -u 'sauron' \
  'https://api.stage.apatil.developers.oracledx.com/v1/selfalerting/alerts/prometheusseriesdropping/actions?action=disable'

Enable one self-alert:

curl --fail --show-error -i -XPUT -u 'sauron' \
  'https://api.stage.apatil.developers.oracledx.com/v1/selfalerting/alerts/prometheusseriesdropping/actions?action=enable'

Check one self-alert's enabled status:

curl --fail --show-error -u 'sauron' \
  'https://api.stage.apatil.developers.oracledx.com/v1/selfalerting/alerts/prometheusseriesdropping/status'

After enabling an alert, open the My Prometheus Grafana dashboard and check that its alert is present. For example, prometheusseriesdropping belongs to that dashboard. If it is missing, check the self-alert status and the Grafana dashboard after the configuration update completes.

Update an individual self-alert

Some self-alert parameters can be modified:

• severity: an arbitrary string that you can set based on your Alertmanager configuration.
• forMinutes: how many minutes the condition must stay true before the alert fires.
• threshold: the threshold value for the alert condition.

Fetch the current configuration for an alert:

curl --fail --show-error -u 'sauron' \
  'https://api.stage.apatil.developers.oracledx.com/v1/selfalerting/alerts/prometheusseriesdropping'

Example response:

{
  "forMinutes": 15,
  "severity": "warning",
  "threshold": "-80"
}

Update the threshold for the same alert:

curl --fail --show-error -i -XPUT -u 'sauron' \
  -H 'Content-Type: application/json' \
  --data '{"severity":"warning","forMinutes":15,"threshold":"-50.0"}' \
  'https://api.stage.apatil.developers.oracledx.com/v1/selfalerting/alerts/prometheusseriesdropping'

Prove it works before production

After changing self-alerting:

1. Confirm the API request returns a successful status.
2. Confirm /v1/selfalerting/status or the alert-specific status endpoint shows the expected state.
3. Open the related built-in Grafana dashboard, such as My Prometheus, and confirm the alert appears when enabled.
4. Confirm Alertmanager has a route and receiver for the alert.
5. If notification delivery is expected, verify the receiver receives a test or real notification.
6. If delivery fails, check Alertmanager logs and receiver errors with secrets removed.

Future self-alert defaults

New self-alerts appear through /v1/selfalerting/alerts. The enableNewAlerts setting controls whether future built-in alerts are enabled when added. Check your instance's current setting before changing it. To disable future self-alerts by default:

curl --fail --show-error -i -XPUT -u 'sauron' \
  -H 'Content-Type: application/json' \
  --data '{"enableNewAlerts":false}' \
  'https://api.stage.apatil.developers.oracledx.com/v1/selfalerting'

View the current future-alert setting:

curl --fail --show-error -u 'sauron' \
  'https://api.stage.apatil.developers.oracledx.com/v1/selfalerting'

Alertmanager routing reference

Sauron self-alerts are routed through Alertmanager. Grafana is configured out of the box with a notification channel for Alertmanager, and that notification channel is used by Sauron self-alerts.

If you already have an Alertmanager route and receiver configured, self-alerts enter that existing receiver stream. To route self-alerts separately from ordinary alerts, use the mysauron label.

For complete receiver examples, see:

• Ocean integration for metrics alerts
• Slack integration for Alertmanager alerts
• Email integration for Alertmanager alerts

This example shows the routing shape for sending alerts with the mysauron label to a separate receiver:

route:
  receiver: Service-Production
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 1m
  repeat_interval: 3m
  routes:
    - receiver: Slack-Receiver
      group_by: ['alertname']
      match_re:
        mysauron: .+
receivers:
  - name: Service-Production
    webhook_configs:
      - url: replace-with-service-webhook-url
  - name: Slack-Receiver
    slack_configs:
      - api_url_file: /etc/alertmanager/auxiliaryfiles/self-alert-slack.url
        send_resolved: true

Ask your Sauron owner to store the approved Slack webhook in that restricted auxiliary file before using this receiver.

If you set custom self-alert severity values, make sure your Alertmanager routes and receivers treat those values the way you expect.

Prometheus self-alert reference

Prometheus Disk Usage High

This alert warns when Prometheus storage usage crosses its configured threshold for the configured time. Check the alert's current settings through the self-alert API. Check current disk usage in the My Prometheus Grafana dashboard. Ask the Sauron owner for the configured storage capacity if you need to assess remaining space.

Customer checks:

• Review your Prometheus configuration. If you are scraping aggressively, consider increasing the scrape interval.
• Check for high-cardinality metrics in the Prometheus status page. Review labels with your service team before dropping them; a label used by a dashboard or alert may be needed.
• Consider shortening the retention period if possible. Sauron-team action is required to change the retention period.

If those checks do not resolve the issue, contact Sauron support.

My Sauron - Time Series Dropping

This alert checks the hourly change in Prometheus's active metric series against its configured threshold. A series is one metric with a particular set of labels. Check the current threshold and hold period before interpreting it; missing sources or a configuration change can reduce the number of series.

Customer checks:

• Recheck scrape configuration.
• Recheck server configuration.
• Recheck connectivity to Prometheus, PushProx, or Shuttleproxy servers.

OpenSearch self-alert reference

OpenSearch Disk Usage High

This alert warns when OpenSearch storage usage crosses its configured threshold. Check the current alert settings and Console Instance before acting on a percentage.

Customer checks:

• Review unnecessary DEBUG logging with your service owner; reduce it without losing logs needed for investigation.
• Check for indices named without a timestamp or with a timestamp outside the supported patterns:
• -%{+yyyy.MM.dd}
• -%{+yyyy-MM-dd}
• -%{+yyyy_MM_dd}
• Fix log shipper client settings to use one of the supported index patterns.
• Review log volume and index management with your OpenSearch owner.
• If several regions send logs here, ask your Sauron owner to review capacity and regional placement before changing the design.
• Consider shortening the retention period if possible. Sauron-team action is required to change the retention period.
• If ingestion has grown because of more clients, ask Sauron support about increasing OpenSearch disk size.

OpenSearch Shard Size Too Large

This alert checks average shard size by index against its configured threshold. Check current alert settings; ask the OpenSearch owner to choose a suitable shard size before changing the layout.

Customer checks:

• Review index and shard sizes with the OpenSearch owner before changing shard layout. A shard-count change may require a new index or reindexing.
• Filter unnecessary messages from the client side.
• Split one large index into several smaller indices.

Alertmanager self-alert reference

OCEAN Token Expiring

An OCEAN token in the customer-supplied Alertmanager configuration has expired or will expire soon.

Customer action:

• Get a new OCEAN token.
• Use the Sauron API to update the Alertmanager configuration with the new token.
• See the OCEAN External Webhook Onboarding Guide for information about how to get a new token.

Troubleshoot failed Alertmanager notifications

A failed notification is not one of the configurable built-in self-alerts listed by /v1/selfalerting/alerts. If an expected notification did not arrive, check the Alertmanager route and receiver, then check Alertmanager logs through the Sauron API Server for a delivery error. Possible causes include invalid SMTP, OCEAN, or Slack credentials; a missing or archived Slack channel; or a network error. Correct the receiver or credential indicated by the actual error, then send a test alert using Defining Alerts.

Tips and checks

• Tips: Inspect current settings before changing thresholds. Change one setting at a time and keep its previous value.
• Check: Read the setting back from the API. For missing notifications, check whether the alert is firing, then its matching route, receiver, silences, and delivery error. Use the approved alert test to check delivery.