Restore OpenSearch Indices

Use this page when you need to restore older log indices from Sauron OpenSearch backups in OCI Object Storage. Small restores can usually be done from OpenSearch Dashboards Dev Tools. Large or many-index restores require Sauron-team help because they can affect capacity, performance, and backups.

Which restore path should I use?
Task Recommended path
Restore a few older indices Use the self-service Dev Tools flow below.
Restore many or very large indices Request Sauron-team help in #sauron-support.
Export OpenSearch data to local files Use Download OpenSearch Data.
Before you restore

Before restoring indices:

  1. Open https://kibana.stage.apatil.developers.oracledx.com and go to Dev Tools.
  2. Confirm the current OpenSearch cluster indices are green.
  3. Identify the exact source index names.
  4. Choose a snapshot that contains the indices you need.
  5. Restore into renamed indices so the restored logs are not treated like the original old indices.
  6. Keep the source index names, snapshot ID, restore request, and restored index names for validation.

Reference for snapshot restore API syntax: https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html

Restore a few indices
1) Check the status of your OpenSearch cluster
GET _cat/indices?v&s=index:asc

Make sure all indices are green before proceeding. If any indices are other than green, contact #sauron-support before proceeding.

2) Get a list of all snapshots in OCI Object Storage
GET _cat/snapshots/object-store-repo?h=id&s=id:desc

This may take a moment, but returns all OpenSearch backups currently in OCI Object Storage for your Sauron.

3) Select a snapshot and verify it has the logs you're interested in

From the list returned in Step 2, determine which snapshot to use. Suppose you want to restore example_index_2021.08_06 and example_index_2021.08_07. Logs can continue to arrive after an official end-of-day boundary, so choose a snapshot that is a day or two after the index dates you want. For this example, you might choose the snapshot 20210809-192148.

Next, confirm that the chosen snapshot contains those indices:

GET _snapshot/object-store-repo/20210809-192148?pretty

Check through the list to make sure the logs you're interested in appear.

4) Restore the indices

Restore the indices with new names. If you do not rename them, they can be identified as old logs by the daily backup system and removed. This example removes the date separators and adds a -restore suffix.

POST _snapshot/object-store-repo/20210809-192148/_restore
{
  "indices": "example_index_2021.08_06",
  "index_settings": { "index.number_of_replicas": 1 },
  "rename_pattern": "example_index_2021.08_06",
  "rename_replacement": "example_index_20210806-restore"
}

POST _snapshot/object-store-repo/20210809-192148/_restore
{
  "indices": "example_index_2021.08_07",
  "index_settings": { "index.number_of_replicas": 1 },
  "rename_pattern": "example_index_2021.08_07",
  "rename_replacement": "example_index_20210807-restore"
}

Upon running the above command, the selected indices will be restored and renamed.

5) Verify the indices have been restored

Depending on the size of the indices, it may take a few minutes to an hour to fully restore. Check the state of the indices:

GET _cat/indices?v&s=index:asc

You should see example_index_20210806-restore and example_index_20210807-restore. Wait for the restored indices to become green before relying on them.

6) Prove the restored data is usable

Before treating the restore as complete:

  1. Confirm the restored index names are present.
  2. Confirm the restored indices are green.
  3. Search the restored index names in OpenSearch Dashboards.
  4. Confirm the restored records include the log time range you expected.
7) Clean up restored indices when finished

When you are finished with the restored data, delete only the restored indices. Before deleting, confirm the index names have the -restore suffix and are not the original source indices.

DELETE example_index_20210806-restore
DELETE example_index_20210807-restore
Large restore requests

An example of a large restore request is: "I want to restore all daily indices from Aug 7-21, 2021." In this case, contact #sauron-support for assistance. Restoring large numbers of indices to your current Sauron can cause a number of problems:

  • Capacity: Your environment must be able to hold your current data plus all restored data.
  • Performance: When a lot of data is being restored, it may degrade performance of your OpenSearch cluster.
  • Backups: Logs are routinely backed up on a daily basis, but restored logs are already backed up. Extra restored data can also make the daily backups take longer.

To solve these problems, the Sauron team may create a new read-only Sauron and restore the data there. The read-only Sauron is temporary and will be retired when you are finished using it.

Self-check before escalation

If you run into trouble, check and collect:

  • Sauron OpenSearch endpoint.
  • Component: OpenSearch restore.
  • Indices to restore.
  • Snapshot ID.
  • Authentication method used.
  • Exact error/status from OpenSearch Dashboards Dev Tools.
  • Start time.
  • Current OpenSearch cluster health.
  • Recent retention, backup, or index changes.
  • Checks completed.
  • Production impact, if any.
  • Whether this is a partial or full restore request.
  • Whether you already verified the snapshot contains the requested indices.

Do not paste passwords, bearer tokens, private keys, or other secrets in Slack.