Download OpenSearch Data

Use this page when you need to export OpenSearch indices from Sauron to local files. Elasticdump is a tool that can be used for moving and saving indices. It can be used to download data from, or load data to, an OpenSearch instance.

Reference: https://www.npmjs.com/package/elasticdump

If you need to restore older indices from Sauron backups in OCI Object Storage, start with Restore OpenSearch Indices instead.

When should I use this?
Task Recommended path
Download logs from one or more OpenSearch indices to local files Use Elasticdump with the index pattern you verified in OpenSearch Dashboards.
Move data between OpenSearch and local files Use Elasticdump dump or load direction as described below.
Restore a few older indices from Sauron backups Use Restore OpenSearch Indices.
Restore many or very large indices Use the restore page guidance and contact #sauron-support when Sauron-team help is required.
Before you run Elasticdump

Before downloading data:

  1. Confirm the index names or index pattern in OpenSearch Dashboards.
  2. Confirm your local host has enough free disk space for the exported data.
  3. Confirm the OpenSearch authentication method and credential you will use.
  4. Decide whether you are dumping a small set of indices or a large/high-volume set.
  5. Keep the command, index pattern, output path, and dump log for validation.

Do not paste OpenSearch passwords, bearer tokens, private keys, or generated credentials in Slack.

Notes and recommendations
  • When dumping data, be sure the target has sufficient storage space.
  • Similarly, when restoring data to an OpenSearch index, be sure that there are a sufficient number of data nodes and shards. Otherwise, data will load extremely slowly.
  • If you get an error: Error Emitted => unable to verify the first certificate, you need to set an environment variable NODE_TLS_REJECT_UNAUTHORIZED=0.
  • Elasticdump has a built-in --fsCompress option for compressing the data with gzip. In testing, it seemed faster to dump the data in a raw format and then just use pigz (or the equivalent) to compress it.
  • Dumping indices in parallel did not seem to work as well for indices with large numbers of documents (ex: over 100 million). Usually, all but one of the indices would error out when dumping. It's possible that one or more of Elasticdump's knobs could be adjusted to keep this from happening / increase performance. Unfortunately, Elasticdump's documentation does not provide much guidance for best performance with very large indices.
Sample Script

Here's a simple Python script which uses multielasticdump to download a set of indices in parallel. The amount of time it takes is dependent on the number of indices and the number of docs in each index. You may need to experiment a bit to find the proper settings for your environment.

Description of selected parameters:

  • --direction: defaults to "dump"; set to "load" if restoring data to an OpenSearch instance
  • --input: source environment (can be an OpenSearch URL or local files)
  • --output: target environment (can be an OpenSearch URL or local files)
  • --match: regex to match indices
  • --noRefresh: disable input refresh; recommended for large indices.
  • --limit: number of docs to move in each batch
  • --parallel: number of forks to run simultaneously.
  • --intervalCap: max requests within a concurrency interval.
  • --scrollTime: dumps will be resumed if scrollTime has not expired
  • --ignoreChildError: operation can continue if child throws an error
  • --prefix: add a prefix to the index being created
  • --suffix: allows you to add a suffix to the index being created

Use the following syntax for HTTP basic authentication: https://replace-with-opensearch-user:replace-with-opensearch-password@elasticsearch.stage.apatil.developers.oracledx.com

Please follow the link above for more information.

#!/usr/local/bin/python
# Helper script to dump OpenSearch data to a local host in raw json format
import os, time

DUMP_PATH   = "/path/to/node_modules/elasticdump/bin"
WORKING_DIR = "/path/to/work/directory"
DUMP_LOG    = WORKING_DIR + "/dump.log"

######################################
# INPUT
######################################
ENV     = "elasticsearch.stage.apatil.developers.oracledx.com"
USER    = "replace-with-opensearch-user"
PWD     = "replace-with-opensearch-password"
INDICES = "^.*myIndex-2019.11.*$"

INPUT    = "https://%s:%s@%s" % (USER, PWD, ENV)

######################################
OUTPUT   = "%s/output" % WORKING_DIR
######################################

if not os.path.isdir(OUTPUT):
  os.mkdir(OUTPUT)

CMD = "%s/multielasticdump --direction=dump --input=%s --output=%s --match=\'%s\' --noRefresh --limit 5000 --parallel=6 --intervalCap=25 --scrollTime=60m --ignoreChildError &> %s" % (DUMP_PATH, INPUT, OUTPUT, INDICES, DUMP_LOG)

# Environment variable required by elasticdump
os.environ["NODE_TLS_REJECT_UNAUTHORIZED"] = "0"

start = time.time()
os.system(CMD)
elapsed = time.time() - start

SEPARATOR = "============================"
log = open(DUMP_LOG, "a")
log.write(SEPARATOR + "\nCommand executed:\n" + CMD + "\n\n")
log.write(SEPARATOR + "\nDump completed.\n")
log.write("Elapsed time: "+str(elapsed)+" seconds\n" + SEPARATOR)
Prove it worked

After the dump finishes:

  1. Check the dump log for errors.
  2. Confirm the output directory contains files for the expected indices.
  3. Spot-check the dumped data before deleting or moving anything.
  4. Keep the index pattern, command, output path, and elapsed time with your records.
Uploading Data to OpenSearch

The script above is easily modified to load data back to an OpenSearch instance. Here's a sample command to upload data to a running OpenSearch instance:

multielasticdump --direction=load \
  --input=/path/to/dumped/data \
  --output=https://replace-with-opensearch-user:replace-with-opensearch-password@elasticsearch.stage.apatil.developers.oracledx.com \
  --match='^.*myIndex-2019.11.*$' \
  --limit=5000 \
  --parallel=6 \
  --intervalCap=25
Self-check before escalation

Most download questions can be resolved by checking the index pattern, credentials, local storage, and dump log. If Elasticdump does not behave as expected, check and collect:

  • Sauron OpenSearch endpoint.
  • Index names or index pattern.
  • Direction: dump or load.
  • Output path.
  • Local disk space available.
  • Authentication method used.
  • Exact Elasticdump command, with secrets removed.
  • Dump log excerpt, with secrets removed.
  • Start time and elapsed time.
  • Recent credential, index, retention, or network changes.
  • Checks completed.
  • Production impact, if any.

Do not paste OpenSearch passwords, bearer tokens, private keys, or other secrets in Slack.