Deleting Old Indices from AWS OpenSearch with Python Curator Package

2023-02-16

🎯 Objectives

  • Delete old data on ES indices using Curator

Prerequisite

  • An AWS account
  • An Opensearch hosted application
  • Create an IAM user specifically for this purpose and copy aws access and secret-keys.

Delete Old Indices on AWS OpenSearch

  1. Click Identity and Access Management (IAM) -> Users -> Add Users -> <specify-username> -> Click Access key - Programmatic access -> Attach existing policies directly -> AmazonOpenSearchServiceFullAccess

To avoid the following error below

1
no permissions for [cluster:monitor/main] and User [name=arn:aws:iam::<AWS_ACCOUNT_ID>:user/<specify-username>, backend_roles=[], requestedTenant=null]
  1. Login to Kibana UI,

Click Security -> Roles -> all_access -> Mapped users -> Manage Mapping, then add your IAM user ARN, to internal user mapping -> Map
Internal User Mapping

  1. Create a virtual enviroment and Install Curator
1
2
3
python3 -m venv env
source venv/bin/activate
pip install elasticsearch-curator
  1. Create a curator configuration file
1
touch curator.yaml

Add the following content below to your curator.yaml file.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
client:
  hosts:
    - "<OpenSearch-name>.<AWS_REGION>.es.amazonaws.com"
  port: <open-search-port-no>
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  aws_key: <AWS_ACCESS_KEY>
  aws_secret_key: <AWS_SECRET_KEY>
  aws_region: <AWS_REGION>
  ssl_no_validate: False
  username: <OpenSearch_username>
  password: <OpenSearch_password>
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile: "info.log"
  logformat: default
  blacklist: ["elasticsearch", "urllib3"]

If you want to add more environment configurations, here is the official documentation

  1. Create an action file for deleting indices

Actions are the tasks which Curator can perform on your indices.

1
2
touch delete_indices.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 10 days (based on index name), for <index-name>
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
      - filtertype: pattern
        kind: prefix
        value: <index-name>-
        exclude:
      - filtertype: age
        source: name
        direction: older
        timestring: "%Y.%m.%d"
        unit: days
        unit_count: 30
        exclude:

If you want to add more environment configurations, here is the official documentation

  1. Run curator
    The command-line arguments are as follows:
1
curator [--config CONFIG.YML] [--dry-run] ACTION_FILE.YML

Note
If --dry-run flag is included, Curator will simulate the action(s) in delete_indices.yaml as closely as possible without actually making any changes. The results will be in the logfile info.log or STDOUT/command-line if no logfile is specified.

Run the following command

1
curator -config curator.yaml delete_indices.yaml

References