Policies

Description

Policies are index management configurations that describe:

• the states an index can be in, representing stages of its lifecycle (e.g., hot, warm, delete, etc.)
• the actions to be performed on indexes in a given state (e.g., relocating to cold storage nodes or deleting the index)
• the conditions that must be met for an index to transition from one state to another. For example, if the index age exceeds eight weeks, it should transition to a state where a delete action is defined

In other words, a policy defines the states an index can be in, the actions that need to be performed in each state, and the conditions that must be met to transition from one state to another.

Full freedom of action is provided when developing policies. You can create any state, transition to any other state, and specify any number of actions in each state.

Parameter	Description	Type	Required	Read Only
policy_id	Policy name	string	Yes	Yes
description	Policy description in human-readable format	string	Yes	No
ism_template	Specified ISM template pattern matching the index for policy application	nested list of objects	No	No
last_updated_time	Time of the last policy update	timestamp	Yes	Yes
error_notification	Destination and message template for error notifications	object	No	No
default_state	Default initial state for each index using this policy	string	Yes	No
states	States defined in the policy	nested list of objects	Yes	No

States

A state is a description of the current status of a managed index. A managed index can be in only one state at a time. Each state has associated actions that are executed sequentially upon entering the state, and transitions that are evaluated after all actions are completed.

The following table describes the parameters that can be configured for a state:

Field	Description	Type	Required
name	State name	string	Yes
actions	Actions to be executed upon entering the state. Additional information is described in the "Actions" section.	nested list of objects	Yes
transitions	The next states and the conditions required to transition into them. If there are no transitions, the policy is considered complete and may stop managing the index. Additional information is described in the "Transitions" section.	nested list of objects	Yes

Actions

Actions are the steps that a policy sequentially performs on an index when it enters a specific state.

ISM (Index State Management) executes actions in the order they are defined. For example, if actions [A, B, C, D] are specified, ISM performs action A and then goes into a sleep period, as configured by the cluster setting plugins.index_state_management.job_interval. After the sleep interval, ISM continues executing the remaining actions one by one. However, if ISM fails to successfully perform action A, the operation stops and actions B, C, and D are not executed. In such cases, a retry mechanism is available (described below).

Optionally, a timeout period can be defined for an action. If the timeout is exceeded, the operation is forcibly terminated. For example, if the timeout is set to 1d and the action is not completed within one day (even after retries), the action fails.

Each action can be configured with timeout and retry settings:

Parameter	Description	Type	Required	Default
timeout	Timeout period for the action. Time units are accepted: minutes, hours, and days.	time unit	No	-
retry	Retry configuration for this action	object	No	Action-dependent

The retry operation has the following parameters:

Parameter	Description	Type	Required	Default
count	Number of retry attempts	number	Yes	-
backoff	Type of backoff policy used for retries. Valid values: Exponential, Constant, Linear	string	No	Exponential
delay	Wait time between retry attempts. Time units are accepted: minutes, hours, and days	time unit	No	1 minute

The following example configures an action with a timeout of one hour. The policy retries the action up to three times using exponential backoff, with an initial delay of 10 minutes:

{
  "actions": {
    "timeout": "1h",
    "retry": {
      "count": 3,
      "backoff": "exponential",
      "delay": "10m"
    }
  }
}

Operations supported by ISM

ISM supports the following actions:

• force_merge
• read_only
• read_write
• replica_count
• shrink
• close
• open
• delete
• rollover
• notification
• snapshot
• index_priority
• allocation
• rollup
• Move to ClickHouse

Force Merge

Reduces the number of index segments by merging shard segments. Before starting the merge, ISM attempts to set the index to read-only.

Parameter	Description	Type	Required
max_num_segments	The number of segments to reduce the shard to	number	Yes
wait_for_completion	Whether to wait for the action to complete or return immediately after start	boolean	No
task_execution_timeout	Timeout for task execution. Only applies if wait_for_completion is false	string (time unit)	No

{
  "force_merge": {
    "max_num_segments": 1
  }
}

Read Only

Sets the managed index to read-only mode.

{
  "read_only": {}
}

Set the index.blocks.write parameter to true for the managed index.

warning

This block does not prevent the index from being updated.

Read Write

Sets the index to read-write mode.

Example configuration:

{
  "read_write": {}
}

Replica Count

Sets the number of replicas for the index.

Parameter	Description	Type	Required
number_of_replicas	Defines the number of replicas to assign to the index.	number	Yes

{
  "replica_count": {
    "number_of_replicas": 2
  }
}

Shrink

Allows reducing the number of primary shards in an index. This action supports the following parameters:

Parameter	Description	Type	Required
num_new_shards	The maximum number of primary shards in the shrunken index.	integer	Yes, but cannot be used with max_shard_size or percentage_of_source_shards
max_shard_size	The maximum shard size in bytes for the target index.	string (size unit, example values: 500mb, 1gb)	Yes, but cannot be used with num_new_shards or percentage_of_source_shards
percentage_of_source_shards	The percentage of the source primary shards to shrink. This parameter specifies the minimum percentage to use when shrinking the number of primary shards. Must be in the range from 0.0 to 1.0 (exclusive).	number	Yes, but cannot be used with max_shard_size or num_new_shards
target_index_name_template	The name of the shrunken index. Accepts strings and Mustache variables (e.g., {"source": "_shrunken"}).	string	No
aliases	Aliases to add to the new index.	array<object>	No
force_unsafe	Whether to perform the action even when no replicas are present.	boolean	No

{
  "shrink": {
    "num_new_shards": 1,
    "target_index_name_template": {
      "source": "_shrunken"
    },
    "aliases": [
      {
        "my-alias": {}
      }
    ],
    "force_unsafe": false
  }
}

If aliases need to be assigned, the aliases parameter should contain an array of alias objects. For example:

"aliases": [
  {
    "my-alias": {}
  },
  {
    "my-second-alias": {
      "is_write_index": false,
      "filter": {
        "multi_match": {
          "query": "QUEEN",
          "fields": ["speaker", "text_entry"]
        }
      },
      "index_routing" : "1",
      "search_routing" : "1"
    }
  },
]

Move to ClickHouse

The Move to ClickHouse action allows for the automatic transfer of data from OpenSearch to ClickHouse, providing efficient management of large data volumes and reducing the load on OpenSearch. This action can be configured as part of an ISM policy for specific indices.

Parameters for configuring the "Move to ClickHouse" action

Transfers data to ClickHouse.

Parameter	Description	Type	Required
table_name_template	A regular expression defining the table name in ClickHouse based on the index name. A named capture group in the regex will be used as the table name. The capture group must be named name.	string (e.g., (?<name>.*?)-\d+)	Yes
connection_id	The ClickHouse connection ID configured in the OpenSearch settings.	string	Yes
batch_size	The number of events sent to ClickHouse in a single batch.	integer	No