Decrease the disk usage of data nodes

On the Elasticsearch Service panel, click the name of your deployment.

In the list of all your indices, click the Replicas column twice to sort the indices based on their number of replicas starting with the one that has the most. Go through the indices and pick one by one the index with the least importance and higher number of replicas.

For each index you chose, click on its name, then on the panel that appears click Edit settings, reduce the value of the index.number_of_replicas to the desired value and then click Save.

First, retrieve the relevant disk thresholds that will indicate how much space should be released. The relevant thresholds are the high watermark for all the tiers apart from the frozen one and the frozen flood stage watermark for the frozen tier. The following example demonstrates disk shortage in the hot tier, so we will only retrieve the high watermark:

resp = client.cluster.get_settings(
    include_defaults=True,
    filter_path="*.cluster.routing.allocation.disk.watermark.high*",
)
print(resp)

response = client.cluster.get_settings(
  include_defaults: true,
  filter_path: '*.cluster.routing.allocation.disk.watermark.high*'
)
puts response

const response = await client.cluster.getSettings({
  include_defaults: "true",
  filter_path: "*.cluster.routing.allocation.disk.watermark.high*",
});
console.log(response);

GET _cluster/settings?include_defaults&filter_path=*.cluster.routing.allocation.disk.watermark.high*

The response will look like this:

{
  "defaults": {
    "cluster": {
      "routing": {
        "allocation": {
          "disk": {
            "watermark": {
              "high": "90%",
              "high.max_headroom": "150GB"
            }
          }
        }
      }
    }
  }
}

The above means that in order to resolve the disk shortage we need to either drop our disk usage below the 90% or have more than 150GB available, read more on how this threshold works here.

The next step is to find out the current disk usage; this will indicate how much space should be freed. For simplicity, our example has one node, but you can apply the same for every node over the relevant threshold.

resp = client.cat.allocation(
    v=True,
    s="disk.avail",
    h="node,disk.percent,disk.avail,disk.total,disk.used,disk.indices,shards",
)
print(resp)

response = client.cat.allocation(
  v: true,
  s: 'disk.avail',
  h: 'node,disk.percent,disk.avail,disk.total,disk.used,disk.indices,shards'
)
puts response

const response = await client.cat.allocation({
  v: "true",
  s: "disk.avail",
  h: "node,disk.percent,disk.avail,disk.total,disk.used,disk.indices,shards",
});
console.log(response);

GET _cat/allocation?v&s=disk.avail&h=node,disk.percent,disk.avail,disk.total,disk.used,disk.indices,shards

The response will look like this:

node                disk.percent disk.avail disk.total disk.used disk.indices shards
instance-0000000000           91     4.6gb       35gb    31.1gb       29.9gb    111

The next step is to list all the indices and choose which replicas to reduce.

resp = client.cat.indices(
    v=True,
    s="rep:desc,pri.store.size:desc",
    h="health,index,pri,rep,store.size,pri.store.size",
)
print(resp)

response = client.cat.indices(
  v: true,
  s: 'rep:desc,pri.store.size:desc',
  h: 'health,index,pri,rep,store.size,pri.store.size'
)
puts response

const response = await client.cat.indices({
  v: "true",
  s: "rep:desc,pri.store.size:desc",
  h: "health,index,pri,rep,store.size,pri.store.size",
});
console.log(response);

GET _cat/indices?v&s=rep:desc,pri.store.size:desc&h=health,index,pri,rep,store.size,pri.store.size

The response will look like:

health index                                                      pri rep store.size pri.store.size
green  my_index                                                     2   3      9.9gb          3.3gb
green  my_other_index                                               2   3      1.8gb        470.3mb
green  search-products                                              2   3    278.5kb         69.6kb
green  logs-000001                                                  1   0      7.7gb          7.7gb

In the list above we see that if we reduce the replicas to 1 of the indices my_index and my_other_index we will release the required disk space. It is not necessary to reduce the replicas of search-products and logs-000001 does not have any replicas anyway. Reduce the replicas of one or more indices with the index update settings API:

resp = client.indices.put_settings(
    index="my_index,my_other_index",
    settings={
        "index.number_of_replicas": 1
    },
)
print(resp)

response = client.indices.put_settings(
  index: 'my_index,my_other_index',
  body: {
    'index.number_of_replicas' => 1
  }
)
puts response

const response = await client.indices.putSettings({
  index: "my_index,my_other_index",
  settings: {
    "index.number_of_replicas": 1,
  },
});
console.log(response);

PUT my_index,my_other_index/_settings
{
  "index.number_of_replicas": 1
}