Logstash无法向Elasticsearch索引事件：分片数达上限求助

Hey there, let's break down how to fix this shard limit issue you're facing. The error message says it all—your Elasticsearch cluster has hit its default maximum shard count (1000), so Logstash can't create new daily Wazuh alerts/archives indices since each new index adds more shards to the cluster. Let's tackle this step by step.

1. Quick Fix: Temporarily Increase Shard Limit #

First, let's get Logstash back up and running right away by raising the cluster's shard limit. Since your cluster is single-node (based on the stats showing total shards = 1000, matching the default single-node limit), we'll adjust the cluster.max_shards_per_node setting:

curl -XPUT "http://206.189.196.214:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster.max_shards_per_node": 2000
  }
}
'

• Using persistent means this change stays after cluster restarts. If you just want a temporary fix for testing, swap persistent with transient.

2. Reduce Existing Shards: Clean Up Old Indices #

Your Wazuh indices are date-based, so old data you don't need can be safely deleted to free up shards.

Delete Specific Old Indices #

For example, delete alerts/archives from 30 days ago:

# Delete 30-day-old alerts index
curl -XDELETE "http://206.189.196.214:9200/wazuh-alerts-3.x-$(date -d "-30 days" +%Y.%m.%d)"

# Delete 30-day-old archives index
curl -XDELETE "http://206.189.196.214:9200/wazuh-archives-3.x-$(date -d "-30 days" +%Y.%m.%d)"

Batch Delete Old Indices (Use with Caution!) #

If you need to clear multiple old indices, use a wildcard (e.g., delete all 2024 May and earlier alerts):

curl -XDELETE "http://206.189.196.214:9200/wazuh-alerts-3.x-2024.0[1-5]*"

3. Long-Term Solution: Automate Index Management #

To avoid hitting this limit again, set up Index Lifecycle Management (ILM) to automatically delete old data, and optimize your index shard settings.

Step 3.1: Create an ILM Retention Policy #

This policy will automatically delete indices after 30 days (adjust the min_age to match your data retention needs):

curl -XPUT "http://206.189.196.214:9200/_ilm/policy/wazuh-retention-policy" -H 'Content-Type: application/json' -d'
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
'

Step 3.2: Apply the Policy to Wazuh Indices #

Create an index template so all new Wazuh alerts/archives indices automatically use this ILM policy, plus optimize shard settings (disable replicas since this is a single-node cluster):

curl -XPUT "http://206.189.196.214:9200/_index_template/wazuh-index-template" -H 'Content-Type: application/json' -d'
{
  "index_patterns": ["wazuh-alerts-3.x-*", "wazuh-archives-3.x-*"],
  "template": {
    "settings": {
      "index.number_of_replicas": 0,
      "index.lifecycle.name": "wazuh-retention-policy"
    }
  }
}
'

Step 3.3: Optimize Existing Indices #

Update your existing Wazuh indices to disable replicas (saves shard count since replicas don't serve a purpose on a single node):

# Disable replicas for all alerts indices
curl -XPUT "http://206.189.196.214:9200/wazuh-alerts-3.x-*/_settings" -H 'Content-Type: application/json' -d'
{
  "index.number_of_replicas": 0
}
'

# Disable replicas for all archives indices
curl -XPUT "http://206.189.196.214:9200/wazuh-archives-3.x-*/_settings" -H 'Content-Type: application/json' -d'
{
  "index.number_of_replicas": 0
}
'

Final Notes #

After making these changes, check your cluster shard count again to confirm it's below the new limit:

curl -XGET "http://206.189.196.214:9200/_cluster/stats?filter_path=indices.shards.total"

You should now see Logstash able to write to Elasticsearch without hitting the 400 error. The ILM policy will keep your shard count in check automatically going forward.

内容的提问来源于stack exchange，提问作者Deb