ELK堆内存不足及稳定性问题咨询（附集群配置详情）

Hey Abhishek, I see you're dealing with heap memory shortages and stability issues in your ELK 6.2.4 cluster—those sky-high shard counts (nearly 30k total!) are definitely putting heavy strain on things. Let's walk through each of your questions with practical, beginner-friendly steps:

1. Reducing Shard Count for Old Indices via Reindexing #

Since your 2019 data is already closed, reindexing is the safest way to shrink their shard counts without losing data. Here's how to do it:

• Step 1: Plan your target shard size
  A good rule of thumb is to keep shards between 10-50GB (adjust based on your data density). For example, if a closed 2019 index is 80GB, 2 shards would be ideal.
• Step 2: Create a new target index with fewer shards
  First, grab the mapping of your old index with GET /old-index-2019-01/_mapping, then use it to create a new index with your desired shard count:

  PUT /new-old-index-2019-01
  {
    "settings": {
      "number_of_shards": 2,  # Your smaller target shard count
      "number_of_replicas": 0  # Disable replicas temporarily to speed up reindex
    },
    "mappings": {
      "_doc": {
        # Paste the exact properties from your old index's mapping here
      }
    }
  }
  

• Step 3: Run the reindex command
  Copy data from the old closed index to the new one:

  POST _reindex
  {
    "source": {
      "index": "old-index-2019-01"
    },
    "dest": {
      "index": "new-old-index-2019-01"
    }
  }
  

• Step 4: Verify and clean up
  Confirm all data transferred with GET /new-old-index-2019-01/_count. Once verified, delete the original old index, then re-enable replicas on the new index if needed:

  PUT /new-old-index-2019-01/_settings
  {
    "number_of_replicas": 1
  }
  

  Pro tip: Run this during low-traffic hours to avoid impacting your live cluster.

2. Archiving Old Indices and Restoring Them On-Demand #

ELK's snapshot functionality is the best way to archive data long-term while keeping it accessible. Here's the process:

• Step 1: Configure a snapshot repository
  First, create a directory on all cluster nodes (or use cloud storage like S3) and update your elasticsearch.yml to allow it:

  path.repo: ["/path/to/your/archive/directory"]
  

  Restart Elasticsearch, then register the repository:

  PUT _snapshot/elk-archive-repo
  {
    "type": "fs",
    "settings": {
      "location": "/path/to/your/archive/directory",
      "compress": true
    }
  }
  

• Step 2: Create a snapshot of your old indices
  Archive all closed 2019 indices with this command:

  PUT _snapshot/elk-archive-repo/2019-data-snapshot
  {
    "indices": "old-index-2019-*",  # Match your 2019 index pattern
    "ignore_unavailable": true,
    "include_global_state": false
  }
  

  Check the snapshot status with GET _snapshot/elk-archive-repo/2019-data-snapshot/_status.
• Step 3: Restore from the archive when needed
  If you need to access archived data later, restore it to a temporary index to avoid cluttering your live cluster:

  POST _snapshot/elk-archive-repo/2019-data-snapshot/_restore
  {
    "indices": "old-index-2019-01",
    "rename_pattern": "old-index-(.+)",
    "rename_replacement": "restored-old-index-$1"
  }
  

  After you're done using it, delete the restored index to free up resources.

3. Switching from Daily to Weekly/Monthly Index Rollover (and Why It Helps) #

Changing your index rotation frequency will directly slash your total shard count—here's how to set it up:

• Step 1: Update your index template
  Your current daily rollover is controlled by an index template. Grab your existing template with GET _template/your-daily-template, then modify it for weekly/monthly rotation:

  PUT _template/weekly-index-template
  {
    "index_patterns": ["logs-*-weekly-*"],  # Use a weekly pattern like `logs-%{+YYYY.ww}`
    "settings": {
      "number_of_shards": 3,  # Align with your data node count
      "number_of_replicas": 1,
      "index.lifecycle.name": "weekly-rollover-policy"  # Attach a new lifecycle policy
    },
    "mappings": {
      # Keep your existing mappings here
    }
  }
  

• Step 2: Create an ILM policy for rollover
  For ELK 6.2.4, Index Lifecycle Management (ILM) automates rollover. Create a policy that triggers rollover after 7 days (weekly) or 30 days (monthly):

  PUT _ilm/policy/weekly-rollover-policy
  {
    "policy": {
      "phases": {
        "hot": {
          "actions": {
            "rollover": {
              "max_age": "7d"  # Change to "30d" for monthly rollover
            }
          }
        },
        "warm": {
          "min_age": "30d",
          "actions": {
            "shrink": {
              "number_of_shards": 1
            },
            "forcemerge": {
              "max_num_segments": 1
            }
          }
        },
        "delete": {
          "min_age": "180d",
          "actions": {
            "delete": {}
          }
        }
      }
    }
  }
  

• Step 3: Update your data pipeline
  If you're using Logstash or Beats, update their output configurations to use the new index pattern (e.g., logs-%{+YYYY.ww} for weekly, logs-%{+YYYY.MM} for monthly).

Why This Adjustment Helps: #

• Cuts total shard count: Instead of 365 indices per year (daily), you'll have 52 (weekly) or 12 (monthly)—drastically reducing cluster metadata overhead.
• Eases heap pressure: Fewer shards mean less memory used for shard metadata, directly addressing your heap shortage issues.
• Simplifies management: Fewer indices are easier to monitor, archive, and maintain long-term.

内容的提问来源于stack exchange，提问作者Abhishek