为何Elasticsearch部分索引事务日志过大？如何预防？

Hey there, let's dive into why those specific Logstash indices have such massive transaction logs (translogs) in your 3-node Elasticsearch cluster, and what you can do to prevent this from happening again.

First, let's recall what the translog does: it's Elasticsearch's safety net to ensure no data is lost between flushes (when in-memory data is written to disk as segments). Normally, ES flushes the translog automatically when it hits a size threshold (default 512MB) or after a set interval (default 5 seconds). Here are the most likely culprits for your oversized translogs:

• Misconfigured flush thresholds: If someone modified the index.translog.flush_threshold_size setting for these Logstash indices (or the template they use) to an unusually large value (like 200GB), ES will wait until the translog hits that size before flushing. That's exactly what it looks like here—your indices have translogs over 150GB, which way exceeds the default.
• Disk IO bottlenecks: If your cluster's storage is slow (e.g., spinning disks instead of SSDs), the flush operation can get delayed. ES can't clear the translog until the flush completes, so the log keeps growing as new writes come in.
• Unusually high write volume: Those indices have tens of millions of operations—maybe there was a sudden spike in log volume on those days (e.g., a system event, increased traffic, or debug logging enabled). If the write rate outpaces ES's ability to flush the translog, it will balloon.
• Resource constraints: If your data nodes are starved for CPU or memory, the threads responsible for flushing translogs might not run in time. This delays the cleanup of old translog data, leading to accumulation.
• Missing or infrequent snapshots: Elasticsearch retains old translog files to support recovery if a node goes down. If you're not taking regular snapshots of your indices, ES might hold onto more translog data than necessary to ensure it can recover shards.

Now, let's talk about how to keep translogs in check going forward:

• Reset flush thresholds to reasonable values: For Logstash-style time-series indices (which are write-heavy but don't need extremely large translogs), stick close to the default index.translog.flush_threshold_size (512MB) or adjust it slightly based on your write rate (e.g., 1GB-5GB). Avoid setting it to tens or hundreds of GB—this increases recovery time if a node fails, and risks data loss if the node crashes before a flush.
  • You can update existing indices with this command:

    PUT /logstash-2018.05.01/_settings
    {
      "index.translog.flush_threshold_size": "1GB"
    }
    

  • For future indices, update your Logstash index template to include this setting so all new daily indices inherit it automatically.
• Upgrade to SSD storage: Translog flushes are disk-intensive operations. SSDs drastically reduce the time it takes to flush data to disk, ensuring ES can clear old translog files quickly even under high write load.
• Monitor translog metrics proactively: Keep an eye on translog size and flush frequency using Elasticsearch's built-in stats:
  • Check translog sizes for all indices at a glance: GET /_cat/indices?v&h=index,translog_size
  • Get detailed translog stats across the cluster: GET /_stats/translog
  • Set up alerts in Kibana (since you're using the ELK stack) to notify you when translog sizes exceed a safe threshold (e.g., 10GB).
• Implement Index Lifecycle Management (ILM): For time-series logs, use ILM to automate rolling over old indices to read-only state, shrink them, or archive them to cheaper storage. Read-only indices don't have active writes, so their translogs won't grow anymore. This also reduces the load on your active data nodes.
• Optimize write operations:
  • Use larger bulk requests in Logstash (adjust the batch_size and batch_delay settings) to reduce the number of small, frequent writes. This makes translog operations more efficient and cuts down on unnecessary flushes.
  • Avoid overloading your cluster with too many concurrent write requests. Tune Logstash's pipeline workers to match your cluster's capacity.
• Ensure sufficient node resources: Make sure your data nodes have enough CPU and memory to handle write operations and flush tasks. For a 3-node cluster handling log data, each node should have at least 8GB of heap (with the rest allocated to the OS for disk caching) and multi-core CPUs.
• Take regular snapshots: Schedule daily snapshots of your indices to a reliable storage location. This tells Elasticsearch it can safely clean up old translog files since it has a backup to recover from if needed.

内容的提问来源于stack exchange，提问作者Nick Peirson