Logstash性能调优咨询：连接Elasticsearch与ntopng时丢包问题

Hey there, let's dig into fixing that traffic loss issue you're seeing with Logstash acting as the middle layer between ntopng and Elasticsearch. Your 20GB RAM/8-core server has plenty of hardware headroom, so tweaking Logstash's core configuration in logstash.yml (and a few critical JVM settings) should get things back on track. Here's a breakdown of the key parameters to adjust:

1. Pipeline Parallelism: pipeline.workers #

This controls the number of worker threads that process your event pipeline. As a general rule, set this to 1-2 times your CPU core count to leverage parallel processing. For your 8-core server, start with:

pipeline.workers: 8

If you still see bottlenecks after testing, bump it up to 16—just make sure you monitor CPU usage to avoid overloading your server.

2. Batch Processing: pipeline.batch.size & pipeline.batch.delay #

Logstash processes events in batches to reduce overhead. Tweaking these two parameters can significantly boost throughput:

• pipeline.batch.size: The number of events each worker processes in one go. Default is 125; try increasing it to 200-500 to cut down on thread-switching costs. Example:

  pipeline.batch.size: 500
  

  Note: Don't set this too high—larger batches consume more heap memory, which can trigger GC pauses or out-of-memory errors if paired with insufficient JVM heap.
• pipeline.batch.delay: The time (in ms) Logstash waits to fill a batch before processing it. If you have steady high traffic, lower this to 2-5ms to process events faster. If traffic is bursty, keep it at the default 5ms or raise to 10ms to ensure batches are full:

  pipeline.batch.delay: 5
  

3. Persistent Queue: queue.type & Related Settings #

By default, Logstash uses an in-memory queue, which can lose data if Logstash crashes or runs out of memory. Switching to a persistent disk queue prevents traffic loss during peaks or restarts:

queue.type: persisted
queue.max_bytes: 10gb  # Allocate 10GB of disk space for the queue (adjust based on your disk capacity)
queue.checkpoint.writes: 1000  # Write a checkpoint to disk every 1000 events to ensure data durability

Make sure the directory specified in path.data is on a fast SSD—slow disk IO will negate the benefits of a persistent queue.

4. Critical JVM Tuning (Not in logstash.yml, but Essential) #

Logstash runs on the JVM, so misconfigured heap memory is a common performance killer. Edit the config/jvm.options file:

• Set Xms and Xmx to 1/2 to 1/3 of your physical RAM (for your 20GB server, 10GB is ideal):

  -Xms10g
  -Xmx10g
  

  Avoid setting this above 16GB—larger heaps lead to longer garbage collection pauses, which disrupt event processing.

Additional Tips #

• Monitor Logstash Metrics: Use the built-in monitoring API (http://<logstash-host>:9600/_node/stats) or Elastic Stack Monitoring to track throughput, queue backlog, GC activity, and worker utilization. This will help you fine-tune parameters further.
• Check Elasticsearch Performance: Sometimes traffic loss isn't Logstash's fault—ensure ES has enough resources, its bulk ingest settings are optimized (e.g., bulk.max_bytes), and disk IO isn't a bottleneck.
• Simplify Filter Logic: If your pipeline has heavy filter processing, streamline it where possible (e.g., use efficient grok patterns, avoid unnecessary lookups) to reduce worker load.

内容的提问来源于stack exchange，提问作者張皓翔