如何优化Elasticsearch日志性能？log4j2异步日志配置指引

Absolutely, Async Logger is perfectly suited for your scenario—here's how to configure it in Elasticsearch's log4j2.properties and why it’ll cut down on log write latency during bulk ingest or high-concurrency search operations:

Why Async Logger Works for Your Use Case #

Synchronous logging forces your ingest/search threads to wait for log events to be written to disk (an IO-bound operation) before resuming work. Async Logger offloads this work to dedicated background threads: log events are added to a fast in-memory ring buffer, and your business threads can immediately return to processing requests. This is a game-changer for bulk ingest or multi-request search scenarios where log volume is high and latency matters.

Step-by-Step Configuration in log4j2.properties #

Elasticsearch uses Log4j2 under the hood, so you’ll modify the log4j2.properties file in your ES config directory. You have two configuration options: global async logging, or targeted async logging for just your plugin.

Option 1: Global Async Logging (All Logs)

If you want all Elasticsearch logs (including your plugin’s) to be processed asynchronously:

1. Keep your existing appender configurations (like rolling file appenders) intact—you just need to enable async behavior on the root logger.
2. Add/update the root logger section to include async = true:

   # Existing appender example (keep your own layout/policy settings)
   appender.rolling.type = RollingFile
   appender.rolling.name = rolling
   appender.rolling.fileName = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}.log
   appender.rolling.layout.type = PatternLayout
   appender.rolling.layout.pattern = [%d{ISO8601}] [%t] [%p] [%c] %m%n
   # ... (keep your rolling policy/trigger strategy settings)
   
   # Enable async for root logger
   rootLogger.level = info
   rootLogger.appenderRef.rolling.ref = rolling
   rootLogger.async = true
   

Option 2: Targeted Async Logging (Only Your Plugin)

If you want to limit async processing to just your plugin’s logs (to avoid altering core ES logging behavior):

1. Define a logger for your plugin’s package, and add async = true to it:

   # Replace with your plugin's actual root package
   logger.yourplugin.name = com.yourcompany.elasticsearch.plugin
   logger.yourplugin.level = info
   logger.yourplugin.appenderRef.rolling.ref = rolling
   logger.yourplugin.async = true
   

Optimize Async Logger Performance #

Tweak these optional settings to get the most out of async logging:

• Adjust ring buffer size: The default buffer size is 262144 events, but you can increase it for ultra-high concurrency to avoid event loss (don’t exceed your JVM’s available memory):

  AsyncLoggerConfig.RingBufferSize = 524288
  

• Disable location tracking: If you don’t need line numbers in your logs, turn this off to reduce overhead:

  AsyncLoggerConfig.includeLocation = false
  

• Avoid immediate flush: Ensure your appenders don’t have immediateFlush=true—this negates async benefits by forcing synchronous disk writes.

Verify Async Logging is Working #

• Check for Log4j2’s dedicated async threads using jstack—look for threads named like AsyncAppender-Dispatcher-rolling (matching your appender name).
• Monitor your ingest/search latency metrics—you should see a noticeable drop in request processing time once async logging is enabled.

Key Notes #

• Elasticsearch’s default Log4j2 setup includes the necessary dependencies for Async Logger (LMAX Disruptor), so no extra installation is needed.
• All built-in ES appenders (like RollingFile) are thread-safe, so they work seamlessly with async logging.

内容的提问来源于stack exchange，提问作者Galet