Kafka 3.8.0中如何配置Log4j实现消费者日志记录？

我在Kafka 3.8.0中尝试通过Log4j开启Producer和Consumer的日志记录，目的是留存新生产者或消费者连接时的日志。目前通过环境变量KAFKA_LOG4J_OPTS覆盖默认的log4j.properties并指定了新路径，能正常看到Broker日志，但看不到Kafka Consumer相关的日志。

当前配置信息 #

环境变量 #

KAFKA_LOG4J_OPTS: "-Dlog4j.configuration=file:/etc/kafka/log4j.properties"  
KAFKA_LOG_DIRS: /var/lib/kafka/data

挂载卷 #

• ./etc/kafka/log4j.properties:/etc/kafka/log4j.properties
• kafka-log:/var/log/kafka

自定义log4j.properties配置 #

log4j.rootLogger=INFO, stdout, kafkaAppender
……

# 调整Broker日志级别（输出到server.log和stdout）
log4j.logger.kafka=INFO
log4j.logger.org.apache.kafka=INFO

# 开启请求日志需改为DEBUG或TRACE
log4j.logger.kafka.request.logger=DEBUG, requestAppender
log4j.additivity.kafka.request.logger=true

# 取消注释并将log4j.logger.kafka.network.RequestChannel$设为TRACE可查看请求处理细节
# log4j.logger.kafka.network.Processor=TRACE, requestAppender
# log4j.logger.kafka.server.KafkaApis=TRACE, requestAppender
# log4j.additivity.kafka.server.KafkaApis=true
# log4j.logger.kafka.network.RequestChannel$=TRACE, requestAppender
# log4j.additivity.kafka.network.RequestChannel$=false


# 尝试开启生产者/消费者组日志
log4j.logger.org.apache.kafka.common.network.Selector=INFO
log4j.logger.org.apache.kafka.clients.producer.KafkaProducer=INFO
log4j.logger.org.apache.kafka.common.requests.ProduceRequest=INFO
log4j.logger.org.apache.kafka.clients.producer=INFO
log4j.logger.org.apache.kafka.clients.consumer=INFO
log4j.logger.kafka.coordinator.group=INFO
log4j.logger.kafka.network.ConnectionHandler=INFO
log4j.logger.kafka.server.KafkaRequestHandler=INFO

解决步骤 #

1. 给消费者Logger绑定输出Appender #

你当前的配置只给消费者相关Logger设置了级别，但没指定输出的Appender，导致日志没有写入文件或stdout。需要给这些Logger绑定Appender，同时关闭additivity避免重复输出。修改log4j.properties中消费者相关配置：

# 消费者分组协调核心日志（捕捉组初始化、重平衡事件）
log4j.logger.kafka.coordinator.group=DEBUG, kafkaAppender
log4j.additivity.kafka.coordinator.group=false

# 客户端消费者操作日志
log4j.logger.org.apache.kafka.clients.consumer=DEBUG, kafkaAppender
log4j.additivity.org.apache.kafka.clients.consumer=false

# 新连接建立日志
log4j.logger.kafka.network.ConnectionHandler=DEBUG, kafkaAppender
log4j.additivity.kafka.network.ConnectionHandler=false

# 组协调器详细操作日志
log4j.logger.kafka.coordinator.group.GroupCoordinator=DEBUG, kafkaAppender
log4j.additivity.kafka.coordinator.group.GroupCoordinator=false

2. 验证Appender配置有效性 #

确保你的kafkaAppender正确指向了日志目录，默认配置类似如下，如果需要单独的消费者日志文件，可以新增一个专属Appender：

# 现有kafkaAppender配置示例
log4j.appender.kafkaAppender=org.apache.log4j.RollingFileAppender
log4j.appender.kafkaAppender.File=/var/log/kafka/server.log
log4j.appender.kafkaAppender.MaxFileSize=100MB
log4j.appender.kafkaAppender.MaxBackupIndex=10
log4j.appender.kafkaAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.kafkaAppender.layout.ConversionPattern=[%d] %p %m (%c)%n

# 新增消费者专属Appender（可选）
log4j.appender.consumerAppender=org.apache.log4j.RollingFileAppender
log4j.appender.consumerAppender.File=/var/log/kafka/consumer.log
log4j.appender.consumerAppender.MaxFileSize=100MB
log4j.appender.consumerAppender.MaxBackupIndex=10
log4j.appender.consumerAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.consumerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n

如果用专属Appender，把消费者Logger绑定到它即可：

log4j.logger.kafka.coordinator.group=DEBUG, consumerAppender
log4j.additivity.kafka.coordinator.group=false

3. 确认Log4j配置加载成功 #

查看Broker启动日志，确认自定义的log4j配置已被加载：

grep "log4j.configuration" /var/log/kafka/server.log

如果输出包含file:/etc/kafka/log4j.properties，说明配置生效。

4. 测试消费者连接 #

启动一个测试消费者，验证日志是否生成：

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-topic --group test-group

此时查看对应日志文件，应该能看到类似以下的连接/组协调日志：

[2024-xx-xx xx:xx:xx] INFO [GroupCoordinator 0]: Preparing to rebalance group test-group with old generation 0 (__consumer_offsets-2) (kafka.coordinator.group.GroupCoordinator)
[2024-xx-xx xx:xx:xx] INFO [GroupCoordinator 0]: Stabilized group test-group generation 1 (__consumer_offsets-2) (kafka.coordinator.group.GroupCoordinator)

内容的提问来源于stack exchange，提问作者shijumax