如何为Hive多进程配置Log4j2：拆分Metastore与HiveServer2日志

问题背景 #

查阅文档后仍无法理清Log4j2的配置逻辑，需要拆分Hive Metastore和HiveServer2的日志。现有Hive默认Log4j2配置如下：

name = HiveLog4j2

# list of properties
property.hive.log.level = INFO
property.hive.root.logger = DRFA
property.hive.log.dir = ${sys:java.io.tmpdir}/${sys:user.name}
property.hive.log.file = hive.log
property.hive.perflogger.log.level = INFO

# console appender
appender.console.type = Console
appender.console.name = console
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{ISO8601} %5p [%t] %c{2}: %m%n

# daily rolling file appender
appender.DRFA.type = RollingRandomAccessFile
appender.DRFA.name = DRFA
appender.DRFA.fileName = ${sys:hive.log.dir}/${sys:hive.log.file}
# Use %pid in the filePattern to append <process-id>@<host-name> to the filename if you want separate log files for different CLI session
appender.DRFA.filePattern = ${sys:hive.log.dir}/${sys:hive.log.file}.%d{yyyy-MM-dd}
appender.DRFA.layout.type = PatternLayout
appender.DRFA.layout.pattern = %d{ISO8601} %5p [%t] %c{2}: %m%n
appender.DRFA.policies.type = Policies
appender.DRFA.policies.time.type = TimeBasedTriggeringPolicy
appender.DRFA.policies.time.interval = 1
appender.DRFA.policies.time.modulate = true
appender.DRFA.strategy.type = DefaultRolloverStrategy
appender.DRFA.strategy.max = 30

仅找到一份基于Log4j（非Log4j2）的拆分示例：

log4j.appender.hiveserver2=org.apache.log4j.RollingFileAppender
  log4j.appender.hiveserver2.File=/var/log/hive/hiveserver2.log
  log4j.appender.hiveserver2.layout=org.apache.log4j.PatternLayout
  log4j.appender.hiveserver2.layout.ConversionPattern=%d{ISO8601} %-5p [%t] %c{2}: %m%n
  log4j.appender.hiveserver2.MaxFileSize=100MB
  log4j.appender.hiveserver2.MaxBackupIndex=10

  log4j.appender.metastore=org.apache.log4j.RollingFileAppender
  log4j.appender.metastore.File=/var/log/hive/metastore.log
  log4j.appender.metastore.layout=org.apache.log4j.PatternLayout
  log4j.appender.metastore.layout.ConversionPattern=%d{ISO8601} %-5p [%t] %c{2}: %m%n

  log4j.logger.org.apache.hadoop.hive.metastore=INFO,metastore
  log4j.logger.org.apache.hadoop.hive.ql=INFO,hiveserver2

疑问点：

• 是否可以通过定义property.metastore.log.file = metastore.log、property.hiveserver2.log.file = hiveserver2.log，再分别配置对应的Appender？
• 多进程场景下，原配置中appender.DRFA.filename该如何适配？

解决方案 #

1. 自定义属性+独立Appender实现日志拆分 #

完全可以通过自定义属性+独立Appender的方式拆分Metastore和HiveServer2的日志，以下是适配Log4j2的完整配置示例：

name = HiveLog4j2

# 基础属性定义
property.hive.log.level = INFO
property.hive.log.dir = /var/log/hive
property.metastore.log.file = metastore.log
property.hiveserver2.log.file = hiveserver2.log
property.hive.perflogger.log.level = INFO

# Console Appender（可选，保留控制台输出）
appender.console.type = Console
appender.console.name = console
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{ISO8601} %5p [%t] %c{2}: %m%n

# Metastore 专属滚动文件Appender
appender.METASTORE.type = RollingRandomAccessFile
appender.METASTORE.name = METASTORE
appender.METASTORE.fileName = ${hive.log.dir}/${metastore.log.file}
appender.METASTORE.filePattern = ${hive.log.dir}/${metastore.log.file}.%d{yyyy-MM-dd}
appender.METASTORE.layout.type = PatternLayout
appender.METASTORE.layout.pattern = %d{ISO8601} %5p [%t] %c{2}: %m%n
appender.METASTORE.policies.type = Policies
appender.METASTORE.policies.time.type = TimeBasedTriggeringPolicy
appender.METASTORE.policies.time.interval = 1
appender.METASTORE.policies.time.modulate = true
appender.METASTORE.strategy.type = DefaultRolloverStrategy
appender.METASTORE.strategy.max = 30

# HiveServer2 专属滚动文件Appender
appender.HIVESERVER2.type = RollingRandomAccessFile
appender.HIVESERVER2.name = HIVESERVER2
appender.HIVESERVER2.fileName = ${hive.log.dir}/${hiveserver2.log.file}
appender.HIVESERVER2.filePattern = ${hive.log.dir}/${hiveserver2.log.file}.%d{yyyy-MM-dd}
appender.HIVESERVER2.layout.type = PatternLayout
appender.HIVESERVER2.layout.pattern = %d{ISO8601} %5p [%t] %c{2}: %m%n
appender.HIVESERVER2.policies.type = Policies
appender.HIVESERVER2.policies.time.type = TimeBasedTriggeringPolicy
appender.HIVESERVER2.policies.time.interval = 1
appender.HIVESERVER2.policies.time.modulate = true
appender.HIVESERVER2.strategy.type = DefaultRolloverStrategy
appender.HIVESERVER2.strategy.max = 30

# 日志路由配置
logger.metastore.name = org.apache.hadoop.hive.metastore
logger.metastore.level = ${hive.log.level}
logger.metastore.appenderRef.METASTORE.ref = METASTORE
logger.metastore.additivity = false  # 避免日志重复输出到根Appender

logger.hiveserver2.name = org.apache.hadoop.hive.ql
logger.hiveserver2.level = ${hive.log.level}
logger.hiveserver2.appenderRef.HIVESERVER2.ref = HIVESERVER2
logger.hiveserver2.additivity = false

# 根日志配置（可选，处理其他未匹配的日志）
rootLogger.level = ${hive.log.level}
rootLogger.appenderRef.console.ref = console

关键说明：

• 新增METASTORE和HIVESERVER2两个独立Appender，分别绑定各自的日志文件属性
• 通过logger.<name>.additivity = false防止日志同时输出到根Appender，避免重复
• 根日志可保留控制台输出，或者根据需求调整为其他通用日志文件

2. 多进程场景下的Appender适配 #

多进程（如同时启动Metastore、HiveServer2、CLI）场景下，原DRFA的fileName可以通过两种方式适配：

方式一：启动时通过系统属性指定日志文件名

启动Metastore或HiveServer2时，通过-Dhive.log.file参数指定专属日志名：

# 启动Metastore
hive --service metastore -Dhive.log.file=metastore.log

# 启动HiveServer2
hive --service hiveserver2 -Dhive.log.file=hiveserver2.log

这种方式不需要修改Log4j2配置，直接复用原DRFA Appender，每个进程会生成对应名称的日志文件。

方式二：在配置中加入进程ID变量

如果希望自动区分同一服务的多实例进程，可以在fileName和filePattern中加入%pid变量：

appender.DRFA.fileName = ${sys:hive.log.dir}/${sys:hive.log.file}.%pid
appender.DRFA.filePattern = ${sys:hive.log.dir}/${sys:hive.log.file}.%pid.%d{yyyy-MM-dd}

这样每个进程的日志文件会自动带上进程ID，避免多个进程写入同一日志文件导致的混乱。

内容的提问来源于stack exchange，提问作者MarcT