Spark流应用打包后Log4J2 CloudWatch Appender类找不到问题

场景 #

在Spark结构化流应用中使用Log4J2 CloudWatch Appender将日志上传到AWS CloudWatch，本地IntelliJ运行正常，但打包成胖JAR/瘦JAR在EC2实例运行时出现类找不到错误。

报错信息 #

初始报错 #

2024-12-03T07:06:05.291599Z main ERROR Error processing element CloudWatchAppender ([Appenders: null]): CLASS_NOT_FOUND

详细报错 #

2024-12-03 07:23:07,843 main ERROR Unable to locate plugin type for CloudWatchAppender
2024-12-03 07:23:07,890 main WARN Level is ignored when levelAndRefs syntax is used.
2024-12-03 07:23:07,891 main WARN Appender references are ignored when levelAndRefs syntax is used
2024-12-03 07:23:07,916 main ERROR Unable to locate plugin for CloudWatchAppender
2024-12-03 07:23:07,917 main ERROR Unable to invoke factory method in class org.apache.logging.log4j.core.config.AppendersPlugin for element Appenders: java.lang.NullPointerException
java.lang.NullPointerException
at org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.visit(PluginElementVisitor.java:52)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.generateParameters(PluginBuilder.java:264)
at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:137)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1122)
at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:1047)
at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:651)
at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:249)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:295)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:621)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:694)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:711)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:245)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:47)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:176)
at org.apache.logging.log4j.LogManager.getLogger(LogManager.java:666)
at org.apache.logging.log4j.LogManager.getRootLogger(LogManager.java:700)
at org.apache.spark.internal.Logging.initializeLogging(Logging.scala:130)
at org.apache.spark.internal.Logging.initializeLogIfNecessary(Logging.scala:115)
at org.apache.spark.internal.Logging.initializeLogIfNecessary$(Logging.scala:109)
at org.apache.spark.deploy.SparkSubmit.initializeLogIfNecessary(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:83)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

2024-12-03 07:23:07,918 main ERROR Unable to locate appender "STDOUT" for logger config "root"
2024-12-03 07:23:07,919 main ERROR Unable to locate appender "cloudwatch" for logger config "root"
2024-12-03 07:23:07,919 main ERROR Unable to locate appender "consoleLogger" for logger config "sparkapp"
2024-12-03 07:23:07,920 main ERROR Unable to locate appender "CLOUDWATCH" for logger config "sparkapp"

Log4j2配置文件 #

status = debug
name= LoggingConfig

# ConsoleAppender 将日志打印到控制台
appender.console.type = Console
appender.console.name = consoleLogger
appender.console.target = SYSTEM_OUT
appender.console.layout.type = PatternLayout
 
 # 指定日志格式
appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss.SSS} %level [%t] [%c] - %msg%n
 
# CloudWatch日志配置
appender.cloudwatch.type= CloudWatchAppender
appender.cloudwatch.name = CLOUDWATCH
appender.cloudwatch.layout.type = PatternLayout
appender.cloudwatch.layout.pattern = %d{yyyy-MM-dd HH:mm:ss.SSS} %level [%t] [%c] - %msg%n
appender.cloudwatch.logGroup= sparklog
appender.cloudwatch.logStream= cloudwatch-{date}-{hostname}-${awslogs:pid}


# 指定包下的类使用ConsoleAppender和CLOUDWATCH
logger.app.name = sparkapp
logger.app.level = info
logger.app.additivity = false
logger.app.appenderRef.console.ref = consoleLogger
logger.app.appenderRef.cloudwatch.ref = CLOUDWATCH
 
 
# 配置根日志记录器，处理其他包的错误日志
rootLogger.level = info
rootLogger.additivity = false
rootLogger.appenderRef.console.ref = consoleLogger
rootLogger.appenderRef.console.level = error
rootLogger.appenderRef.cloudwatch.ref = CLOUDWATCH

依赖配置（Gradle） #

dependencies {
    // Logging
    implementation 'org.apache.logging.log4j:log4j-core:2.23.1'
    implementation 'org.apache.logging.log4j:log4j-api:2.23.1'

    // AWS CloudWatch日志依赖
    implementation 'com.amazonaws:aws-java-sdk-logs:1.12.765'
    implementation 'software.amazon.awssdk:cloudwatchlogs:2.26.30'

    // Log4J2 CloudWatch Appender依赖
    implementation 'com.kdgregory.logging:log4j2-aws-appenders:3.2.1'
    implementation 'com.kdgregory.logging:aws-facade-v2:3.2.1'
    implementation 'com.kdgregory.logging:logwriters:3.2.1'
}

解决方法 #

1. 确保Log4J2插件元数据正确打包 #

Log4J2的自定义Appender通过Plugin注解注册，打包时必须包含META-INF/org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat文件：

• 用Gradle Shadow插件时，添加配置：

shadowJar {
    mergeServiceFiles()
    append 'META-INF/org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat'
}

• 用Maven Shade插件时，添加Transformer：

<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/>
<transformer implementation="org.apache.maven.plugins.shade.resource.Log4j2PluginsDatTransformer"/>

2. 瘦JAR模式下传递依赖 #

如果使用瘦JAR，Spark运行时需要拉取相关依赖，可通过spark-submit的--packages参数指定：

spark-submit --packages com.kdgregory.logging:log4j2-aws-appenders:3.2.1,com.kdgregory.logging:aws-facade-v2:3.2.1,com.kdgregory.logging:logwriters:3.2.1 ...

也可以将依赖包放到Spark的jars目录下。

3. 修正Log4j2配置冗余项 #

原配置中appender.cloudwatch= com.kdgregory.log4j2.aws.CloudWatchAppender是冗余的，Log4J2通过type字段识别插件类型，删除该行即可避免解析冲突。

4. 验证JAR包内容 #

解压打包后的JAR，检查以下内容是否存在：

• com/kdgregory/log4j2/aws/CloudWatchAppender.class
• META-INF/org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat（确认文件内包含CloudWatchAppender的注册信息）

内容的提问来源于stack exchange，提问作者Karthik