Spark 3.3+无法使用胖包内置Log4j2插件动态重配置日志

问题核心原因 #

Spark的Driver类加载机制与普通Java应用不同，它会优先加载自身依赖库，导致胖包中嵌入的Log4j2插件（如log4j-layout-template-json）的元数据未被正确识别，进而无法加载JsonTemplateLayout等插件类型。以下是无需修改集群部署流程的解决办法：

1. 调整Maven Shade插件配置，确保Log4j2插件元数据合并 #

Log4j2通过META-INF/org/apache/logging/log4j/core/config/plugins/下的Log4j2Plugins.dat和Log4j2Plugins.xml文件识别插件，打包时必须合并这些文件而非覆盖。修改Shade插件配置如下：

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>3.4.1</version>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                <transformers>
                    <!-- 处理Log4j2插件缓存 -->
                    <transformer implementation="org.apache.maven.plugins.shade.resource.Log4j2PluginsCacheTransformer"/>
                    <!-- 合并插件元数据文件 -->
                    <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                        <resource>META-INF/org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat</resource>
                    </transformer>
                    <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                        <resource>META-INF/org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.xml</resource>
                    </transformer>
                    <!-- 设置主类（按需添加） -->
                    <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                        <mainClass>com.yourcompany.YourMainClass</mainClass>
                    </transformer>
                </transformers>
            </configuration>
        </execution>
    </executions>
    <dependencies>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
            <version>你的Log4j2版本</version>
        </dependency>
    </dependencies>
</plugin>

2. Spring Boot打包场景的配置调整 #

若使用Spring Boot插件打包，需确保Log4j2插件元数据不被过滤，并添加Log4j2专用Transformer：

<plugin>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-maven-plugin</artifactId>
    <version>你的Spring Boot版本</version>
    <executions>
        <execution>
            <goals>
                <goal>repackage</goal>
            </goals>
            <configuration>
                <transformers>
                    <transformer implementation="org.apache.maven.plugins.shade.resource.Log4j2PluginsCacheTransformer"/>
                    <transformer implementation="org.springframework.boot.maven.PropertiesMergingResourceTransformer"/>
                    <transformer implementation="org.springframework.boot.maven.ManifestResourceTransformer"/>
                    <transformer implementation="org.springframework.boot.maven.MainClassResourceTransformer"/>
                </transformers>
                <!-- 强制包含Json模板布局插件 -->
                <includes>
                    <include>
                        <groupId>org.apache.logging.log4j</groupId>
                        <artifactId>log4j-layout-template-json</artifactId>
                    </include>
                </includes>
            </configuration>
        </execution>
    </executions>
</plugin>

3. 代码层面强制指定Log4j2类加载器 #

在动态加载Log4j2配置前，强制让Log4j2使用应用自身的类加载器（而非Spark的类加载器）：

import org.apache.logging.log4j.core.config.Configurator;
import org.apache.logging.log4j.core.util.Loader;

// 在初始化Log4j2配置前执行
Loader.setThreadContextClassLoader(YourMainClass.class.getClassLoader());
// 加载自定义配置文件
Configurator.initialize(null, "classpath:your-log4j2-config.xml");

4. Spark提交时强制优先加载用户类 #

通过Spark提交参数，让Driver和Executor优先加载胖包中的依赖，避免Spark自带依赖覆盖：

spark-submit \
    --conf spark.driver.userClassPathFirst=true \
    --conf spark.executor.userClassPathFirst=true \
    --class com.yourcompany.YourMainClass \
    your-fat-jar.jar

注意：该参数可能引发其他依赖冲突，需结合实际场景测试验证。

内容的提问来源于stack exchange，提问作者jeanfrancis