Spark Submit运行Spring Boot 3.4胖JAR时遇MalformedURLException错误

我构建了一个Spring Boot 3.4胖JAR，使用Spark Submit运行时出现如下错误：

Exception in thread "main" java.lang.ExceptionInInitializerError
    at org.springframework.boot.loader.net.protocol.jar.Handler.openConnection(Handler.java:46)
    at java.base/java.net.URL.openConnection(URL.java:1094)
    at org.springframework.boot.loader.net.protocol.jar.JarUrlClassLoader.getJarFile(JarUrlClassLoader.java:183)
    at org.springframework.boot.loader.net.protocol.jar.JarUrlClassLoader.definePackage(JarUrlClassLoader.java:146)
    at org.springframework.boot.loader.net.protocol.jar.JarUrlClassLoader.definePackageIfNecessary(JarUrlClassLoader.java:129)
    at org.springframework.boot.loader.net.protocol.jar.JarUrlClassLoader.loadClass(JarUrlClassLoader.java:102)
    at org.springframework.boot.loader.launch.LaunchedClassLoader.loadClass(LaunchedClassLoader.java:91)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Class.java:467)
    at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:99)
    at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:64)
    at org.springframework.boot.loader.launch.JarLauncher.main(JarUrlClassLoader.java:40)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:569)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1034)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:199)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:222)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1125)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1134)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.IllegalStateException: java.net.MalformedURLException: unknown protocol: nested
    at org.springframework.boot.loader.net.protocol.jar.JarUrlConnection.<clinit>(JarUrlConnection.java:67)
    ... 25 more
Caused by: java.net.MalformedURLException: unknown protocol: nested
    at java.base/java.net.URL.<init>(URL.java:681)
    at java.base/java.net.URL.<init>(URL.java:569)
    at java.base/java.net.URL.<init>(URL.java:516)
    at java.base/java.net.JarURLConnection.parseSpecs(JarURLConnection.java:175)
    at java.base/java.net.JarURLConnection.<init>(JarURLConnection.java:158)
    at org.springframework.boot.loader.net.protocol.jar.JarUrlConnection.<init>(JarUrlConnection.java:92)
    at org.springframework.boot.loader.net.protocol.jar.JarUrlConnection.<clinit>(JarUrlConnection.java:64)

我推测问题出在Spring加载application-properties的阶段，尝试生成shadow jar修复但无效，执行的Spark Submit命令如下：

spark-submit --class org.springframework.boot.loader.launch.JarLauncher  app.jar --arg1 --arg2 --arg3

问题根源 #

该错误源于Spring Boot 3.4的JarUrlConnection无法识别Spark提交环境中的nested: URL协议，Spark的类加载机制与Spring Boot胖JAR的嵌套类加载器存在冲突，导致协议解析失败。

解决方案 #

1. 直接指定应用主类而非Spring Boot的JarLauncher #

Spark Submit无需通过Spring Boot的JarLauncher启动，直接指定你的Spring Boot应用主类（即带有@SpringBootApplication注解的类）：

spark-submit --class com.your.package.YourSpringBootMainClass app.jar --arg1 --arg2 --arg3

2. 修改Spring Boot构建配置，生成非嵌套结构的JAR #

如果需要保持胖JAR形式，调整构建插件配置，生成普通可执行JAR而非嵌套JAR：

Maven配置

<plugin>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-maven-plugin</artifactId>
    <configuration>
        <layout>JAR</layout>
        <mainClass>com.your.package.YourSpringBootMainClass</mainClass>
    </configuration>
</plugin>

Gradle配置

springBoot {
    mainClass = 'com.your.package.YourSpringBootMainClass'
    layout = 'JAR'
}

3. 配置Spark类加载优先级 #

添加Spark提交参数，强制使用父类加载器优先加载类，避免类加载冲突：

spark-submit --class com.your.package.YourSpringBootMainClass \
    --conf spark.driver.userClassPathFirst=true \
    --conf spark.executor.userClassPathFirst=true \
    app.jar --arg1 --arg2 --arg3

内容的提问来源于stack exchange，提问作者Siva Pratheep