You need to enable JavaScript to run this app.
优惠活动
大模型
产品
解决方案
定价
更多

Windows下SpringBoot+Maven加载Spark Pipeline模型遇UnsatisfiedLinkError求助

avatar
算力阿峰
2026-6-25
问题描述

需要加载保存在路径C:\Users\ASUS\Desktop\kaggle\classification\dt下的Spark Pipeline模型（该路径包含metadata和stages两个目录），但在SpringBoot+Maven项目中加载时抛出如下异常：

Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$POSIX.stat(Ljava/lang/String;)Lorg/apache/hadoop/io/nativeio/NativeIO$POSIX$Stat;
    at org.apache.hadoop.io.nativeio.NativeIO$POSIX.stat(Native Method)
    at org.apache.hadoop.io.nativeio.NativeIO$POSIX.getStat(NativeIO.java:608)
    at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfoByNativeIO(RawLocalFileSystem.java:934)
    at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:848)
    at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getPermission(RawLocalFileSystem.java:816)
    at org.apache.hadoop.fs.LocatedFileStatus.<init>(LocatedFileStatus.java:52)
    at org.apache.hadoop.fs.FileSystem$4.next(FileSystem.java:2199)
    at org.apache.hadoop.fs.FileSystem$4.next(FileSystem.java:2179)
    at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:287)
    at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:244)
    at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:332)
    at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:205)
    at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:300)
    at scala.Option.getOrElse(Option.scala:189)
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:296)
    at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
    at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:300)
    at scala.Option.getOrElse(Option.scala:189)
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:296)
    at org.apache.spark.rdd.RDD.$anonfun$take$1(RDD.scala:1428)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
    at org.apache.spark.rdd.RDD.take(RDD.scala:1422)
    at org.apache.spark.rdd.RDD.$anonfun$first$1(RDD.scala:1463)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
    at org.apache.spark.rdd.RDD.first(RDD.scala:1463)
    at org.apache.spark.ml.util.DefaultParamsReader$.loadMetadata(ReadWrite.scala:587)
    at org.apache.spark.ml.Pipeline$SharedReadWrite$.$anonfun$load$3(Pipeline.scala:269)
    at org.apache.spark.ml.util.Instrumentation$.$anonfun$instrumented$1(Instrumentation.scala:191)
    at scala.util.Try$.apply(Try.scala:213)
    at org.apache.spark.ml.util.Instrumentation$.instrumented(Instrumentation.scala:191)
    at org.apache.spark.ml.Pipeline$SharedReadWrite$.load(Pipeline.scala:268)
    at org.apache.spark.ml.PipelineModel$PipelineModelReader.$anonfun$load$7(Pipeline.scala:356)
    at org.apache.spark.ml.MLEvents.withLoadInstanceEvent(events.scala:160)
    at org.apache.spark.ml.MLEvents.withLoadInstanceEvent$(events.scala:155)
    at org.apache.spark.ml.util.Instrumentation.withLoadInstanceEvent(Instrumentation.scala:42)
    at org.apache.spark.ml.PipelineModel$PipelineModelReader.$anonfun$load$6(Pipeline.scala:355)
    at org.apache.spark.ml.util.Instrumentation$.$anonfun$instrumented$1(Instrumentation.scala:191)
    at scala.util.Try$.apply(Try.scala:213)
    at org.apache.spark.ml.util.Instrumentation$.instrumented(Instrumentation.scala:191)
    at org.apache.spark.ml.PipelineModel$PipelineModelReader.load(Pipeline.scala:355)
    at org.apache.spark.ml.PipelineModel$PipelineModelReader.load(Pipeline.scala:349)
    at org.apache.spark.ml.util.MLReadable.load(ReadWrite.scala:355)
    at org.apache.spark.ml.util.MLReadable.load$(ReadWrite.scala:355)
    at org.apache.spark.ml.PipelineModel$.load(Pipeline.scala:337)

项目的pom依赖如下：

<properties>
    <java.version>1.8</java.version>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
    <spring-boot.version>2.6.13</spring-boot.version>
</properties>
<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>

    <dependency>
        <groupId>com.baomidou</groupId>
        <artifactId>mybatis-plus-boot-starter</artifactId>
        <version>3.5.3.1</version>
    </dependency>
    <dependency>
        <groupId>com.alibaba</groupId>
        <artifactId>druid</artifactId>
        <version>1.1.16</version>
    </dependency>

    <dependency>
        <groupId>com.mysql</groupId>
        <artifactId>mysql-connector-j</artifactId>
        <scope>runtime</scope>
    </dependency>
    <dependency>
        <groupId>org.projectlombok</groupId>
        <artifactId>lombok</artifactId>
        <optional>true</optional>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>

    <!--spark-->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.12</artifactId>
        <version>3.2.0</version>
    </dependency>

    <!--spark_sql-->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.12</artifactId>
        <version>3.2.0</version>
    </dependency>
    <!--hadoop-->
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-client</artifactId>
        <version>3.2.0</version>
    </dependency>
    <!--scala-->
    <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-library</artifactId>
        <version>2.12.13</version>
    </dependency>
    <!--spark解析xml依赖包-->
    <dependency>
        <groupId>com.databricks</groupId>
        <artifactId>spark-xml_2.12</artifactId>
        <version>0.12.0</version>
    </dependency>
    <!--spark-mlib-->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-mllib_2.12</artifactId>
        <version>3.2.0</version>
    </dependency>
    <!-- janio-->
    <dependency>
        <groupId>org.codehaus.janino</groupId>
        <artifactId>commons-compiler</artifactId>
        <version>3.0.16</version>
    </dependency>
    <dependency>
        <groupId>org.codehaus.janino</groupId>
        <artifactId>janino</artifactId>
        <version>3.0.16</version>
    </dependency>
</dependencies>

直接训练并使用Spark MLlib模型时程序运行正常，已添加Hadoop依赖但问题仍存在，推测和winutils.exe相关，之前本地Spark是将其放入Spark的bin目录解决，但现在通过Maven引入Spark，不清楚放置路径，寻求解决办法。

解决方案

问题根源

Windows环境下，Hadoop依赖的本地系统库（如winutils.exe、hadoop.dll）缺失，导致加载模型时Hadoop文件系统操作调用NativeIO本地方法失败。

具体步骤

1. 获取对应版本的本地库

下载与你Hadoop版本（3.2.0）匹配的winutils.exe和hadoop.dll文件。

2. 放置路径与配置

有两种可选方案：

  • 方案一：系统级配置
    • 创建目录C:\hadoop\bin，将winutils.exe和hadoop.dll放入该目录。
    • 将C:\hadoop\bin添加到系统PATH环境变量中；或者在项目启动时添加JVM参数：
      -Djava.library.path=C:\hadoop\bin
  • 方案二：项目级配置
    • 在项目src/main/resources下创建hadoop/bin目录，放入winutils.exe和hadoop.dll。
    • 在加载Spark模型前，添加代码设置系统属性：
      // 设置本地库路径
String hadoopBinPath = System.getProperty("user.dir") + "/src/main/resources/hadoop/bin";
System.setProperty("java.library.path", hadoopBinPath);

// 强制刷新类加载器的库路径（避免JVM缓存旧路径）
try {
    Field sysPathsField = ClassLoader.class.getDeclaredField("sys_paths");
    sysPathsField.setAccessible(true);
    sysPathsField.set(null, null);
} catch (Exception e) {
    e.printStackTrace();
}

3. 补充配置（可选）

在SpringBoot配置文件application.properties中添加Hadoop本地模式配置：

# 指定Hadoop主目录
hadoop.home.dir=C:\hadoop
# 设置本地文件系统
fs.defaultFS=file:///

4. 验证

重启项目后重新加载Spark Pipeline模型，异常即可解决。

内容的提问来源于stack exchange，提问作者DianleJy

相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分)， 限时 9.9 元起

产品介绍相关文档
相关产品推荐
方舟 Agent Plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列、Doubao-Seedream-5.0-pro、Kimi-K3 (部分)， 限时 9.9 元起

产品介绍相关文档
火山引擎 最新活动
全面开放
Doubao-Seedance-2.5全新模型升级

模型再升级,30秒超长叙事, 模态参考扩容

HOT
方舟Codingplan

模型自由，工具不限，最新支持 Deepseek-V4 系列、GLM-5.3 系列

HOT
方舟Agent plan

超全模态模型 × Harness 升级，最新支持 Deepseek-V4.1-Flash、GLM-5.3 系列

HOT
新用户特惠专场

大模型19元起，Al应用9.9元畅享，新人首购爆款尽享优惠

最近更新时间：2026.06.25 21:54:54