Spark Streaming写入Kafka抛出java.lang.UnsatisfiedLinkError求助

使用Spark 3.4.0编写测试程序，将本地CSV文件的数据流式写入Kafka。已配置VM参数，HADOOP_HOME环境变量指向包含winutils.exe的bin目录，build.sbt依赖配置正确，但运行时抛出java.lang.UnsatisfiedLinkError异常。搜索相关解决方案未获有效帮助，尝试切换至Spark 2.4.8后出现其他错误，请求协助排查。

程序代码 #

object SparkTest {
  def main(args : Array[String]): Unit ={

    val schema = StructType(
      List(
        StructField("id", StringType, true),
        StructField("applicablechannel", StringType, true),
        StructField("value", StringType, true),
        StructField("associatedoffer", StringType, true),
        StructField("pyissue", StringType, true),
        StructField("controlgrouppercentage", StringType, true),
        StructField("direction", StringType, true)
      )
    )

    val spark = SparkSession.builder().appName("LoadFile")
      .master("local[*]")
      .config("spark.driver.memory", "1g")
      .getOrCreate()
    import spark.implicits._
    val csvDF = spark.readStream.format("csv")
      .option("header", true)
      .schema(schema)
      .load("file:////C:\\Users\\XYZ\\Downloads\\files")
      .select("id", "value")

    csvDF.writeStream
      .format("kafka")
      .option("kafka.bootstrap.servers", "myserver.mydomain.com:9092")
      .option("topic", "KafkaDS")
      .option("checkpointLocation", "C:\\Users\\XYZ\\AppData\\Local\\Temp")
      .start()
      .awaitTermination()

    spark.close()
  }
}

VM参数 #

--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.lang.invoke=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.base/java.net=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
--add-opens=java.base/sun.nio.cs=ALL-UNNAMED
--add-opens=java.base/sun.security.action=ALL-UNNAMED
--add-opens=java.base/sun.util.calendar=ALL-UNNAMED
--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED

build.sbt配置 #

ThisBuild / version := "0.1.0-SNAPSHOT"

ThisBuild / scalaVersion := "2.13.11"
val sparkVersion = "3.4.0"

val sparkAndDependencies = Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-sql" % sparkVersion,
  "org.apache.spark" %% "spark-sql-kafka-0-10" % sparkVersion
)

libraryDependencies ++= sparkAndDependencies

lazy val root = (project in file("."))
  .settings(
    name := "SparkTestTake3"
  )

异常信息 #

Exception in thread "main" java.lang.UnsatisfiedLinkError: 'boolean org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(java.lang.String, int)'
    at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
    at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:793)
    at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:1249)
    at org.apache.hadoop.fs.FileUtil.list(FileUtil.java:1454)
    at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:601)
    at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1972)
    at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:2014)
    at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:761)
    at org.apache.spark.util.HadoopFSUtils$.listLeafFiles(HadoopFSUtils.scala:225)
    at org.apache.spark.util.HadoopFSUtils$.$anonfun$parallelListLeafFilesInternal$1(HadoopFSUtils.scala:95)
    at scala.collection.immutable.List.map(List.scala:246)
    at scala.collection.immutable.List.map(List.scala:79)
    at org.apache.spark.util.HadoopFSUtils$.parallelListLeafFilesInternal(HadoopFSUtils.scala:85)
    at org.apache.spark.util.HadoopFSUtils$.parallelListLeafFiles(HadoopFSUtils.scala:69)
    at org.apache.spark.sql.execution.datasources.InMemoryFileIndex$.bulkListLeafFiles(InMemoryFileIndex.scala:162)
    at org.apache.spark.sql.execution.datasources.InMemoryFileIndex.listLeafFiles(InMemoryFileIndex.scala:133)
    at org.apache.spark.sql.execution.datasources.InMemoryFileIndex.refresh0(InMemoryFileIndex.scala:96)
    at org.apache.spark.sql.execution.datasources.InMemoryFileIndex.<init>(InMemoryFileIndex.scala:68)
    at org.apache.spark.sql.execution.datasources.DataSource.createInMemoryFileIndex(DataSource.scala:539)
    at org.apache.spark.sql.execution.datasources.DataSource.$anonfun$sourceSchema$2(DataSource.scala:265)
    at org.apache.spark.sql.execution.datasources.DataSource.tempFileIndex$lzycompute$1(DataSource.scala:162)
    at org.apache.spark.sql.execution.datasources.DataSource.tempFileIndex$1(DataSource.scala:162)
    at org.apache.spark.sql.execution.datasources.DataSource.getOrInferFileFormatSchema(DataSource.scala:167)
    at org.apache.spark.sql.execution.datasources.DataSource.sourceSchema(DataSource.scala:259)
    at org.apache.spark.sql.execution.datasources.DataSource.sourceInfo$lzycompute(DataSource.scala:118)
    at org.apache.spark.sql.execution.datasources.DataSource.sourceInfo(DataSource.scala:118)
    at org.apache.spark.sql.execution.streaming.StreamingRelation$.apply(StreamingRelation.scala:35)
    at org.apache.spark.sql.streaming.DataStreamReader.loadInternal(DataStreamReader.scala:197)
    at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:211)
    at SparkTest$.main(SparkTest.scala:43)
    at SparkTest.main(SparkTest.scala)

排查与解决方案 #

这个UnsatisfiedLinkError是Windows环境下Hadoop本地库版本不匹配或配置错误导致的，按以下步骤处理：

1. 匹配Hadoop与Spark版本
   Spark 3.4.0默认依赖Hadoop 3.3.4，必须确保下载的winutils.exe是对应Hadoop 3.3.4版本的，版本不兼容会直接触发该异常。

2. 修正HADOOP_HOME配置

• HADOOP_HOME必须指向Hadoop的根目录，而非bin目录。例如winutils.exe在D:\hadoop-3.3.4\bin，则HADOOP_HOME设为D:\hadoop-3.3.4
• 将%HADOOP_HOME%\bin添加到系统PATH环境变量
• 重启IDE或终端，确保环境变量生效

3. 禁用Hadoop本地库（临时 workaround）
   如果暂时找不到匹配的winutils，可在SparkSession中添加配置禁用本地库依赖：

val spark = SparkSession.builder().appName("LoadFile")
  .master("local[*]")
  .config("spark.driver.memory", "1g")
  .config("spark.hadoop.io.native.lib.available", "false")
  .getOrCreate()

4. 简化文件路径
   CSV加载路径可简化为C:/Users/XYZ/Downloads/files（Spark在Windows下支持直接识别本地路径），避免多斜杠导致的解析问题。

5. Spark 2.4.8的兼容性问题
   Spark 2.4.8默认依赖Hadoop 2.7，与3.x版本的winutils不兼容，切换版本会触发新的依赖冲突，建议继续使用Spark 3.4.0并解决Hadoop本地库问题。

内容的提问来源于stack exchange，提问作者Krisrettiwt