Databricks 10.4 LTS+Spark3读取Snappy压缩PB文件遇UnsatisfiedLinkError

环境与代码 #

运行环境：Databricks 10.4 LTS（Spark 3）
读取并解析文件的Scala代码：

sc.sequenceFile[NullWritable, BytesWritable](concatUris)
        .map(b => {
          val msg: Array[Byte] = b._2.copyBytes()
          val feed: a_feed = a_feed.parseFrom(msg)
          val properties = feed.toPMessage.value
            .map { case (key, value) =>
              key.name -> (value match {
                case i: PInt        => i.value
                case l: PLong       => l.value
                case s: PString     => s.value
                case d: PDouble     => d.value
                case f: PFloat      => f.value
                case b: PByteString => b.value
                case c: PBoolean    => c.value
                case e: PEnum       => e.value.name
                case other          => other.toString()
              })
            }

          (uid, properties + ("user_id" -> uid))
        })

错误堆栈 #

Caused by: UnsatisfiedLinkError: org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawUncompress(Ljava/nio/ByteBuffer;IILjava/nio/ByteBuffer;I)I
    at org.apache.hadoop.shaded.org.xerial.snappy.SnappyNative.rawUncompress(Native Method)
    at org.apache.hadoop.shaded.org.xerial.snappy.Snappy.uncompress(Snappy.java:551)
    at org.apache.hadoop.io.compress.snappy.SnappyDecompressor.decompressDirectBuf(SnappyDecompressor.java:267)
    at org.apache.hadoop.io.compress.snappy.SnappyDecompressor.decompress(SnappyDecompressor.java:217)
    at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:88)
    at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:105)
    at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:92)

已尝试操作 #

• 安装不同版本的org.xerial.snappy:snappy-java:<version>:jar库
• 通过Databricks的Compute>Libraries标签上传安装库，但不确定是否部署到Driver和Executor所有节点

解决方案 #

1. 移除手动引入的snappy-java库 #

Databricks Runtime已内置适配Hadoop的Snappy库，手动引入外部snappy-java会与Hadoop shaded版本冲突，导致Native方法找不到。直接删除集群中手动添加的snappy-java库。

2. 确保集群库部署范围正确 #

通过Compute>Libraries添加的库默认会部署到Driver和Executor节点，但如果是Notebook级别临时安装（如%scala libraryDependencies += ...），仅Driver生效。需在集群级别添加依赖。

3. 配置Hadoop Snappy参数（可选） #

读取文件前设置Hadoop配置，确保使用正确的解压器：

sc.hadoopConfiguration.set("io.compression.codecs", "org.apache.hadoop.io.compress.SnappyCodec")
sc.hadoopConfiguration.set("io.compression.codec.snappy.native", "true")

4. 区分Hadoop Snappy与标准Snappy格式 #

如果文件是标准Snappy而非Hadoop Snappy压缩，需手动读取二进制文件后解压缩：

import org.xerial.snappy.Snappy
sc.binaryFiles(concatUris)
  .map{ case (path, bytes) => Snappy.uncompress(bytes.toArray) }
  .map(msg => a_feed.parseFrom(msg))
  // 后续处理逻辑...

此时需使用与Databricks 10.4 LTS兼容的snappy-java版本（推荐1.1.8.4），并确保库部署到所有节点。

内容的提问来源于stack exchange，提问作者Niko