将Spark 3 DataFrame写入AWS S3时出现NoSuchMethodError错误

问题描述 #

使用Spark 3将DataFrame写入AWS S3时出现方法不存在错误，但调整代码后在Spark 2环境下可正常执行。

Spark 3 代码 #

spark.conf.set("spark.hadoop.fs.s3.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
spark.conf.set("spark.hadoop.fs.AbstractFileSystem.s3a.impl", "org.apache.hadoop.fs.s3a.S3A")
spark.conf.set("spark.hadoop.fs.s3a.access.key", access_key)
spark.conf.set("spark.hadoop.fs.s3a.secret.key", secret_key)
spark.conf.set("spark.hadoop.fs.s3a.endpoint", "s3.amazonaws.com")

df.coalesce(1).write.mode("overwrite").parquet(path)

错误信息 #

py4j.protocol.Py4JJavaError: An error occurred while calling o88.parquet.
: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V
    at org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword(S3AUtils.java:816)
    at org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword(S3AUtils.java:792)
    at org.apache.hadoop.fs.s3a.S3AUtils.getAWSAccessKeys(S3AUtils.java:747)
    at org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider.<init>(SimpleAWSCredentialsProvider.java:58)
    at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:600)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:257)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3303)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
    at org.apache.spark.sql.execution.datasources.DataSource.planForWritingFileFormat(DataSource.scala:461)
    at org.apache.spark.sql.execution.datasources.DataSource.planForWriting(DataSource.scala:556)
    at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:382)
    at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:355)
    at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
    at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:781)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
    at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
    at java.lang.Thread.run(Thread.java:745)

原因分析 #

该错误由Guava版本冲突导致：Spark 3自带的Guava版本与Hadoop S3A客户端依赖的Guava版本不兼容，S3A代码调用的Preconditions.checkArgument方法在Spark 3的Guava版本中不存在。

解决方案 #

1. 排除冲突依赖（构建工具场景） #

如果用Maven/Gradle管理项目，需排除Spark自带的旧版Guava，引入与Hadoop 3.x兼容的版本（推荐27.0-jre及以上）：

Maven示例：

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.12</artifactId>
    <version>3.x.x</version>
    <exclusions>
        <exclusion>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
        </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>com.google.guava</groupId>
    <artifactId>guava</artifactId>
    <version>27.0-jre</version>
</dependency>

2. 提交时指定兼容依赖 #

通过spark-submit的--packages参数直接引入兼容的Guava版本：

spark-submit --packages com.google.guava:guava:27.0-jre your_script.py

3. 简化S3配置 #

Spark 3对S3的支持已优化，无需手动指定fs.s3.impl和AbstractFileSystem，简化配置即可：

spark.conf.set("spark.hadoop.fs.s3a.access.key", access_key)
spark.conf.set("spark.hadoop.fs.s3a.secret.key", secret_key)

df.coalesce(1).write.mode("overwrite").parquet(path)

（若使用非us-east-1区域，再添加spark.conf.set("spark.hadoop.fs.s3a.endpoint", "对应区域端点")）

4. 切换到AWS SDK v2客户端（Spark 3.2+） #

使用S3A的AWS SDK v2实现，避免旧版Guava冲突：

spark.conf.set("spark.hadoop.fs.s3a.access.key", access_key)
spark.conf.set("spark.hadoop.fs.s3a.secret.key", secret_key)
# 启用SDK v2
spark.conf.set("spark.hadoop.fs.s3a.impl.factory", "org.apache.hadoop.fs.s3a.impl.DefaultS3ClientFactory")

内容的提问来源于stack exchange，提问作者Atharva Patil