PySpark读取S3存储桶文件触发Py4JJavaError错误求助

现象 #

在EC2（含/不含Jupyter Lab）、本地笔记本的PySpark环境中，读取S3存储桶文件均出现相同的Py4JJavaError，但boto3可正常读取目标文件。

配置代码 #

# My spark configuration
conf = SparkConf()
conf.set('spark.jars.packages', 'org.apache.hadoop:hadoop-aws:3.3.0')
#conf.set('spark.hadoop.fs.s3a.aws.credentials.provider', 'org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider')
conf.set('spark.hadoop.fs.s3a.access.key', key)
conf.set('spark.hadoop.fs.s3a.secret.key', secret)
spark = SparkSession.builder.config(conf=conf).getOrCreate()
# path to my test file (which a can read in local with same code
path = "s3a://bucket-name/folder/test.csv"

csv = spark.read.format("csv").load(path)

报错信息 #

Py4JJavaError: An error occurred while calling o37.load.
: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;Ljava/lang/Object;)V
    at org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword(S3AUtils.java:893)
    at org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword(S3AUtils.java:869)
    at org.apache.hadoop.fs.s3a.S3AUtils.getEncryptionAlgorithm(S3AUtils.java:1580)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:341)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469)
    at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
    at org.apache.spark.sql.execution.streaming.FileStreamSink$.hasMetadata(FileStreamSink.scala:53)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:370)
    at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:228)
    at org.apache.spark.sql.DataFrameReader.$anonfun$load$2(DataFrameReader.scala:210)
    at scala.Option.getOrElse(Option.scala:189)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:210)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:185)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
    at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
    at java.lang.Thread.run(Thread.java:750)

SparkSession构建警告 #

:: loading settings :: url = jar:file:/home/ubuntu/.local/lib/python3.10/site-packages/pyspark/jars/ivy-2.5.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: /home/ubuntu/.ivy2/cache
The jars for the packages stored in: /home/ubuntu/.ivy2/jars
org.apache.hadoop#hadoop-aws added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-840765ed-4adc-4453-b354-a3a8093d3776;1.0
    confs: [default]
    found org.apache.hadoop#hadoop-aws;3.3.0 in central
    found com.amazonaws#aws-java-sdk-bundle;1.11.563 in central
    found org.wildfly.openssl#wildfly-openssl;1.0.7.Final in central
downloading https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.11.563/aws-java-sdk-bundle-1.11.563.jar ...
    [SUCCESSFUL ] com.amazonaws#aws-java-sdk-bundle;1.11.563!aws-java-sdk-bundle.jar (4888ms)
downloading https://repo1.maven.org/maven2/org/wildfly/openssl/wildfly-openssl/1.0.7.Final/wildfly-openssl-1.0.7.Final.jar ...
    [SUCCESSFUL ] org.wildfly.openssl#wildfly-openssl;1.0.7.Final!wildfly-openssl.jar (22ms)
:: resolution report :: resolve 697ms :: artifacts dl 4998ms
    :: modules in use:
    com.amazonaws#aws-java-sdk-bundle;1.11.563 from central in [default]
    org.apache.hadoop#hadoop-aws;3.3.0 from central in [default]
    org.wildfly.openssl#wildfly-openssl;1.0.7.Final from central in [default]
    ---------------------------------------------------------------------
    |                  |            modules            ||   artifacts   |
    |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
    ---------------------------------------------------------------------
    |      default     |   3   |   0   |   0   |   0   ||   3   |   2   |
    ---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent-840765ed-4adc-4453-b354-a3a8093d3776
    confs: [default]
    3 artifacts copied, 0 already retrieved (128050kB/319ms)
22/10/08 13:55:28 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).

已尝试操作 #

• 核对hadoop-aws与AWS Java SDK版本兼容性
• 切换为binaryFiles、images格式读取文件
• 验证boto3可正常读取目标文件

解决方案 #

该错误源于Guava版本冲突：Spark自带的Guava版本与hadoop-aws 3.3.0依赖的版本不一致，缺少Preconditions.checkArgument的特定重载方法。

解决步骤： #

1. 指定兼容的Guava版本：修改Spark配置，同时引入hadoop-aws和匹配的Guava jar（hadoop-aws 3.3.0对应Guava 27.0-jre），并强制优先加载该版本：

conf = SparkConf()
# 同时添加hadoop-aws和兼容的Guava依赖
conf.set('spark.jars.packages', 'org.apache.hadoop:hadoop-aws:3.3.0,com.google.guava:guava:27.0-jre')
conf.set('spark.hadoop.fs.s3a.access.key', key)
conf.set('spark.hadoop.fs.s3a.secret.key', secret)
# 强制指定Guava jar的加载路径，避免版本冲突
conf.set('spark.driver.extraClassPath', '/home/ubuntu/.ivy2/jars/com.google.guava_guava-27.0-jre.jar')
conf.set('spark.executor.extraClassPath', '/home/ubuntu/.ivy2/jars/com.google.guava_guava-27.0-jre.jar')
spark = SparkSession.builder.config(conf=conf).getOrCreate()

2. 清理旧缓存：删除~/.ivy2/cache和~/.ivy2/jars中的旧依赖包，重新拉取正确版本的jar文件。
3. 验证版本一致性：检查Spark自带的Guava版本，若低于27.0-jre，直接替换$SPARK_HOME/jars中的对应jar包为27.0-jre版本：

# 查看Spark自带的Guava版本
ls $SPARK_HOME/jars | grep guava

可选优化（EC2环境） #

使用IAM角色替代硬编码密钥，减少配置复杂度：

conf = SparkConf()
conf.set('spark.jars.packages', 'org.apache.hadoop:hadoop-aws:3.3.0,com.google.guava:guava:27.0-jre')
conf.set('spark.hadoop.fs.s3a.aws.credentials.provider', 'org.apache.hadoop.fs.s3a.InstanceProfileCredentialsProvider')
spark = SparkSession.builder.config(conf=conf).getOrCreate()

内容的提问来源于stack exchange，提问作者nhey4Aterno