通过Livy提交PySpark任务读取S3代码时S3AFileSystem找不到的解决方法

已在AWS部署Apache Livy与PySpark环境，PySpark代码存储在S3桶中，通过Livy提交任务的代码如下：

import requests
import json

livy_url_battches = "http://56.43.345.87:8998/batches"

payload = {
    "file": "s3a://abctesting/sparkcode.py",
    "className": "org.apache.spark.examples.SparkPi",

    "conf": {
        "spark.executor.memory": "3g",
        "spark.driver.memory": "1g",
        "spark.hadoop.fs.s3a.access.key": "myAccessKey",
        "spark.hadoop.fs.s3a.secret.key": "mySecreteKey",
        "spark.hadoop.fs.s3a.endpoint": "s3.amazonaws.com",  
        "spark.jars": "https://abctesting.s3.amazonaws.com/aws-java-sdk-bundle-1.12.262.jar,https://abctesting.s3.amazonaws.com/hadoop-aws-3.3.4.jar",
    },
}

headers = {
    "Content-Type": "application/json",
}
try:
    response = requests.post(livy_url_battches, headers=headers, data=json.dumps(payload))
    response.status_code == 201
    print("Batch job submitted successfully.")
    print(response.json())
except Exception as error:
    print("Error ===",error)

提交任务后出现以下错误：

24/09/03 10:59:32 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using pySpark-java classes where applicable
Exception in thread "main" java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2688)
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3431)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466)
    at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
    at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1831)
    at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:727)
    at org.apache.spark.util.DependencyUtils$.downloadFile(DependencyUtils.scala:264)
    at org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$8(SparkSubmit.scala:404)
    at scala.Option.map(Option.scala:230)
    at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:404)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:964)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2592)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2686)
    ... 19 more

怀疑配置的hadoop-aws-3.3.4.jar和aws-java-sdk-bundle-1.12.262.jar未被Spark加载，需解决该问题。

1. 校验JAR包版本兼容性 #

hadoop-aws版本必须与集群Hadoop版本完全一致，若集群Hadoop为3.3.4，当前hadoop-aws-3.3.4版本匹配；aws-java-sdk-bundle-1.12.262与hadoop-aws-3.3.4版本兼容，需确认集群Hadoop版本是否确实为3.3.4。

2. 全局配置Livy加载JAR包 #

修改Livy关联的Spark配置文件（如spark-defaults.conf），添加以下配置，让所有Livy提交的任务自动加载依赖JAR：

spark.jars.packages org.apache.hadoop:hadoop-aws:3.3.4,com.amazonaws:aws-java-sdk-bundle:1.12.262

若不想依赖Maven仓库，可将JAR包下载到Livy服务器本地目录，配置改为：

spark.jars /path/to/hadoop-aws-3.3.4.jar,/path/to/aws-java-sdk-bundle-1.12.262.jar

修改完成后重启Livy服务。

3. 调整提交任务的JAR路径格式 #

提交任务时，将spark.jars的HTTP URL改为S3A协议路径，确保Spark可正常访问：

"spark.jars": "s3a://abctesting/aws-java-sdk-bundle-1.12.262.jar,s3a://abctesting/hadoop-aws-3.3.4.jar"

同时确保Livy/Spark实例拥有该S3桶的访问权限，优先使用IAM角色而非硬编码密钥。

4. 修正密钥配置与权限 #

• 检查密钥拼写：代码中mySecreteKey存在拼写错误，应为mySecretKey，错误拼写会导致权限验证失败，影响JAR包加载。
• 优先使用IAM角色：给Livy/Spark实例赋予S3访问权限，移除配置中的密钥参数，Spark会自动使用实例角色权限，更安全可靠。

5. 验证JAR包存在性 #

确认S3桶中JAR包路径与文件名无拼写错误，可直接访问对应URL验证文件是否存在。

内容的提问来源于stack exchange，提问作者arbind Sah