Spark集成Hadoop连接MinIO遭遇403 Forbidden错误求助

错误日志 #

java.nio.file.AccessDeniedException: s3a://hudi/env.txt: getFileStatus on s3a://hudi/env.txt: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 1778F9AB820DCD94; S3 Extended Request ID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8; Proxy: null), S3 Extended Request ID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8:403 Forbidden
  at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:249)
  at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:170)
  at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3286)
  at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3185)
  at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:3053)
  at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:115)
  at org.apache.hadoop.fs.Globber.doGlob(Globber.java:349)
  at org.apache.hadoop.fs.Globber.glob(Globber.java:202)
  at org.apache.hadoop.fs.s3a.S3AFileSystem.globStatus(S3AFileSystem.java:4253)
  at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:276)
  at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:244)
  at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:332)
  at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:205)
  at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:300)
  at scala.Option.getOrElse(Option.scala:189)
  at org.apache.spark.rdd.RDD.partitions(RDD.scala:296)
  at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)
  at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:300)
  at scala.Option.getOrElse(Option.scala:189)
  at org.apache.spark.rdd.RDD.partitions(RDD.scala:296)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:2289)
  at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1030)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
  at org.apache.spark.rdd.RDD.collect(RDD.scala:1029)
  ... 47 elided
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 1778F9AB820DCD94; S3 Extended Request ID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8; Proxy: null)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1828)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1412)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1374)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
  at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
  at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
  at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
  at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5227)
  at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5173)
  at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1360)
  at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$6(S3AFileSystem.java:2066)
  at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:412)
  at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:375)
  at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2056)
  at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2032)
  at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3273)
  ... 70 more

环境与启动命令 #

• Spark版本：spark-3.2.4-bin-hadoop3.2.tgz
• 已添加依赖：将aws-java-sdk-bundle-1.11.901.jar和hadoop-aws-3.3.1.jar放入Spark的jars目录
• Spark Shell启动命令：

./spark-shell \
--conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' \
--conf 'spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem' \
--conf 'spark.hadoop.fs.s3a.endpoint=http://127.0.0.1:9000' \
--conf 'spark.hadoop.fs.s3a.access.key=minioadmin' \
--conf 'spark.hadoop.fs.s3a.secret.key=minioadmin'\
--conf 'spark.hadoop.fs.s3a.path.style.access=true' \
--conf 'spark.hadoop.fs.s3a.connection.ssl.enabled=true' \
--conf 'spark.hadoop.fs.s3a.signing-algorithm=S3SignerType'

排查解决方案 #

• 修正SSL配置：MinIO使用HTTP协议（endpoint为http://127.0.0.1:9000），但当前配置开启了SSL，将spark.hadoop.fs.s3a.connection.ssl.enabled改为false
• 匹配依赖版本：Spark集成的Hadoop版本为3.2，使用的hadoop-aws-3.3.1版本不兼容，换成与Hadoop版本一致的hadoop-aws-3.2.x系列版本，避免签名逻辑或API冲突
• 验证Bucket权限：确认MinIO中hudi桶已存在，且minioadmin用户拥有该桶的读权限，可通过MinIO控制台检查桶的访问策略
• 调整签名算法：若MinIO版本较新，尝试将spark.hadoop.fs.s3a.signing-algorithm设置为AWS4-HMAC-SHA256，同时检查endpoint地址无拼写错误
• 清理冲突依赖：检查Spark jars目录，删除旧版本或重复的AWS、hadoop-aws相关jar包，确保依赖唯一性

内容的提问来源于stack exchange，提问作者Dodge_X