在EKS集群运行spark-sql时遭遇AWS认证失败问题

使用Kubernetes作为集群管理器和调度器运行Spark工作负载时，spark-submit可正常配合IAM访问，从S3存储桶获取jar包并使用指定服务账号；但执行spark-sql时出现AWS认证失败问题。

成功运行的spark-submit命令 #

spark-submit \
  --master k8s://<k8s-master-url> \
  --name spark-pi \
  --deploy-mode cluster \
  --class org.apache.spark.examples.SparkPi \
  --conf spark.kubernetes.container.image=1234567890.dkr.ecr.ap-south-1.amazonaws.com/spark:3.3.1 \
  --conf spark.executor.instances=2 \
  --conf spark.kubernetes.namespace=spark \
  --conf spark.kubernetes.file.upload.path=s3a://<bucket-name>/ \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=prod-spark-sa \
  --conf spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.WebIdentityTokenCredentialsProvider \
  --conf spark.eventLog.enabled=true \
  --conf spark.eventLog.dir=s3a://<bucket-name>/logs/ \
  --conf spark.history.fs.logDirectory=s3a://<bucket-name>/logs/ \
s3a://<bucket-name>/spark-examples_2.12-3.5.0.jar

失败的spark-sql命令 #

spark-sql \
--master k8s://<k8s-master-url> \
--conf spark.kubernetes.container.image=1234567890.dkr.ecr.ap-south-1.amazonaws.com/spark:3.3.1 \
--conf spark.kubernetes.namespace=spark \
--packages io.delta:delta-core_2.12:2.4.0,org.apache.hadoop:hadoop-aws:3.3.4 \
--conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
--conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog \
--conf spark.hadoop.hive.metastore.uris=thrift://172.xx.xx.xx:10000 \
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
--conf spark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
--conf spark.sql.warehouse.dir=s3a://<bucket-name>/warehouse/ \
--conf spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.WebIdentityTokenCredentialsProvider \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=prod-spark-sa \
--conf spark.kubernetes.authenticate.executor.serviceAccountName=prod-spark-sa

错误日志 #

24/05/14 12:47:38 WARN DeltaLog: Failed to parse s3://<bucket-name>/587f8193-b09a-41a5-9d1d-a14f84911567/tables/e59dc872-36d5-4409-8ee9-dfcbd6f3d63d/_delta_log/_last_checkpoint. This may happen if there was an error during read operation, or a file appears to be partial. Sleeping and trying again.
java.lang.NullPointerException: You must specify a value for roleArn and roleSessionName
        at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$Builder.<init>(STSAssumeRoleSessionCredentialsProvider.java:390)
        at com.amazonaws.services.securitytoken.internal.STSProfileCredentialsService.getAssumeRoleCredentialsProvider(STSProfileCredentialsService.java:31)
        at com.amazonaws.auth.profile.internal.securitytoken.STSProfileCredentialsServiceProvider.getProfileCredentialsProvider(STSProfileCredentialsServiceProvider.java:39)

问题原因 #

1. 错误日志显示STSAssumeRoleSessionCredentialsProvider要求指定roleArn和roleSessionName，说明spark-sql的凭证认证链触发了默认AWS profile配置（比如本地~/.aws/credentials），而非优先使用配置的WebIdentityTokenCredentialsProvider。
2. 错误中出现s3://路径，而非配置的s3a://，说明Delta Lake或Hive metastore在访问存储时未使用s3a客户端，导致凭证配置不生效。
3. spark-sql默认使用client模式，而spark-submit用的是cluster模式，client模式下本地环境的AWS配置可能干扰driver的凭证获取。

解决步骤 #

1. 强制所有S3路径使用s3a协议 #

• 添加配置将s3://和s3n://协议映射到s3a客户端：

  --conf spark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
  --conf spark.hadoop.fs.s3n.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
  

• 指定Delta Lake使用s3a协议的日志存储类：

  --conf spark.delta.logStore.class=org.apache.spark.sql.delta.storage.S3SingleDriverLogStore \
  

2. 锁定凭证提供者，避免 fallback 到默认profile #

• 明确指定凭证提供者链，并禁用AWS profile：

  --conf spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.WebIdentityTokenCredentialsProvider,com.amazonaws.auth.EnvironmentVariableCredentialsProvider \
  --conf spark.hadoop.fs.s3a.aws.credentials.provider.disableAwsProfile=true \
  

3. 确保K8s服务账号token正确挂载 #

• 显式指定driver和executor的token路径：

  --conf spark.kubernetes.authenticate.driver.tokenFile=/var/run/secrets/kubernetes.io/serviceaccount/token \
  --conf spark.kubernetes.authenticate.executor.tokenFile=/var/run/secrets/kubernetes.io/serviceaccount/token \
  

4. 使用cluster模式运行spark-sql #

• 添加--deploy-mode cluster，避免本地环境配置干扰：

  --deploy-mode cluster \
  

调整后的spark-sql命令示例 #

spark-sql \
--master k8s://<k8s-master-url> \
--deploy-mode cluster \
--conf spark.kubernetes.container.image=1234567890.dkr.ecr.ap-south-1.amazonaws.com/spark:3.3.1 \
--conf spark.kubernetes.namespace=spark \
--packages io.delta:delta-core_2.12:2.4.0,org.apache.hadoop:hadoop-aws:3.3.4 \
--conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension \
--conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog \
--conf spark.hadoop.hive.metastore.uris=thrift://172.xx.xx.xx:10000 \
--conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
--conf spark.hadoop.fs.s3.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
--conf spark.hadoop.fs.s3n.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
--conf spark.sql.warehouse.dir=s3a://<bucket-name>/warehouse/ \
--conf spark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.WebIdentityTokenCredentialsProvider,com.amazonaws.auth.EnvironmentVariableCredentialsProvider \
--conf spark.hadoop.fs.s3a.aws.credentials.provider.disableAwsProfile=true \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=prod-spark-sa \
--conf spark.kubernetes.authenticate.executor.serviceAccountName=prod-spark-sa \
--conf spark.kubernetes.authenticate.driver.tokenFile=/var/run/secrets/kubernetes.io/serviceaccount/token \
--conf spark.kubernetes.authenticate.executor.tokenFile=/var/run/secrets/kubernetes.io/serviceaccount/token \
--conf spark.delta.logStore.class=org.apache.spark.sql.delta.storage.S3SingleDriverLogStore

内容的提问来源于stack exchange，提问作者Nitish