PySpark集成MinIO+Iceberg+Nessie认证失败，无法查询表

问题现象 #

使用MinIO搭配PySpark、Iceberg和Nessie目录时，能正常访问目录、数据库及表，但执行表查询时触发S3认证错误：

py4j.protocol.Py4JJavaError: An error occurred while calling o41.sql.
: software.amazon.awssdk.services.s3.model.S3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: S3, Status Code: 403, Request ID: MYBC49K73C1T64ZW, Extended Request ID: H/WT1VriLv0ppiW7fPMFIPRS0QERqMd0X4ooLYpnqrGrJ3DyA05NREqVIVC7u4VXpw9iQetwQlA=)

观察到脚本中的S3凭据配置被忽略：移除脚本凭据配置时，程序会读取AWS环境变量；环境变量缺失则提示凭据未找到。

环境配置 #

Docker Compose配置 #

version: '3.8'

x-common-variables: &aws_env
  AWS_ACCESS_KEY_ID: minioadmin
  AWS_SECRET_ACCESS_KEY: minioadmin
  AWS_REGION: us-west-1
  AWS_DEFAULT_REGION: us-west-1

services:
  minio:
    image: minio/minio
    container_name: minio
    environment:
      - MINIO_ROOT_USER=minioadmin
      - MINIO_ROOT_PASSWORD=minioadmin
      - MINIO_DOMAIN=minio
      - MINIO_REGION=us-west-1
    ports:
      - 9001:9001
      - 9000:9000
    command: ["server", "/data", "--console-address", ":9001"]
    volumes:
      - minio-data:/data 
  mc:
    depends_on:
      - minio
    image: minio/mc
    container_name: mc
    environment:
      - AWS_ACCESS_KEY_ID=minioadmin
      - AWS_SECRET_ACCESS_KEY=minioadmin
      - AWS_REGION=us-west-1
      - MINIO_USER=minioadmin
      - MINIO_PASSWORD=minioadmin
      - MINIO_DOMAIN=minio
      - MINIO_REGION=us-west-1
    entrypoint: >
      /bin/sh -c "
      until (/usr/bin/mc config host add minio http://minio:9000 minioadmin minioadmin) do echo '...waiting...' && sleep 1; done;
      /usr/bin/mc mb minio/sensors-lakehouse;
      /usr/bin/mc policy set public minio/sensors-lakehouse;
      tail -f /dev/null
      "      
  postgres:
    image: postgres:13
    environment:
      POSTGRES_USER: nessie
      POSTGRES_PASSWORD: nessie
      POSTGRES_DB: nessie
    ports:
      - "5432:5432"
    volumes:
      - postgres-data:/var/lib/postgresql/data
  nessie:
    image: projectnessie/nessie
    ports:
      - "19120:19120"
    environment:
      QUARKUS_HTTP_PORT: 19120
      NESSIE_VERSION_STORE_TYPE: JDBC
      QUARKUS_DATASOURCE_JDBC_URL: jdbc:postgresql://postgres:5432/nessie
      QUARKUS_DATASOURCE_USERNAME: nessie
      QUARKUS_DATASOURCE_PASSWORD: nessie
      QUARKUS_OIDC_ENABLED: "false"
      <<: *aws_env  
    depends_on:
      - postgres

  pyspark:
    build:
      context: ./pyspark
    volumes:
      - ./pyspark/scripts:/pyspark/scripts
    depends_on:
      - minio
      - mc
      - nessie
    environment:
       AWS_S3_ENDPOINT: "http://minio:9000"
       <<: *aws_env  

volumes:
  postgres-data:
    driver: local
  minio-data:   
    driver: local

PySpark Dockerfile配置 #

FROM bitnami/spark:3.5

ADD https://repo.maven.apache.org/maven2/org/apache/iceberg/iceberg-spark-runtime-3.5_2.12/1.5.2/iceberg-spark-runtime-3.5_2.12-1.5.2.jar /opt/bitnami/spark/jars
ADD https://repo.maven.apache.org/maven2/org/projectnessie/nessie-integrations/nessie-spark-extensions-3.5_2.12/0.99.0/nessie-spark-extensions-3.5_2.12-0.99.0.jar /opt/bitnami/spark/jars
ADD https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-aws-bundle/1.5.2/iceberg-aws-bundle-1.5.2.jar /opt/bitnami/spark/jars
ADD https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.12.773/aws-java-sdk-bundle-1.12.773.jar /opt/bitnami/spark/jars
ADD https://repo1.maven.org/maven2/software/amazon/awssdk/url-connection-client/2.28.16/url-connection-client-2.28.16.jar /opt/bitnami/spark/jars
ADD https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/3.3.6/hadoop-aws-3.3.6.jar /opt/bitnami/spark/jars

USER root

RUN pip3 install py4j

PySpark脚本原配置 #

spark = SparkSession.builder \
    .appName("test") \
    .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions,org.projectnessie.spark.extensions.NessieSparkSessionExtensions") \
    .config("spark.sql.catalog.sensors_catalog", "org.apache.iceberg.spark.SparkCatalog") \
    .config("spark.sql.catalog.sensors_catalog.warehouse", "s3a://sensors-lakehouse") \
    .config("spark.sql.warehouse.dir", "s3a://sensors-lakehouse") \
    .config("spark.sql.catalog.sensors_catalog.catalog-impl", "org.apache.iceberg.nessie.NessieCatalog") \
    .config("spark.sql.catalog.sensors_catalog.uri", "http://nessie:19120/api/v1") \
    .config("spark.sql.catalog.sensors_catalog.ref", "main") \
    .config("spark.sql.catalog.sensors_catalog.io-impl", "org.apache.iceberg.aws.s3.S3FileIO") \
    .getOrCreate()
spark._jsc.hadoopConfiguration().unset("fs.s3a.aws.credentials.provider")
# Explicitly set the credentials in Hadoop configuration
spark._jsc.hadoopConfiguration().set("fs.s3a.access.key", "minioadmin")
spark._jsc.hadoopConfiguration().set("fs.s3a.secret.key", "minioadmin")
spark._jsc.hadoopConfiguration().set("fs.s3a.path.style.access", "true")
spark._jsc.hadoopConfiguration().set("fs.s3a.connection.ssl.enabled", "false")
spark._jsc.hadoopConfiguration().set("fs.s3a.proxy.host", "minio")
spark._jsc.hadoopConfiguration().set("fs.s3a.endpoint", "minio")
spark._jsc.hadoopConfiguration().set("fs.s3a.proxy.port", "9000")
spark._jsc.hadoopConfiguration().set("fs.s3a.endpoint.region", "us-west-1")

已尝试操作 #

• 在Spark脚本中显式设置Hadoop s3a凭据
• 环境变量和脚本均使用minioadmin作为密钥
• 将MinIO桶sensors-lakehouse设置为公共访问

问题原因 #

你使用的是Iceberg的S3FileIO，它依赖AWS SDK v2，不会读取Hadoop的fs.s3a.*配置项。脚本里修改的是Hadoop的s3a配置，但Iceberg直接用SDK访问MinIO，所以这些配置完全没生效，导致认证失败。

解决方案 #

直接在SparkSession构建阶段，为Iceberg catalog添加专属的S3配置项，替换原来修改Hadoop配置的代码：

修改后的PySpark脚本 #

spark = SparkSession.builder \
    .appName("test") \
    .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions,org.projectnessie.spark.extensions.NessieSparkSessionExtensions") \
    .config("spark.sql.catalog.sensors_catalog", "org.apache.iceberg.spark.SparkCatalog") \
    .config("spark.sql.catalog.sensors_catalog.warehouse", "s3a://sensors-lakehouse") \
    .config("spark.sql.warehouse.dir", "s3a://sensors-lakehouse") \
    .config("spark.sql.catalog.sensors_catalog.catalog-impl", "org.apache.iceberg.nessie.NessieCatalog") \
    .config("spark.sql.catalog.sensors_catalog.uri", "http://nessie:19120/api/v1") \
    .config("spark.sql.catalog.sensors_catalog.ref", "main") \
    .config("spark.sql.catalog.sensors_catalog.io-impl", "org.apache.iceberg.aws.s3.S3FileIO") \
    # 添加Iceberg S3FileIO专属配置
    .config("spark.sql.catalog.sensors_catalog.s3.access-key-id", "minioadmin") \
    .config("spark.sql.catalog.sensors_catalog.s3.secret-access-key", "minioadmin") \
    .config("spark.sql.catalog.sensors_catalog.s3.endpoint", "http://minio:9000") \
    .config("spark.sql.catalog.sensors_catalog.s3.path-style-access", "true") \
    .config("spark.sql.catalog.sensors_catalog.s3.region", "us-west-1") \
    .getOrCreate()

# 移除所有修改Hadoop s3a配置的代码

额外说明 #

• 如果环境变量中的AWS配置正确，也可以不用显式设置access-key-id和secret-access-key，Iceberg的S3FileIO会自动读取环境变量
• 确保MinIO的endpoint配置是完整的http://minio:9000，不要只写主机名，避免SDK解析错误

内容的提问来源于stack exchange，提问作者A_A