Spark中Hadoop S3配置覆盖冲突：写入MinIO实例错误

MinIO部署配置 #

使用Docker Compose部署两个独立的MinIO实例，配置如下：

version: "3.9"
services:
  minio:
    image: quay.io/minio/minio:latest
    ports:
      - "9000:9000"
      - "9001:9001"
    environment:
      MINIO_ROOT_USER: "user"
      MINIO_ROOT_PASSWORD: "pwd"
    volumes:
      - ./data/minio:/data # 持久化存储数据
    command: server /data --console-address ":9001"

  minio2:
    image: quay.io/minio/minio:latest
    ports:
      - "9002:9000"
      - "9003:9001"
    environment:
      MINIO_ROOT_USER: "user2"
      MINIO_ROOT_PASSWORD: "pwd2"
    volumes:
      - ./data/minio2:/data # 持久化存储数据
    command: server /data --console-address ":9001"

Spark Scala代码 #

尝试从第一个MinIO读取CSV文件并写入第二个MinIO的代码如下：

import org.apache.spark.sql.SparkSession

object Main extends App {

  // 初始化SparkSession
  val spark = SparkSession
    .builder()
    .appName("O2SS_SparkServer")
    .master("local[*]")
    .config("spark.ui.port", "4040")
    .getOrCreate()
  spark.sparkContext.setLogLevel("WARN")

  // 从第一个MinIO读取CSV
  val endpoint  = "http://127.0.0.1:9000"
  val accessKey = "user"
  val secretKey = "pwd"
  val bucket    = "test_bucket"
  spark.sparkContext.hadoopConfiguration.set(s"fs.s3a.bucket.$bucket.endpoint", endpoint)
  spark.sparkContext.hadoopConfiguration.set(s"fs.s3a.bucket.$bucket.access.key", accessKey)
  spark.sparkContext.hadoopConfiguration.set(s"fs.s3a.bucket.$bucket.secret.key", secretKey)
  val input     = spark.read.csv(s"s3a://$bucket/<someFile>.csv")

  // 写入第二个MinIO
  val endpoint2  = "http://127.0.0.1:9002"
  val accessKey2 = "user2"
  val secretKey2 = "pwd2"
  val bucket2    = "test_bucket"
  spark.sparkContext.hadoopConfiguration.set(s"fs.s3a.bucket.$bucket2.endpoint", endpoint2)
  spark.sparkContext.hadoopConfiguration.set(s"fs.s3a.bucket.$bucket2.access.key", accessKey2)
  spark.sparkContext.hadoopConfiguration.set(s"fs.s3a.bucket.$bucket2.secret.key", secretKey2)
  input.write.parquet(s"s3a://$bucket2/IfNotInS3_2_THENBUG.parquet")

}

问题现象 #

执行上述代码后，数据并未写入第二个MinIO实例，而是错误地写入了第一个MinIO实例。

备注 #

• 已知按Bucket配置的机制以及S3 Bucket命名唯一性规则，在S3中不会出现此问题，此点无需讨论
• 当前使用Spark 3.3.1版本

内容的提问来源于stack exchange，提问作者James JIANG