基于Spark、外部Hive Metastore与S3搭建Open Data Lakehouse报错排查

错误核心是Hive Metastore（HMS）的默认本地仓库路径覆盖了Spark中配置的S3路径，Iceberg使用HMS返回的本地路径生成数据文件URI，但你指定的S3FileIO无法识别file://协议，导致报错。

1. 修改HMS的仓库配置与依赖 #

在Docker Compose的metastore和hiveserver2服务中添加环境变量，指定S3为默认仓库，并添加Hadoop AWS依赖以支持S3路径：

# 修改metastore服务配置
metastore:
  image: apache/hive:4.0.1
  container_name: metastore
  depends_on:
    - postgres
  environment:
    SERVICE_NAME: metastore
    DB_DRIVER: postgres
    # 添加S3仓库路径配置
    HIVE_METASTORE_WAREHOUSE_DIR: s3a://bucket-fs-686190543346/dwh/
    SERVICE_OPTS: >
      -Djavax.jdo.option.ConnectionDriverName=org.postgresql.Driver
      -Djavax.jdo.option.ConnectionURL=jdbc:postgresql://postgres:5432/metastore_db
      -Djavax.jdo.option.ConnectionUserName=hive
      -Djavax.jdo.option.ConnectionPassword=hivepassword
      # 添加Hadoop S3相关配置
      -Dhadoop.fs.s3a.endpoint=s3.ap-south-1.amazonaws.com
      -Dhadoop.fs.s3a.path.style.access=true
      -Dhadoop.fs.s3a.connection.ssl.enabled=true
  ports:
    - "9083:9083"
  volumes:
    - ./postgresql-42.7.7.jar:/opt/hive/lib/postgres.jar
    # 添加Hadoop AWS依赖jar包（版本需与Spark中hadoop-aws一致，这里是3.3.4）
    - ./hadoop-aws-3.3.4.jar:/opt/hive/lib/hadoop-aws-3.3.4.jar
    - ./aws-java-sdk-bundle-1.12.592.jar:/opt/hive/lib/aws-java-sdk-bundle-1.12.592.jar

# 修改hiveserver2服务配置
hiveserver2:
  image: apache/hive:4.0.1
  container_name: hiveserver2
  depends_on:
    - metastore
  environment:
    SERVICE_NAME: hiveserver2
    IS_RESUME: "true"
    HIVE_METASTORE_WAREHOUSE_DIR: s3a://bucket-fs-686190543346/dwh/
    SERVICE_OPTS: >
      -Dhive.metastore.uris=thrift://metastore:9083
      -Djavax.jdo.option.ConnectionDriverName=org.postgresql.Driver
      -Djavax.jdo.option.ConnectionURL=jdbc:postgresql://postgres:5432/metastore_db
      -Djavax.jdo.option.ConnectionUserName=hive
      -Djavax.jdo.option.ConnectionPassword=hivepassword
      -Dhadoop.fs.s3a.endpoint=s3.ap-south-1.amazonaws.com
      -Dhadoop.fs.s3a.path.style.access=true
      -Dhadoop.fs.s3a.connection.ssl.enabled=true
  ports:
    - "10000:10000"
    - "10002:10002"
  volumes:
    - ./postgresql-42.7.7.jar:/opt/hive/lib/postgres.jar
    - ./hadoop-aws-3.3.4.jar:/opt/hive/lib/hadoop-aws-3.3.4.jar
    - ./aws-java-sdk-bundle-1.12.592.jar:/opt/hive/lib/aws-java-sdk-bundle-1.12.592.jar

2. 调整SparkSession配置 #

• 移除enableHiveSupport()，避免Spark内置Hive客户端与Iceberg Hive Catalog冲突
• 明确指定Iceberg Catalog读取HMS的配置，确保优先级高于HMS默认值

SparkSession.builder.appName("IcebergPySpark")
    .config(
        "spark.jars.packages",
        "org.apache.iceberg:iceberg-spark-runtime-3.3_2.12:1.3.0,org.apache.hadoop:hadoop-aws:3.3.4,software.amazon.awssdk:bundle:2.17.257,software.amazon.awssdk:url-connection-client:2.17.257",
    )
    .config("spark.sql.catalog.my_catalog", "org.apache.iceberg.spark.SparkCatalog")
    .config("spark.sql.catalog.my_catalog.type", "hive")
    .config("spark.sql.catalog.my_catalog.warehouse", "s3a://bucket-fs-686190543346/dwh/")
    .config("spark.sql.catalog.my_catalog.hive.metastore.uris", "thrift://172.17.0.1:9083")
    # 强制指定HMS仓库路径，覆盖HMS内部配置
    .config("spark.sql.catalog.my_catalog.hive.conf.hive.metastore.warehouse.dir", "s3a://bucket-fs-686190543346/dwh/")
    .config("spark.sql.catalog.my_catalog.io-impl", "org.apache.iceberg.aws.s3.S3FileIO")
    .config("spark.hadoop.fs.s3a.endpoint", "s3.ap-south-1.amazonaws.com")
    .config("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
    .config("spark.hadoop.fs.s3a.path.style.access", "true")
    .config("spark.hadoop.fs.s3a.connection.ssl.enabled", "true")
    .config("spark.hadoop.fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider")
    .config("spark.hadoop.fs.s3a.multiobjectdelete.enable", "false")
    # 移除enableHiveSupport()
    .getOrCreate()

3. 重启服务并验证 #

• 重启HMS服务：

  docker-compose down
  docker-compose up -d
  

• 在Spark中执行SQL验证表存储路径：

  DESCRIBE EXTENDED my_catalog.my_table;
  

  检查Location字段是否为指定的S3路径。

内容的提问来源于stack exchange，提问作者Zafar