无法从Nessie访问兼容S3的MINIO服务器上的Iceberg表元数据

已通过PySpark结合Nessie和MINIO完成Iceberg表的读写操作，PySpark代码如下：

#!/usr/bin/python3.9

from pyspark.sql import SparkSession

iceberg_spark_jar = '/AKE/iceberg-spark-runtime-3.5_2.12-1.9.0.jar'
hadoop_aws_jar = '/AKE/hadoop-aws-3.3.4.jar'
aws_java_sdk_jar = '/AKE/aws-java-sdk-bundle-1.11.1026.jar'

warehouse_path = '/user/hive/warehouse'
catalog = 'nessie'
namespace = 'ake_database'
table = 'ake_iceberg_table'

# Initialize Spark session with Hive support and Iceberg configuration
spark = SparkSession.builder \
.appName("Iceberg with Nessie Catalog") \
    .config("spark.sql.catalog.nessie", "org.apache.iceberg.spark.SparkCatalog") \
    .config("spark.sql.catalog.nessie.catalog-impl", "org.apache.iceberg.nessie.NessieCatalog") \
    .config("spark.sql.catalog.nessie.uri", "http://localhost:19120/api/v1") \
    .config("spark.sql.catalog.nessie.ref", "main") \
    .config("spark.sql.catalog.nessie.warehouse", "s3a://ake-bucket/ake-warehouse") \
    .config("spark.hadoop.fs.s3a.endpoint", "http://localhost:9000") \
    .config("spark.hadoop.fs.s3a.access.key", "minioadmin") \
    .config("spark.hadoop.fs.s3a.secret.key", "minioadmin123") \
    .config("spark.hadoop.fs.s3a.path.style.access", "true") \
    .config("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem") \
    .config("spark.jars", f"{iceberg_spark_jar},{hadoop_aws_jar},{aws_java_sdk_jar}") \
    .getOrCreate()

# Example DataFrame
data = [("Alice", 34), ("Bob", 45), ("Cathy", 29)]
columns = ["Name", "Age"]
df = spark.createDataFrame(data, columns)

# Drop an Iceberg table from database
spark.sql(f"DROP TABLE IF EXISTS {catalog}.{namespace}.{table}")

# Drop the database
spark.sql(f"DROP DATABASE IF EXISTS {catalog}.{namespace}")

# Create the database
spark.sql(f"CREATE NAMESPACE IF NOT EXISTS {catalog}.{namespace}")

spark.sql(f"SHOW NAMESPACES IN {catalog}").show()
spark.sql(f"SHOW TABLES IN {catalog}.{namespace}").show()

# Create an Iceberg table directly in the database
spark.sql(f"""
CREATE TABLE IF NOT EXISTS {catalog}.{namespace}.{table} (
    name STRING, 
    age INT
)
USING iceberg
""")

df.write.format("iceberg").mode("overwrite").saveAsTable(f"{catalog}.{namespace}.{table}")

iceberg_df = spark.read.format("iceberg").load(f"{catalog}.{namespace}.{table}")
iceberg_df.show()

通过MINIO WebUI可以确认表已成功创建，但访问Nessie WebUI的表地址时，出现以下异常：

{
"error" : {
"message" : "Failed to read table metadata from s3a://ake-bucket/ake-warehouse/ake_database/ake_iceberg_table_9b043614-15b0-4b84-91b0-0d31a056632e/metadata/00001-4988857f-b819-4a96-806f-0fe6f163dd1a.metadata.json",
"type" : "RuntimeException",
"code" : 500
}
}

MINIO的Docker启动命令：

docker run -it --name minio-local -d -p 9000:9000 -p 9001:9001 \
-e "MINIO_ROOT_USER=minioadmin" \
-e "MINIO_ROOT_PASSWORD=minioadmin123" \
-v /ake-data/minio-data:/data \
quay.io/minio/minio server /data --console-address ":9001"

Nessie的Docker启动命令：

sudo docker run -d --name nessie01041 -p 19120:19120 -v /ake-data/nessie/warehouse:/warehouse -e nessie.catalog.warehouses='ake-warehouse' -e nessie.catalog.warehouses.ake-warehouse.location="s3a://ake-bucket/ake-warehouse" -e nessie.catalog.default-warehouse=ake-warehouse \
-e nessie.catalog.service.s3.default-options.endpoint=http://localhost:9000 \
-e nessie.catalog.service.s3.default-options.access-key=urn:nessie-secret:quarkus:nessie.catalog.secrets.access-key \
-e nessie.catalog.secrets.access-key.name=minioadmin \
-e nessie.catalog.secrets.access-key.secret=minioadmin123 \
-e nessie.catalog.service.s3.default-options.path-style-access=true \
ghcr.io/projectnessie/nessie:0.104.1

请求解决此无法通过Nessie访问MINIO上Iceberg表元数据的问题。

内容的提问来源于stack exchange，提问作者Ahmed Kamal ELSaman