从EMR Serverless提交Hudi任务后无法在AWS Glue注册表/库

使用EMR 6.15和Hudi 0.14提交了一份Hudi任务，预期在AWS Glue中创建数据库和表。EMR Serverless绑定的IAM角色已具备S3和Glue的必要权限，Hudi表已成功创建，但数据库和表未在AWS Glue中注册。

提交的任务命令 #

aws emr-serverless start-job-run \
  --application-id xx \
  --execution-role-arn arn:aws:iam::xxx:role/EmrExecutionRole \
  --name 'create_huditable' \
  --execution-timeout-minutes 15 \
  --job-driver '{
    "sparkSubmit": {"entryPoint": "/usr/lib/hudi/hudi-utilities-bundle.jar",
    "entryPointArguments": [
            "--table-type","COPY_ON_WRITE",
            "--op","INSERT",
            "--target-base-path","s3://bucket-dev-emr-data/test/dbhome/rjdemo/employee",
            "--target-table","employee",
            "--source-class","org.apache.hudi.utilities.sources.CsvDFSSource",
            "--source-ordering-field","name",
            "--hoodie-conf","hoodie.streamer.source.dfs.root=s3://bucket-dev-emr-data/test/sampleCsvHudi.csv",
            "--hoodie-conf","hoodie.datasource.write.recordkey.field=name",
            "--hoodie-conf","hoodie.datasource.hive_sync.support_timestamp=true",
            "--hoodie-conf","hoodie.datasource.hive_sync.database=rjdemo",
            "--hoodie-conf","hoodie.datasource.hive_sync.table=employee",
            "--hoodie-conf","hoodie.datasource.write.schema.allow.auto.evolution.column.drop=true",
            "--hoodie-conf","hoodie.streamer.csv.inferSchema=true",
            "--hoodie-conf","hoodie.streamer.csv.header=true",
            "--hoodie-conf","hoodie.streamer.csv.sep=,",
            "--hoodie-conf","hoodie.datasource.hive_sync.use_jdbc=false",
            "--hoodie-conf","hoodie.datasource.hive_sync.enable=true",
            "--hoodie-conf","hoodie.datasource.hive_sync.mode=hms"
        ],
    "sparkSubmitParameters": "--conf spark.jars=/usr/lib/hudi/hudi-spark-bundle.jar --class org.apache.hudi.utilities.streamer.HoodieStreamer --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.hadoop.hive.metastore.client.factory.class=com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"}}'

EMR Serverless默认Spark配置 #

spark.sql.catalogImplementation = hive
spark.hadoop.hive.metastore.client.factory.class = com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory

针对Hudi表已创建但未同步到Glue的问题，需补充以下关键配置：

• 自动创建Glue数据库：Hudi默认不会自动创建不存在的Glue数据库，需添加hoodie.datasource.hive_sync.create_database_if_not_exists=true配置项，确保目标数据库不存在时自动生成。
• 指定Glue Catalog关联：在Spark提交参数中添加--conf spark.sql.catalog.glue_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog，同时补充Hudi配置hoodie.datasource.hive_sync.catalog=glue_catalog，明确同步目标为Glue数据目录。
• 配置HMS连接信息：补充hoodie.datasource.hive_sync.metastore.uris=thrift://localhost:9083，EMR Serverless中Hive Metastore默认绑定本地端口，该配置确保Hudi能正确连接HMS进而同步到Glue。
• 指定Hive仓库路径：在sparkSubmitParameters中添加--conf spark.hadoop.hive.metastore.warehouse.dir=s3://bucket-dev-emr-data/test/dbhome/，让Glue能正确关联表的存储位置。

调整后的关键配置补充示例 #

在entryPointArguments中新增：

"--hoodie-conf","hoodie.datasource.hive_sync.create_database_if_not_exists=true",
"--hoodie-conf","hoodie.datasource.hive_sync.catalog=glue_catalog",
"--hoodie-conf","hoodie.datasource.hive_sync.metastore.uris=thrift://localhost:9083"

在sparkSubmitParameters中新增：

--conf spark.sql.catalog.glue_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog --conf spark.hadoop.hive.metastore.warehouse.dir=s3://bucket-dev-emr-data/test/dbhome/

内容的提问来源于stack exchange，提问作者Roobal Jindal