Spark集群模式下Kerberos认证连接Metastore失败求助

核心问题定位 #

你遇到的GSSException: No valid credentials provided本质是Worker节点无法自动加载Kerberos凭证，默认 fallback 到SIMPLE认证，主要原因是配置未正确分发、核心Kerberos参数缺失，以及JAAS配置细节错误。

具体修正步骤 #

1. 分发必要的认证文件到所有Worker节点

当前spark-submit仅传递了core-site.xml，需要补充JAAS配置和keytab文件，Spark会自动将这些文件同步到Worker的工作目录：

spark-submit --class Main \
--conf "spark.cores.max=4" \
--deploy-mode cluster \
--master spark://sparkmaster01.domain:7090 \
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
--conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
--conf spark.sql.warehouse.dir=s3a://hits/ \
--conf spark.sql.catalog.spark_catalog.type=hive \
--conf spark.sql.catalog.iceberg=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.iceberg.warehouse=s3a://hits/ \
--packages org.apache.iceberg:iceberg-spark-runtime-3.3_2.12:1.2.0,org.apache.hadoop:hadoop-aws:3.3.2,com.amazonaws:aws-java-sdk-bundle:1.11.874 \
--files /etc/spark/core-site.xml,/etc/spark/security/client_jaas.conf,/etc/spark/security/spark.keytab \
--conf spark.hadoop.hadoop.security.authentication=kerberos \
--conf spark.hadoop.hive.metastore.kerberos.keytab.file=spark.keytab \
--conf spark.hadoop.hive.metastore.kerberos.principal=spark/edge01.domain@REALM.LAN \
--conf spark.executor.extraJavaOptions="-Djava.security.auth.login.config=client_jaas.conf" \
--conf spark.driver.extraJavaOptions="-Djava.security.auth.login.config=client_jaas.conf" \
KafkaToIceberg-1.0-SNAPSHOT-jar-with-dependencies.jar

2. 修正JAAS配置文件

调整client_jaas.conf的Client块，将serviceName改为Hive Metastore标准的hive，并使用分发后的相对路径：

KafkaClient {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    storeKey=true
    refreshKrb5Config=true
    serviceName="kafka"
    debug=true
    doNotPrompt=false
    keyTab="spark.keytab"
    principal="spark/edge01.domain@REALM.LAN";
};

Client {
    com.sun.security.auth.module.Krb5LoginModule required
    useKeyTab=true
    storeKey=true
    refreshKrb5Config=true
    serviceName="hive"
    debug=true
    doNotPrompt=false
    keyTab="spark.keytab"
    principal="spark/edge01.domain@REALM.LAN";
};

3. 补充Iceberg Catalog的Kerberos配置

确保Iceberg的Hive Catalog继承Kerberos认证规则，添加以下配置到spark-submit：

--conf spark.sql.catalog.iceberg.catalog-impl=org.apache.iceberg.hive.HiveCatalog \
--conf spark.sql.catalog.iceberg.hive.metastore.uris=thrift://metastore01.domain:9083 \
--conf spark.sql.catalog.iceberg.hive.metastore.sasl.enabled=true \
--conf spark.sql.catalog.iceberg.hive.metastore.kerberos.principal=metastore/metastore01.domain@GEMIUSPOC.LAN

4. 验证Worker节点环境

• 所有Worker节点需安装krb5-libs、krb5-workstation包
• 确保/etc/krb5.conf正确指向FreeIPA的KDC和REALM

为什么之前的配置无效 #

1. 文件未分发：Worker节点没有client_jaas.conf和spark.keytab，无法加载Kerberos凭证
2. 核心参数缺失：未设置spark.hadoop.hadoop.security.authentication=kerberos，Hadoop客户端默认使用SIMPLE认证
3. JAAS服务名错误：原配置中serviceName="metastore"不符合Hive Metastore的SASL服务名规范，导致认证协商失败
4. Iceberg配置不完整：未明确指定Iceberg Catalog使用Hive的Kerberos认证逻辑，绕过了凭证加载流程

内容的提问来源于stack exchange，提问作者notaras