Kerberos认证HDP集群中Standalone Spark无法读取HDFS及Hive外部表

环境信息 #

• Spark 3.x/2.3
• HDP 2.7（已启用Kerberos）
• 集群节点：
  • HDP节点：master1、node1
  • Spark Standalone集群：node2（master）、node1（worker）

现象 #

Spark Standalone集群无法读取HDFS和Hive外部表，但能读取Hive元数据；使用local模式提交spark-submit可正常读取，指定master为spark://node2:7077时触发认证报错。

报错信息 #

An error occurred while calling o29.csv.
: java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "node2.hadoop.com/10.x.x.x"; destination host is: "master1.hadoop.com":8020; 
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
    at org.apache.hadoop.ipc.Client.call(Client.java:1480)
    at org.apache.hadoop.ipc.Client.call(Client.java:1413)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
    at com.sun.proxy.$Proxy20.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:776)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy21.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2108)
    at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
    at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
    at org.apache.hadoop.fs.FileSystem.isDirectory(FileSystem.java:1439)
    at org.apache.spark.sql.execution.streaming.FileStreamSink$.hasMetadata(FileStreamSink.scala:47)
    at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:376)
    at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:326)
    at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:308)
    at scala.Option.getOrElse(Option.scala:189)
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:308)
    at org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:796)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:750)
Caused by: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
    at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:688)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
    at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:651)
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:738)
    at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:376)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1529)
    at org.apache.hadoop.ipc.Client.call(Client.java:1452)
    ... 35 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
    at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
    at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:414)
    at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:561)
    at org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:376)
    at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:730)
    at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:726)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:726)
    ... 38 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
    at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:162)
    at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
    at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:189)
    at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
    at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
    at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
    at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
    ... 47 more

已尝试的提交方式 #

• spark-submit --master spark://node2:7077 --deploy-mode client abc.py
• spark-submit --master spark://node2:7077 --keytab /tmp/sudhir.keytab --principal sudhir@EXAMPLE.COM ~/abc.py
• spark-submit --master spark://node2:7077 --deploy-mode client --conf spark.kerberos.keytab=/tmp/sudhir.keytab --conf spark.kerberos.principal=sudhir@EXAMPLE.COM ~/abc.py
• 额外添加配置--conf spark.kerberos.access.hadoopFileSystems=hdfs://master1:8020，问题仍未解决

核心原因 #

Standalone模式下，Spark Worker节点上的Executor无法获取Kerberos TGT凭证，导致访问Kerberos认证的HDFS时失败。Client模式下Driver在本地运行有凭证，但Executor运行在Worker节点上未同步凭证信息。

解决步骤 #

1. 统一Worker节点Kerberos配置

   • 所有Spark Worker节点（node1）必须配置与HDP集群一致的krb5.conf，放置在/etc/krb5.conf路径下。
   • 在Worker节点执行kinit sudhir@EXAMPLE.COM -kt /tmp/sudhir.keytab测试，确认能正常获取TGT凭证。

2. 修改Spark集群全局配置
   在所有节点的spark-defaults.conf中添加以下配置：

   spark.executor.extraJavaOptions -Djava.security.krb5.conf=/etc/krb5.conf
   spark.driver.extraJavaOptions -Djava.security.krb5.conf=/etc/krb5.conf
   spark.security.credentials.hdfs.enabled true
   

3. 提交任务时传递完整凭证参数
   使用以下命令提交任务，确保凭证能传递给所有Executor：

   spark-submit \
     --master spark://node2:7077 \
     --deploy-mode client \
     --conf spark.kerberos.keytab=/tmp/sudhir.keytab \
     --conf spark.kerberos.principal=sudhir@EXAMPLE.COM \
     --conf spark.executorEnv.KRB5_CONFIG=/etc/krb5.conf \
     ~/abc.py
   

4. 确保Keytab文件在所有节点可用

   • 将/tmp/sudhir.keytab同步到所有Worker节点的相同路径下，设置权限为600（仅所有者可读可写）。

5. 验证Worker节点HDFS访问权限
   在Worker节点通过kinit获取凭证后，执行hdfs dfs -ls hdfs://master1:8020/测试，确认能正常访问HDFS资源，排除权限配置问题。

内容的提问来源于stack exchange，提问作者Sudhir K Bansal