Spark应用连接远程Hadoop集群HBase报错求助：无有效认证方法

问题 #

运行基于Spark的数据对账应用，需连接两个不同集群的HBase表做数据对比。本地集群HBase数据可正常连接获取，但连接远程集群时抛出以下错误：

failed on local exception:exception is Failed after attempts=4, exceptions: 2023-04-19T17:11:13.491Z, RpcRetryingCaller{globalStartTime=1681924273451, pause=1000, maxAttempts=4}, java.io.IOException: Call to Host:16020 failed on local exception: java.io.IOException: java.lang.RuntimeException: Found no valid authentication method from options 2023-04-19T17:11:14.505Z, RpcRetryingCaller{globalStartTime=1681924273451, pause=1000, maxAttempts=4}, java.io.IOException: Call to host:16020 failed on local exception: java.io.IOException: java.lang.RuntimeException: Found no valid authentication method from options 2023-04-19T17:11:16.522Z, RpcRetryingCaller{globalStartTime=1681924273451, pause=1000, maxAttempts=4}, java.io.IOException: Call to host:16020 failed on local exception: java.io.IOException: java.lang.RuntimeException: Found no valid authentication method from options 2023-04-19T17:11:19.549Z, RpcRetryingCaller{globalStartTime=1681924273451, pause=1000, maxAttempts=4}, java.io.IOException: Call to host:16020 failed on local exception: java.io.IOException: java.lang.RuntimeException: Found no valid authentication method from options

已配置UserGroupInformation.loginUserFromKeytab(userLoginString, keytabPath);指定用户名和keytab文件路径，求解决办法。

解决方法 #

• 匹配远程集群的Kerberos配置
  确保应用使用的krb5.conf是远程集群的配置文件，而非本地集群的。远程集群的KDC地址、Realm信息必须正确，可通过JVM参数-Djava.security.krb5.conf=/path/to/remote/krb5.conf强制指定。

• 验证Keytab与用户权限

  1. 用kinit -kt /path/to/keytab user@REALM手动测试keytab能否获取远程集群的Kerberos票据，失败则说明keytab无效或用户名/Realm不匹配。
  2. 检查远程HBase是否给该用户授权，执行HBase Shell命令grant 'user', 'RW', 'target_table'确认权限配置。

• 隔离双集群的认证上下文
  同时连接两个集群时，需避免本地认证上下文覆盖远程的，用UserGroupInformation.doAs为远程HBase操作单独指定认证：

  Configuration remoteHBaseConf = HBaseConfiguration.create();
  remoteHBaseConf.set("hbase.zookeeper.quorum", "remote-zk-host-list");
  remoteHBaseConf.set("hbase.security.authentication", "kerberos");
  remoteHBaseConf.set("hbase.master.kerberos.principal", "hbase/master@REMOTE_REALM");
  remoteHBaseConf.set("hbase.regionserver.kerberos.principal", "hbase/regionserver@REMOTE_REALM");
  
  UserGroupInformation ugi = UserGroupInformation.loginUserFromKeytab(userLoginString, keytabPath);
  ugi.doAs(new PrivilegedAction<Void>() {
      @Override
      public Void run() {
          try (Connection conn = ConnectionFactory.createConnection(remoteHBaseConf)) {
              // 执行远程HBase数据读取逻辑
          } catch (IOException e) {
              e.printStackTrace();
          }
          return null;
      }
  });
  

• 配置Spark的Kerberos传递
  若Spark运行在Yarn集群，提交任务时需添加以下参数确保认证传递：

  --conf spark.yarn.principal=user@REMOTE_REALM \
  --conf spark.yarn.keytab=/path/to/keytab \
  --conf spark.security.credentials.hbase.enabled=true
  

• 排查网络连通性
  用telnet remote-host 16020测试应用机器到远程HBase Master端口的连通性，若不通需协调运维开放对应端口。

内容的提问来源于stack exchange，提问作者Sudarsana Kasireddy