非CDH环境Airflow Docker向Kerberos CDH集群提交Spark作业报Hive实例化错误

基础场景 #

• 运行环境：未纳入Cloudera Manager管控的Airflow Docker容器
• 提交目标：开启Kerberos认证的远程CDH YARN集群
• 作业内容：简单Hive表查询PySpark脚本
• 作业运行状态：可被YARN集群正常接收调度，但启动后两次运行均触发初始化失败

异常提交命令 #

airflow@41166b660d82:~$ spark-submit --master yarn --deploy-mode cluster --keytab keytab_name.keytab --principal --jars keytab_name@REALM --jars /path/to/spark-hive_2.11-2.3.0.jar sranje.py

错误日志 #

携带对应hive jar包提交时，核心报错栈如下：

...
2020-12-31 10:11:43 INFO  StateStoreCoordinatorRef:54 - Registered StateStoreCoordinator endpoint
Traceback (most recent call last):
  File "sranje.py", line 21, in <module>
    source_df = hiveContext.table(hive_source).na.fill("")
  File "/dfs/dn4/yarn/nm/usercache/etladmin/appcache/application_1608187067076_0150/container_e29_1608187067076_0150_02_000001/pyspark.zip/pyspark/sql/context.py", line 366, in table
  File "/dfs/dn4/yarn/nm/usercache/etladmin/appcache/application_1608187067076_0150/container_e29_1608187067076_0150_02_000001/pyspark.zip/pyspark/sql/session.py", line 721, in table
  File "/dfs/dn4/yarn/nm/usercache/etladmin/appcache/application_1608187067076_0150/container_e29_1608187067076_0150_02_000001/py4j-0.10.6-src.zip/py4j/java_gateway.py", line 1160, in __call__
  File "/dfs/dn4/yarn/nm/usercache/etladmin/appcache/application_1608187067076_0150/container_e29_1608187067076_0150_02_000001/pyspark.zip/pyspark/sql/utils.py", line 79, in deco
pyspark.sql.utils.IllegalArgumentException: u"Error while instantiating 'org.apache.spark.sql.hive.HiveExternalCatalog':"
2020-12-31 10:11:43 ERROR ApplicationMaster:70 - User application exited with status 1
2020-12-31 10:11:43 INFO  ApplicationMaster:54 - Final app status: FAILED, exitCode: 1, (reason: User application exited with status 1)
...

去掉--jars /path/to/spark-hive_2.11-2.3.0.jar参数提交时，报错会变为：

...
pyspark.sql.utils.IllegalArgumentException: u"Error while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder':"
...

已完成排查记录 #

• 提交命令执行前已确认存在有效Kerberos票据
• 本地环境版本对齐：Spark 2.3.0、Hadoop 2.6.0、Java版本与CDH集群对应版本完全一致
• 已确认提交时附带的hive-site.xml、yarn-site.xml等配置文件内容有效
• 相同提交命令与作业脚本在CDH集群内部节点运行可正常执行
• 已尝试追加多组依赖提交：--jars spark-hive_2.11-2.3.0.jar,spark-core_2.11-2.3.0.jar,spark-sql_2.11-2.3.0.jar,hive-hcatalog-core-2.3.0.jar,spark-hive-thriftserver_2.11-2.3.0.jar，问题仍未解决
• 测试脚本1（HiveContext实现）：

# -*- coding: utf-8 -*-
import sys
reload(sys)
sys.setdefaultencoding('utf-8')
from pyspark.context import SparkContext
from pyspark.sql import SparkSession, SQLContext, HiveContext, functions as F
from pyspark.sql.utils import AnalysisException
from datetime import datetime

sc = SparkContext.getOrCreate()
spark = SparkSession(sc)
sqlContext = SQLContext(sc)
hiveContext = HiveContext(sc)

current_date = str(datetime.now().strftime('%Y-%m-%d'))
hive_source = "lnz_ch.lnz_cfg_codebook"
source_df = hiveContext.table(hive_source).na.fill("")

print("Number of records: {}".format(source_df.count()))
print("First 20 rows of the table:")
source_df.show(20)

• 测试脚本2（SparkSession开启Hive支持实现），运行仍复现相同报错：

# -*- coding: utf-8 -*-
import sys
reload(sys)
sys.setdefaultencoding('utf-8')
from pyspark.sql.types import *
from pyspark.sql.functions import *
from pyspark.sql import SparkSession

if __name__ == "__main__":
    spark = SparkSession.builder.appName("ZekoTest").enableHiveSupport().getOrCreate()
    data = spark.sql("SELECT * FROM lnz_ch.lnz_cfg_codebook")
    data.show(20)
    spark.close()

根因定位方向 #

• 提交命令参数格式错误：原命令中--principal参数未绑定实际主体值，错误将principal字符串放在--jars参数值列表中，导致Spark启动阶段Kerberos认证上下文初始化不完整，后续Hive元数据访问阶段因认证上下文缺失触发实例化失败
• 依赖版本不兼容：非CDH管控环境使用的是开源社区版Spark 2.3.0，手动上传的开源版spark-hive jar与CDH集群内置的Hive类版本存在冲突，仅上传单个jar无法覆盖HiveExternalCatalog初始化所需的全量依赖，会触发类加载链路断裂
• 网络与解析缺失：集群模式下ApplicationMaster运行在CDH集群节点上，若Docker提交环境未将Hive Metastore主机、集群Kerberos KDC主机的解析配置同步分发到容器，会导致初始化HiveExternalCatalog时无法连通服务端，且错误日志未打印底层连通性异常

可行解决方案 #

• 修正spark-submit参数格式，正确写法参考：

spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --keytab /path/to/keytab_name.keytab \
  --principal keytab_name@REALM \
  --conf spark.sql.hive.metastore.jars=builtin \
  --conf spark.security.credentials.hive.files=hive-site.xml \
  sranje.py

注意：不要手动使用开源社区版spark-hive jar包，优先使用Spark内置的Hive元数据访问依赖，避免类加载冲突

• 同步集群DNS配置到Docker容器，确保可正常解析NameNode、ResourceManager、Hive Metastore、Kerberos KDC的主机名
• 若必须使用外部jar包，需直接从CDH集群节点的Spark安装路径下拷贝对应版本的hive相关jar，禁止使用Maven仓库下载的开源版依赖

内容的提问来源于stack exchange，提问作者adrijand