CDP 7.1.9环境下Spark-Solr连接执行无报错但停滞问题

环境配置 #

CDP version: 7.1.9
Spark: Spark3
Solr: 8.11
Spark-Solr Connector: opt/cloudera/parcels/SPARK3/lib/spark3/spark-solr/spark-solr-3.9.3000.3.3.7191000.0-78-shaded.jar

问题现象 #

执行Spark读写Solr的基础任务时，进程无限期停滞，无报错也无结果。Hive、HBase等其他组件与Spark集成正常，Kerberos凭证能正常认证其他Hadoop组件，curl和Python requests调用Solr REST API也能正常获取数据。

指定不存在的Solr集合时会收到「集合不存在」的错误，说明ZooKeeper初始连接正常，但Spark Executor节点无法与Solr节点建立有效连接。Spark UI的stderr日志无有效信息。

所用Spark代码 #

solr_options = {
    "zkhost": "zkURL-01.orgis.ie:2181,zkURL-02.orgis.ie:2181,zkURL.orgis.ie:2181/solr",
    "collection": "collection_phoectic_test2"
}

# Read data from Solr
df = spark.read.format("solr").options(**solr_options).load()
df.show()

错误日志详情 #

DEBUG http.wire: [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)]: http-outgoing-0 >> "POST /solr/collection_phonectic_test2_shard1_replica_n1/select HTTP/1.1[\r][\n]"
 DEBUG http.wire: [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)]: http-outgoing-0 >> "Content-Type: application/x-www-form-urlencoded; charset=UTF-8[\r][\n]"
 DEBUG http.wire: [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)]: http-outgoing-0 >> "User-Agent: Solr[org.apache.solr.client.solrj.impl.HttpSolrClient] 1.0[\r][\n]"
 DEBUG http.wire: [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)]: http-outgoing-0 >> "Content-Length: 652[\r][\n]"
 DEBUG http.wire: [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)]: http-outgoing-0 >> "Host: worker-02.xx:8985[\r][\n]"
 DEBUG http.wire: [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)]: http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
 DEBUG http.wire: [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)]: http-outgoing-0 >> "[\r][\n]"
 DEBUG http.wire: [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)]: http-outgoing-0 >> "q=*%3A*&rows=5000&qt=%2Fselect&fq=_version_%3A%5B*+TO+1812630352655548416%5D&fq=%7B%21hash+workers%3D2+worker%3D0%7D&collection=collection_phonectic_test2&fl=address%2Cmade%2Ccategory%2Ccompanyname%2Cuserfeedback&distrib=false&start=0&sort=id+asc&partitionKeys=_version_&cursorMark=*&wt=javabin&version=2"
 DEBUG http.wire: [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)]: http-outgoing-0 << "HTTP/1.1 401 Authentication required[\r][\n]"
 DEBUG http.wire: [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)]: http-outgoing-0 << "Content-Security-Policy: default-src 'none'; base-uri 'none'; connect-src 'self'; form-action 'self'; font-src 'self'; frame-ancestors 'none'; img-src 'self'; media-src 'self'; style-src 'self' 'unsafe-inline'; script-src 'self'; worker-src 'self';[\r][\n]"
 DEBUG http.wire: [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)]: http-outgoing-0 << "X-Content-Type-Options: nosniff[\r][\n]"
 DEBUG http.wire: [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)]: http-outgoing-0 << "X-Frame-Options: SAMEORIGIN[\r][\n]"
 DEBUG http.wire: [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)]: http-outgoing-0 << "X-XSS-Protection: 1; mode=block[\r][\n]"
 DEBUG http.wire: [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)]: http-outgoing-0 << "Strict-Transport-Security: max-age=31536000; includeSubDomains[\r][\n]"
 DEBUG http.wire: [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)]: http-outgoing-0 << "WWW-Authenticate: Negotiate[\r][\n]"
 DEBUG http.wire: [Executor task launch worker for task 0.0 in stage 0.0 (TID 0)]: http-outgoing-0 << "Set-Cookie: hadoop.auth=; Secure; HttpOnly[\r][\n]"

排查建议 #

从日志看，Spark请求Solr时返回了401 Authentication required，且要求Negotiate认证，说明核心问题是Spark-Solr Connector没有正确传递Kerberos凭证到Solr节点。建议按以下步骤排查：

• 确认Spark Executor节点的Kerberos ticket有效：在Executor节点上执行klist检查凭证是否存在且未过期
• 配置Spark-Solr Connector的Kerberos参数：在solr_options中添加authentication=kerberos、kerberos.principal、kerberos.keytab（或依赖Spark的全局Kerberos配置）
• 检查Solr的Kerberos配置：确保Solr节点启用了SPNEGO认证，且Spark使用的主体有权限访问Solr集合
• 验证Executor节点能解析Solr节点的主机名：日志中显示的worker-02.xx是否能被Executor正确解析，避免DNS问题导致连接停滞
• 调整Connector的日志级别：开启更详细的SolrJ日志，查看认证过程中的具体报错

内容的提问来源于stack exchange，提问作者BigD