K8s上SparkContext初始化报错：Py4j网关不安全及Java网关进程退出

错误原因分析及修复方案 #

1. Py4j安全认证错误 #

错误提示：You are trying to pass an insecure Py4j gateway to Spark. This is not allowed as it is a security risk

• 原因：Spark 3.0及以上版本默认启用Py4j网关认证，禁止显式设置spark.authenticate=false，这会带来安全风险。
• 修复：删除set("spark.authenticate", "false")这行配置，Spark会默认启用合规的认证机制。

2. Java Gateway进程退出错误 #

错误提示：RuntimeError: Java gateway process exited before sending its port number
常见触发原因及修复：

• spark.driver.host配置错误：原配置中添加了https://前缀，Spark Driver的主机地址应为纯域名或IP，不需要协议前缀，否则Java网关无法正确解析通信地址，修正为driver-service.jupyterhub.svc.cluster.local。
• K8s API地址协议错误：原配置中k8s://http://需改为k8s://https://，Kubernetes API服务器默认使用HTTPS，HTTP协议会导致认证失败。
• Spark镜像验证：确保pyspark:latest镜像包含完整的Spark运行环境（JDK、Spark二进制文件等），自定义镜像需验证构建逻辑是否正确。
• 端口与Service匹配：确认spark.driver.port=2222与对应的Kubernetes Service端口一致，保证Executor能正常访问Driver。

修正后的完整配置 #

conf = (SparkConf().setMaster("k8s://https://kubernetes.default.svc.cluster.local:443") 
    .set("spark.kubernetes.container.image", "pyspark:latest") # Spark image name
    .set("spark.driver.port", "2222") # Needs to match svc
    .set("spark.driver.blockManager.port", "7777")
    .set("spark.driver.host", "driver-service.jupyterhub.svc.cluster.local") 
    .set("spark.driver.bindAddress", "0.0.0.0")
    .set("spark.kubernetes.namespace", "spark")
    .set("spark.kubernetes.authenticate.driver.serviceAccountName", "spark")
    .set("spark.kubernetes.authenticate.serviceAccountName", "spark")
    .set("spark.kubernetes.authenticate.submission.caCertFile","/var/run/secrets/kubernetes.io/serviceaccount/ca.crt")
    .set("spark.kubernetes.authenticate.submission.oauthTokenFile","/var/run/secrets/kubernetes.io/serviceaccount/token")
    .set("spark.executor.instances", "2")
    .set("spark.kubernetes.container.image.pullPolicy", "IfNotPresent")
    .set("spark.app.name", "spark-test"))

内容的提问来源于stack exchange，提问作者KUNAL DAS