如何将动态IP转换为字符串写入kernel.json的Spark conf配置

提供两种可直接落地的方案，按需选择：

方案1：用包装脚本替代kernel.json中直接调用的ipykernel启动命令 #

这种方案无需改现有spark初始化逻辑，仅需加一层壳做IP替换：

1. 先创建一个启动脚本比如/opt/conda/bin/ipykernel_with_spark_ip.sh，内容如下：

#!/bin/bash
# 先获取本机IP，注意如果有多个网卡可以调整grep/awk规则取你需要的IP
export DYNAMIC_IP=$(hostname -i | awk '{print $1}')
# 拼接完整的PYSPARK_SUBMIT_ARGS，替换掉IP占位
export PYSPARK_SUBMIT_ARGS="--master spark://spark-master-svc.xxxxx.svc.cluster.local:7077 --conf spark.driver.host=${DYNAMIC_IP} --conf spark.driver.port=8888 --conf spark.driver.bindAddress=${DYNAMIC_IP} pyspark-shell"
# 执行原本的ipykernel启动命令
exec /opt/conda/bin/python3 -m ipykernel_launcher "$@"

2. 给脚本加执行权限：chmod +x /opt/conda/bin/ipykernel_with_spark_ip.sh
3. 修改kernel.json的argv字段，替换原本的启动命令，同时删掉原env中写死的PYSPARK_SUBMIT_ARGS配置即可：

{
  "argv": [
    "/opt/conda/bin/ipykernel_with_spark_ip.sh",
    "-f",
    "{connection_file}"
  ],
  "display_name": "python",
  "language": "python",
  "env": {
    "SPARK_HOME": "/usr/local/spark",
    "JAVA_HOME":"/usr/lib/jvm/java-11-openjdk-amd64",
    "PYSPARK_DRIVER_PYTHON": "/opt/conda/bin/python3",
    "PYSPARK_PYTHON": "/usr/bin/python3",
    "PYLIB":"/usr/local/spark/python/lib",
    "PYTHONPATH": "/usr/local/spark/python/:/usr/local/spark/python/lib/py4j-0.10.9-src.zip",
    "PYTHONSTARTUP": "/usr/local/spark/python/pyspark/shell.py"
 }
}

注：该方案适合多用户共用同一个kernel配置的场景，不需要修改上层业务代码

方案2：代码层面动态初始化SparkSession，不依赖PYSPARK_SUBMIT_ARGS #

如果不想改动内核配置，可以把spark配置的逻辑挪到代码里，初始化的时候动态获取IP填充：

import socket
from pyspark.sql import SparkSession

# 动态获取本机出口IP
def get_host_ip():
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    try:
        # 不需要真的建立连接，仅用于获取当前网络路由对应的出口IP
        s.connect(('10.255.255.255', 1))
        ip = s.getsockname()[0]
    except Exception:
        ip = '127.0.0.1'
    finally:
        s.close()
    return ip

local_ip = get_host_ip()

# 初始化SparkSession的时候传入所有动态配置
spark = SparkSession.builder \
    .master("spark://spark-master-svc.xxxxx.svc.cluster.local:7077") \
    .config("spark.driver.host", local_ip) \
    .config("spark.driver.port", "8888") \
    .config("spark.driver.bindAddress", local_ip) \
    .getOrCreate()

注：用该方案需要把kernel.json里的PYSPARK_SUBMIT_ARGS和PYTHONSTARTUP配置删掉，避免默认启动的spark shell加载错误的固定配置

内容的提问来源于stack exchange，提问作者mam667