Docker环境下Spark任务执行失败：无法连接host.docker.internal

使用以下Docker Compose配置部署包含Spark、Kafka、ZooKeeper和MongoDB的集群：

version: "3.8"

services:
  kafka:
    image: bitnami/kafka:3.3.2
    container_name: kafka
    hostname: kafka
    restart: "no"
    links:
      - zookeeper
    ports:
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: INTERNAL://:29092,EXTERNAL://:9092
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,EXTERNAL://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      ALLOW_PLAINTEXT_LISTENER: yes

  zookeeper:
    image: bitnami/zookeeper:3.7.1
    container_name: zookeeper
    hostname: zookeeper
    ports:
      - "2181:2181"
      - "2888:2888"
      - "3888:3888"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      ALLOW_ANONYMOUS_LOGIN: yes

  spark-master:
    image: bitnami/spark:latest
    volumes:
      - ./data:/opt/spark-data
    environment:
      - SPARK_MODE=master
      - SPARK_MASTER_PORT=7077
      - SPARK_MASTER_WEBUI_PORT=8080 # Master UI port
      - SPARK_RPC_AUTHENTICATION_ENABLED=no
      - SPARK_RPC_ENCRYPTION_ENABLED=no
      - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
      - SPARK_SSL_ENABLED=no
    ports:
      - "8080:8080" # Exposing Spark master UI to host
      - "7077:7077" # Exposing Spark master communication port to host

  spark-worker:
    image: bitnami/spark:latest
    volumes:
      - ./data:/opt/spark-data
    environment:
      - SPARK_MODE=worker
      - SPARK_MASTER_URL=spark://spark-master:7077
      - SPARK_WORKER_CORES=3
      - SPARK_WORKER_MEMORY=6G
      - SPARK_RPC_AUTHENTICATION_ENABLED=no
      - SPARK_RPC_ENCRYPTION_ENABLED=no
      - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
      - SPARK_SSL_ENABLED=no
    depends_on:
      - spark-master
    ports:
      - "8081:8081" # Worker UI port (optional, if you want to access worker UI)

  database:
    image: mongo:latest
    environment:
      MONGO_INITDB_DATABASE: database
      MONGO_INITDB_ROOT_USERNAME: user
      MONGO_INITDB_ROOT_PASSWORD: password
    ports:
      - "27017:27017"
    volumes:
      - db-data:/data/db
volumes:
  db-data:
    driver: local

可通过以下代码正常启动SparkSession：

spark = SparkSession.builder \
    .appName("MySparkApp") \
    .master("spark://localhost:7077") \
    .getOrCreate()

但执行任何任务时，spark-worker容器的stderr中反复出现如下错误：

[...]
Caused by: java.io.IOException: Failed to connect to host.docker.internal/192.168.65.254:52833
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:294)
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:214)
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:226)
    at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:204)
    at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:202)
    at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:198)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: host.docker.internal/192.168.65.254:52833
Caused by: java.net.ConnectException: Connection refused
    at java.base/sun.nio.ch.Net.pollConnect(Native Method)
    at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
    at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
    at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:840)

问题根源 #

Spark Driver运行在本地主机，Worker运行在Docker容器内。Worker尝试连接Driver时，默认使用host.docker.internal解析主机地址，但Driver未监听该地址，或使用了随机端口导致容器无法访问，最终连接被拒绝。

解决步骤 #

1. 固定Spark Driver的地址与端口 #

修改SparkSession启动代码，明确指定Driver的可访问地址和固定端口，让Worker能准确定位：

spark = SparkSession.builder \
    .appName("MySparkApp") \
    .master("spark://localhost:7077") \
    .config("spark.driver.host", "host.docker.internal") \
    .config("spark.driver.port", "4040") \
    .getOrCreate()

• spark.driver.host设为host.docker.internal，确保Docker容器能解析到主机
• spark.driver.port指定固定端口（如4040，Spark UI默认端口），避免随机端口带来的连接问题

2. 开放主机防火墙端口 #

检查主机防火墙规则，允许Docker容器访问指定的spark.driver.port（如4040），或临时关闭防火墙验证是否为防火墙导致的连接阻断。

3. 可选：将Driver部署到Docker容器 #

若希望所有组件在同一网络内通信，可在Docker Compose中添加spark-driver服务：

spark-driver:
    image: bitnami/spark:latest
    volumes:
      - ./data:/opt/spark-data
      - ./your-app-code:/opt/spark-app
    environment:
      - SPARK_MODE=driver
      - SPARK_MASTER_URL=spark://spark-master:7077
      - SPARK_DRIVER_MEMORY=4G
      - SPARK_RPC_AUTHENTICATION_ENABLED=no
      - SPARK_RPC_ENCRYPTION_ENABLED=no
      - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
      - SPARK_SSL_ENABLED=no
    depends_on:
      - spark-master
    ports:
      - "4040:4040" # Spark UI port

之后在容器内运行Spark应用，Driver与Worker处于同一Docker网络，无需跨网络通信，可彻底避免此类连接问题。

内容的提问来源于stack exchange，提问作者Omar El Atyqy