Windows环境下Docker部署Spark创建Delta Table时容器循环重启求助

在Windows环境通过Docker Compose部署Spark 3.3集群，尝试写入Delta Table时，Worker节点的Executor会不断退出并被Master重启，循环往复无有效执行结果，Linux环境下无此异常。

相关日志 #

biocloud-core-dockercompose-spark-1         | 23/03/22 16:56:41 INFO Master: Removing executor app-20230322165616-0000/23 because it is EXITED
biocloud-core-dockercompose-spark-1         | 23/03/22 16:56:41 INFO Master: Launching executor app-20230322165616-0000/25 on worker worker-20230322165555-172.21.0.4-39781
biocloud-core-dockercompose-spark-worker-2  | 23/03/22 16:56:41 INFO Worker: Asked to launch executor app-20230322165616-0000/25 for create-Raw-tables
biocloud-core-dockercompose-spark-worker-2  | 23/03/22 16:56:41 INFO SecurityManager: Changing view acls to: spark
biocloud-core-dockercompose-spark-worker-2  | 23/03/22 16:56:41 INFO SecurityManager: Changing modify acls to: spark
biocloud-core-dockercompose-spark-worker-2  | 23/03/22 16:56:41 INFO SecurityManager: Changing view acls groups to:
biocloud-core-dockercompose-spark-worker-2  | 23/03/22 16:56:41 INFO SecurityManager: Changing modify acls groups to:
biocloud-core-dockercompose-spark-worker-2  | 23/03/22 16:56:41 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(spark); groups with view permissions: Set(); users  with modify permissions: Set(spark); groups with modify permissions: Set()
biocloud-core-dockercompose-spark-worker-2  | 23/03/22 16:56:41 INFO ExecutorRunner: Launch command: "/opt/bitnami/java/bin/java" "-cp" "/opt/bitnami/spark/conf/:/opt/bitnami/spark/jars/*" "-Xmx1024M" "-Dspark.driver.port=54879" "-XX:+IgnoreUnrecognizedVMOptions" "--add-opens=java.base/java.lang=ALL-UNNAMED" "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED" "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED" "--add-opens=java.base/java.io=ALL-UNNAMED" "--add-opens=java.base/java.net=ALL-UNNAMED" "--add-opens=java.base/java.nio=ALL-UNNAMED" "--add-opens=java.base/java.util=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED" "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED" "--add-opens=java.base/sun.security.action=ALL-UNNAMED" "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED" "--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@host.docker.internal:54879" "--executor-id" "25" "--hostname" "172.21.0.4" "--cores" "1" "--app-id" "app-20230322165616-0000" "--worker-url" "spark://Worker@172.21.0.4:39781"
biocloud-core-dockercompose-spark-worker-1  | 23/03/22 16:56:43 INFO Worker: Executor app-20230322165616-0000/24 finished with state EXITED message Command exited with code 1 exitStatus 1
biocloud-core-dockercompose-spark-worker-1  | 23/03/22 16:56:43 INFO ExternalShuffleBlockResolver: Clean up non-shuffle and non-RDD files associated with the finished executor 24
biocloud-core-dockercompose-spark-worker-1  | 23/03/22 16:56:43 INFO ExternalShuffleBlockResolver: Executor is not registered (appId=app-20230322165616-0000, execId=24)

Spark连接代码 #

builder = SparkSession.builder.appName(app_name).master(self.spark_master)  \
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
    .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \
    .config("spark.jars.packages", "{maven_packages}") \
    .config("spark.hadoop.fs.s3a.access.key", Config.S3_ACCESS_KEY_ID) \
    .config("spark.hadoop.fs.s3a.secret.key", Config.S3_SECRET_ACCESS_KEY) \
    .config("spark.hadoop.fs.s3a.path.style.access", "true") \
    .config("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem") \
    .config("spark.shuffle.service.enabled", "false") \
    .config("spark.dynamicAllocation.enabled", "false")

self.spark_session = configure_spark_with_delta_pip(builder).enableHiveSupport().getOrCreate()

Docker Compose配置 #

version: '2'

services:

  spark:
    image: docker.io/bitnami/spark:3.3
    hostname: spark
    environment:
      - SPARK_MODE=master
      - SPARK_RPC_AUTHENTICATION_ENABLED=no
      - SPARK_RPC_ENCRYPTION_ENABLED=no
      - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
      - SPARK_SSL_ENABLED=no
    ports:
      - '8080:8080'
      - '7077:7077'
    volumes:
      - /data/spark:/bitnami/spark

  spark-worker:
    image: docker.io/bitnami/spark:3.3
    deploy:
      replicas: 2
    environment:
      - SPARK_MODE=worker
      - SPARK_MASTER_URL=spark://spark:7077
      - SPARK_WORKER_MEMORY=1G
      - SPARK_WORKER_CORES=1
      - SPARK_RPC_AUTHENTICATION_ENABLED=no
      - SPARK_RPC_ENCRYPTION_ENABLED=no
      - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
      - SPARK_SSL_ENABLED=no

排查方向 #

• 文件系统与权限问题：Windows下Docker卷挂载的/data/spark:/bitnami/spark路径，可能存在权限不兼容问题。Bitnami镜像默认使用spark用户运行，Windows挂载的目录权限可能导致Executor无法读写临时文件或Delta表数据。可尝试更换为Windows本地路径（如./data/spark:/bitnami/spark），或在容器中添加user: root环境变量临时提升权限测试。
• Delta依赖兼容性：确保spark.jars.packages中指定的Delta版本与Spark 3.3兼容（Delta 2.2.0适配Spark 3.3）。Windows环境下动态下载依赖可能出现缓存或网络问题，建议预先在Docker镜像中安装Delta包，避免跨平台依赖差异。
• 网络通信问题：日志中Driver地址使用host.docker.internal，Windows下Docker的该地址解析可能存在异常，导致Worker节点的Executor无法连接Driver。可在Spark配置中添加spark.driver.bindAddress=0.0.0.0，让Driver监听所有网卡，确保Worker能正常访问。
• 资源不足问题：Windows下Docker默认内存配额较低，Worker节点1G内存可能不足以支撑Delta表写入的内存开销，导致Executor OOM退出。可调整Docker Desktop的资源分配（增加内存），或调高SPARK_WORKER_MEMORY至2G测试。

内容的提问来源于stack exchange，提问作者Leon Cullens