Airflow-Spark-Docker环境下Client模式无法创建Spark文件

问题描述 #

在Airflow中运行Spark任务时出现以下错误：

org.apache.spark.sql.delta.DeltaIOException: [DELTA_CANNOT_CREATE_LOG_PATH] Cannot create file:/opt/bitnami/spark/my_lakehouse/staging/customers/_delta_log

但直接在Spark Docker容器中运行相同任务时，文件可正常创建，且Airflow与Spark容器已挂载同一份lakehouse卷。

用户提供的docker-compose配置如下：

version: '3'

x-spark-common: &spark-common
  build:
    context: .
    dockerfile: Dockerfile.spark
  volumes:
    - ./jobs:/opt/bitnami/spark/jobs
    - lakehouse:/opt/bitnami/spark/my_lakehouse
  networks:
    - ani

x-airflow-common: &airflow-common
  build:
    context: .
    dockerfile: Dockerfile.airflow
  env_file:
    - airflow.env
  volumes:
    - ./jobs:/opt/airflow/jobs
    - ./dags:/opt/airflow/dags
    - ./logs:/opt/airflow/logs
    - ./spark_jar_deps:/opt/bitnami/spark/jars
    - lakehouse:/opt/bitnami/spark/my_lakehouse
  depends_on:
    - postgres
  networks:
    - ani

services:
  spark-master:
    <<: *spark-common
    command: bin/spark-class org.apache.spark.deploy.master.Master
    ports:
      - "9090:8080"
      - "7077:7077"

  spark-worker-1:
    <<: *spark-common
    command: bin/spark-class org.apache.spark.deploy.worker.Worker spark://spark-master:7077
    depends_on:
      - spark-master
    environment:
      SPARK_MODE: worker
      SPARK_WORKER_CORES: 2
      SPARK_WORKER_MEMORY: 1g
      SPARK_MASTER_URL: spark://spark-master:7077
  

  postgres:
    image: postgres:14.0
    environment:
      - POSTGRES_USER=airflow
      - POSTGRES_PASSWORD=airflow
      - POSTGRES_DB=airflow
    networks:
      - ani

  webserver:
    <<: *airflow-common
    command: webserver
    ports:
      - "8080:8080"
    depends_on:
      - scheduler

  scheduler:
    <<: *airflow-common
    command: bash -c "airflow db migrate && airflow users create --username admin --firstname abc --lastname qwe --role Admin --email abc@gmail.com --password admin@123 && airflow scheduler"

  mssql:
    image: mcr.microsoft.com/mssql/server:2022-latest
    environment:
      - ACCEPT_EULA=Y
      - SA_PASSWORD=pasS_123
      - MSSQL_AGENT_ENABLED=true
    ports:
      - "1433:1433"
    networks:
      - ani

networks:
  ani:

volumes:
  lakehouse:
    driver: local

排查与解决方案 #

1. 容器权限不匹配 #

Spark容器通常以spark用户运行，而Airflow容器默认以airflow用户运行，两个用户对挂载卷的权限可能不一致，导致Airflow触发任务时无法写入路径。

• 检查权限：

  # 查看Spark容器内目录权限
  docker exec -it spark-master ls -ld /opt/bitnami/spark/my_lakehouse
  # 查看Airflow容器内目录权限
  docker exec -it scheduler ls -ld /opt/bitnami/spark/my_lakehouse
  

• 修复权限：
  在Airflow的Dockerfile中添加权限配置：

  RUN chown -R airflow:airflow /opt/bitnami/spark/my_lakehouse
  

  或在Spark的Dockerfile中开放目录权限：

  RUN chmod -R 775 /opt/bitnami/spark/my_lakehouse && chown -R spark:spark /opt/bitnami/spark/my_lakehouse
  

2. Spark任务执行方式错误 #

若Airflow任务是在本地执行Spark（而非提交到Spark集群），Airflow容器内的Spark环境可能缺少Delta Lake依赖，或路径配置错误。

• 确保任务通过spark-submit提交到集群：

  spark-submit --master spark://spark-master:7077 --packages io.delta:delta-core_2.12:2.4.0 /opt/airflow/jobs/your_job.py
  

• 检查Airflow容器内的Spark依赖：确认spark_jar_deps挂载目录包含Delta Lake的jar包，且Spark配置中启用Delta扩展。

3. 卷挂载有效性验证 #

虽然配置中已挂载同一份卷，但需确认Airflow容器内的路径实际可访问：

docker exec -it scheduler ls /opt/bitnami/spark/my_lakehouse

若目录不存在或无法访问，需检查卷挂载配置是否正确，重启容器重新挂载。

4. Spark Worker节点权限检查 #

集群模式下，Spark Worker节点需要对挂载路径有读写权限，可在Worker容器中执行权限检查：

docker exec -it spark-worker-1 ls -ld /opt/bitnami/spark/my_lakehouse

若权限不足，同方案1修复Worker容器内的目录权限。

内容的提问来源于stack exchange，提问作者Aniketh Deshpande