Docker环境下Airflow执行spark-submit失败，报错找不到Java路径

在Docker环境中使用Airflow自动化PySpark代码时执行报错，完整错误信息如下：

[2024-05-30, 09:53:15 UTC] {spark_submit.py:401} INFO - Spark-Submit cmd: spark-submit --master spark://***-spark-master-1:7077 --name arrow-spark --deploy-mode cluster jobs/python/ETL.py
[2024-05-30, 09:53:15 UTC] {spark_submit.py:571} INFO - /home/***/.local/lib/python3.8/site-packages/pyspark/bin/spark-class: line 71: /usr/lib/jvm/java-11-openjdk-amd64/bin/java: No such file or directory
[2024-05-30, 09:53:15 UTC] {spark_submit.py:571} INFO - /home/***/.local/lib/python3.8/site-packages/pyspark/bin/spark-class: line 97: CMD: bad array subscript
[2024-05-30, 09:53:16 UTC] {taskinstance.py:2698} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 433, in _execute_task
    result = execute_callable(context=context, **execute_callable_kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/apache/spark/operators/spark_submit.py", line 174, in execute
    self._hook.submit(self.application)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/apache/spark/hooks/spark_submit.py", line 502, in submit
    raise AirflowException(
airflow.exceptions.AirflowException: Cannot execute: spark-submit --master spark://***-spark-master-1:7077 --name arrow-spark --deploy-mode cluster jobs/python/ETL.py. Error code is: 1.
[2024-05-30, 09:53:16 UTC] {taskinstance.py:1138} INFO - Marking task as FAILED. dag_id=sparking_flow, task_id=python_job, execution_date=20240530T095308, start_date=20240530T095315, end_date=20240530T095316
[2024-05-30, 09:53:16 UTC] {standard_task_runner.py:107} ERROR - Failed to execute job 1368 for task python_job (Cannot execute: spark-submit --master spark://***-spark-master-1:7077 --name arrow-spark --deploy-mode cluster jobs/python/ETL.py. Error code is: 1.; 73)
[2024-05-30, 09:53:16 UTC] {local_task_job_runner.py:234} INFO - Task exited with return code 1
[2024-05-30, 09:53:16 UTC] {taskinstance.py:3280} INFO - 0 downstream tasks scheduled from follow-on schedule check

已配置Airflow的Spark连接，对应的DAG代码：

import airflow
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.providers.apache.spark.operators.spark_submit import SparkSubmitOperator

dag = DAG(
    dag_id = "sparking_flow",
    default_args = {
        "owner": "airflow",
        "start_date": airflow.utils.dates.days_ago(1)
    },
    schedule_interval = "@yearly"
)

start = PythonOperator(
    task_id="start",
    python_callable = lambda: print("Jobs started"),
    dag=dag
)

python_job = SparkSubmitOperator(
    task_id="python_job",
    conn_id="spark_default",
    application="jobs/python/ETL.py",
    dag=dag
)

end = PythonOperator(
    task_id="end",
    python_callable = lambda: print("completed successfully"),
    dag=dag
)

start >> python_job >> end

原因分析 #

• 核心错误是/usr/lib/jvm/java-11-openjdk-amd64/bin/java: No such file or directory，说明执行spark-submit的Airflow容器中未安装Java 11环境，或Java路径配置错误。
• Spark依赖Java运行环境，当Airflow容器缺失Java时，spark-class脚本找不到Java可执行文件，进而引发后续数组下标错误和任务失败。

解决方法 #

方法1：自定义Airflow镜像并安装Java 11 #

在Airflow的Dockerfile中添加Java 11安装步骤：

# 基于官方Airflow镜像
FROM apache/airflow:2.x.x-python3.8

# 切换root用户安装Java
USER root
RUN apt-get update && apt-get install -y openjdk-11-jdk && rm -rf /var/lib/apt/lists/*

# 设置JAVA_HOME环境变量
ENV JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
ENV PATH=$PATH:$JAVA_HOME/bin

# 切回airflow用户
USER airflow

重新构建镜像并启动容器。

方法2：通过Docker Compose挂载宿主机Java环境（临时方案） #

如果不想重新构建镜像，可在docker-compose.yml中将宿主机的Java 11目录挂载到Airflow容器：

services:
  airflow-worker:
    volumes:
      - /usr/lib/jvm/java-11-openjdk-amd64:/usr/lib/jvm/java-11-openjdk-amd64
    environment:
      - JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
      - PATH=$PATH:$JAVA_HOME/bin

注意：宿主机必须已安装Java 11，且路径与挂载路径一致。

方法3：调整SparkSubmitOperator的执行模式 #

当前使用deploy-mode cluster模式，若Airflow仅作为任务提交客户端，可切换为client模式（前提是Spark集群已配置好Java环境，且能访问到ETL.py文件）。修改DAG中的SparkSubmitOperator：

python_job = SparkSubmitOperator(
    task_id="python_job",
    conn_id="spark_default",
    application="jobs/python/ETL.py",
    deploy_mode="client",
    dag=dag
)

若使用集群模式，需确保Spark Worker节点能访问到ETL.py文件（如上传至HDFS或共享存储）。

额外检查项 #

• 确认Airflow的Spark连接配置中，spark_home是否正确指向Spark安装路径。
• 检查ETL.py文件在Airflow容器或Spark集群中的可访问性。

内容的提问来源于stack exchange，提问作者rahes