Apache Airflow Docker容器中Git命令缺失及权限问题求助
- 使用镜像:Apache Airflow latest版本镜像(SHA256: 3e0ab13f9525974867cbb612ff632d332b838b05fac4e59bf83f700827744d7d)
- 执行BashOperator任务时触发报错:
代码片段:
报错信息:# Create a BashOperator to sync the Git repositories git_sync_task = BashOperator( task_id='git_sync_task', bash_command='cd /opt/airflow/service1 && git pull origin main', dag=dag, )[2023-01-04, 13:09:30 UTC] {subprocess.py:75} INFO - Running command: ['/bin/bash', '-c', 'cd /opt/***/service1 && git pull origin main']
[2023-01-04, 13:09:30 UTC] {subprocess.py:86} INFO - Output:
[2023-01-04, 13:09:30 UTC] {subprocess.py:93} INFO - /bin/bash: line 1: git: command not found
[2023-01-04, 13:09:30 UTC]{subprocess.py:97} INFO - Command exited with return code 127
[2023-01-04, 13:09:30 UTC] {taskinstance.py:1772} ERROR - Task failed with exception - 手动进入容器安装git时触发权限错误:
default@3df7bafd3a37:/opt/airflow$ apt-get install git
E: Could not open lock file /var/lib/dpkg/lock-frontend - open (13: Permission denied)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), are you root?
方法1:自定义Docker镜像(生产环境首选)
官方Airflow镜像默认未预装Git,构建自定义镜像可永久解决问题:
- 创建
Dockerfile:
FROM apache/airflow:latest USER root # 安装Git并清理缓存 RUN apt-get update && apt-get install -y --no-install-recommends git \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* # 切回airflow用户 USER airflow
- 构建镜像:
docker build -t my-airflow-with-git .
- 修改部署配置(如
docker-compose.yml),将镜像替换为my-airflow-with-git - 重启容器:
docker-compose up -d --build
方法2:临时以Root身份安装Git(仅测试环境使用)
该方式仅适用于临时测试,容器重启后Git会丢失:
- 以Root身份进入目标容器:
docker exec -u root -it airflow_airflow-webserver_1 bash
- 安装Git:
apt-get update && apt-get install -y git
- 切回airflow用户继续操作:
su airflow
方法3:使用Airflow内置GitOperator(最佳实践)
Airflow提供专门的GitOperator用于Git仓库操作,无需手动编写Bash命令:
- 确保环境已安装Git Provider(若未安装,可在自定义镜像中添加
RUN pip install apache-airflow-providers-git) - 替换原BashOperator代码:
from airflow.providers.git.operators.git import GitOperator git_sync_task = GitOperator( task_id='git_sync_task', repo='<你的Git仓库地址>', dest_dir='/opt/airflow/service1', branch='main', pull=True, dag=dag, )
内容的提问来源于stack exchange,提问作者A l w a y s S u n n y

