Docker部署Airflow时安装HDFS Provider遇krb5-config权限拒绝

问题详情 #

在Airflow-Docker 2.5.3环境中尝试安装apache-airflow-providers-apache-hdfs==3.2.1库，已以root身份安装所有必要的Kerberos组件，但切换为airflow用户执行pip install -r requirements.txt时，出现“krb5-config: Permission denied”错误，且在容器内找不到krb5-config文件。

错误日志 #

#0 5.236 Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /home/airflow/.local/lib/python3.10/site-packages (from aiohttp->apache-airflow-providers-http->apache-airflow>=2.3.0->apache-airflow-providers-apache-hdfs==3.2.1->-r /requirements.txt (line 2)) (4.0.2)
#0 5.311 Collecting krb5>=0.3.0
#0 5.335   Downloading krb5-0.5.0.tar.gz (220 kB)
#0 5.353      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 221.0/221.0 kB 13.2 MB/s eta 0:00:00
#0 5.394   Installing build dependencies: started
#0 9.611   Installing build dependencies: finished with status 'done'
#0 9.618   Getting requirements to build wheel: started
#0 9.945   Getting requirements to build wheel: finished with status 'error'
#0 9.951   error: subprocess-exited-with-error
#0 9.951   
#0 9.951   × Getting requirements to build wheel did not run successfully.
#0 9.951   │ exit code: 1
#0 9.951   ╰─> [22 lines of output]
#0 9.951       /bin/sh: 1: krb5-config: Permission denied
#0 9.951       Using krb5-config at 'krb5-config'
#0 9.951       Traceback (most recent call last):
#0 9.951         File "/home/airflow/.local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
#0 9.951           main()
... (part of the log is removed) ...
#0 9.951         File "/usr/local/lib/python3.10/subprocess.py", line 526, in run
#0 9.951           raise CalledProcessError(retcode, process.args,
#0 9.951       subprocess.CalledProcessError: Command '('krb5-config --cflags krb5',)' returned non-zero exit status 127.
#0 9.951       [end of output]
#0 9.951   
#0 9.951   note: This error originates from a subprocess, and is likely not a problem with pip.
#0 9.953 error: subprocess-exited-with-error
#0 9.953 
#0 9.953 × Getting requirements to build wheel did not run successfully.
#0 9.953 │ exit code: 1
#0 9.953 ╰─> See above for output.

原Dockerfile #

FROM apache/airflow:2.5.3-python3.10

ENV DEBIAN_FRONTEND=noninteractive
ENV TERM linux

USER root

RUN set -ex \
    && buildDeps=' \
        krb5-config \
        krb5-user \
        libpam-krb5 \
        libkrb5-dev \
    ' \
    && apt-get -qq update \
    && apt-get -yqq install --no-install-recommends $buildDeps \
    && apt-get purge --auto-remove -yqq $buildDeps \
    && apt-get -yqq clean


USER airflow

# Copy requirement.txt into Docker
COPY requirements.txt /

## the code failed on the following line ##
RUN pip install --no-cache-dir -r /requirements.txt

requirements.txt #

apache-airflow-providers-apache-hdfs==3.2.1

问题原因 #

核心问题是Kerberos编译依赖被提前删除：原Dockerfile中安装krb5-config等编译依赖后，立刻执行apt-get purge将其删除，导致后续以airflow用户执行pip install时，python的krb5包（apache-airflow-providers-apache-hdfs的依赖）需要调用krb5-config获取编译参数，但该工具已被删除，因此报错“Permission denied”（实际是命令不存在，shell返回错误码127的误提示）。

解决方案 #

调整Dockerfile的执行顺序，先保留编译依赖完成pip install，再删除依赖：

FROM apache/airflow:2.5.3-python3.10

ENV DEBIAN_FRONTEND=noninteractive
ENV TERM linux

USER root

# 安装编译依赖但暂不删除
RUN set -ex \
    && buildDeps=' \
        krb5-config \
        krb5-user \
        libpam-krb5 \
        libkrb5-dev \
    ' \
    && apt-get -qq update \
    && apt-get -yqq install --no-install-recommends $buildDeps

USER airflow

# 复制并安装Python依赖
COPY requirements.txt /
RUN pip install --no-cache-dir -r /requirements.txt

USER root

# 完成编译后再清理依赖
RUN apt-get purge --auto-remove -yqq $buildDeps \
    && apt-get -yqq clean

# 切回airflow用户
USER airflow

说明 #

1. 先以root身份安装所有Kerberos编译依赖，确保krb5-config存在
2. 切换到airflow用户执行pip install，此时krb5包可以正常调用krb5-config完成编译
3. 最后再切换回root用户清理编译依赖，减少镜像体积

内容的提问来源于stack exchange，提问作者Donny