Airflow WebServer以systemd服务运行时OAuth认证失败排查

问题背景 #

在Ubuntu 22.04上运行Airflow 2.3.3，WebServer配置了Azure AD的OAuth认证授权机制。以ubuntu用户身份通过命令行airflow webserver -D启动时，认证功能完全正常；但通过systemd服务设置开机自启后，WebServer能正常启动并显示登录界面，却在Azure AD认证环节出现超时。

当前配置 #

systemd服务文件 #

路径：/lib/systemd/system/airflow-webserver.service

[Unit]
Description=Airflow webserver daemon
After=network.target
Before=airflow-scheduler.service

[Service]
EnvironmentFile=/home/ubuntu/airflow/airflow.env
User=ubuntu
Group=ubuntu
Type=simple
ExecStart=/usr/bin/python /home/ubuntu/.local/bin/airflow webserver -D
Restart=on-failure
RestartSec=5s
PrivateTmp=false
StandardOutput=file:/home/ubuntu/airflow/logs/webserver/systemd-stdout.log
StandardError=file:/home/ubuntu/airflow/logs/webserver/systemd-errout.log

[Install]
WantedBy=multi-user.target

环境变量文件airflow.env #

export AIRFLOW_CONFIG=/home/ubuntu/airflow/airflow.cfg
export AIRFLOW_HOME=/home/ubuntu/airflow

排查信息（2022-10-04更新） #

系统日志无异常 #

执行sudo journalctl -f -u airflow-webserver仅输出第三方插件的提示信息，与命令行启动时一致，可忽略：

ubuntu@xxx:~/airflow$ sudo journalctl -f -u airflow-webserver
Oct 04 10:07:48 ip-10-194-84-28 airflow[1508]: [2022-10-04 10:07:48,734] {init_appbuilder.py:515} INFO - Registering class RedocView on menu
Oct 04 10:07:48 ip-10-194-84-28 airflow[1508]: [2022-10-04 10:07:48,734] {init_appbuilder.py:515} INFO - Registering class RedocView on menu
Oct 04 10:07:48 ip-10-194-84-28 airflow[1508]: [2022-10-04 10:07:48,735] {baseviews.py:302} INFO - Registering route /redoc ('GET',)
Oct 04 10:07:48 ip-10-194-84-28 airflow[1508]: [2022-10-04 10:07:48,735] {baseviews.py:302} INFO - Registering route /redoc ('GET',)
Oct 04 10:07:48 ip-10-194-84-28 airflow[1508]: /home/ubuntu/.local/lib/python3.10/site-packages/airflow/plugins_manager.py:256 DeprecationWarning: This decorator is deprecated.
Oct 04 10:07:48 ip-10-194-84-28 airflow[1508]: In previous versions, all subclasses of BaseOperator must use apply_default decorator for the `default_args` feature to work properly.
Oct 04 10:07:48 ip-10-194-84-28 airflow[1508]: In current version, it is optional. The decorator is applied automatically using the metaclass.
Oct 04 10:07:48 ip-10-194-84-28 airflow[1508]: /home/ubuntu/.local/lib/python3.10/site-packages/airflow/providers_manager.py:614 DeprecationWarning: The provider airflow-provider-vaultspeed uses `hook-class-names` property in provider-info and has no `connection-types` one. The 'hook-class-names' property has been deprecated in favour of 'connection-types' in Airflow 2.2. Use **both** in case you want to have backwards compatibility with Airflow < 2.2
Oct 04 10:07:48 ip-10-194-84-28 airflow[1508]: [2022-10-04 10:07:48,780] {providers_manager.py:623} WARNING - The connection_type 'snowflake' has been already registered by provider 'airflow-provider-vaultspeed.'
Oct 04 10:07:48 ip-10-194-84-28 airflow[1508]: [2022-10-04 10:07:48,801] {providers_manager.py:623} WARNING - The connection_type 'snowflake' has been already registered by provider 'airflow-provider-vaultspeed.'

进程参数差异 #

通过htop对比发现gunicorn子进程的配置参数存在关键差异：

• 命令行启动（认证正常）：gunicorn进程携带--config /home/ubuntu/airflow/airflow.cfg参数，正确加载Azure AD认证配置
• systemd服务启动（认证失败）：gunicorn进程未携带该配置参数，导致加载的配置不符合预期

解决方案 #

1. 修正systemd服务的ExecStart命令 #

去掉-D参数（systemd本身负责管理守护进程，无需Airflow自行后台运行，否则会导致进程脱离管控，环境变量传递异常），同时直接调用airflow可执行脚本（无需通过python，脚本已关联正确Python环境）：

ExecStart=/home/ubuntu/.local/bin/airflow webserver

2. 修正环境变量文件格式 #

systemd的EnvironmentFile不支持export前缀，修改airflow.env：

AIRFLOW_CONFIG=/home/ubuntu/airflow/airflow.cfg
AIRFLOW_HOME=/home/ubuntu/airflow

3. 添加工作目录配置 #

在[Service]段添加WorkingDirectory，确保进程运行目录与命令行启动时一致：

[Service]
...
WorkingDirectory=/home/ubuntu/airflow
...

4. 重新加载配置并重启服务 #

sudo systemctl daemon-reload
sudo systemctl restart airflow-webserver

验证 #

重启后通过htop查看gunicorn子进程，确认已携带正确的--config参数，再测试Azure AD认证是否正常。

内容的提问来源于stack exchange，提问作者dovregubben