Apache Airflow WebServer SSL权限错误排查求助

问题现象 #

调度器和Worker运行正常，但Web门户无法访问，报错SSL权限拒绝。

报错日志 #

[2023-06-27 02:09:34 +0000] [1710631] [ERROR] Socket error processing request.
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.10/site-packages/gunicorn/workers/sync.py", line 131, in handle
    client = ssl.wrap_socket(client, server_side=True,
  File "/usr/lib/python3.10/ssl.py", line 1439, in wrap_socket
    context.load_cert_chain(certfile, keyfile)
PermissionError: [Errno 13] Permission denied

Airflow SSL配置（airflow.cfg） #

# Paths to the SSL certificate and key for the web server. When both are
# provided SSL will be enabled. This does not change the web server port.
web_server_ssl_cert = /etc/letsencrypt/live/mysite.com/cert.pem

# Paths to the SSL certificate and key for the web server. When both are
# provided SSL will be enabled. This does not change the web server port.
web_server_ssl_key = /etc/letsencrypt/live/mysite.com/privkey.pem

SSL证书目录权限（/etc/letsencrypt/live/mysite.com） #

airflow@etl:/etc/letsencrypt/live/mysite.com$ ls -al
total 12
drwxrw-rw- 2 airflow root 4096 May 13 09:27 .
drwxrw-rw- 3 airflow root 4096 Nov 14  2022 ..
-rwxrw-rw- 1 airflow root  692 Nov 14  2022 README
lrwxrwxrwx 1 root    root   36 May 13 09:27 cert.pem -> ../../archive/mysite.com/cert4.pem
lrwxrwxrwx 1 root    root   37 May 13 09:27 chain.pem -> ../../archive/mysite.com/chain4.pem
lrwxrwxrwx 1 root    root   41 May 13 09:27 fullchain.pem -> ../../archive/mysite.com/fullchain4.pem
lrwxrwxrwx 1 root    root   39 May 13 09:27 privkey.pem -> ../../archive/mysite.com/privkey4.pem

SSL证书归档目录权限（/etc/letsencrypt/archive/mysite.com） #

airflow@etl:/etc/letsencrypt/archive/mysite.com$ ls -al
total 88
drwxr-xr-x 2 airflow root 4096 May 13 09:27 .
drwx------ 3 airflow root 4096 Nov 14  2022 ..
-rw-r--r-- 1 airflow root 1838 Nov 14  2022 cert1.pem
-rw-r--r-- 1 airflow root 1834 Jan 13 03:31 cert2.pem
-rw-r--r-- 1 root    root 1838 Mar 14 03:59 cert3.pem
-rw-r--r-- 1 root    root 1834 May 13 09:27 cert4.pem
-rw-r--r-- 1 airflow root 3749 Nov 14  2022 chain1.pem
-rw-r--r-- 1 airflow root 3749 Jan 13 03:31 chain2.pem
-rw-r--r-- 1 root    root 3749 Mar 14 03:59 chain3.pem
-rw-r--r-- 1 root    root 3749 May 13 09:27 chain4.pem
-rw-r--r-- 1 airflow root 5587 Nov 14  2022 fullchain1.pem
-rw-r--r-- 1 airflow root 5583 Jan 13 03:31 fullchain2.pem
-rw-r--r-- 1 root    root 5587 Mar 14 03:59 fullchain3.pem
-rw-r--r-- 1 root    root 5583 May 13 09:27 fullchain4.pem
-rw------- 1 airflow root 1704 Nov 14  2022 privkey1.pem
-rw------- 1 airflow root 1704 Jan 13 03:31 privkey2.pem
-rw------- 1 root    root 1704 Mar 14 03:59 privkey3.pem
-rw------- 1 root    root 1704 May 13 09:27 privkey4.pem

WebServer守护进程配置（/usr/lib/systemd/system/airflow-webserver.service） #

#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.

[Unit]
Description=Airflow webserver daemon
After=network.target postgresql.service rabbitmq-server.service
Wants=postgresql.service mysql.service rabbitmq-server.service

[Service]
EnvironmentFile=/home/airflow/airflow/airflow.conf
User=airflow
Group=airflow
Type=simple
ExecStart=/home/airflow/.local/bin/airflow webserver --pid /run/airflow/webserver.pid
Restart=on-failure
RestartSec=5s
PrivateTmp=true

[Install]
WantedBy=multi-user.target

问题原因与解决步骤 #

原因分析 #

Airflow WebServer以airflow用户运行，但归档目录下的最新证书文件（cert4.pem、privkey4.pem等）属主为root，airflow用户无读取权限，导致SSL加载失败。

解决步骤 #

1. 调整归档目录下最新证书文件的属主，让airflow用户拥有读取权限：

   sudo chown airflow:root /etc/letsencrypt/archive/mysite.com/cert4.pem
   sudo chown airflow:root /etc/letsencrypt/archive/mysite.com/privkey4.pem
   sudo chown airflow:root /etc/letsencrypt/archive/mysite.com/chain4.pem
   sudo chown airflow:root /etc/letsencrypt/archive/mysite.com/fullchain4.pem
   

2. 确保私钥文件权限符合安全要求（仅所有者可读）：

   sudo chmod 600 /etc/letsencrypt/archive/mysite.com/privkey4.pem
   

3. 重启Airflow WebServer服务：

   sudo systemctl restart airflow-webserver
   

避免后续证书更新重复触发问题 #

配置certbot自动更新后的钩子脚本，确保证书权限自动调整：

1. 创建post更新钩子脚本：

   sudo nano /etc/letsencrypt/renewal-hooks/post/fix-airflow-cert-perms.sh
   

   写入以下内容：

   #!/bin/bash
   chown airflow:root /etc/letsencrypt/archive/mysite.com/*.pem
   chmod 600 /etc/letsencrypt/archive/mysite.com/privkey*.pem
   systemctl restart airflow-webserver
   

2. 赋予脚本执行权限：

   sudo chmod +x /etc/letsencrypt/renewal-hooks/post/fix-airflow-cert-perms.sh
   

内容的提问来源于stack exchange，提问作者arcee123