Docker部署Airflow邮件发送失败：[smtp/smtp_user]未找到及Errno99错误

问题背景 #

在本地Docker部署的Airflow中，使用EmailOperator发送邮件时任务持续失败，日志中出现[smtp/smtp_user] not found in config警告，最终抛出OSError: [Errno 99] Cannot assign requested address错误。

相关代码与配置 #

DAG代码 #

default_args = {
        "owner": "Airflow",
        "start_date": pendulum.today('UTC').add(days=-1),
        "depends_on_past": False,
        "email_on_failure": True,
        "email_on_retry": False,
        "email": "myemail@outlook.com",
        "retries": 1,
        "retry_delay": timedelta(minutes=5)
    }
with DAG(dag_id="send_emails", schedule="@daily", default_args=default_args, catchup=False) as dag:

    sending_email_notification = EmailOperator(
        task_id="sending_email",
        to="myemail@outlook.com",
        subject=f"New email",
        html_content=f"""
            <h3>New Data is available</h3>
        """
        )

    sending_email_notification

已配置的SMTP参数 #

Outlook配置（airflow.cfg）

[smtp]
smtp_host = smtp-mail.outlook.com
smtp_starttls = True
smtp_ssl = False
smtp_user = myemail@outlook.com
smtp_password = mypassword
smtp_port = 587
smtp_mail_from = myemail@outlook.com

Gmail配置（airflow.cfg）

[smtp]
smtp_host = smtp.gmail.com
smtp_starttls = True
smtp_ssl = False
smtp_user = myemail@gmail.com
smtp_password = appssecret
smtp_port = 587
smtp_mail_from = myemail@gmail.com
smtp_timeout = 30
smtp_retry_limit = 5

报错日志 #

[2024-01-31, 21:24:42 UTC] {taskinstance.py:1400} INFO - Marking task as FAILED. dag_id=send_emails, task_id=sending_email, execution_date=20240131T211339, start_date=20240131T212442, end_date=20240131T212442
[2024-01-31, 21:24:42 UTC] {configuration.py:1050} WARNING - section/key [smtp/smtp_user] not found in config
[2024-01-31, 21:24:42 UTC] {email.py:270} INFO - Email alerting: attempt 1
[2024-01-31, 21:24:42 UTC] {configuration.py:1050} WARNING - section/key 
[smtp/smtp_user] not found in config
[2024-01-31, 21:24:42 UTC] {email.py:270} INFO - Email alerting: attempt 1
[2024-01-31, 21:24:42 UTC] {taskinstance.py:2007} ERROR - Failed to send email to: myemail@outlook.com


During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 2005, in handle_failure
self.email_alert(error, task)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 2422, in email_alert
send_email(task.email, subject, html_content_err)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/email.py", line 80, in send_email
 return backend(
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/email.py", line 154, in send_email_smtp
send_mime_email(e_from=mail_from, e_to=recipients, mime_msg=msg, conn_id=conn_id, dryrun=dryrun)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/email.py", line 272, in send_mime_email
smtp_conn = _get_smtp_connection(smtp_host, smtp_port, smtp_timeout, smtp_ssl)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/email.py", line 316, in _get_smtp_connection
return smtplib.SMTP(host=host, port=port, timeout=timeout)
  File "/usr/local/lib/python3.8/smtplib.py", line 255, in __init__
 (code, msg) = self.connect(host, port)
  File "/usr/local/lib/python3.8/smtplib.py", line 339, in connect
self.sock = self._get_socket(host, port, self.timeout)
  File "/usr/local/lib/python3.8/smtplib.py", line 310, in _get_socket
return socket.create_connection((host, port), timeout,
  File "/usr/local/lib/python3.8/socket.py", line 808, in create_connection
raise err
  File "/usr/local/lib/python3.8/socket.py", line 796, in create_connection
sock.connect(sa)
OSError: [Errno 99] Cannot assign requested address
[2024-01-31, 21:24:42 UTC] {standard_task_runner.py:104} ERROR - Failed to execute job 105 for task sending_email ([Errno 99] Cannot assign requested address; 557)
[2024-01-31, 21:24:42 UTC] {local_task_job_runner.py:228} INFO - Task exited with return code 1
[2024-01-31, 21:24:42 UTC] {taskinstance.py:2778} INFO - 0 downstream tasks scheduled from follow-on schedule check

原因分析 #

1. 配置文件未被容器加载：Docker环境中，若本地airflow.cfg未通过卷挂载到容器内部的默认配置路径，容器会使用内置的默认配置，导致自定义SMTP参数无法生效，触发[smtp/smtp_user] not found警告。
2. 容器网络连接受限：Errno 99错误表明容器无法建立到SMTP服务器的网络连接，可能是本地防火墙拦截、容器网络模式配置问题，或代理设置干扰了外部访问。
3. 邮件服务配置细节遗漏：部分邮箱（如Gmail）要求必须开启两步验证并使用应用密码，Outlook也可能需要开启POP3/IMAP服务才能通过SMTP发送邮件。

解决方法 #

1. 确保配置文件正确挂载到容器 #

• 编辑docker-compose.yaml，添加本地airflow.cfg到容器的卷映射，Airflow容器默认配置路径为/opt/airflow/airflow.cfg：

  services:
    airflow-webserver:
      volumes:
        - ./airflow.cfg:/opt/airflow/airflow.cfg
    airflow-worker:
      volumes:
        - ./airflow.cfg:/opt/airflow/airflow.cfg
    # scheduler等其他服务也需添加相同卷映射
  

• 重启Airflow容器使配置生效：

  docker-compose down && docker-compose up -d
  

2. 验证容器网络连通性 #

• 进入Airflow worker容器，测试SMTP服务器的连通性：

  # 替换为你的worker容器名称
  docker exec -it airflow-worker bash
  # 测试Outlook SMTP
  telnet smtp-mail.outlook.com 587
  # 测试Gmail SMTP
  telnet smtp.gmail.com 587
  

  若无法连通，检查：
  • 本地防火墙是否允许容器访问587端口
  • 容器是否使用默认bridge网络模式（该模式默认允许访问外部网络）
  • 本地是否有代理配置，需确保容器能绕过代理或正确配置代理参数

3. 检查邮件服务配置细节 #

• Outlook：登录Outlook账户，开启POP3/IMAP服务；若开启两步验证，需使用应用密码作为smtp_password。
• Gmail：必须开启账户两步验证，然后在Google账户设置中创建专门的应用密码，将该密码填入smtp_password（Gmail已停用"不太安全的应用"选项，必须使用应用密码）。

4. 验证容器内配置是否生效 #

• 进入容器查看airflow.cfg中的SMTP配置：

  docker exec -it airflow-worker cat /opt/airflow/airflow.cfg | grep -A 10 "\[smtp\]"
  

  确认输出内容与本地配置一致，说明配置已正确加载。

5. 可选：使用Airflow UI配置SMTP连接 #

• 若不想修改airflow.cfg，可在Airflow UI的Admin > Connections中创建SMTP类型的连接，配置相关参数，然后在EmailOperator中指定conn_id参数：

  sending_email_notification = EmailOperator(
      task_id="sending_email",
      to="myemail@outlook.com",
      subject=f"New email",
      html_content=f"""<h3>New Data is available</h3>""",
      conn_id="my_smtp_conn"  # 对应UI中创建的连接ID
  )
  

内容的提问来源于stack exchange，提问作者geds133