Apache Airflow集成Office 365邮件告警SMTP配置故障求助

一、解决BashOperator执行docker run失败（Docker daemon未运行） #

场景1：Airflow以容器方式部署 #

• 问题根源：容器内默认未安装Docker引擎，也无法直接访问宿主机的Docker daemon
• 解决方案：
  1. 在docker-compose.yml中挂载宿主机Docker资源，让容器能调用宿主机Docker服务：

     volumes:
       - /var/run/docker.sock:/var/run/docker.sock
       - /usr/bin/docker:/usr/bin/docker
     

  2. 确保Airflow容器内用户拥有Docker操作权限，可临时设置user: root，或把容器用户加入宿主机docker组

场景2：Airflow在宿主机直接运行 #

• 问题根源：Docker服务未启动，或Airflow运行用户无Docker操作权限
• 解决方案：
  1. 启动并设置Docker服务开机自启：

     sudo systemctl start docker
     sudo systemctl enable docker
     

  2. 将Airflow运行用户加入docker组，重启Airflow服务生效：

     sudo usermod -aG docker airflow_user
     

二、Apache Airflow集成Office 365邮件告警的正确配置 #

1. 核心SMTP配置（修改airflow.cfg） #

找到[smtp]配置段，替换为以下内容：

[smtp]
smtp_host = smtp.office365.com
smtp_starttls = True
smtp_ssl = False
smtp_port = 587
smtp_user = your_office365_email@domain.com
smtp_password = your_app_password  # 注意：启用两步验证的账号需用Office 365应用密码，未启用可直接用账号密码
smtp_mail_from = your_office365_email@domain.com
smtp_timeout = 30
smtp_retry_limit = 5

提示：Office 365建议开启两步验证并使用应用密码，避免账号密码直接暴露。

2. 带邮件告警的DAG示例代码 #

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime, timedelta

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'email': ['recipient_email@domain.com'],
    'email_on_failure': True,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
}

with DAG(
    'office365_failure_alert_dag',
    default_args=default_args,
    description='DAG触发Office 365邮件告警',
    schedule_interval=timedelta(days=1),
    start_date=datetime(2024, 1, 1),
    catchup=False,
    tags=['email-alert', 'office365'],
) as dag:

    # 故意设置失败任务触发告警
    fail_task = BashOperator(
        task_id='trigger_failure_alert',
        bash_command='exit 1',
    )

    success_task = BashOperator(
        task_id='normal_task',
        bash_command='echo "Task executed successfully"',
    )

    fail_task >> success_task

3. 任务失败时的正常执行日志示例 #

[2024-05-20 14:30:00,215] {taskinstance.py:1849} INFO - Marking task as FAILED. dag_id=office365_failure_alert_dag, task_id=trigger_failure_alert, execution_date=20240520T123000, start_date=20240520T143000, end_date=20240520T143000
[2024-05-20 14:30:00,562] {email.py:142} INFO - Email alert sent to: ['recipient_email@domain.com']
[2024-05-20 14:30:00,563] {taskinstance.py:2729} INFO - 0 downstream tasks scheduled from follow-on schedule check

常见错误排查 #

• 错误[smtp/smtp_user]配置缺失：检查airflow.cfg中smtp_user是否填写正确，修改后需重启Airflow服务加载新配置
• 错误OSError: [Errno 99] Cannot assign requested address：确认smtp_host=smtp.office365.com和smtp_port=587配置正确，同时测试Airflow服务器与该端口的连通性：telnet smtp.office365.com 587

内容的提问来源于stack exchange，提问作者Austin Jackson