如何解决Airflow apache-airflow-providers-ssh的SSH命令超时问题

问题描述 #

安装Airflow v2.5.1后，使用apache-airflow-providers-ssh配置SSH连接，日志显示连接与认证均成功，但执行远程命令时触发超时错误：

[2023-02-27, 23:42:57 CST] {taskinstance.py:1768} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/airflow/providers/ssh/operators/ssh.py", line 171, in execute
    result = self.run_ssh_client_command(ssh_client, self.command, context=context)
  File "/usr/local/lib/python3.10/dist-packages/airflow/providers/ssh/operators/ssh.py", line 156, in run_ssh_client_command
    exit_status, agg_stdout, agg_stderr = self.ssh_hook.exec_ssh_client_command(
  File "/usr/local/lib/python3.10/dist-packages/airflow/providers/ssh/hooks/ssh.py", line 521, in exec_ssh_client_command
    raise AirflowException("SSH command timed out")
airflow.exceptions.AirflowException: SSH command timed out

连接成功日志：

[2023-02-27, 23:42:43 CST] {ssh.py:133} INFO - Creating ssh_client
[2023-02-27, 23:42:43 CST] {ssh.py:283} WARNING - No Host Key Verification. This won't protect against Man-In-The-Middle attacks
[2023-02-27, 23:42:43 CST] {transport.py:1871} INFO - Connected (version 2.0, client OpenSSH_for_Windows_8.9)
[2023-02-27, 23:42:44 CST] {transport.py:1871} INFO - Authentication (password) successful!

解决方案 #

• 调整SSH命令超时参数
  在SSHOperator或SSHHook中显式设置cmd_timeout参数，延长命令执行的超时时间（单位：秒）。示例：

  from airflow.providers.ssh.operators.ssh import SSHOperator
  
  ssh_task = SSHOperator(
      task_id="run_remote_cmd",
      ssh_conn_id="your_ssh_connection",
      command="your_long_running_command",
      cmd_timeout=360  # 设置为6分钟
  )
  

• 验证远程命令执行时长
  手动在Airflow节点上通过SSH执行相同命令，确认命令本身的运行时间。如果命令确实需要长时间运行，要么优化命令逻辑，要么进一步延长cmd_timeout。

• 检查网络与服务器负载

  • 测试Airflow节点与远程服务器之间的网络稳定性，比如持续ping或传输大文件，排查是否存在丢包或延迟过高的情况。
  • 查看远程服务器的CPU、内存负载，若资源占用过高，会导致命令执行缓慢甚至超时。

• 设置任务级执行超时
  确保Airflow任务本身的execution_timeout大于SSH命令的超时时间，避免任务整体先触发超时。示例：

  from datetime import timedelta
  
  ssh_task = SSHOperator(
      task_id="run_remote_cmd",
      ssh_conn_id="your_ssh_connection",
      command="your_long_running_command",
      cmd_timeout=360,
      execution_timeout=timedelta(minutes=10)
  )
  

• 优化SSH连接配置

  • 修改远程服务器的sshd_config，添加或调整以下参数，保持连接活跃：

    ClientAliveInterval 30
    ClientAliveCountMax 10
    

    重启sshd服务生效。
  • 若不需要命令输出，可在命令末尾添加>/dev/null 2>&1，减少数据传输量。

内容的提问来源于stack exchange，提问作者obautista