Airflow中SSHExecuteOperator执行失败：无法创建远程临时文件

我来帮你分析下这个问题，先把你的场景和排查信息整理清楚，再给出针对性的解决建议：

你的DAG代码 #

from airflow.contrib.hooks.ssh_hook import SSHHook
from datetime import timedelta
default_args = {
    'owner': 'airflow',
    'start_date':airflow.utils.dates.days_ago(2),
    'retries': 3
}
dag = DAG('Nas_Hdfs', description='Simple tutorial DAG', schedule_interval=None,default_args=default_args, catchup=False)
sshHook = SSHHook(conn_id='101')
sshHook.no_host_key_check = True
t2 = SSHExecuteOperator(task_id="NAS_TO_HDFS_FILE_COPY", bash_command="hostname ", ssh_hook=sshHook, dag=dag )
t2

SSH连接配置 #

你在Airflow UI中配置的Connection ID为101的SSH连接信息：

• 主机：目标远程服务器IP
• 端口：22
• 登录用户：root
• 认证方式：密码认证（已填写对应密码）

完整错误日志 #

INFO - Subtask: --------------------------------------------------------------------------------
INFO - Subtask: Starting attempt 1 of 4
INFO - Subtask: --------------------------------------------------------------------------------
INFO - Subtask:
INFO - Subtask: [2018-05-28 08:54:22,812] {models.py:1342} INFO - Executing <Task(SSHExecuteOperator): NAS_TO_HDFS_FILE_COPY> on 2018-05-28 08:54:12.876538
INFO - Subtask: [2018-05-28 08:54:23,303] {models.py:1417} ERROR - Failed to create remote temp file
INFO - Subtask: Traceback (most recent call last):
INFO - Subtask:   File "/opt/miniconda3/lib/python2.7/site-packages/airflow/models.py", line 1374, in run
INFO - Subtask:     result = task_copy.execute(context=context)
INFO - Subtask:   File "/opt/miniconda3/lib/python2.7/site-packages/airflow/contrib/operators/ssh_execute_operator.py", line 128, in execute
INFO - Subtask:     self.task_id) as remote_file_path:
INFO - Subtask:   File "/opt/miniconda3/lib/python2.7/site-packages/airflow/contrib/operators/ssh_execute_operator.py", line 64, in __enter__
INFO - Subtask:     raise AirflowException("Failed to create remote temp file")
INFO - Subtask: AirflowException: Failed to create remote temp file
INFO - Subtask: [2018-05-28 08:54:23,304] {models.py:1433} INFO - Marking task as UP_FOR_RETRY
INFO - Subtask: [2018-05-28 08:54:23,342] {models.py:1462} ERROR - Failed to create remote temp file
INFO - Subtask: Traceback (most recent call last):
INFO - Subtask:   File "/opt/miniconda3/bin/airflow", line 28, in <module>
INFO - Subtask:     args.func(args)
INFO - Subtask:   File "/opt/miniconda3/lib/python2.7/site-packages/airflow/bin/cli.py", line 422, in run
INFO - Subtask:     pool=args.pool,
INFO - Subtask:   File "/opt/miniconda3/lib/python2.7/site-packages/airflow/utils/db.py", line 53, in wrapper
INFO - Subtask:     result = func(*args, **kwargs)
INFO - Subtask:   File "/opt/miniconda3/lib/python2.7/site-packages/airflow/models.py", line 1374, in run
INFO - Subtask:     result = task_copy.execute(context=context)
INFO - Subtask:   File "/opt/miniconda3/lib/python2.7/site-packages/airflow/contrib/operators/ssh_execute_operator.py", line 128, in execute
INFO - Subtask:     self.task_id) as remote_file_path:
INFO - Subtask:   File "/opt/miniconda3/lib/python2.7/site-packages/airflow/contrib/operators/ssh_execute_operator.py", line 64, in __enter__
INFO - Subtask:     raise AirflowException("Failed to create remote temp file")
INFO - Subtask: airflow.exceptions.AirflowException: Failed to create remote temp file
INFO - Task exited with return code 1

补充测试情况 #

你在airflow用户的Python Shell中执行了以下测试代码：

from airflow.contrib.hooks.ssh_hook import SSHHook
sshHook = SSHHook(conn_id='101')
sshHook.no_host_key_check = True
sshHook.Popen(["-q", "mktemp", "--tmpdir", "tmp_XXXXXX"])

测试结果显示mktemp命令执行失败，提示：mktemp: 无法创建临时文件"/tmp_XXXXXX": 没有那个文件或目录

解决建议 #

从测试结果来看，问题核心在于临时文件创建的命令参数错误，同时可能涉及远程目录权限问题，你可以按以下步骤排查修复：

1. 修正mktemp命令参数
   你测试里的--tmpdir参数用法有误，--tmpdir后面需要跟一个已存在的目录路径，比如指定系统默认临时目录：

   sshHook.Popen(["-q", "mktemp", "--tmpdir=/tmp", "tmp_XXXXXX"])
   

   或者直接省略--tmpdir，让mktemp使用系统默认的临时目录：

   sshHook.Popen(["-q", "mktemp", "tmp_XXXXXX"])
   

2. 在SSHExecuteOperator中指定远程临时目录
   Airflow的SSHExecuteOperator默认会尝试在远程主机的临时目录创建文件，你可以显式指定一个存在且有权限的目录（比如/tmp），避免默认路径的问题：

   t2 = SSHExecuteOperator(
       task_id="NAS_TO_HDFS_FILE_COPY",
       bash_command="hostname ",
       ssh_hook=sshHook,
       remote_tmp_dir='/tmp',  # 明确指定远程主机的临时目录
       dag=dag
   )
   

3. 验证远程主机的目录权限
   手动用root用户SSH到目标服务器，执行mktemp命令，确认是否能正常创建临时文件，同时检查/tmp目录的权限是否允许root读写（一般默认是允许的，但不排除特殊配置）。

4. 检查SSH连接的完整性
   确保Airflow服务器能正常SSH到目标主机，没有防火墙拦截、密码错误或网络连通性问题。

内容的提问来源于stack exchange，提问作者Deepesh Rehi