执行Ansible Tower作业模板时出现设备无剩余空间错误求助

环境配置 #

• AWX版本：17.1.0（Ansible Tower）
• Ansible版本：2.9.18
• 目标主机：Java应用服务器

问题现象 #

执行Java应用部署作业模板时，每次任务在不同步骤随机失败，报错「设备无剩余空间（Errno 28）」及模块故障；目标主机磁盘空间已确认充足，作业偶尔能成功，重启目标主机无法解决。

错误日志 #

场景1：yum操作失败 #

"msg": "
Traceback (most recent call last):
  File "/bin/yum", line 29, in 
    yummain.user_main(sys.argv[1:], exit_code=True)
  File "/usr/share/yum-cli/yummain.py", line 375, in user_main
    errcode = main(args)
  File "/usr/share/yum-cli/yummain.py", line 281, in main
    return_code = base.doTransaction()
  File "/usr/share/yum-cli/cli.py", line 740, in doTransaction
    testcb = RPMTransaction(self, test=True)
  File "/usr/lib/python2.7/site-packages/yum/rpmtrans.py", line 199, in __init__
    self._setupOutputLogging(base.conf.rpmverbosity)
  File "/usr/lib/python2.7/site-packages/yum/rpmtrans.py", line 222, in _setupOutputLogging
    io_r = tempfile.NamedTemporaryFile()
  File "/usr/lib64/python2.7/tempfile.py", line 458, in NamedTemporaryFile
    (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags)
  File "/usr/lib64/python2.7/tempfile.py", line 239, in _mkstemp_inner
    fd = _os.open(file, flags, 0600)
OSError: [Errno 28] No space left on device: '/tmp/tmpOR9pdD'

Unable to upload Enabled Repositories Report
"

场景2：copy模块失败 #

FAILED! => {"ansible_loop_var": "item", "changed": false, "item": "application.properties", "module_stderr": "Shared connection to 198.19.20.91 closed.\r\n", "module_stdout": "\r\nTraceback (most recent call last):\r\n  File \"/var/SP/dyna_adm/.ansible/tmp/ansible-tmp-1672301679.9486687-78304-254181291127392/AnsiballZ_copy.py\", line 102, in \r\n    _ansiballz_main()\r\n  File \"/var/SP/dyna_adm/.ansible/tmp/ansible-tmp-1672301679.9486687-78304-254181291127392/AnsiballZ_copy.py\", line 89, in _ansiballz_main\r\n    with open(zipped_mod, 'wb') as modlib:\r\nIOError: [Errno 28] No space left on device: '/tmp/ansible_copy_payload_wGKWRZ/ansible_copy_payload.zip'\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

场景3：setup模块失败 #

FAILED! => {"ansible_facts": {}, "changed": false, "failed_modules": {"setup": {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "failed": true, "module_stderr": "Shared connection to 198.19.20.90 closed.\r\n", "module_stdout": "\r\nTraceback (most recent call last):\r\n  File \"/var/SP/dyna_adm/.ansible/tmp/ansible-tmp-1672145315.0321684-55457-255515902570625/AnsiballZ_setup.py\", line 102, in \r\n    _ansiballz_main()\r\n  File \"/var/SP/dyna_adm/.ansible/tmp/ansible-tmp-1672145315.0321684-55457-255515902570625/AnsiballZ_setup.py\", line 89, in _ansiballz_main\r\n    with open(zipped_mod, 'wb') as modlib:\r\nIOError: [Errno 28] No space left on device: '/tmp/ansible_setup_payload_RrXk84/ansible_setup_payload.zip'\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}}, "msg": "The following modules failed to execute: setup\n"}

已确认信息 #

• 部署模板在其他服务器运行正常，脚本无问题
• 目标主机磁盘空间充足，无磁盘满情况
• 目标主机重启后问题依旧
• Ansible主节点无异常，未做变更

排查方向与解决建议 #

1. 检查inode使用情况：磁盘空间充足但报无空间，大概率是inode耗尽。执行df -i查看目标主机/tmp所在分区的inode使用率，若接近100%，清理/tmp下的小文件或临时文件。
2. 检查tmpfs配置：若/tmp是tmpfs（内存文件系统），查看其大小配置（mount | grep tmpfs），可能是tmpfs容量不足导致。可临时扩容：mount -o remount,size=4G /tmp，或修改/etc/fstab永久调整。
3. 清理Ansible临时文件：目标主机/var/SP/dyna_adm/.ansible/tmp/下可能残留大量未清理的临时文件，执行rm -rf /var/SP/dyna_adm/.ansible/tmp/*清理，同时确认Ansible配置中cleanup参数是否开启（默认开启）。
4. 检查用户磁盘配额：确认执行作业的用户（dyna_adm）是否有磁盘配额限制，执行quota -u dyna_adm查看，若有配额限制，调整配额或清理该用户下的文件。
5. 排查文件系统错误：执行fsck检查/tmp所在分区的文件系统是否有错误（需先umount分区），修复可能存在的文件系统异常。

内容的提问来源于stack exchange，提问作者jeevan1494