nmcli触发dbus OOM及sender inactive错误的根因与解决问询

问题描述 #

我编写了一个bash脚本，通过nmcli c s $VLAN检查VLAN的活跃状态，运行过程中偶尔会触发dbus在系统日志中生成OOM相关记录，频率约1-2个月一次。核心错误为dbus-daemon transaction failed (OOM), sending error to sender inactive，同时nmcli报错Could not create NMClient object: GDBus.Error:org.freedesktop.DBus.Error.NoMemory。

系统日志 #

Nov 30 05:14:41 nyx-master-2 dbus-daemon[692]: [system] dbus-daemon transaction failed (OOM), sending error to sender inactive
Nov 30 05:14:41 nyx-master-2 bash[1113268]: Error: Could not create NMClient object: GDBus.Error:org.freedesktop.DBus.Error.NoMemory: Error return with empty body: .
Nov 30 05:14:41 nyx-master-2 bash[1113232]:  is inactivated now,try to up VLAN.STG
Nov 30 05:14:41 nyx-master-2 NetworkManager[446553]: <info>  [1764450881.4727] agent-manager: agent[5b8a32d9e0f6e5e0,:1.578712/nmcli-connect/0]: agent registered
Nov 30 05:14:41 nyx-master-2 NetworkManager[446553]: <info>  [1764450881.4736] device (VLAN.STG): state change: activated -> deactivating (reason 'new-activation', sys-iface-state: 'managed')
Nov 30 05:14:41 nyx-master-2 NetworkManager[446553]: <info>  [1764450881.4746] device (VLAN.STG): disconnecting for new activation request.
Nov 30 05:14:41 nyx-master-2 audit[446553]: USYS_CONFIG pid=446553 uid=0 auid=4294967295 ses=4294967295 msg='op=connection-activate uuid=6774e461-bfdf-4d9a-a751-550e3d92ecf1 name="VLAN.STG" pid=1113279 uid=0 result=success exe="/usr/sbin/NetworkManager" hostname=? addr=? terminal=? res=success'
Nov 30 05:14:41 nyx-master-2 NetworkManager[446553]: <info>  [1764450881.4747] audit: op="connection-activate" uuid="6774e461-bfdf-4d9a-a751-550e3d92ecf1" name="VLAN.STG" pid=1113279 uid=0 result="success"
Nov 30 05:14:41 nyx-master-2 dbus-daemon[692]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.482634' (uid=0 pid=446553 comm="/usr/sbin/NetworkManager --no-daemon")
Nov 30 05:14:41 nyx-master-2 systemd[1]: Starting Network Manager Script Dispatcher Service...
Nov 30 05:14:41 nyx-master-2 dbus-daemon[692]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Nov 30 05:14:41 nyx-master-2 audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Nov 30 05:14:41 nyx-master-2 systemd[1]: Started Network Manager Script Dispatcher Service.
Nov 30 05:14:41 nyx-master-2 NetworkManager[446553]: <info>  [1764450881.4976] device (VLAN.STG): state change: deactivating -> disconnected (reason 'new-activation', sys-iface-state: 'managed')

脚本内容 #

#!/bin/bash

status=$(systemctl is-active NetworkManager)
status2=$(systemctl is-enabled NetworkManager)


if [ "$status" == "inactive" ]; then
    echo "NetworkManager is inactive. Restarting..."
    systemctl restart NetworkManager
elif [ "$status" == "failed" ]; then
    echo "NetworkManager is failed. Restarting..."
    systemctl restart NetworkManager
fi

if [ "$status2" == "masked" ]; then
    echo "NetworkManager is mask. Restarting..."
    systemctl unmask NetworkManager
    systemctl restart NetworkManager
elif [ "$status2" == "disabled" ]; then
    echo "NetworkManager is disabled. Restarting..."
    systemctl enable NetworkManager
    systemctl restart NetworkManager
fi

# VLAN names
vlans=("VLAN.K8S" "VLAN.STG" "VLAN.LB")

# Loop over VLANs
for vlan in "${vlans[@]}"
do
  # Check VLAN status
  status1=$(nmcli c s $vlan | grep GENERAL.STATE | awk '{print $2}')
  status2=$(ip a s $vlan | grep 'inet\b' | awk '{print $2}' )

  # Activate VLAN if not already activated
  if [[ $status1 != "activated" ]] || [[ $status2 == "" ]]; then
    echo "$VLAN is inactivated now,try to up $vlan"
    nmcli c up $vlan
  fi
done

exit 0

环境版本 #

• dbus：1.14.10
• nmcli：1.44.0

复现场景与根本原因 #

复现场景 #

该问题仅在系统内存资源极度紧张时触发：

• 系统存在内存泄漏，长期运行后可用内存耗尽
• 业务高峰时段，大量进程占用内存，剩余可用内存不足以支撑dbus完成事务处理
• OOM Killer机制触发前，系统处于内存临界状态，此时调用依赖dbus的工具（如nmcli）会触发内存分配失败

根本原因 #

1. OOM导致dbus事务失败：dbus-daemon在处理nmcli的请求时，无法分配足够内存完成消息事务，因此抛出transaction failed (OOM)错误。
2. sender变为inactive的原因：当dbus无法处理请求时，nmcli进程可能因等待响应超时、被OOM Killer杀死，或自身内存分配失败退出，导致dbus无法找到请求发送方，因此标记为inactive。
3. 脚本小问题：日志中的空值 is inactivated now是因为误用了大写的$VLAN变量，实际应为小写$vlan，但这并非触发OOM的核心原因。

解决方法 #

1. 系统层面优化 #

• 监控内存使用：定期通过top、htop、vmstat工具检查内存状态，排查内存泄漏进程并及时修复。
• 调整OOM策略：修改/etc/sysctl.conf添加vm.oom_kill_allocating_task=1，让OOM Killer优先杀死触发内存分配失败的进程；为dbus-daemon设置OOM分数（echo -1000 > /proc/$(pidof dbus-daemon)/oom_score_adj），降低被OOM Killer选中的概率。
• 升级硬件内存：若系统长期处于高内存负载状态，考虑增加物理内存。

2. 工具与脚本优化 #

• 简化nmcli调用：使用t参数输出简洁格式，直接过滤状态，减少内存占用：

  status1=$(nmcli -t -f GENERAL.STATE c show "$vlan")
  

• 替换部分检查逻辑：用ip link替代nmcli检查接口状态，减少对dbus的依赖：

  if ip link show "$vlan" | grep -q "UP"; then
      # 接口已启用
  fi
  

• 修复脚本变量错误：将echo "$VLAN is inactivated now,try to up $vlan"改为echo "$vlan is inactivated now,try to up $vlan"，避免日志空值。
• 增加错误处理与重试：对nmcli调用增加错误捕获，失败时重试：

  if ! nmcli c up "$vlan"; then
      echo "Failed to up $vlan, retrying..."
      sleep 2
      nmcli c up "$vlan"
  fi
  

3. 组件版本升级 #

检查dbus和NetworkManager的更新版本，部分OOM相关bug可能在后续版本中被修复。

内容的提问来源于stack exchange，提问作者Vke