执行systemctl daemon-reload后Hadoop-YARN cgroup devices目录丢失求助

问题背景 #

CentOS 8（cgroup v1）环境下，Hadoop 3.3.4手动管理cgroup（yarn.nodemanager.linux-container-executor.cgroups.mount=false），配置了/hadoop-yarn层级的cpu、cpuacct、devices三个cgroup组。通过cgconfig或自定义systemd服务创建目录后，执行systemctl daemon-reload会导致/sys/fs/cgroup/devices/hadoop-yarn/被删除，YARN NodeManager依赖该目录需重建后才能恢复。

核心原因 #

CentOS 8的systemd对cgroup v1的devices子系统存在自动清理机制，当检测到非systemd管理的cgroup目录（或与systemd slice冲突的目录）时，会在daemon-reload过程中移除该目录。自定义服务中添加Slice=hadoop-yarn.slice会让systemd接管该slice对应的cgroup，进一步触发清理逻辑。

解决方法 #

方法1：移除自定义systemd服务的Slice配置，改用cgconfig持久化管理 #

1. 停止并禁用之前的自定义systemd服务：

systemctl stop hadoop-yarn-cgroup.service
systemctl disable hadoop-yarn-cgroup.service
rm -f /etc/systemd/system/hadoop-yarn-cgroup.service
systemctl daemon-reload

2. 确保cgconfig和cgred服务开机自启，并配置在systemd初始化完成后启动：

systemctl enable cgconfig.service
systemctl enable cgred.service
# 配置cgconfig在systemd初始化完成后启动
echo "After=systemd-remount-fs.service systemd-sysctl.service" >> /etc/systemd/system/cgconfig.service.d/override.conf
systemctl daemon-reload

3. 修改/etc/cgconfig.conf，添加devices子系统的显式权限配置（避免systemd认为该目录无使用价值）：

group hadoop-yarn {
        perm {
                admin {
                        uid = yarn;
                        gid = hadoop;
                }
                task {
                        uid = yarn;
                        gid = hadoop;
                }
        }
        cpu {
        }
        cpuacct {
        }
        devices {
                # 添加默认设备权限，防止systemd判定该cgroup无活跃资源
                devices.allow = *;
        }
}

4. 重启cgconfig服务验证：

systemctl restart cgconfig.service
ll /sys/fs/cgroup/{cpu,cpuacct,devices}/hadoop-yarn/ -d

执行systemctl daemon-reload后再次检查目录是否存在。

方法2：让systemd托管hadoop-yarn的cgroup slice #

如果必须用systemd管理，可直接创建systemd slice而非手动创建cgroup目录：

1. 创建/etc/systemd/system/hadoop-yarn.slice：

[Unit]
Description=Hadoop YARN CGroup Slice
DefaultDependencies=no
Before=slices.target

[Slice]
CPUAccounting=yes
CPUQuota=100%
DeviceAllow=block-* rwm
DeviceAllow=char-* rwm

2. 配置YARN使用该slice对应的cgroup层级：
   在yarn-site.xml中修改：

<property>
  <name>yarn.nodemanager.linux-container-executor.cgroups.hierarchy</name>
  <value>/system.slice/hadoop-yarn.slice</value>
</property>

3. 启用slice并重启YARN NodeManager：

systemctl daemon-reload
systemctl start hadoop-yarn.slice
systemctl restart yarn-nodemanager.service

此时systemctl daemon-reload不会删除systemd托管的slice目录。

方法3：添加systemd钩子，在daemon-reload后重建devices目录 #

如果以上方法都无法生效，可添加一个systemd钩子脚本，在daemon-reload完成后自动重建目录：

1. 创建/usr/lib/systemd/system-shutdown/hadoop-yarn-cgroup-fix.sh：

#!/bin/bash
if [ "$1" = "reload" ]; then
    mkdir -p /sys/fs/cgroup/devices/hadoop-yarn
    chown yarn:hadoop /sys/fs/cgroup/devices/hadoop-yarn
    # 恢复设备权限
    echo "*:* rwm" > /sys/fs/cgroup/devices/hadoop-yarn/devices.allow
fi

2. 赋予脚本执行权限：

chmod +x /usr/lib/systemd/system-shutdown/hadoop-yarn-cgroup-fix.sh

验证步骤 #

每次操作后执行以下命令验证：

systemctl daemon-reload
ll /sys/fs/cgroup/{cpu,cpuacct,devices}/hadoop-yarn/ -d
systemctl status yarn-nodemanager.service

内容的提问来源于stack exchange，提问作者Rémi Viboud