Slurm控制进程启动时报'unrecognized key: OverSubscribe'错误排查

我在大学单节点服务器上部署Slurm，想让大家交替运行实验（避免长实验阻塞短实验），配置了GANG调度系统，但启动执行systemctl status slurmctld时失败，报错信息如下：

× slurmctld.service - Slurm controller daemon
     Loaded: loaded (/lib/systemd/system/slurmctld.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2024-01-17 12:11:57 -03; 24s ago
       Docs: man:slurmctld(8)
    Process: 1288791 ExecStart=/usr/sbin/slurmctld -D -s $SLURMCTLD_OPTIONS (code=exited, status=1/FAILURE)
   Main PID: 1288791 (code=exited, status=1/FAILURE)
        CPU: 22ms

Jan 17 12:11:57 lab04 systemd[1]: Started Slurm controller daemon.
Jan 17 12:11:57 lab04 slurmctld[1288791]: slurmctld: error: _parse_next_key: Parsing error at unrecognized key: OverSubscribe
Jan 17 12:11:57 lab04 slurmctld[1288791]: error: _parse_next_key: Parsing error at unrecognized key: OverSubscribe
Jan 17 12:11:57 lab04 slurmctld[1288791]: fatal: Unable to process configuration file
Jan 17 12:11:57 lab04 slurmctld[1288791]: slurmctld: fatal: Unable to process configuration file
Jan 17 12:11:57 lab04 systemd[1]: slurmctld.service: Main process exited, code=exited, status=1/FAILURE
Jan 17 12:11:57 lab04 systemd[1]: slurmctld.service: Failed with result 'exit-code'.

我的slurm.conf配置：

ClusterName=localcluster
SlurmctldHost=localhost
ProctrackType=proctrack/linuxproc
ReturnToService=1
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmctldPort=6817
SlurmdPidFile=/var/run/slurmd.pid
SlurmdPort=6818
SlurmdSpoolDir=/var/lib/slurm/slurmd
SlurmUser=slurm
StateSaveLocation=/var/lib/slurm/slurmctld
TaskPlugin=task/none
# TIMERS
InactiveLimit=0
KillWait=30
MinJobAge=300
SlurmctldTimeout=120
SlurmdTimeout=300
Waittime=0
# SCHEDULING
DefMemPerNode=245760
MaxMemPerNode=245760
#设置为每半小时交替一次任务
SchedulerTimeSlice=15
SchedulerType=sched/builtin
SelectType=select/linear
SelectTypeParameters=CR_Memory
PreemptMode=GANG
#控制同时运行的任务数，假设任务不占用重复内存
OverSubscribe=FORCE:6
# LOGGING AND ACCOUNTING
JobCompType=jobcomp/none
JobAcctGatherFrequency=30
SlurmctldDebug=info
SlurmctldLogFile=/var/log/slurm/slurmctld.log
SlurmdDebug=info
SlurmdLogFile=/var/log/slurm/slurmd.log
# COMPUTE NODES
NodeName=localhost CPUs=2 State=UNKNOWN
PartitionName=LocalQ Nodes=ALL Default=YES MaxTime=INFINITE State=UP

错误原因 #

报错明确指出OverSubscribe是未被识别的配置项，因为当前使用的SelectType=select/linear不支持全局的OverSubscribe参数，该参数需要绑定到具体节点的配置中，而非全局设置。

修改步骤 #

1. 删除slurm.conf中的全局OverSubscribe=FORCE:6行
2. 将OverSubscribe参数添加到NodeName配置行中，修改后的节点配置为：

   NodeName=localhost CPUs=2 State=UNKNOWN OverSubscribe=FORCE:6
   

修改后的完整slurm.conf：

ClusterName=localcluster
SlurmctldHost=localhost
ProctrackType=proctrack/linuxproc
ReturnToService=1
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmctldPort=6817
SlurmdPidFile=/var/run/slurmd.pid
SlurmdPort=6818
SlurmdSpoolDir=/var/lib/slurm/slurmd
SlurmUser=slurm
StateSaveLocation=/var/lib/slurm/slurmctld
TaskPlugin=task/none
# TIMERS
InactiveLimit=0
KillWait=30
MinJobAge=300
SlurmctldTimeout=120
SlurmdTimeout=300
Waittime=0
# SCHEDULING
DefMemPerNode=245760
MaxMemPerNode=245760
#设置为每半小时交替一次任务
SchedulerTimeSlice=15
SchedulerType=sched/builtin
SelectType=select/linear
SelectTypeParameters=CR_Memory
PreemptMode=GANG
# LOGGING AND ACCOUNTING
JobCompType=jobcomp/none
JobAcctGatherFrequency=30
SlurmctldDebug=info
SlurmctldLogFile=/var/log/slurm/slurmctld.log
SlurmdDebug=info
SlurmdLogFile=/var/log/slurm/slurmd.log
# COMPUTE NODES
NodeName=localhost CPUs=2 State=UNKNOWN OverSubscribe=FORCE:6
PartitionName=LocalQ Nodes=ALL Default=YES MaxTime=INFINITE State=UP

3. 重启Slurm控制器服务：

   systemctl restart slurmctld
   

内容的提问来源于stack exchange，提问作者Francisco José Letterio