Slurm集群UID/GID一致仍报安全违规错误的解决请求

Hey there, let's break down your Slurm security violation issue step by step. First off, yes—compiling compute nodes with --enable-front-end is absolutely a contributing factor to your problems. That flag is specifically intended for front-end (controller) nodes only; compute nodes should never be built with it, as it enables front-end-specific functionality that conflicts with slurmd's role as a compute daemon.

1. Why --enable-front-end on Compute Nodes Breaks Things #

• The --enable-front-end flag configures Slurm to operate as a controller/front-end component. When you compile slurmd (the compute node daemon) with this flag, it will expect front-end-level permissions and security checks, which clash with the normal compute node security model.
• This misconfiguration leads to incorrect RPC validation, which is likely triggering those "Security violation" errors even when your UIDs/GIDs appear to match.

2. Core Issues From Your Logs & Setup #

Beyond the compile flag, two critical problems stand out:

• SlurmUser Misconfiguration: The slurmd log explicitly asks: Do you have SlurmUser configured as uid 1000?. Slurm requires a dedicated system user (usually slurm with a low, non-personal UID) to run its daemons. Using your personal user (UID 1000) violates Slurm's security policies and directly triggers these errors.
• Stale GID Cache: Even though you updated the GID on compute nodes, slurmd might not have picked up the new values because it was running before the change. Daemons cache user/group info at startup, so a simple restart won't help if you didn't first sync the passwd/group files properly.

3. Step-by-Step Resolution #

A. Recompile Compute Nodes Correctly #

On every compute node, rebuild Slurm without the front-end flag:

cd slurm
make clean
./configure --enable-debug  # Omit --enable-front-end entirely
sudo make install

This builds a compute-node-only version of Slurm that adheres to the correct security model.

B. Create & Sync a Dedicated Slurm User #

You need a consistent slurm user across all nodes with matching UID/GID:

1. On your front-end node, create the user (if it doesn't exist):

   sudo groupadd -g 998 slurm
   sudo useradd -u 998 -g slurm -m -d /var/lib/slurm -s /bin/bash slurm
   

2. Sync this user/group to all compute nodes (use scp or a config management tool like Ansible):

   # On front-end:
   sudo scp /etc/passwd /etc/group <compute-node-ip>:/tmp/
   # On each compute node:
   sudo cp /tmp/passwd /etc/passwd
   sudo cp /tmp/group /etc/group
   

3. Update your slurm.conf (on front-end, then sync to compute nodes) to set the dedicated user:

   SlurmUser=slurm
   

C. Fully Restart All Slurm Services #

After syncing users and updating configs, do a full restart to clear cached user data:

• Front-end node:

  sudo killall slurmctld slurmdbd slurmd munged
  sudo munged -f
  sudo /etc/init.d/munge start
  sudo slurmdbd &
  sudo slurmctld -cDvvvvvv
  

• Compute nodes:

  sudo killall slurmd munged
  sudo munged -f
  sudo /etc/init.d/munge start
  sudo slurmd -Dvvvvv
  

D. Verify Consistency Across Nodes #

Run these commands on every node to confirm everything is synced:

# Check slurm user UID/GID
id slurm
# Verify SlurmUser configuration
scontrol show config | grep SlurmUser

Ensure the UID/GID for slurm is identical across all nodes, and SlurmUser is set to slurm.

4. Test the Fix #

Submit a simple test job to confirm the errors are resolved:

sbatch --wrap="echo Hello World from compute node"

Check the slurmd and slurmctld logs—you should no longer see "Security violation" messages, and the job should complete successfully.

内容的提问来源于stack exchange，提问作者alper