使用Slurm运行WindNinja报slurmstepd mkdir路径错误求助

在HPC集群运行WindNinja时，批量提交sbatch脚本反复触发错误：slurmstepd: error: execve(): mkdir: No such file or directory。手动逐行执行sbatch脚本无异常，但批量提交就报错。

运行涉及三个文件：

• windninja_minimal.sbatch：通过srun循环动态分配任务
• run.sh：配置环境并执行WindNinja
• base_cli.cfg：WindNinja参数配置文件

路径结构 #

/rds/projects/h/hemidah10/Kaiyuan/WindNinja/momWindNinja/temp/Shap/
├── 0/
├── 1/
├── 2/
├── 3/
├── test/
│   ├── run.sh
│   └── base_cli.cfg
└── windninja_minimal.sbatch

sbatch脚本 #

#!/bin/bash
#SBATCH --job-name=WindNinja_Minimal
#SBATCH --output=windninja_%j.log
#SBATCH --error=windninja_%j.err
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=5
#SBATCH --cpus-per-task=5
#SBATCH --mem-per-cpu=2000

### === USER-DEFINED PATHS === ###
WINDNINJA_SIF="/rds/bear-apps/container-images/singularity/x86_64/w/windninja/windninja_latest.sif"    # Path to .sif container
SHARED_STORAGE="/rds/projects/h/hemidah10/Kaiyuan/WindNinja/momWindNinja/temp/Shap"                  # Input/Output directory (shared across nodes)
LOCAL_DIR="/rds/projects/h/hemidah10/Kaiyuan/WindNinja/momWindNinja/temp/data/Shap"                  # Temporary working directory (local to node)
SCRIPT_DIR="/rds/projects/h/hemidah10/Kaiyuan/WindNinja/momWindNinja/temp/Shap/test"                 # Where run.sh and config file will go

### === CREATE LOCAL DIRS ON ALL NODES === ###
srun --exclusive mkdir -p ${LOCAL_DIR}
srun --exclusive mkdir -p ${SCRIPT_DIR}
srun --exclusive rsync -av --progress ${WINDNINJA_SIF} ${LOCAL_DIR}/
srun --exclusive bash -c "rsync -av --progress ${SHARED_STORAGE}/run.sh ${SCRIPT_DIR}/"
srun --exclusive bash -c "rsync -av --progress ${SHARED_STORAGE}/base_cli.cfg ${SCRIPT_DIR}/"
echo ".sif file and scripts copied to all compute nodes."

### === DEFINE WHICH FOLDERS TO RUN === ###
task_queue=()
FOLDERS=(0 1 2 3)  # <<< USER: Update this list based on folder names

for folder in "${FOLDERS[@]}"; do
    if [ -d "${SHARED_STORAGE}/${folder}" ]; then
        task_queue+=("$folder")
    fi
done

total_jobs=${#task_queue[@]}
echo "Total folders to process: $total_jobs"

### === PROCESS TASK QUEUE === ###
running_jobs=0
job_index=0

while [ $job_index -lt $total_jobs ] || [ $running_jobs -gt 0 ]; do
    if [ $running_jobs -lt $((SLURM_NNODES * 4)) ] && [ $job_index -lt $total_jobs ]; then
        echo "Launching jobs..."
        for i in {0..3}; do
            if [ $job_index -lt $total_jobs ]; then
                folder="${task_queue[$job_index]}"
                job_index=$((job_index + 1))

                srun --exclusive -N1 -n1 bash -c "
                    cp -r \"${SHARED_STORAGE}/${folder}\" \"${LOCAL_DIR}/\" &&
                    singularity exec -B ${LOCAL_DIR}/${folder}:/output -B ${SCRIPT_DIR}:${SCRIPT_DIR} ${LOCAL_DIR}/windninja_latest.sif ${SCRIPT_DIR}/run.sh ${folder}
                " &
                ((running_jobs+=1))
            fi
        done
    else
        wait -n
        ((running_jobs-=1))
    fi
done
wait
echo "All tasks completed."

run.sh脚本 #

#!/bin/bash

### === USER-DEFINED VARIABLES === ###
FOLDER=$1
OUTPUT_FOLDER="./output"                          # << Do not change
CONFIG_FILE="./test/base_cli.cfg"                 # << Do not change unless renamed
MODIFIED_CONFIG="./test/base_cli_${FOLDER}.cfg"
LOG_FILE="${OUTPUT_FOLDER}/simulation.log"

### === ENV SETUP (FOR MOMENTUM SOLVER) === ###
export CPL_DEBUG=NINJAFOAM
source /opt/openfoam8/etc/bashrc                  # << Need to check
export OMPI_ALLOW_RUN_AS_ROOT=1
export OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1
export FOAM_USER_LIBBIN=/usr/local/lib/           

### === TEMP UNSET SLURM ENV TO AVOID FOAM CONFLICT === ###
SLURM_ENV=$(env | grep ^SLURM_)
unset $(env | grep ^SLURM_ | cut -d= -f1)

mkdir -p ${OUTPUT_FOLDER}

### === USER: MODIFY IF DEM PATH CHANGES === ###
DEM_PATH="./output/dems_folder/dem0/dem0.tif"

sed \
  -e "s|\$dem_file|${DEM_PATH}|g" \
  -e "s|\$output_directory|${OUTPUT_FOLDER}|g" \
  ${CONFIG_FILE} > ${MODIFIED_CONFIG}

echo "Starting WindNinja CLI run for folder ${FOLDER}..."
WindNinja_cli ${MODIFIED_CONFIG} > ${LOG_FILE} 2>&1
EXIT_CODE=$?

export ${SLURM_ENV}

if [ $EXIT_CODE -eq 0 ]; then
    echo "WindNinja CLI run completed for folder ${FOLDER}."
else
    echo "WindNinja CLI run failed for folder ${FOLDER} with exit code $EXIT_CODE."
    exit 1
fi

### === SYNC AND CLEANUP === ###
start_time=$(date +%s)
rm -r "${LOCAL_DIR}/${FOLDER}/dems_folder/dem0/NINJA"* 2>/dev/null
cp -r "${LOCAL_DIR}/${FOLDER}/dems_folder/dem0/mass" "${SHARED_STORAGE}/${FOLDER}/dems_folder/dem0/"
cp -r "${LOCAL_DIR}/${FOLDER}/dems_folder/dem0/momentum" "${SHARED_STORAGE}/${FOLDER}/dems_folder/dem0/"
cp -r "${LOCAL_DIR}/${FOLDER}/simulation.log" "${SHARED_STORAGE}/${FOLDER}/simulation.log"

end_time=$(date +%s)
echo "Execution time = $((end_time - start_time)) seconds"
rm -rf "${LOCAL_DIR}/${FOLDER}"

核心原因分析 #

错误execve(): mkdir: No such file or directory本质是Slurm执行srun时，找不到mkdir命令或其依赖的路径，核心问题集中在环境变量丢失和路径引用不一致两点。

1. run.sh中SLURM环境变量清空逻辑破坏系统PATH #

run.sh中清空SLURM变量的代码写法存在缺陷，会意外清空系统核心环境变量（比如PATH），导致mkdir等基础命令找不到：

unset $(env | grep ^SLURM_ | cut -d= -f1)

当grep输出包含特殊字符时，unset会误操作其他变量，直接导致系统命令路径丢失。

修复方案：
替换为精准的SLURM变量清理逻辑，同时保留原始PATH：

# 保存原始PATH避免丢失
ORIGINAL_PATH=$PATH
# 仅收集SLURM开头的变量名
SLURM_VARS=$(env | grep ^SLURM_ | cut -d= -f1)
# 仅清空SLURM变量（避免误操作）
if [ -n "$SLURM_VARS" ]; then
    unset $SLURM_VARS
fi
# 恢复原始PATH
export PATH=$ORIGINAL_PATH

2. run.sh未获取LOCAL_DIR变量 #

run.sh末尾清理部分用到的${LOCAL_DIR}是sbatch脚本中的自定义变量，但并未传递到容器内部，导致容器内的mkdir/rm命令引用了不存在的路径。

修复方案：
在sbatch调用run.sh时，将LOCAL_DIR作为参数传递：

# 修改sbatch中的srun命令
srun --exclusive -N1 -n1 bash -c "
    cp -r \"${SHARED_STORAGE}/${folder}\" \"${LOCAL_DIR}/\" &&
    singularity exec -B ${LOCAL_DIR}/${folder}:/output -B ${SCRIPT_DIR}:${SCRIPT_DIR} ${LOCAL_DIR}/windninja_latest.sif ${SCRIPT_DIR}/run.sh ${folder} ${LOCAL_DIR}
" &

同时在run.sh开头接收参数：

FOLDER=$1
LOCAL_DIR=$2

3. srun创建目录时的节点一致性问题 #

sbatch开头的srun --exclusive mkdir -p ${LOCAL_DIR}可能因部分节点存储不可用或权限不足导致创建失败，后续任务引用该路径时触发错误。

修复方案：
添加错误检查确保目录创建成功：

# 替换原mkdir命令
srun --exclusive bash -c "mkdir -p ${LOCAL_DIR} || { echo 'Failed to create LOCAL_DIR on node' \$HOSTNAME; exit 1; }"

4. 容器内OpenFOAM环境修改PATH #

run.sh中source /opt/openfoam8/etc/bashrc会修改容器内PATH，可能覆盖系统默认命令路径，导致mkdir找不到。

修复方案：
在source前保存PATH，之后恢复：

ORIGINAL_PATH=$PATH
source /opt/openfoam8/etc/bashrc
# 保留原始PATH优先级
export PATH=$ORIGINAL_PATH:$PATH

验证步骤 #

1. 先修改run.sh中的环境变量清理逻辑，确保PATH不丢失
2. 传递LOCAL_DIR参数给run.sh，修正路径引用
3. 提交单个文件夹的测试任务，查看错误日志是否消失

内容的提问来源于stack exchange，提问作者Kaiyuan Zheng