DIA-NN蛋白质组学分析HPC并行化时出现std::bad_alloc内存错误

背景 #

基于已发表工具DIA-NN搭建并行化蛋白质组学数据分析流程，按开发者说明可将流程拆分为5个独立组件，其中两个最慢的部分支持并行运行。

问题现象 #

• 单样本或3-4个样本并行时可正常运行（4个样本偶尔失败），样本量超过此数量必然失败；1节点运行3个样本可行，2节点运行6个样本失败，排除核心分配问题。
• 脚本中设置pwait为3时可运行，设为更大值则触发失败。

错误提示与排查方向 #

• 核心错误输出：

  Loading spectral library timsTOF/libs//lib.predicted.speclib
  [0:17] Library annotated with sequence database(s): timsTOF/fasta/Combined_proteins_48970.fasta; timsTOF/fasta/Mouse_uniprot-proteome_UP000000589_2023.04.27.fasta
  [0:17] Protein names missing for some isoforms
  [0:17] Gene names missing for some isoforms
  [0:17] Library contains 17129 proteins, and 16776 genes
  terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
  
  /var/spool/slurm/slurmd/job923716/slurm_script: line 20: [: -ge: unary operator expected
  /var/spool/slurm/slurmd/job923716/slurm_script: fork: Cannot allocate memory
  

• 猜测流程代码存在内存泄漏，正常场景下少量样本不应触发内存分配失败；同时正在排查HPC集群的基础内存限制。

示例运行脚本（1节点96线程） #

#!/bin/bash
#SBATCH -J diann
#SBATCH -N 1
#SBATCH --partition=XXX
#SBATCH --qos=XXX
#SBATCH --time=0-00:05:00 #D-HH:MM:SS
#SBATCH --mail-type=ALL 
#SBATCH --mail-user=XXX
#SBATCH --account=XXX

echo "starting dia-nn run"

module purge

module load anaconda3/2022.05-gcc-12.2.0-oqiw76n
source activate $DATA/myenv

module load openjdk/11.0.17_8-gcc-12.2.0-o2utqnb #updated java

# function to specify max number of jobs
function pwait() {
    while [ $(jobs -p | wc -l) -ge $1 ]; do
        sleep 1
    done
}

# set working directories 
mkdir "full-diann-trial"

LIB="timsTOF/libs/"
INPUT="timsTOF/2023/20231030"
OUT="full-diann-trial"

mkdir "$OUT/part2"
mkdir "$OUT/part4"

PART2="$OUT/part2"
PART4="$OUT/part4"
FILES=("$INPUT"/*.d)

F_OPTIONS=""
for FILE_PATH in "${FILES[@]}"; do
    FILENAME=$(basename "$FILE_PATH" .d)
    F_OPTIONS+=" --f $FILE_PATH"
done

PART2_PARAMS="--cut K*,R* --var-mod UniMod:35,15.994915,M --var-mod UniMod:1,42.010565,*n --monitor-mod UniMod:1 \
        --lib $LIB/lib.predicted.speclib \
        --min-pr-mz 400 --max-pr-mz 1000 --min-fr-mz 100 --max-fr-mz 1700 \
        --threads 12 \
        --missed-cleavages 2 --min-pep-len 7 --max-pep-len 50 \
        --min-pr-charge 2 --max-pr-charge 4 --var-mods 2 --verbose 3 \
        --individual-windows \
        --temp $PART2 \
        --min-corr 2.0 --corr-diff 1.0 \
        --quick-mass-acc \
        --individual-mass-acc \
        --time-corr-only"

### step2 - analysing each run separately with the in silico library generated in step 1, generate quant files

echo "starting part2"

for FILE in "${FILES[@]}"; do
    ./bin/diann-1.8.1 $PART2_PARAMS --f "$FILE" --out "$OUT/step2-$(basename "$FILE" .d)" &
    pwait 6
done

wait

echo "finished part 2"

内容的提问来源于stack exchange，提问作者aliibarry