CentOS7集群SLURM任务中Conda环境tf_gpu无法激活求助

问题背景 #

在CentOS7计算集群提交SLURM任务，运行依赖tensorflow-gpu 2.8.1的Python文件cifar100-vgg16.py，该环境已安装在conda环境tf_gpu中。提交的bash脚本如下：

#!/bin/bash --login
########## SBATCH Lines for Resource Request ##########
 
#SBATCH --time=00:10:00             # limit of wall clock time - how long the job will run (same as -t)
#SBATCH --nodes=1                   # the number of node requested.
#SBATCH --ntasks=1                  # the number of task to run
#SBATCH --cpus-per-task=1           # the number of CPUs (or cores) per task (same as -c)
#SBATCH --mem-per-cpu=2G            # memory required per allocated CPU (or core) - amount of memory (in bytes)
#SBATCH --job-name test2            # you can give your job a name for easier identification (same as -J)
 
########## Command Lines to Run ##########
conda activate tf_gpu
python cifar100-vgg16.py

报错情况 #

SLURM输出显示任务实际调用了系统base conda环境（Python/3.6.4-foss-2018a，内置tensorflow 1.10.1），而非目标tf_gpu环境，导致版本不兼容报错：

> /opt/software/Python/3.6.4-foss-2018a/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/opt/software/Python/3.6.4-foss-2018a/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:524: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/opt/software/Python/3.6.4-foss-2018a/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/opt/software/Python/3.6.4-foss-2018a/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/opt/software/Python/3.6.4-foss-2018a/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/opt/software/Python/3.6.4-foss-2018a/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:532: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
Tensorflow version 1.10.1
Keras version 2.1.6-tf
Scikit learn version 0.20.0
Traceback (most recent call last):
  File "cifar100-vgg16.py", line 39, in <module>
    print("Number of GPUs Available:", len(tensorflow.config.experimental.list_physical_devices('GPU')))
AttributeError: module 'tensorflow' has no attribute 'config'

解决方法 #

1. 手动加载conda初始化脚本 #

集群的非交互式SLURM任务shell通常不会自动加载conda的初始化配置，导致conda activate命令无法识别。在脚本中先加载conda的初始化脚本（路径需根据集群实际conda安装位置调整，常见路径为/opt/conda/etc/profile.d/conda.sh或用户home目录下的~/.bashrc）：

修改后的完整脚本：

#!/bin/bash --login
########## SBATCH Lines for Resource Request ##########
 
#SBATCH --time=00:10:00
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --mem-per-cpu=2G
#SBATCH --job-name test2
 
########## Command Lines to Run ##########
# 加载conda初始化配置
source /opt/conda/etc/profile.d/conda.sh
# 若为用户级conda安装，替换为：source ~/.bashrc

# 激活目标环境
conda activate tf_gpu

# 可选：验证环境是否正确激活（用于调试）
echo "当前Python路径: $(which python)"
echo "当前TensorFlow版本: $(python -c 'import tensorflow as tf; print(tf.__version__)')"

# 运行任务脚本
python cifar100-vgg16.py

2. 使用conda run直接指定环境执行 #

跳过手动激活环境步骤，直接用conda run命令指定目标环境运行脚本：

修改脚本的执行命令部分：

########## Command Lines to Run ##########
conda run -n tf_gpu python cifar100-vgg16.py

3. 卸载冲突的系统Python模块 #

部分集群会通过module命令预加载系统Python环境，抢占conda环境的优先级。可在激活conda环境前先卸载系统Python模块：

添加到脚本执行命令开头：

# 卸载系统预加载的Python模块
module unload Python

conda activate tf_gpu
python cifar100-vgg16.py

4. 直接调用环境中Python的绝对路径 #

如果已知tf_gpu环境中Python的绝对路径，直接使用该路径执行脚本，无需激活环境：

替换执行命令为：

# 替换为你的tf_gpu环境Python实际路径，示例路径为~/.conda/envs/tf_gpu/bin/python
~/.conda/envs/tf_gpu/bin/python cifar100-vgg16.py

内容的提问来源于stack exchange，提问作者Snehal Patel