如何解决Conda环境中Intel OpenMP与LLVM OpenMP的冲突

问题背景 #

运行Conda环境时出现如下警告：

/opt/miniconda/lib/python3.10/site-packages/threadpoolctl.py:546: RuntimeWarning:
Found Intel OpenMP ('libiomp') and LLVM OpenMP ('libomp') loaded at
the same time. Both libraries are known to be incompatible and this
can cause random crashes or deadlocks on Linux when loaded in the
same Python program.

已排查出依赖LLVM OpenMP的包：

grep llvm ~/anaconda3/pkgs/*/info/index.json
/home/user/anaconda3/pkgs/libclang-10.0.1-default_hb85057a_2/info/index.json:    "libllvm10 >=10.0.1,<10.1.0a0",
/home/user/anaconda3/pkgs/libllvm10-10.0.1-hbcb73fb_5/info/index.json:  "name": "libllvm10",
/home/user/anaconda3/pkgs/libllvm11-11.1.0-h9e868ea_6/info/index.json:  "name": "libllvm11",
/home/user/anaconda3/pkgs/libllvm14-14.0.6-hdb19cb5_3/info/index.json:  "name": "libllvm14",
/home/user/anaconda3/pkgs/llvmlite-0.39.1-py310he621ea3_0/info/index.json:    "libllvm11 >=11.1.0,<11.2.0a0",
/home/user/anaconda3/pkgs/llvmlite-0.39.1-py310he621ea3_0/info/index.json:  "name": "llvmlite",
/home/user/anaconda3/pkgs/llvmlite-0.40.0-py310he621ea3_0/info/index.json:    "libllvm14 >=14.0.6,<14.1.0a0",
/home/user/anaconda3/pkgs/llvmlite-0.40.0-py310he621ea3_0/info/index.json:  "name": "llvmlite",
/home/user/anaconda3/pkgs/numba-0.56.4-py310h1128e8f_0/info/index.json:    "llvmlite >=0.39.*,<0.40",
/home/user/anaconda3/pkgs/numba-0.57.0-py310h1128e8f_0/info/index.json:    "libllvm14 >=14.0.6,<14.1.0a0",
/home/user/anaconda3/pkgs/numba-0.57.0-py310h1128e8f_0/info/index.json:    "llvmlite >=0.40.0,<0.41.0a0",
/home/user/anaconda3/pkgs/pynndescent-0.5.10-py310h06a4308_0/info/index.json:    "llvmlite >=0.34",

明确依赖Intel OpenMP的包：

grep intel ~/anaconda3/pkgs/*/info/index.json
/home/user/anaconda3/pkgs/intel-openmp-2021.4.0-h06a4308_3561/info/index.json:  "name": "intel-openmp",
/home/user/anaconda3/pkgs/intel-openmp-2023.1.0-hdb19cb5_46305/info/index.json:  "name": "intel-openmp",
/home/user/anaconda3/pkgs/mkl-2021.4.0-h06a4308_640/info/index.json:    "intel-openmp 2021.*"
/home/user/anaconda3/pkgs/mkl-2023.1.0-h6d00ec8_46342/info/index.json:    "intel-openmp 2023.*",
/home/user/anaconda3/pkgs/pytorch-1.12.1-cpu_py310hb1f1ab4_1/info/index.json:    "intel-openmp >=2021.4.0,<2022.0a0",
/home/user/anaconda3/pkgs/scikit-learn-intelex-2023.0.2-py310h06a4308_0/info/index.json:  "name": "scikit-learn-intelex",
/home/user/anaconda3/pkgs/scipy-1.10.0-py310hd5efca6_1/info/index.json:    "intel-openmp >=2021.4.0,<2022.0a0",
/home/user/anaconda3/pkgs/scipy-1.10.1-py310h5f9d8c6_1/info/index.json:    "intel-openmp >=2023.1.0,<2024.0a0",

解决方法 #

方案一：统一使用LLVM OpenMP #

1. 卸载依赖Intel OpenMP的包：

   conda remove intel-openmp mkl pytorch scipy scikit-learn-intelex --force
   

2. 从conda-forge频道重新安装兼容LLVM OpenMP的版本：

   conda install -c conda-forge pytorch scipy scikit-learn numba pynndescent
   

   注：conda-forge频道的包默认依赖LLVM OpenMP，安装时指定该频道可避免引入Intel OpenMP。

方案二：统一使用Intel OpenMP #

1. 卸载依赖LLVM OpenMP的包：

   conda remove libllvm* llvmlite numba pynndescent libclang --force
   

2. 从默认conda频道重新安装兼容Intel OpenMP的版本：

   conda install numba pynndescent libclang
   

   注：默认频道多数科学计算包依赖MKL/Intel OpenMP，安装后可保证环境仅加载Intel OpenMP。

方案三：临时屏蔽单个库（仅应急测试用） #

若不想重新安装包，可在运行Python前设置环境变量强制优先加载某一个OpenMP库：

• 优先加载Intel OpenMP：

  export LD_PRELOAD=/opt/miniconda/lib/libiomp5.so
  python your_script.py
  

• 优先加载LLVM OpenMP：

  export LD_PRELOAD=/opt/miniconda/lib/libomp.so
  python your_script.py
  

  注：此方法可能导致依赖另一库的包出现性能下降或运行错误，仅作为临时方案。

方案四：创建分离的Conda环境 #

若需同时使用两类依赖，可创建两个独立环境：

1. 用于运行Intel OpenMP依赖包的环境：

   conda create -n intel_env python=3.10 pytorch scipy scikit-learn-intelex
   conda activate intel_env
   

2. 用于运行LLVM OpenMP依赖包的环境：

   conda create -n llvm_env -c conda-forge python=3.10 numba pynndescent libclang
   conda activate llvm_env
   

内容的提问来源于stack exchange，提问作者SinisterMJ