Linux Environment Modules与Conda Environment差异及环境隔离方案咨询

Hey there! Let's break this down step by step since you're trying to juggle OpenMPI and MPICH on Ubuntu while keeping them properly isolated—totally get why you want to avoid conflicts with those overlapping compiler wrappers.

First, let's clear up the core distinctions between these two tools:

• Use Case Focus: Modules are built for managing shared, system-level software (think HPC clusters where multiple users need access to different tool versions). They tweak your shell's PATH, LD_LIBRARY_PATH, and other variables to switch between existing installations without duplicating files. Conda, by contrast, is a package manager that creates self-contained environments with full copies of binaries, libraries, and dependencies—ideal for user-specific, reproducible setups, even for non-compiled languages like Python.
• Isolation Level: Modules don't duplicate software; they just redirect your shell to pre-existing installs. Conda environments are fully independent (mostly—we'll unpack your base env confusion later) because every dependency lives in the env's directory.
• Compiled Software Flexibility: Modules shine with pre-compiled system tools (like MPI implementations) that you don't want to recompile for every use case. Conda handles compiled software too, but it's more optimized for packages in its official repository; compiling custom binaries in a Conda env requires a bit more intentional setup.

For your specific goal of switching between OpenMPI and MPICH on Ubuntu, here's the breakdown:

• Linux Environment Modules is the most straightforward pick if you want to use Ubuntu's pre-built packages. Install both via sudo apt install openmpi-bin mpich, then create simple module files to toggle between them. Each module will adjust your PATH to prioritize either OpenMPI's or MPICH's mpicc/mpirun, eliminating wrapper conflicts. It's lightweight too—no wasted disk space on duplicate binaries.
• Conda Environments work if you prefer self-contained, user-managed setups. You can install each MPI implementation into its own env:

  conda create -n openmpi_env openmpi
  conda create -n mpich_env mpich
  

  Activating either env will restrict you to that MPI's tools and libraries.
• Alternative: If you want maximum control, compile each MPI from source into separate directories (e.g., /opt/openmpi and /opt/mpich) and write simple shell scripts to adjust your PATH/LD_LIBRARY_PATH when switching. But Modules or Conda are more scalable and less error-prone.

This is a super common gotcha! By default, Conda adds the base environment's bin directory to your system PATH even when you activate other envs—that's why you can access base-installed tools elsewhere.

To enforce strict isolation:

• Create new envs with the --no-default-packages flag to block base package leakage:

  conda create --no-default-packages -n my_isolated_env
  

• Or disable auto-activation of the base env entirely:

  conda config --set auto_activate_base false
  

  After this, you'll only access base env software when you explicitly run conda activate base.

If you compile custom software in a Conda env, make sure to:

1. Activate the env first before compiling.
2. Use the env's built-in compiler wrappers (e.g., the $CC variable set by Conda) instead of system compilers. This ensures your compiled binaries link against the env's libraries and are only accessible when the env is active.

内容的提问来源于stack exchange，提问作者Porcupine