Singularity中shell与exec行为差异及HPC环境运行问题排查求助

Let's break down your problem step by step—this is a classic case of environment variable discrepancies between Singularity execution modes and host environments, especially with a dependency like NEURON that relies heavily on runtime configuration.

1. Core Differences Between singularity shell and singularity exec with --containall #

The key gap lies in how environment initialization is handled:

• When you run singularity shell --containall, you launch an interactive shell session inside the container. This triggers the container's shell initialization scripts (like /etc/profile, ~/.bashrc, or ~/.profile depending on your shell), which often set up critical variables for packages like NEURON.
• singularity exec --containall runs your command directly without loading interactive shell configurations by default. Singularity skips sourcing those init files unless you explicitly wrap your command in a shell invocation. That’s why your script works in the shell but fails in exec: the shell is loading NEURON’s required environment variables that exec isn’t.

2. Why Shell Mode Works But Exec Doesn’t #

NEURON depends on several non-default environment variables to find its binaries, libraries, and Python modules:

• NRN_HOME: Points to NEURON’s installation directory
• PATH: Must include $NRN_HOME/bin for NEURON executables (like nrniv)
• LD_LIBRARY_PATH: Needs $NRN_HOME/lib to load shared libraries
• PYTHONPATH: May require $NRN_HOME/lib/python to locate NEURON’s Python bindings

In your shell session, these variables are likely set by a script like nrn_env.sh (often sourced in ~/.bashrc during container setup). When you use exec directly, that sourcing step never happens—so NEURON can’t locate its dependencies, triggering your error.

3. Local vs. HPC Cluster Differences #

The root of this split is how host environment variables are propagated (or blocked):

• On your local machine, without --containall, Singularity shares most host environment variables with the container. Even with --containall, an interactive shell might still pick up residual variables or load container-side init scripts that fill gaps.
• On the HPC cluster, stricter defaults are common:
  • The cluster’s Singularity config might enforce --cleanenv or filter more host variables by default, stripping out any accidental local fixes.
  • HPC user home directories may be mounted differently (or not at all) with --containall, so even if your local shell relied on host-side configs, those aren’t available on the cluster.
  • Cluster nodes might have kernel/filesystem setups that interact with Singularity’s containerization more strictly than your local machine.

Troubleshooting Steps #

Here’s how to diagnose and resolve the issue:

a. Compare Environment Variables Across Modes #

Capture the environment in working and failing scenarios to spot missing variables:

• Working (local shell):

  singularity shell --containall --bind ./:/mnt container.sif
  # Inside the shell:
  env > /mnt/shell_env.txt
  exit
  

• Failing (exec):

  singularity exec --containall --bind ./:/mnt container.sif env > /mnt/exec_env.txt
  

Use diff shell_env.txt exec_env.txt to identify NEURON-related variables missing in the exec mode.

b. Force Shell Initialization in Exec Mode #

Wrap your script command in a shell that sources the necessary init files:

singularity exec --containall --bind ./:/mnt container.sif bash -c "source ~/.bashrc && python3 /mnt/my_script.py"

If this works, it confirms the shell init scripts are the missing piece.

c. Manually Set NEURON Environment Variables #

If sourcing shell configs isn’t feasible, explicitly define the required variables in your exec command:

singularity exec --containall --bind ./:/mnt container.sif bash -c "
export NRN_HOME=/path/to/neuron/inside/container
export PATH=\$NRN_HOME/bin:\$PATH
export LD_LIBRARY_PATH=\$NRN_HOME/lib:\$LD_LIBRARY_PATH
export PYTHONPATH=\$NRN_HOME/lib/python3.6/site-packages:\$PYTHONPATH
python3 /mnt/my_script.py
"

Adjust paths to match NEURON’s installation location inside your container.

d. Check HPC Singularity Configuration #

On the cluster, verify if Singularity uses default flags that modify environment propagation:

singularity exec --help

Look for mentions of --cleanenv or default environment filtering. You may need to explicitly allow variables with --env or request adjustments to the cluster’s Singularity settings (if you have permissions).

e. Validate Container Setup #

Ensure your container has NEURON properly installed with its environment scripts. Check if nrn_env.sh exists (usually in $NRN_HOME/bin) and that it’s sourced in the container’s ~/.bashrc or /etc/profile.

内容的提问来源于stack exchange，提问作者bidby