共享Linux服务器并行计算资源：最优MPI进程数选择咨询

Let's break this down clearly, since you're working with a shared compute resource and want to get the fastest simulation speed possible with your meshfree solver.

First, Understand Your Hardware & Available Resources #

From your system info, here's the key hardware context:

• Total physical cores: 2 sockets × 20 cores per socket = 40 physical cores (the 80 "CPU(s)" listed are logical threads from hyper-threading)
• NUMA nodes: 2 (one per socket) – cross-NUMA memory access has higher latency, so we want to avoid unnecessary cross-node scheduling
• Your colleague is using 10 cores – assuming this refers to physical cores (standard for most engineering solvers), you have 30 physical cores available

Is 30 MPI Processes the Optimal Choice? #

Short answer: 30 is a very strong starting point, but here's why and how to confirm:

1. Hyper-threading vs physical cores: Engineering solvers (like meshfree methods) are almost always compute-bound. Hyper-threading (using logical threads) rarely gives meaningful speedups here—two threads sharing a physical core's execution units will compete for resources, often leading to slower performance. So you should prioritize using one MPI process per available physical core.
2. NUMA awareness: To get the best performance, make sure your 30 MPI processes are distributed evenly across the remaining cores without crossing NUMA nodes unnecessarily. For example, if your colleague is using 10 cores on one NUMA node, you could use the remaining 10 on that node plus all 20 on the other (total 30). If their usage is split, adjust accordingly.
3. Test small variations: Since solver parallel efficiency can vary by problem size, run quick test simulations with 28, 30, and maybe 32 processes (if 32 uses some hyper-threads) to see which gives the fastest wall-clock time. The difference might be small, but it's worth checking for your specific workload.

Additional Tips to Maximize Speed #

• Bind MPI processes to physical cores: Use tools like numactl to lock your processes to specific cores, preventing the OS from moving them around (which causes latency). For example, if your colleague is using cores 0-9, you could run:

  numactl --physcpubind=10-39 /path/to/meshfree/installation/folder/meshfree_run.sh 30
  

  (Verify your colleague's core usage first with htop or ps -eo pid,pcpu,core.)
• Avoid oversubscribing: Never run more MPI processes than available physical cores—this will cause context switching and slow down your simulation.
• Check memory per process: Ensure each MPI process has enough memory allocated on its NUMA node. Use numactl --hardware to see memory distribution across nodes, and use --membind with numactl to pin memory to the same node as your cores.

内容的提问来源于stack exchange，提问作者vikingd