如何在多Docker容器及Docker Swarm多节点运行MPI进程？

Great job building that Ubuntu 16.04 + MPI Docker image! Let's walk through how to scale your MPI workload across multiple containers on a single host first, then move to distributed execution with Docker Swarm.

Running MPI Across Multiple Containers on a Single Host #

MPI relies on reliable network connectivity between all participating nodes, so we’ll start by setting up a dedicated Docker network to ensure seamless DNS resolution between containers.

1. Create a custom Docker bridge network
   This network lets containers communicate using their names as hostnames:

   docker network create mpi-net
   

2. Launch multiple MPI containers
   Spin up as many containers as you need (we’ll use 4 for this example) and attach them to the mpi-net network:

   docker run -d --name mpi-node-1 --network mpi-net orwel84/ubuntu-16-mpi
   docker run -d --name mpi-node-2 --network mpi-net orwel84/ubuntu-16-mpi
   docker run -d --name mpi-node-3 --network mpi-net orwel84/ubuntu-16-mpi
   docker run -d --name mpi-node-4 --network mpi-net orwel84/ubuntu-16-mpi
   

3. Execute MPI across containers
   Enter one of the containers (we’ll use mpi-node-1 as the "head" node):

   docker exec -it mpi-node-1 bash
   

   Now run mpirun, specifying the list of hostnames (container names) to distribute processes across:

   mpirun -np 4 --host mpi-node-1,mpi-node-2,mpi-node-3,mpi-node-4 --allow-run-as-root ./mpi_hello_world
   

   Alternatively, you can create a hostfile to manage nodes more cleanly:

   cat > hostfile << EOF
   mpi-node-1 slots=1
   mpi-node-2 slots=1
   mpi-node-3 slots=1
   mpi-node-4 slots=1
   EOF
   mpirun -np 4 --hostfile hostfile --allow-run-as-root ./mpi_hello_world
   

   You’ll see output from each container’s unique hostname (instead of a single container ID like your initial test).

Running MPI Across Multiple Nodes with Docker Swarm #

To scale across a cluster, Docker Swarm provides overlay networking for cross-node container communication. Here’s how to set it up:

1. Initialize your Swarm cluster
   On your manager node, initialize the swarm (replace <YOUR_MANAGER_NODE_IP> with your actual manager IP):

   docker swarm init --advertise-addr <YOUR_MANAGER_NODE_IP>
   

   Follow the printed command to join worker nodes to the swarm.

2. Create an overlay network
   Overlay networks enable communication between containers across different Swarm nodes:

   docker network create --driver overlay mpi-swarm-net
   

3. Deploy MPI worker containers
   Create a Swarm service to launch MPI containers across your cluster (we’ll use 4 replicas):

   docker service create --name mpi-node --network mpi-swarm-net --replicas 4 orwel84/ubuntu-16-mpi sleep infinity
   

   The sleep infinity command keeps containers running since we need them alive for MPI coordination.

4. Prepare passwordless SSH access between nodes
   MPI uses SSH to coordinate processes, so we need to set up passwordless SSH between all containers. Enter one of the containers to act as the head node:

   docker exec -it $(docker ps -q -f name=mpi-node) bash
   

   Generate an SSH key and copy it to all MPI containers (using Swarm’s built-in DNS discovery):

   # Generate a passwordless SSH key
   ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa
   # Copy the public key to all mpi-node containers
   for host in $(dig +short mpi-node); do
     ssh-copy-id -o StrictHostKeyChecking=no root@$host
   done
   # Start SSHd on all containers
   for host in $(dig +short mpi-node); do
     ssh $host "service ssh start"
   done
   

5. Run MPI across the Swarm cluster
   Create a hostfile with all MPI container addresses, then launch the job:

   dig +short mpi-node > hostfile
   mpirun -np 4 --hostfile hostfile --allow-run-as-root ./mpi_hello_world
   

   You’ll see output from containers running on different Swarm nodes, each with their unique hostnames/IPs.

内容的提问来源于stack exchange，提问作者revolutionary