如何在ipyparallel集群从机上指定启动多个ipengine实例？

Yes, you absolutely can launch multiple ipengine instances on a single worker machine—even though the ipengine command itself doesn’t have a -n flag like ipcluster start does. Here are the most straightforward methods to make this work:

Method 1: Launch multiple instances via a shell loop #

The simplest approach is to run the ipengine command multiple times in the background using a shell loop. For example, to spin up 4 engine instances on your worker machine:

# Bash/zsh loop to start 4 background ipengine processes
for i in {1..4}; do
    ipengine &
done

All these instances will automatically connect to your running ipyparallel controller—just make sure the worker can reach the controller’s IP address (you may need to use --ip=<master-public-ip> when starting ipcluster on the master node to bind it to a publicly accessible address instead of localhost).

To stop all running engines later, you can use:

pkill ipengine

If you want to track logs for each engine separately, modify the loop to redirect output:

for i in {1..4}; do
    ipengine >> "/tmp/ipengine_$i.log" 2>&1 &
done

Method 2: Configure ipcluster to auto-launch multiple engines per worker #

If you’re managing a multi-node cluster (e.g., via SSH), you can set up an ipyparallel profile to specify how many engines should run on each worker node automatically.

1. First, generate a custom profile (if you don’t have one already):

ipcluster configure --profile=my_cluster

2. Open the configuration file (located at ~/.ipython/profile_my_cluster/ipcluster_config.py) in a text editor.
3. Set the appropriate launcher options based on your cluster setup:
   • For SSH-managed clusters, use SSHClusterLauncher to define node-specific engine counts:

     c.IPClusterEngines.engine_launcher_class = 'SSHClusterLauncher'
     # Map worker hostnames to the number of engines to start on each
     c.SSHClusterLauncher.nodes = {
         'worker-1': 4,
         'worker-2': 2,
         'localhost': 3
     }
     

   • If you’re launching engines locally on the worker and want ipcluster to manage them, use LocalEngineSetLauncher:

     c.IPClusterEngines.engine_launcher_class = 'LocalEngineSetLauncher'
     # Number of engines to start on this local machine
     c.LocalEngineSetLauncher.engines = 4
     

4. Start the cluster with your custom profile:

ipcluster start --profile=my_cluster

This will automatically launch the specified number of engines on each configured worker node.

Key Notes to Remember #

• Controller Access: Double-check that the worker machine can reach the controller’s IP. Binding the controller to a public IP (instead of localhost) is critical for multi-node setups.
• Resource Limits: Avoid launching more engines than your worker’s CPU cores can handle (unless testing oversubscription). A safe default is to match the number of physical cores on the machine.
• Task Distribution: Each ipengine instance will appear as a separate engine in your ipyparallel client, so you can distribute tasks across all of them just like engines started via ipcluster start -n.

内容的提问来源于stack exchange，提问作者Vasily