如何配置future包实现远程计算服务器多任务并行提交？

Hey there! Let's tackle this future remote parallel submission issue you're facing. The core problem here is that your current plan is using a single remote worker, which queues tasks sequentially. Here's how to fix it with proper topology configuration—no repeated plan() calls needed:

Step 1: Define Your Reusable Remote Worker Template #

First, make sure your login object is a configured remote worker template (using future.remote). It might look something like this (fill in your server-specific details):

library(future)
library(future.remote)

# Reusable remote login configuration
login <- tweak(remote,
               user = "your_username",
               host = "your_compute_server",
               rscript = "/path/to/remote/Rscript", # Path to R on your server
               ssh.cmd = "ssh", # Use custom path if your SSH client isn't in PATH
               ssh.args = c("-i", "~/.ssh/your_private_key.pem") # Add SSH auth params
)

Step 2: Configure a Multi-Worker Parallel Plan #

Instead of using a single worker, set up a plan that uses multiple independent remote workers. This lets future submit tasks simultaneously to your server. Choose one of these two straightforward approaches:

Option A: Quick Multi-Worker Setup with multisession

This is the simplest way to spin up identical parallel remote workers:

# Configure 3 parallel remote workers (adjust the number to match your server's capacity)
plan(multisession, workers = list(login, login, login))

Each entry in the workers list creates a separate remote connection, so tasks are sent in parallel instead of waiting for the previous one to complete.

Option B: Fine-Grained Control with a PSOCK Cluster

If you need more control (e.g., specifying different servers, resource limits, or worker-specific settings), use a PSOCK cluster:

# Create 3 remote workers on your compute server
cl <- makeClusterPSOCK(
  rep("your_compute_server", 3), # Spin up 3 workers on the same server
  user = "your_username",
  rscript = "/path/to/remote/Rscript",
  ssh.cmd = "ssh",
  ssh.args = c("-i", "~/.ssh/your_private_key.pem")
)

# Bind the cluster to your future plan
plan(cluster, workers = cl)

Step 3: Verify Parallel Execution #

Test with a set of time-consuming tasks to confirm simultaneous submission:

# Create 3 tasks that each sleep for 5 seconds
futures <- list(
  future({ Sys.sleep(5); paste("Task 1 completed at", Sys.time()) }),
  future({ Sys.sleep(5); paste("Task 2 completed at", Sys.time()) }),
  future({ Sys.sleep(5); paste("Task 3 completed at", Sys.time()) })
)

# Collect and print results
results <- value(futures)
print(results)

If working correctly, all three tasks will finish at nearly the same time (not 5 seconds apart).

Why Your Original Setup Was Serial #

When you used plan(login), you were only configuring one remote worker. Future queues all tasks to this single worker, so it runs them sequentially. By defining multiple workers in your plan topology, you enable simultaneous task submission without needing to re-run plan() between tasks.

内容的提问来源于stack exchange，提问作者Ruben