Slurm环境下使用furrr时，能否按变量修改future的集群资源？

Great question! Being able to dynamically assign Slurm resources based on task-specific variables is super useful when running heterogeneous workloads with future, future.batchtools, and furrr—no sense wasting 32GB of RAM on a tiny task, or starving a big job of CPU cores.

The key issue with your current setup is that the global plan(batchtools_slurm(...)) applies the same resources to every future. To fix this, we need to create customized futures for each task that use resources tailored to the variables passed into your workflow. Here's a step-by-step solution that plays nicely with your existing toolset:

Step 1: Set up a base Slurm plan (without fixed resources) #

First, define a base batchtools_slurm plan that references your template file, but don't hardcode the resources parameter—we'll tweak this per task later:

library(future)
library(future.batchtools)
library(furrr)
library(purrr)

# Base plan with your Slurm template, no fixed resources
base_slurm_plan <- batchtools_slurm(template = "path/to/your/slurm_template.tmpl")

Step 2: Prepare your task data with resource parameters #

Make sure your input data (the one you pass to your workflow) includes columns/values for the resources each task needs—like memory, CPU cores, or wall time. For example:

# Example task data: each row is a task with compute params + resource needs
task_df <- tibble(
  data_path = c("small_data.csv", "large_data.csv", "medium_data.csv"),
  analysis_type = c("summary", "model_train", "visualization"),
  # Resource parameters tailored to each task
  mem_gb = c(8, 32, 16),
  cpus = c(2, 8, 4),
  wall_time = c("1:00:00", "4:00:00", "2:00:00")
)

Step 3: Use pmap + dynamic futures (instead of future_pwalk) #

Instead of relying on future_pwalk (which uses a global plan), we'll use purrr::pmap to iterate over your tasks, create a customized future for each one, and then collect results. We use future::tweak() to modify the base plan's resources for each task:

# Create a future for each task with custom resources
task_futures <- pmap(task_df, function(data_path, analysis_type, mem_gb, cpus, wall_time) {
  # Tweak the base plan to use task-specific resources
  custom_plan <- tweak(base_slurm_plan, resources = list(
    mem = paste0(mem_gb, "G"),
    cpus_per_task = cpus,
    time = wall_time
  ))
  
  # Define and launch the future with the custom plan
  future({
    # Your task code goes here—use the task-specific params
    data <- read.csv(data_path)
    if (analysis_type == "summary") {
      summary(data)
    } else if (analysis_type == "model_train") {
      glm(y ~ ., data = data)
    } else {
      ggplot(data, aes(x, y)) + geom_point()
    }
  }, plan = custom_plan)
})

# Collect results from all futures
task_results <- map(task_futures, value)

Step 4: Verify your Slurm template uses the dynamic resources #

Make sure your Slurm template file references the resources variables you're passing. A minimal template might look like this:

#!/bin/bash
#SBATCH --job-name={{ job.name }}
#SBATCH --output={{ log.file }}
#SBATCH --error={{ log.file }}
#SBATCH --mem={{ resources$mem }}
#SBATCH --cpus-per-task={{ resources$cpus_per_task }}
#SBATCH --time={{ resources$time }}

Rscript -e 'batchtools::doJobCollection("{{ collection }}")'

Why this works #

• tweak() lets us modify the base batchtools_slurm plan on the fly, overriding the resources parameter for each task.
• By creating a separate future for each task with its own customized plan, we ensure Slurm gets the exact resource specs for every job.
• This approach keeps your workflow flexible—you can adjust resource logic (e.g., calculate memory based on file size) directly in the pmap iteration.

If you really want to stick with future_pwalk, you'd need to refactor to pass resource params into the task function and use a dynamic plan that pulls resources from the task context, but the pmap + custom futures method is cleaner and more explicit for heterogeneous workloads.

内容的提问来源于stack exchange，提问作者Johannes Koch