Slurm数组任务从TXT取参数并在R脚本设为对象的实现咨询

Step 1: Prepare the Parameter File #

Your params.txt should have one parameter set per line, formatted as comma-separated key=value assignments:

a=5,b=10,c=0.75,d="SL",e=1
a=5,b=10,c=0.5,d="TL",e=5
...

Step 2: Write the R Simulation Script (simulate.R) #

This script handles parameter loading, simulation execution, and output saving tied to the Slurm task ID:

# Fetch Slurm array task ID from environment
task_id <- as.integer(Sys.getenv("SLURM_ARRAY_TASK_ID"))

# Read all parameter lines from file
params_lines <- readLines("params.txt")

# Extract the line matching the current task ID
selected_line <- params_lines[task_id]

# Convert comma-separated assignments to valid R code (replace commas with semicolons)
r_code <- gsub(",", ";", selected_line)

# Evaluate code to create variables a, b, c, d, e in the workspace
eval(parse(text = r_code))

# --------------------------
# Insert your simulation code here
# Example simulation:
set.seed(42 + task_id)  # Unique seed per task to avoid correlated results
out <- data.frame(
  sim_values = rnorm(100, mean = a, sd = b),
  group = rep(d, 100),
  param_c = c,
  param_e = e
)
# --------------------------

# Save output with task ID in filename
write.csv(out, paste0("out_", task_id, ".csv"), row.names = FALSE)

Step 3: Write the Slurm Submission Script (submit.sh) #

This script defines the array job configuration and triggers the R script:

#!/bin/bash
#SBATCH --job-name=sim_array
#SBATCH --array=1-$(wc -l < params.txt)  # Auto-set task count to match parameter lines
#SBATCH --output=sim_%A_%a.out  # Log file: %A = job ID, %a = task ID
#SBATCH --error=sim_%A_%a.err    # Error log file
#SBATCH --cpus-per-task=1        # Adjust CPU cores per task as needed
#SBATCH --mem=2G                 # Adjust memory per task based on simulation needs
#SBATCH --time=01:00:00          # Time limit per task (HH:MM:SS)

# Load R module (adjust version to match your HPC's setup)
module load R/4.3.2

# Execute the R script
Rscript simulate.R

Step 4: Submit the Job #

Run this command on the HPC login node to launch the array job:

sbatch submit.sh

Alternative: Use CSV Parameter File #

For a safer, more structured parameter format, switch to a CSV file (params.csv) with headers:

a,b,c,d,e
5,10,0.75,SL,1
5,10,0.5,TL,5

Modify the parameter loading section in simulate.R to:

params_df <- read.csv("params.csv")
selected_params <- params_df[task_id, ]

# Assign variables directly
a <- selected_params$a
b <- selected_params$b
c <- selected_params$c
d <- selected_params$d
e <- selected_params$e

内容的提问来源于stack exchange，提问作者Fumia