R中foreach与doParallel调用XGBoost的方法及卡顿问题求助

Hey there! Let's break down how to solve your foreach + doParallel + XGBoost issues, fix that lag on your AMD Naples system, and get your results stored in an ordered list properly.

First, let's make sure you're structuring the code to avoid common pitfalls. The key thing to note here is that XGBoost has its own internal parallelism (via the nthread parameter), which can clash with the parallel workers spawned by doParallel—this is almost certainly contributing to your lag.

Here's the core workflow:

• Load all required packages upfront
• Set up your cluster with a sensible number of workers (more on that for your AMD setup below)
• Disable XGBoost's internal parallelism in each worker to avoid resource contention
• Use a combine function that builds an ordered list of results

Your 32-core/64-thread system is powerful, but oversubscribing it (spawning more workers than physical cores) leads to constant CPU context switching, which causes that frustrating lag. Here's how to fix it:

• Limit workers to physical cores: Use 32 workers (or 28-30, to leave some headroom for system processes) instead of 20—you're underutilizing your hardware right now, but overdoing it would make things worse.
• Disable XGBoost internal parallel: Set nthread = 1 in your xgb.train() call. Each foreach worker will use one physical core, so letting XGBoost spawn its own threads just creates unnecessary competition.
• Optimize data sharing: If your p object is large, use clusterExport() to send it to workers once instead of letting each worker copy it independently—this reduces memory overhead and speeds things up.

To get your loop results stored in order in a list, you need to use a combine function that appends each worker's output to a list. Your original .combine=combine is likely undefined, so let's fix that:

Full Working Code Example #

# Load all required packages
library(doParallel)
library(foreach)
library(xgboost)
library(data.table)
library(plyr)
library(Matrix)
library(dplyr)

# Set up cluster: Use physical cores to avoid oversubscription
num_workers <- 32  # Adjust to 28-30 if you still see lag
c1 <- makeCluster(num_workers)
registerDoParallel(c1)

# Export large objects to workers (only if they're not in the global env)
clusterExport(c1, varlist = c("p"))  # Add any other objects your loop needs here

# Custom combine function to build an ordered list
combine_to_list <- function(current_list, new_result) {
  c(current_list, list(new_result))
}

# Run the parallel loop
pg <- foreach(i = 1:100,
              .combine = combine_to_list,
              .init = list(),  # Start with an empty list
              .inorder = TRUE,  # Critical for keeping results in sequence
              .packages = c("data.table", "plyr", "Matrix", "dplyr", "xgboost")) %dopar% {
                # Your existing logic to extract indices
                temp <- unlist(p[i])
                index_1 <- temp[1]
                index_2 <- temp[2]  # Fill in the rest of your index logic here
                
                # Example XGBoost training with nthread=1
                # Replace this with your actual data preparation and training code
                dtrain <- xgb.DMatrix(data = your_feature_matrix[index_1:index_2, ], 
                                     label = your_labels[index_1:index_2])
                xgb_model <- xgb.train(
                  data = dtrain,
                  max_depth = 6,
                  eta = 0.1,
                  nthread = 1,  # No internal parallelism to avoid contention
                  nrounds = 100,
                  objective = "binary:logistic"  # Adjust to your task type
                )
                
                # Return the result you want to store in the list
                # This could be the model, predictions, metrics, etc.
                list(model = xgb_model, 
                     predictions = predict(xgb_model, dtrain),
                     iteration = i)
              }

# Clean up the cluster when done
stopCluster(c1)

Quick Tips for Further Optimization #

• Monitor CPU usage with htop or Windows Task Manager: If you see high context switching (red bars in htop), reduce the number of workers slightly.
• Use gc() inside the loop if you're dealing with large datasets: This frees up memory between iterations.
• Preprocess your data outside the loop: Avoid repeating data cleaning/transformation in each worker to save time.

内容的提问来源于stack exchange，提问作者igormr