R新手求助：基于mlr包实现svyglm的10折交叉验证

Hey there! As someone who's worked with complex survey models and mlr, I totally get where you're coming from—svyglm doesn't play nice with standard ML packages out of the box, but we can absolutely build a solution using mlr's resampling tools plus some custom code. Let's break this down into two approaches: a manual loop (great for understanding the mechanics) and building a custom mlr learner (the cleaner, more scalable ideal you mentioned).

First, we'll use makeResampleInstance to generate our 10-fold splits, then loop through each fold to fit svyglm, make predictions, and calculate our metrics.

Step 1: Load Required Packages #

We'll need the survey package for svyglm, mlr for resampling, and pROC to compute AUC:

library(survey)
library(mlr)
library(pROC)

Step 2: Prepare Your Survey Data & Design #

Let's use the built-in api dataset from the survey package as an example (swap this with your own data):

# Load sample survey data
data(api)

# Create the full survey design object (adjust id/weights/fpc to match your data)
dclus1 <- svydesign(id = ~dnum, weights = ~pw, data = apiclus1, fpc = ~fpc)

Step 3: Generate 10-Fold Resampling Indices #

Use mlr to create consistent train/test splits across all folds:

# Define 10-fold cross-validation
cv_desc <- makeResampleDesc("CV", iters = 10)

# Generate the resampling instance (this gives us train/test indices for each fold)
cv_inst <- makeResampleInstance(cv_desc, size = nrow(apiclus1))

Step 4: Loop Through Folds & Calculate Metrics #

We'll store misclassification errors and AUC values for each fold, then take the average:

# Initialize vectors to store results
misclass_errors <- numeric(10)
auc_values <- numeric(10)

# Loop through each fold
for (i in 1:10) {
  # Get train/test indices for the current fold
  train_idx <- cv_inst$train[[i]]
  test_idx <- cv_inst$test[[i]]
  
  # Create a subset survey design for the training data
  train_design <- subset(dclus1, subset = rownames(apiclus1) %in% rownames(apiclus1)[train_idx])
  
  # Fit the svyglm model (adjust your formula to match your target/predictors)
  svy_model <- svyglm(sch.wide ~ api00 + api99, design = train_design, family = binomial())
  
  # Extract test data and generate predictions (probabilities for classification)
  test_data <- apiclus1[test_idx, ]
  pred_probs <- predict(svy_model, newdata = test_data, type = "response")
  
  # Calculate misclassification error (using 0.5 as the default threshold)
  pred_class <- ifelse(pred_probs > 0.5, 1, 0)
  true_class <- test_data$sch.wide
  misclass_errors[i] <- mean(pred_class != true_class)
  
  # Calculate AUC using pROC
  roc_obj <- roc(true_class ~ pred_probs)
  auc_values[i] <- auc(roc_obj)
}

# Compute average metrics across all folds
mean_misclass <- mean(misclass_errors)
mean_auc <- mean(auc_values)

# Print results
cat(sprintf("Average Misclassification Error: %.3f\n", mean_misclass))
cat(sprintf("Average AUC: %.3f\n", mean_auc))

Building a custom mlr learner lets you leverage mlr's built-in resampling, metric calculation, and parallelization tools—no manual loops needed. Here's how to do it:

Step 1: Define Custom Train & Predict Functions #

We'll create functions that mlr can use to train svyglm models and generate predictions:

# Custom train function: Fits svyglm on the training subset
train_svyglm <- function(task, subset, weights = NULL, ...) {
  # Extract training data from the task
  train_data <- getTaskData(task, subset = subset)
  
  # Recreate the survey design for the training subset (adjust to your design specs)
  train_design <- svydesign(id = ~dnum, weights = ~pw, data = train_data, fpc = ~fpc)
  
  # Fit the svyglm model (update formula to match your task)
  svyglm(sch.wide ~ api00 + api99, design = train_design, family = binomial(), ...)
}

# Custom predict function: Generates class predictions and probabilities
predict_svyglm <- function(model, task, subset, ...) {
  # Extract test data from the task
  test_data <- getTaskData(task, subset = subset)
  
  # Generate predicted probabilities
  pred_probs <- predict(model, newdata = test_data, type = "response")
  
  # Return predictions in mlr's required format: class labels + probability matrix
  list(
    response = factor(ifelse(pred_probs > 0.5, "1", "0"), levels = getTaskClassLevels(task)),
    prob = cbind(`0` = 1 - pred_probs, `1` = pred_probs)
  )
}

Step 2: Create the Custom Learner #

Use makeLearner to wrap our functions into a mlr-compatible learner:

# Create a classification learner for svyglm
lrn_svyglm <- makeLearner(
  "classif.svyglm",
  train = train_svyglm,
  predict = predict_svyglm,
  predict.type = "prob",  # We need probabilities for AUC calculation
  type = "classif"
)

Step 3: Run Cross-Validation with mlr #

Now we can use mlr's resample function to handle the 10-fold CV automatically:

# Create a classification task from your data (adjust target to your variable)
svy_task <- makeClassifTask(data = apiclus1, target = "sch.wide")

# Define the metrics we want to calculate: MMCE (mean misclassification error) and AUC
metrics <- list(mmce, auc)

# Run 10-fold CV
cv_results <- resample(learner = lrn_svyglm, task = svy_task, resampling = cv_desc, measures = metrics)

# View aggregated results
print(cv_results$aggr)

• Survey Design Consistency: Always recreate the survey design for each training subset—don't reuse the full design, as this will skew your results.
• Threshold Adjustment: The 0.5 threshold for class predictions is arbitrary. You might want to optimize it based on your specific use case (e.g., using Youden's J statistic).
• Parallelization: mlr supports parallel resampling with parallelStart()—great for speeding up CV on large datasets.

内容的提问来源于stack exchange，提问作者Anuja Majmundar