Hey there! Let's work through your confusion matrix and cross-validation challenges together. I see you've already built a glmnet model and used ROCR to check its performance—great start! The issue with confusionMatrix() is almost certainly due to input format, so let's fix that first, then move on to structured k-fold cross-validation.

Fixing the Confusion Matrix with Kappa #

Why Your confusionMatrix() Failed #

Caret's confusionMatrix() requires categorical class labels (e.g., "0"/"1" or "Yes"/"No") as input, but you're currently passing predicted probabilities from predict(model, type = 'response'). That mismatch is why the function throws an error.

Step-by-Step Code to Generate the Confusion Matrix #

Let's convert your probability predictions to class labels, format your true labels correctly, and then generate the matrix with kappa:

# Load the caret package if you haven't already
library(caret)

# 1. Format your true test labels as a factor (caret requires this)
# Adjust levels to match your actual class values (e.g., c("negative", "positive") if needed)
true_test_labels <- factor(first.round[test.set], levels = c(0, 1))

# 2. Convert predicted probabilities to class labels
# Use a threshold (0.5 is standard, but you can adjust later)
predicted_test_probs <- training$sparse.fr.hat[test.set]
predicted_test_labels <- ifelse(predicted_test_probs > 0.5, 1, 0)
# Match the factor levels to your true labels
predicted_test_labels <- factor(predicted_test_labels, levels = c(0, 1))

# 3. Generate the confusion matrix (kappa is included automatically!)
cm <- confusionMatrix(predicted_test_labels, true_test_labels, positive = "1")
# Print the full results (includes accuracy, kappa, sensitivity, specificity, etc.)
print(cm)

Bonus: Optimize the Classification Threshold #

Instead of using the default 0.5 threshold, you can pick a threshold that maximizes model performance (e.g., using Youden's J statistic from your ROCR results):

# Calculate Youden's J statistic to find the optimal threshold
perf_youden <- performance(predictions, "sens", "spec")
youden_index <- which.max(perf_youden@y.values[[1]] + perf_youden@x.values[[1]] - 1)
optimal_threshold <- perf_youden@alpha.values[[1]][youden_index]

# Use this optimized threshold for predictions
predicted_test_labels_opt <- ifelse(predicted_test_probs > optimal_threshold, 1, 0)
predicted_test_labels_opt <- factor(predicted_test_labels_opt, levels = c(0, 1))

# Generate the optimized confusion matrix
cm_opt <- confusionMatrix(predicted_test_labels_opt, true_test_labels, positive = "1")
print(cm_opt)

K-Fold Cross-Validation with Caret #

While cv.glmnet() handles internal cross-validation for lambda tuning, using Caret's train() function gives you a unified framework to run k-fold CV, track metrics like kappa, and easily access results.

Step-by-Step Cross-Validation Code #

# 1. Set up cross-validation controls
train_control <- trainControl(
  method = "cv",          # Use k-fold cross-validation
  number = 10,            # 10 folds (adjust to your preference)
  classProbs = TRUE,      # Enable probability calculations
  summaryFunction = twoClassSummary # Track binary classification metrics (AUC, sensitivity, specificity)
)

# 2. Train the glmnet model with cross-validation
# Use the alpha value from your original model, and tune lambda (or use your existing lambda.min)
caret_glmnet_model <- train(
  x = sparesemx[train.set,],
  y = factor(first.round[train.set], levels = c(0, 1)),
  method = "glmnet",
  trControl = train_control,
  tuneGrid = expand.grid(alpha = 0.05, lambda = model$lambda.min), # Use your pre-tuned lambda
  family = "binomial"
)

# 3. View cross-validated metrics (includes kappa!)
print(caret_glmnet_model)

# 4. Generate a confusion matrix using the cross-validated model on your test set
cv_predicted_labels <- predict(caret_glmnet_model, newdata = sparesemx[test.set,])
cv_cm <- confusionMatrix(cv_predicted_labels, true_test_labels, positive = "1")
print(cv_cm)

内容的提问来源于stack exchange，提问作者Ellen O'Callaghan