Caret库KNN重复交叉验证结果异常，自定义Class库KNN结果稳定

Let’s break down the key differences between your caret implementation and custom class::knn() code that’s causing the divergent results, plus fix the critical issues in your caret script.

1. Critical Indexing & Vector Length Mismatch in Your Caret Code #

Your biggest problem is a broken mapping between k values and your accuracy tracking, caused by vector length mismatches and incorrect index calculations:

• You initialize accuracy_data with 10 zeros, but tuneLength = 20 tells caret to test 20 different k values (by default, caret’s KNN uses odd k values starting at 1: 1, 3, 5, ..., 39 for 20 values).
• When you run accuracy_data <- accuracy_data + knnFit$results$Accuracy, R uses vector recycling: it repeats your 10-element accuracy_data twice to match the 20-element accuracy vector. This corrupts your accumulated accuracy values, making which.max(accuracy_data) completely unreliable.
• Your calculation 3 + which.max(accuracy_data)*2 is wrong: the first k value tested is 1 (not 3), so you should directly pull the k value from knnFit$results$k instead of guessing.

2. Unstable Cross-Validation Setup in Caret #

Your caret code is prone to random fluctuations because:

• You use repeats = 1 in trainControl(method="repeatedcv"): this means only one round of cross-validation per training run, which has high variance. Small shuffles in your training data lead to big swings in accuracy for each k.
• You change random seeds every loop (set.seed(i*10) and set.seed(i*100)): while seeds are good for reproducibility, changing them every iteration means each train() run uses completely independent CV splits and data shuffles, amplifying variability in the optimal k.

In contrast, your custom class::knn() code:

• Averages accuracy across 20 repetitions of 10-fold CV per k value, which drastically reduces random noise.
• Tracks consistency in the optimal k across multiple outer loops, stopping only when it stabilizes—this filters out one-off fluctuations.

3. Fixed Caret Code That Converges #

Here’s a revised version of your caret script that aligns with your custom approach and produces stable results:

# Initialize accuracy storage for 20 k values (matches our explicit k sequence)
accuracy_data <- numeric(20)
current_op <- 0
count_same <- 0
col_label_name <- "diagnosis" # Replace with your actual label column name
fold_n <- 10

# Use a single base seed for consistent reproducibility
set.seed(123)

for (i in 1:50) {
  cat('\n Current repetition is', i)
  
  # Shuffle training data
  training_now <- training[sample(nrow(training)),]
  
  # Use repeated CV with more repeats to reduce variance
  ctrl <- trainControl(method="repeatedcv", repeats = 5, number = fold_n)
  
  formula <- as.formula(paste(col_label_name, ' ~ .' ))
  
  # Explicitly define k values to match your custom code (1, 3, ..., 39)
  knnFit <- train(
    formula, 
    data=training_now, 
    method = "knn", 
    trControl = ctrl, 
    preProcess = c("center","scale"), 
    tuneGrid = data.frame(k = seq(1, 39, 2)) # No ambiguity about k values
  )
  
  # Accumulate mean accuracy across CV runs
  accuracy_data <- accuracy_data + knnFit$results$Accuracy
  
  # Get the actual optimal k from the results (no manual calculation)
  current_k <- knnFit$results$k[which.max(accuracy_data)]
  
  cat('\n Current optimal k:', current_k, '\n')
  
  # Check for convergence
  if (current_op == current_k) {
    count_same <- count_same + 1
  } else {
    current_op <- current_k
    count_same <- 1 # Reset to 1, since we've observed this k once
  }
  
  if (count_same == 3) {
    cat('\n Converged after', i, 'repetitions. Optimal k:', current_op, '\n')
    break
  }
}

Key Fixes Explained: #

• Explicit tuneGrid: Defines exactly the k values to test, matching your custom code and eliminating ambiguity.
• Vector length match: accuracy_data starts as 20 elements, so no recycling errors when accumulating accuracy.
• More CV repeats: repeats=5 reduces variance in each train() run’s accuracy estimates.
• Correct k mapping: Pulls the optimal k directly from knnFit$results$k instead of guessing with manual math.
• Better convergence check: Resets count_same to 1 (not 0) when a new k is found, since we’ve already observed that k once.

Why Your Custom Code Converges #

Your class::knn() implementation stays consistent because:

• It averages accuracy across multiple CV runs for each k, smoothing out random noise.
• It uses a reliable mapping between loop index and k value (kvalue=2*j-1).
• It stops only when the optimal k is consistent across multiple outer loops, ensuring the result isn’t a one-off fluke.

With these fixes, your caret code should converge to the same optimal k as your custom implementation.

内容的提问来源于stack exchange，提问作者Hojin Jeong