使用caret训练随机森林时ROC指标全缺失的问题排查与解决

错误现象 #

训练随机森林模型时出现以下错误，所有ROC相关指标均为NA，且伴随大量警告：

Something is wrong; all the ROC metric values are missing:
      ROC           Sens          Spec    
 Min.   : NA   Min.   : NA   Min.   : NA  
 1st Qu.: NA   1st Qu.: NA   1st Qu.: NA  
 Median : NA   Median : NA   Median : NA  
 Mean   :NaN   Mean   :NaN   Mean   :NaN  
 3rd Qu.: NA   3rd Qu.: NA   3rd Qu.: NA  
 Max.   : NA   Max.   : NA   Max.   : NA  
 NA's   :10    NA's   :10    NA's   :10   
Error: Stopping
In addition: There were 50 or more warnings (use warnings() to see the first 50)

已知条件：

• 实际数据特征更多，但仍出现相同错误；
• 数据无NA或Inf值；
• 已尝试重装caret、pROC包及调整参数，无效。

复现代码 #

ctrlCV = trainControl(method = 'cv', number = 10 , classProbs = TRUE , savePredictions = TRUE, summaryFunction = twoClassSummary )
rfGRID <- expand.grid(.mtry = c(1 : 10))

rfFit <- train(response~., data = dimdum,
               method = "rf",
               metric="ROC",
               importance = TRUE,
               trControl = ctrlCV,
               tuneGrid = rfGRID,
               ntree = c(50,100,150,200,300,400,500),
               nodesize = c(1:10)
)

rfROC = roc(dimdum$response,predict(rfFit,dimdum, type='prob')[,1])
plot(rfROC)

示例数据 #

structure(list(response = c("NoResponse", "Response", "NoResponse", 
"NoResponse", "NoResponse", "Response", "NoResponse", "NoResponse", 
"NoResponse", "NoResponse", "NoResponse", "Response", "NoResponse", 
"Response", "NoResponse", "NoResponse", "NoResponse", "NoResponse", 
"Response", "Response", "NoResponse", "NoResponse", "NoResponse", 
"NoResponse", "NoResponse", "NoResponse", "NoResponse", "Response", 
"NoResponse", "NoResponse", "NoResponse", "NoResponse", "NoResponse", 
"NoResponse", "NoResponse", "Response", "NoResponse", "NoResponse", 
"NoResponse", "Response", "Response", "NoResponse", "Response", 
"Response", "NoResponse", "Response", "Response", "NoResponse", 
"Response", "Response"), CD4..Tem = c(0.206146305909711, 0.38344530718027, 
0.111171710498514, -0.024620418652091, -0.024620418652091, 0.241356282324198, 
0.190401963339481, 0.0381564797828905, 0.00208479038732372, -0.024620418652091, 
0.0345146685774692, 0.0047448955916752, 0.0954402244646442, 0.216170609750478, 
0.0679837609588422, 0.1087338604344, 0.0307783462567513, -0.024620418652091, 
0.00930882669937516, 0.228984175232275, 0.198029266287967, 0.00849741399216577, 
0.167122425878708, -0.0185389752646852, 0.0349285293854749, -0.015164453751509, 
0.0530002007752186, -0.00464766527016771, 0.229228539194469, 
-0.024620418652091, -0.024620418652091, 0.00909241866793368, 
0.00959645779130966, -0.0169517988930254, 0.130416251320013, 
-0.0226155780862924, -0.0226155780862924, -0.00708354014661853, 
-0.0226155780862924, 0.0236200668251617, -0.0226155780862924, 
-0.0226155780862924, -0.0226155780862924, -0.0226155780862924, 
-0.0226155780862924, -0.00226780446329141, -0.00703727203694584, 
-0.0226155780862924, 0.040447933249888, -0.0226155780862924), 
    Epithelial.cells = c(0.213818759771441, 0.224884228557244, 
    0.213818759771441, 0.564636116181376, 0.213818759771441, 
    0.213818759771441, 0.256571259511661, 0.213818759771441, 
    0.225551386999972, 0.213818759771441, 0.320498217450289, 
    0.213818759771441, 0.213818759771441, 0.213818759771441, 
    0.250752952186148, 0.54432086478806, 0.213818759771441, 0.213818759771441, 
    0.213818759771441, 0.224579338204213, 0.244604368723937, 
    0.239048638424405, 0.213818759771441, 0.213818759771441, 
    0.213818759771441, 0.213818759771441, 0.755246080444261, 
    0.213818759771441, 0.224789200187943, 0.213818759771441, 
    0.213818759771441, 0.213818759771441, 0.602551670320415, 
    0.221211285726714, 0.528154858032774, 0.191393513022707, 
    0.204473730554233, 0.199140565064947, 0.191393513022707, 
    0.191393513022707, 0.208364515830724, 0.199044034497245, 
    0.218231966624601, 0.191393513022707, 0.199037583564646, 
    0.233310726880044, 0.257245920265987, 0.2245306029313, 0.30356359401388, 
    0.202283902795669)), row.names = c("Pt1", "Pt101", "Pt106", 
"Pt11", "Pt17", "Pt18", "Pt24", "Pt26", "Pt27", "Pt28", "Pt29", 
"Pt3", "Pt31", "Pt34", "Pt36", "Pt37", "Pt38", "Pt39", "Pt44", 
"Pt49", "Pt5", "Pt52", "Pt59", "Pt62", "Pt65", "Pt66", "Pt67", 
"Pt72", "Pt77", "Pt78", "Pt84", "Pt85", "Pt89", "Pt9", "Pt90", 
"EA595454", "EA595500", "EA595522", "EA595529", "EA595597", "EA595624", 
"EA595635", "EA595647", "EA595654", "EA595719", "EA595720", "EA632133", 
"EA632171", "EA632174", "EA632234"), class = "data.frame")

原因分析 #

1. 参数传递错误：ntree和nodesize是randomForest包的模型参数，不能直接以向量形式传入train()函数。train()不会自动对这两个参数做网格搜索（除非手动加入tuneGrid），向量形式会导致模型在交叉验证过程中参数混乱，无法生成有效概率输出，进而导致ROC计算失败。
2. 交叉验证折叠中类别缺失：示例数据中类别分布不平衡（16个Response，34个NoResponse），10折交叉验证时部分折叠可能仅包含单一类别，twoClassSummary无法计算ROC、灵敏度等指标，直接返回NA。
3. 类别水平顺序不明确：twoClassSummary默认以响应变量因子的第二个水平为正类，若因子顺序未明确指定，可能因折叠内类别分布问题导致指标计算异常。

解决方法 #

1. 修正参数传递方式 #

• 若无需对ntree和nodesize做网格搜索，直接传入单一固定值（ntree建议设为较大值，如500，提升模型稳定性）：

ctrlCV = trainControl(method = 'cv', number = 10 , classProbs = TRUE , savePredictions = TRUE, summaryFunction = twoClassSummary )
rfGRID <- expand.grid(.mtry = c(1 : 10))

rfFit <- train(response~., data = dimdum,
               method = "rf",
               metric="ROC",
               importance = TRUE,
               trControl = ctrlCV,
               tuneGrid = rfGRID,
               ntree = 500,
               nodesize = 5
)

• 若需对这两个参数做网格搜索，需将其加入tuneGrid：

rfGRID <- expand.grid(.mtry = 1:10, .ntree = c(100, 300, 500), .nodesize = c(3,5,7))

2. 避免交叉验证折叠中类别缺失 #

• 使用分层交叉验证，确保每个折叠的类别分布与整体数据一致：

ctrlCV = trainControl(method = 'cv', number = 10 , classProbs = TRUE , savePredictions = TRUE, 
                      summaryFunction = twoClassSummary, stratify = TRUE)

• 若数据极度不平衡，可尝试减少交叉验证折数（如5折），或使用downSample/upSample处理数据：

library(caret)
dimdum_down <- downSample(x = dimdum[, -which(names(dimdum) == "response")],
                          y = dimdum$response)

3. 明确指定正类 #

自定义summary函数，明确指定类别水平与正类，避免因因子顺序导致的计算异常：

customTwoClassSummary <- function(data, lev = NULL, model = NULL) {
  lev <- c("NoResponse", "Response")  # 第二个水平为正类
  twoClassSummary(data, lev = lev, model = model)
}

ctrlCV = trainControl(method = 'cv', number = 10 , classProbs = TRUE , savePredictions = TRUE, 
                      summaryFunction = customTwoClassSummary, stratify = TRUE)

4. 验证模型概率输出 #

先训练单一随机森林模型，检查概率输出是否正常，排除特征与响应变量无相关性的可能：

library(randomForest)
test_rf <- randomForest(response~., data = dimdum, ntree=500, mtry=2)
prob <- predict(test_rf, type="prob")
head(prob)

若概率全为0或1，需重新审视特征的有效性。

内容的提问来源于stack exchange，提问作者Programming Noob