基于Random Forest的多分类任务：如何衡量结果的稳定性

Great question! When working with Random Forests— which have built-in randomness from bootstrap sampling and feature subset selection— checking model stability alongside raw performance is crucial, especially for ordinal classification tasks like yours (0 to 4, balanced classes). Let’s break down how to measure this stability and build an automated check for it:

1. Quantify Prediction Consistency (Tailored for Ordinal Classes) #

Since your categories are ordered, a small jump (e.g., 0 → 1) is far less problematic than a drastic shift (e.g., 0 → 4). Use these metrics to capture this nuance:

Weighted Kappa Coefficient #

This is the gold standard for ordinal classification consistency. It penalizes cross-category mismatches more heavily than adjacent ones, returning a score from -1 (complete disagreement) to 1 (perfect agreement). Use sklearn’s implementation with quadratic weighting (harsher penalties for large jumps):

from sklearn.metrics import cohen_kappa_score

# Compare two sets of predictions (y_pred1 from first run, y_pred2 from second)
quadratic_kappa = cohen_kappa_score(y_pred1, y_pred2, weights="quadratic")

• A score > 0.8 means predictions are highly consistent (mostly adjacent shifts)
• A score < 0.3 indicates severe, random shifts between non-adjacent classes

Consistency Confusion Matrix #

Visualize where prediction shifts happen to spot patterns. A good model will have most mismatches along the diagonal (perfect agreement) or adjacent cells:

import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix

cm = confusion_matrix(y_pred1, y_pred2)
sns.heatmap(cm, annot=True, fmt="d", cmap="Blues")
plt.xlabel("Second Run Predictions")
plt.ylabel("First Run Predictions")
plt.title("Prediction Consistency Matrix")
plt.show()

If you see large values in off-diagonal, non-adjacent cells (like 0 ↔ 4), your model is unstable.

2. Combine Stability with Performance #

Stability alone doesn’t mean much— you also need to ensure your model is accurate and its performance doesn’t swing wildly.

Track Performance Across Multiple Runs #

Train your model 10-20 times (with random_state=None each time) and record key metrics. Look for a high mean performance and low standard deviation (small swings):

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, mean_absolute_error

n_repeats = 20
acc_scores = []
mae_scores = []  # MAE works here since classes are ordered (measures average shift size)

for _ in range(n_repeats):
    rf = RandomForestClassifier(random_state=None)
    rf.fit(X_train, y_train)
    y_pred = rf.predict(X_test)
    acc_scores.append(accuracy_score(y_test, y_pred))
    mae_scores.append(mean_absolute_error(y_test, y_pred))

print(f"Average Accuracy: {np.mean(acc_scores):.4f} ± {np.std(acc_scores):.4f}")
print(f"Average Ordinal Error (MAE): {np.mean(mae_scores):.4f} ± {np.std(mae_scores):.4f}")

• A low standard deviation for accuracy/MAE means performance is consistent
• A high mean accuracy + low MAE means your model is both accurate and stable

Sample-Level Stability #

Check how individual test samples behave across runs. For each sample, calculate the standard deviation of its predicted classes— high std means the model can’t decide on that sample:

# Collect all predictions across runs
all_preds = []
for _ in range(n_repeats):
    rf = RandomForestClassifier(random_state=None)
    rf.fit(X_train, y_train)
    all_preds.append(rf.predict(X_test))

all_preds = np.array(all_preds)  # Shape: (n_repeats, n_test_samples)
sample_stds = all_preds.std(axis=0)

# Report sample stability
print(f"% of samples with consistent predictions (std ≤ 0.5): {np.mean(sample_stds <= 0.5)*100:.2f}%")
print(f"% of samples with highly inconsistent predictions (std ≥ 2): {np.mean(sample_stds >= 2)*100:.2f}%")

3. Automated Stability Check #

Wrap these metrics into a function to automatically assess your model’s stability. Adjust thresholds based on your tolerance for variability:

def assess_rf_stability(X_train, y_train, X_test, y_test, n_repeats=20):
    # Baseline prediction
    rf_base = RandomForestClassifier(random_state=None)
    rf_base.fit(X_train, y_train)
    y_pred_base = rf_base.predict(X_test)
    
    # Track metrics across runs
    kappa_scores = []
    acc_scores = [accuracy_score(y_test, y_pred_base)]
    
    for _ in range(n_repeats - 1):
        rf = RandomForestClassifier(random_state=None)
        rf.fit(X_train, y_train)
        y_pred = rf.predict(X_test)
        kappa_scores.append(cohen_kappa_score(y_pred_base, y_pred, weights="quadratic"))
        acc_scores.append(accuracy_score(y_test, y_pred))
    
    # Calculate summary stats
    avg_kappa = np.mean(kappa_scores)
    acc_std = np.std(acc_scores)
    
    # Print results and judge stability
    print(f"Average Weighted Kappa vs Baseline: {avg_kappa:.4f}")
    print(f"Accuracy Standard Deviation: {acc_std:.4f}")
    
    # Thresholds adjust based on your needs
    if avg_kappa >= 0.8 and acc_std <= 0.03:
        print("✅ Model is stable and performs consistently!")
        return True
    else:
        print("❌ Model has poor stability or inconsistent performance.")
        return False

# Run the check
assess_rf_stability(X_train, y_train, X_test, y_test)

Extra Tips #

• If you want reproducibility while testing randomness, use different fixed random_state values (e.g., random_state=i for each run in the loop) instead of None.
• For ordinal tasks, consider using metrics like cumulative gain curves or ordinal ROC AUC alongside accuracy to get a more complete picture of performance.

内容的提问来源于stack exchange，提问作者sqrt2sqrt2