调整梯度提升分类自定义损失函数：兼顾误分类与额外惩罚

Hey Sarah, let's work through how to adjust your gradient boosting loss function to handle both misclassification penalties and your precomputed extra penalty values. Here's a step-by-step solution with modified code and explanations:

Key Idea #

We'll amplify the loss (and corresponding gradients/hessians) for misclassified samples by incorporating your precomputed penalties. This ensures the model prioritizes correcting these samples more heavily than correctly classified ones, while adding your custom penalty values directly to the loss calculation.

Modified Code with Penalty Integration #

First, we'll use closures to pass your precomputed penalty array cleanly (no messy global variables). We'll adjust both the objective function (for training) and validation function (for evaluation):

import numpy as np
from sklearn.preprocessing import OneHotEncoder

def softmax(mat):
    res = np.exp(mat)
    res = res / np.sum(res, axis=1, keepdims=True)
    return res

# Closure to create a custom objective with precomputed penalties
def create_custom_objective(extra_penalties):
    def custom_asymmetric_objective(y_true, y_pred_encoded):
        # Reshape predictions and compute softmax probabilities
        pred = y_pred_encoded.reshape((-1, 3), order='F')
        pred_probs = softmax(pred)
        
        # One-hot encode true labels (optimize: fit OHE once outside this function!)
        ohe = OneHotEncoder(sparse=False, categories='auto')
        y_true_ohe = ohe.fit_transform(y_true.reshape(-1, 1))
        
        # Identify misclassified samples
        pred_classes = np.argmax(pred_probs, axis=1)
        true_classes = np.argmax(y_true_ohe, axis=1)
        misclassified_mask = pred_classes != true_classes
        
        # Build penalty weights: scale misclassified samples by (1 + extra penalty)
        penalty_weights = np.ones(len(y_true), dtype=np.float32)
        penalty_weights[misclassified_mask] = 1 + extra_penalties[misclassified_mask]
        
        # Expand weights to match the shape of gradients/hessians (per-class)
        penalty_weights_expanded = penalty_weights[:, np.newaxis]
        
        # Compute base gradients and hessians (original cross-entropy logic)
        grad = (pred_probs - y_true_ohe).astype(np.float32)
        hess = 2.0 * pred_probs * (1.0 - pred_probs)
        
        # Apply penalty weights to gradients and hessians
        grad *= penalty_weights_expanded
        hess *= penalty_weights_expanded
        
        return grad.flatten('F'), hess.flatten('F')
    return custom_asymmetric_objective

# Closure to create a matching validation function
def create_custom_valid(extra_penalties):
    def custom_asymmetric_valid(y_true, y_pred_encoded):
        pred = y_pred_encoded.reshape((-1, 3), order='F')
        pred_probs = softmax(pred)
        
        ohe = OneHotEncoder(sparse=False, categories='auto')
        y_true_ohe = ohe.fit_transform(y_true.reshape(-1, 1))
        
        # Identify misclassified samples
        pred_classes = np.argmax(pred_probs, axis=1)
        true_classes = np.argmax(y_true_ohe, axis=1)
        misclassified_mask = pred_classes != true_classes
        
        # Original validation loss logic
        y_true_flat = y_true_ohe.flatten('F')
        margin = (y_true_flat - y_pred_encoded).astype(np.float32)
        base_loss = margin * 10
        
        # Add extra penalty to misclassified samples
        penalty_loss = np.zeros_like(base_loss)
        # Map sample-level penalties to the flattened loss array
        for idx in np.where(misclassified_mask)[0]:
            penalty_loss[idx*3 : idx*3+3] = extra_penalties[idx]
        
        total_loss = base_loss + penalty_loss
        
        return "custom_asymmetric_eval", np.mean(total_loss), False
    return custom_asymmetric_valid

How to Use This #

1. Prepare your precomputed penalty array (one value per sample, e.g., 0.05 for all or varying values):

   # Example: 100 samples, all with an extra penalty of 0.05
   extra_penalties = np.full(100, 0.05)
   

2. Create your custom objective and evaluation functions:

   custom_obj = create_custom_objective(extra_penalties)
   custom_eval = create_custom_valid(extra_penalties)
   

3. Plug these into your gradient boosting framework (e.g., LightGBM/XGBoost):

   import lightgbm as lgb
   
   train_data = lgb.Dataset(X_train, y_train)
   valid_data = lgb.Dataset(X_valid, y_valid, reference=train_data)
   
   params = {
       'objective': custom_obj,
       'metric': 'None',  # Disable default metrics, use our custom eval
       'num_leaves': 31,
       'learning_rate': 0.05
   }
   
   model = lgb.train(
       params,
       train_data,
       num_boost_round=100,
       valid_sets=[valid_data],
       feval=custom_eval
   )
   

Important Notes #

• Optimize OHE: For efficiency, fit the OneHotEncoder once on your training labels outside the objective/validation functions, then just call transform inside. This avoids redundant fitting on every iteration.
• Adjust Penalty Logic: If you want to apply penalties to specific classes instead of all misclassified samples, modify the misclassified_mask to check for specific class mismatches (e.g., (true_classes == 2) & (pred_classes != 2)).
• Penalty Type: We scaled gradients/hessians by (1 + penalty) to amplify the model's focus on misclassified samples. If you prefer to add the penalty directly to the loss instead of scaling, adjust the gradient calculation to include the penalty as an additive term.

内容的提问来源于stack exchange，提问作者Sarah