LightGBM Python API：自定义评估函数的best_iteration与best_score获取

Absolutely! You can absolutely retrieve the best iteration and corresponding score for your custom evaluation metric when using lightgbm.train for multiclass tasks. The key is to properly define your custom metric, configure LightGBM to use it for early stopping, and capture the evaluation results. Here's a detailed breakdown:

1. Correctly Define Your Custom Evaluation Metric #

For multiclass problems, LightGBM passes predictions as a 1D array (shape: n_samples * num_classes), so you'll need to reshape it first to calculate your metric. Your function must return a tuple of three values:

• Metric name (string)
• Metric value (float)
• is_higher_better (boolean): Tells LightGBM whether a higher value means better performance (e.g., accuracy = True, custom loss = False)

Example custom metric (multiclass accuracy):

import numpy as np

def custom_multiclass_acc(preds, train_data):
    labels = train_data.get_label()
    num_classes = len(np.unique(labels))
    # Reshape 1D preds to (n_samples, num_classes)
    preds_reshaped = preds.reshape(-1, num_classes)
    # Get predicted class labels
    pred_labels = np.argmax(preds_reshaped, axis=1)
    # Calculate accuracy
    acc = np.mean(pred_labels == labels)
    return "custom_acc", acc, True

2. Train the Model with Proper Configuration #

When calling lightgbm.train, you need to:

• Pass your custom metric via feval
• Specify your custom metric as the basis for early stopping using eval_metric
• Use evals_result to capture all evaluation results across iterations
• Enable early_stopping_rounds as usual

Full training example:

import lightgbm as lgb
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

# Generate sample multiclass data
X, y = make_classification(n_samples=1000, n_features=20, n_informative=10, n_classes=3, random_state=42)
X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.2, random_state=42)

# Create LightGBM datasets
train_set = lgb.Dataset(X_train, label=y_train)
valid_set = lgb.Dataset(X_valid, label=y_valid, reference=train_set)

# Model parameters
params = {
    "objective": "multiclass",
    "num_class": 3,
    "boosting_type": "gbdt",
    "num_leaves": 31,
    "learning_rate": 0.05,
    "verbose": 0
}

# Dictionary to store evaluation results
evals_result = {}

# Train the model
booster = lgb.train(
    params,
    train_set,
    num_boost_round=1000,
    valid_sets=[valid_set],
    valid_names=["valid"],
    feval=custom_multiclass_acc,
    early_stopping_rounds=50,
    evals_result=evals_result,
    verbose_eval=10,
    # Critical: Tell LightGBM to use your custom metric for early stopping
    eval_metric="custom_acc"
)

3. Retrieve Best Iteration & Custom Metric Score #

After training, you can access the best iteration directly from the booster object, and pull the corresponding custom metric score from evals_result:

# Get best iteration based on custom metric
best_iter = booster.best_iteration
print(f"Best iteration for custom_acc: {best_iter}")

# Get best custom metric score (note: evals_result uses 0-based indexing)
best_custom_score = evals_result["valid"]["custom_acc"][best_iter - 1]
print(f"Best custom_acc score: {best_custom_score:.4f}")

Key Notes #

• If you want to keep the default multi_logloss metric alongside your custom one, leave metric="multi_logloss" in the params. Just ensure eval_metric="custom_acc" is set to prioritize your metric for early stopping.
• Double-check the is_higher_better value: if your custom metric is a loss (lower = better), set this to False.
• The evals_result dictionary will contain all metrics tracked during training, so you can analyze trends for both default and custom metrics if needed.

内容的提问来源于stack exchange，提问作者arvelek