Scikit-learn自定义评分函数报错：求解Kaggle泰坦尼克号数据集问题遇阻

Hey there! Let’s dig into that custom scoring function error you hit while working on the Titanic dataset with Scikit-learn. I’ve run into these issues plenty of times, so let’s break down the most common pitfalls and how to fix them:

1. Your Function Has the Wrong Signature #

Scikit-learn expects custom scorers to accept three parameters: estimator, X (feature matrix), and y_true (ground truth labels). A lot of folks accidentally write functions that only take y_true and y_pred—which will throw an error when used with cross-validation or grid search.

Example of what not to do:

# ❌ Wrong: Missing estimator and X parameters
def bad_custom_score(y_true, y_pred):
    return accuracy_score(y_true, y_pred)

Fixed version:

from sklearn.metrics import accuracy_score

# ✅ Correct: Matches Scikit-learn's required signature
def good_custom_scorer(estimator, X, y_true):
    y_pred = estimator.predict(X)  # Let the estimator generate predictions
    return accuracy_score(y_true, y_pred)

2. You’re Returning a Loss Instead of a Score #

Scikit-learn defaults to expecting higher values = better performance. If your custom function calculates a loss (like mean squared error, where lower is better), you need to explicitly tell Scikit-learn this using make_scorer with greater_is_better=False.

Example:

from sklearn.metrics import mean_squared_error
from sklearn.metrics import make_scorer

# A function that calculates a loss (lower = better)
def custom_loss(estimator, X, y_true):
    y_pred = estimator.predict(X)
    return mean_squared_error(y_true, y_pred)

# ✅ Convert loss to a scorer with the correct direction
custom_scorer = make_scorer(custom_loss, greater_is_better=False)

3. You Forgot to Wrap Your Function with make_scorer #

Even if your function has the right signature, it’s still best practice to wrap it with make_scorer when passing it to tools like cross_val_score or GridSearchCV. This ensures Scikit-learn handles it correctly under the hood.

Example of the fix:

from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier

# ✅ Wrap your custom scorer
my_scorer = make_scorer(good_custom_scorer)

# Use it in cross-validation
scores = cross_val_score(RandomForestClassifier(), X, y, scoring=my_scorer, cv=5)

4. Data Type/Format Mismatches #

For the Titanic dataset (a classification problem), common issues here include:

• Using regression metrics (like MSE) on classification labels
• Returning probability values from predict_proba() instead of class labels from predict() when using classification metrics (like accuracy)

Example of a fix for this:

# ❌ Wrong: Using predict_proba() with accuracy_score
def wrong_scorer(estimator, X, y_true):
    y_pred = estimator.predict_proba(X)[:, 1]  # Returns probabilities, not 0/1 labels
    return accuracy_score(y_true, y_pred)  # Will throw an error

# ✅ Correct: Use predict() to get class labels
def correct_scorer(estimator, X, y_true):
    y_pred = estimator.predict(X)
    return accuracy_score(y_true, y_pred)

Let’s put this all together with a custom scorer that focuses on predicting survival for female passengers (a key pattern in the Titanic dataset):

import pandas as pd
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
from sklearn.metrics import make_scorer

# Load and prep Titanic data
data = pd.read_csv('titanic_train.csv')
X = data[['Pclass', 'Age', 'SibSp', 'Parch', 'Fare']].fillna(0)
y = data['Survived']
gender = data['Sex']  # We'll use this to filter female passengers

# Custom scorer: Accuracy only for female passengers
def female_survival_accuracy(estimator, X, y_true):
    y_pred = estimator.predict(X)
    # Filter to only female samples
    female_mask = gender == 'female'
    return accuracy_score(y_true[female_mask], y_pred[female_mask])

# Wrap and use the scorer
custom_scorer = make_scorer(female_survival_accuracy)
scores = cross_val_score(RandomForestClassifier(), X, y, scoring=custom_scorer, cv=5)

print(f"Female Survival Accuracy (CV Scores): {scores.round(3)}")
print(f"Mean Female Survival Accuracy: {scores.mean().round(3)}")

If you share the exact error message and your code snippet, I can help pinpoint the exact issue—but these fixes cover most common problems with custom scorers in Scikit-learn.

内容的提问来源于stack exchange，提问作者Sam16