能否为使用LBFGS求解器的MLPRegressor生成损失曲线？

Great question! You’re right that scikit-learn’s MLPRegressor only natively tracks loss curves for the adam solver by default. But with a couple of custom workarounds, we absolutely can generate a loss curve for the lbfgs solver—let’s walk through how to do it.

Method 1: Custom Callback Function #

The cleanest approach is to use a custom callback that captures the training loss after each LBFGS optimization step. Scikit-learn’s MLPRegressor accepts a callback parameter that triggers functions after every iteration, which we can leverage to record losses.

Here’s a complete implementation:

from sklearn.neural_network import MLPRegressor
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

# Custom callback to track training loss at each iteration
class LossTracker:
    def __init__(self):
        self.training_loss = []
    
    def __call__(self, model):
        # Use the model's internal loss function to compute current training loss
        current_loss = model.loss_(model.coefs_, model.intercepts_, model.X_, model.y_)
        self.training_loss.append(current_loss)

# Generate sample regression data
X, y = make_regression(n_samples=1000, n_features=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

# Initialize the loss tracker callback
loss_tracker = LossTracker()

# Set up the MLPRegressor with LBFGS solver
mlp = MLPRegressor(
    solver='lbfgs',
    hidden_layer_sizes=(64, 32),
    max_iter=100,
    random_state=42,
    callback=[loss_tracker]  # Pass our callback to the model
)

# Train the model
mlp.fit(X_train, y_train)

# Plot the loss curve
plt.figure(figsize=(10, 6))
plt.plot(loss_tracker.training_loss, label='Training Loss')
plt.xlabel('LBFGS Optimization Steps')
plt.ylabel('Mean Squared Error Loss')
plt.title('Loss Curve for MLPRegressor (LBFGS Solver)')
plt.legend()
plt.show()

How this works: #

• The LossTracker class uses the model’s internal loss_ method to calculate the training loss after each LBFGS iteration.
• The callback is passed to the MLPRegressor during initialization, so it runs automatically during training.
• After training, we plot the collected losses to visualize how the model’s performance improves over time.

Method 2: Warm Start Iterative Training #

If you prefer a more explicit approach, you can use warm_start=True to train the model in small increments (1 iteration at a time) and record the loss after each step. This mimics the iterative training flow of Adam while working with LBFGS.

Here’s the code:

# Initialize MLPRegressor with warm_start enabled
mlp_warm = MLPRegressor(
    solver='lbfgs',
    hidden_layer_sizes=(64, 32),
    max_iter=1,  # Train 1 iteration per fit call
    warm_start=True,  # Preserve model weights between fits
    random_state=42
)

warm_start_losses = []
max_total_iterations = 100
loss_tolerance = 1e-4  # Stop early if loss stops improving
previous_loss = float('inf')

for iteration in range(max_total_iterations):
    mlp_warm.fit(X_train, y_train)
    # Calculate current training loss
    current_loss = mlp_warm.loss_(mlp_warm.coefs_, mlp_warm.intercepts_, mlp_warm.X_, mlp_warm.y_)
    warm_start_losses.append(current_loss)
    
    # Early stopping to avoid unnecessary iterations
    if abs(previous_loss - current_loss) < loss_tolerance:
        print(f"Early stopping at iteration {iteration + 1}")
        break
    previous_loss = current_loss

# Plot the loss curve
plt.figure(figsize=(10, 6))
plt.plot(warm_start_losses, label='Training Loss')
plt.xlabel('Iterations')
plt.ylabel('Mean Squared Error Loss')
plt.title('Loss Curve for MLPRegressor (LBFGS) with Warm Start')
plt.legend()
plt.show()

Key Notes #

• Validation Loss: To track validation loss instead of training loss, modify the code to compute loss on the test set (e.g., using mean_squared_error(y_test, model.predict(X_test))).
• Iteration Differences: LBFGS iterations are full-dataset optimization steps, unlike Adam’s mini-batch iterations. Your loss curve will reflect progress over complete passes of the training data.
• Efficiency: The callback method is more efficient, as it avoids reinitializing the model multiple times like the warm start approach.

内容的提问来源于stack exchange，提问作者Krishna Moorthy