多轮迭代下最优性能模型的选择及代码优化问询

Great question! Your current approach to training multiple model runs and tracking performance is a solid starting point, but we can tweak it to be more efficient—especially when it comes to memory usage and directly isolating your best-performing model. Let's break down the improvements step by step.

Key Optimizations #

Instead of storing every trained model in a list (which can eat up a lot of memory if your model is large), we can track only the best model as we iterate. This way, you don't have to hold all 10 models in memory at once, and you'll have your optimal model ready as soon as the loop finishes.

We'll base the "best" model on your validation metrics—either the lowest MSE (test loss) or highest R² score (whichever aligns better with your goals). Let's use both metrics for clarity, but you can prioritize whichever matters most to you.

Revised Code Implementation #

Here's how to adjust your existing code to implement this:

import numpy as np
from sklearn.metrics import r2_score
from tensorflow.keras.layers import Input, Dense, BatchNormalization, Concatenate
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import Adam

def build_model(input1_shape, input2_shape):
    """ Creates the a multi-channel ANN, capable of accepting multiple inputs.
    :param input1_shape: Shape of input1 (excluding batch dimension)
    :param input2_shape: Shape of input2 (excluding batch dimension)
    :return: Compiled ANN model with a single output
    """
    # Define Inputs for ANN (using passed shapes instead of modifying input data here)
    input1 = Input(shape=input1_shape, name="input1")
    input2 = Input(shape=input2_shape, name="input2")
    
    # First Branch of ANN (Weight)
    x = Dense(units=1, activation="relu")(input1)
    x = BatchNormalization()(x)
    
    # Second Branch of ANN (Word Embeddings)
    y = Dense(units=36, activation="relu")(input2)
    y = BatchNormalization()(y)
    
    # Merge branches
    combined = Concatenate()([x, y])
    
    # Final Output Layer
    outputs = Dense(1, name="output")(combined)
    
    # Create and compile model
    model = Model(inputs=[input1, input2], outputs=outputs)
    model.compile(loss='mse', optimizer=Adam(lr=0.01), metrics=['mse'])
    
    # Optional: Uncomment if you want to see the model summary once, not every iteration
    # model.summary()
    return model

# Initialize tracking variables for the best model
best_mse = float('inf')  # Start with a very high value (we want to minimize MSE)
best_r2 = -float('inf')  # Start with a very low value (we want to maximize R²)
best_model = None

test_outcomes = []  # Keep if you want to analyze all run metrics later
r2_outcomes = []

# Get input shapes once (instead of recalculating in each loop)
input1_shape = x_train['input1'].shape[1:]
input2_shape = x_train['input2'].shape[1:]

for i in range(10):
    print(f"--- Training Run {i+1}/10 ---")
    # Build a fresh model for each run
    model = build_model(input1_shape, input2_shape)
    
    # Train the model
    print("Training model...")
    model.fit(
        [x_train['input1'], x_train['input2']], 
        y_train, 
        batch_size=25, 
        epochs=60, 
        verbose=0,
        validation_data=([x_valid['input1'], x_valid['input2']], y_valid)
    )
    
    # Evaluate on validation set
    print("Evaluating model...")
    score = model.evaluate([x_valid['input1'], x_valid['input2']], y_valid, verbose=0)
    test_loss = round(score[0], 3)
    test_outcomes.append(test_loss)
    
    # Calculate R² score
    y_pred = model.predict([x_valid['input1'], x_valid['input2']], verbose=0).flatten()
    r_squared = r2_score(y_valid, y_pred)
    r2_outcomes.append(r_squared)
    
    # Print current run metrics
    print(f"Run {i+1}: Validation MSE = {test_loss}, R² = {round(r_squared, 3)}")
    
    # Update best model if current run is better
    # You can prioritize either MSE or R² here—adjust the condition as needed
    if test_loss < best_mse or (test_loss == best_mse and r_squared > best_r2):
        print(f"New best model found! Updating...")
        best_mse = test_loss
        best_r2 = r_squared
        best_model = model  # Save the entire model, or just weights with model.save_weights()

# After all runs, analyze results
mean_test = np.mean(test_outcomes)
r2_means = np.mean(r2_outcomes)

print("\n--- Final Results ---")
print(f"Average Validation MSE across runs: {round(mean_test, 3)}")
print(f"Average R² across runs: {round(r2_means, 3)}")
print(f"Best Model Validation MSE: {best_mse}, R²: {round(best_r2, 3)}")

What Changed & Why #

1. Separate Input Shape Calculation: We now pass input shapes to build_model instead of modifying the input data inside the function. This is cleaner and avoids unintended side effects with your training data.
2. Track Only Best Model: Instead of storing all 10 models, we keep updating best_model whenever we find a run with better performance. This saves significant memory, especially for larger models.
3. Metric Comparison Logic: The condition checks if the current run has a lower MSE (or same MSE but higher R²) to decide if it's the new best. You can adjust this to prioritize R² directly if that's your main metric.
4. Reduced Verbosity: We set verbose=0 for predict and evaluate to clean up output, but you can toggle this back if you want more detail.
5. Optional: Save Weights Instead of Entire Model: If memory is still a concern, replace best_model = model with best_model.save_weights('best_model_weights.h5'), then load it later with model.load_weights('best_model_weights.h5') after building the model architecture again.

Extra Tips #

• Add Early Stopping: To prevent overfitting during training, add EarlyStopping to your model.fit() call:

  from tensorflow.keras.callbacks import EarlyStopping
  early_stop = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)
  # Then add callbacks=[early_stop] to model.fit()
  

  This will stop training early if validation loss doesn't improve, and restore the weights from the best epoch.
• Seed Randomness: For more reproducible results, set random seeds for NumPy, TensorFlow, and Python at the start of your script:

  import random
  random.seed(42)
  np.random.seed(42)
  tf.random.set_seed(42)
  

  This ensures each training run starts with the same initial weights, making performance differences more meaningful.

内容的提问来源于stack exchange，提问作者machine_apprentice