TensorFlow中CNN绘制学习率-准确率曲线及Cyclic LR实现问题

Hey there! Let's troubleshoot why your Cyclic LR learning rate vs accuracy plot is coming up empty or returning empty lists. I’ve run into similar headaches with TensorFlow before, so let’s break this down step by step.

1. First: Verify You’re Actually Collecting Data Correctly #

Empty lists usually mean your code isn’t appending acc and lr values during training. Let’s fix that depending on whether you’re using a custom training loop or model.fit():

Custom Training Loop Example

If you’re writing your own training loop, you need to explicitly calculate accuracy and grab the current learning rate at each step:

# Assuming you've already initialized acc_list = [], lr_list = []
for epoch in range(epochs):
    for step, (x_batch, y_batch) in enumerate(train_dataset):
        with tf.GradientTape() as tape:
            y_pred = model(x_batch, training=True)
            loss = loss_fn(y_batch, y_pred)
        
        # Update weights
        grads = tape.gradient(loss, model.trainable_variables)
        optimizer.apply_gradients(zip(grads, model.trainable_variables))
        
        # Calculate batch accuracy
        batch_acc = tf.reduce_mean(tf.cast(
            tf.equal(tf.argmax(y_pred, axis=1), tf.argmax(y_batch, axis=1)),
            tf.float32
        ))
        
        # Get current learning rate (critical for TensorFlow 2.x)
        # For most optimizers, use this to get the decayed/active LR:
        current_lr = optimizer._decayed_lr(tf.float32).numpy()
        
        # Append to your lists
        acc_list.append(batch_acc.numpy())
        lr_list.append(current_lr)

Using model.fit() with a Custom Callback

If you’re using Keras’ built-in fit() method, create a callback to log values at each batch:

class LRAccLogger(tf.keras.callbacks.Callback):
    def on_train_batch_end(self, batch, logs=None):
        # Grab current accuracy from logs
        batch_acc = logs.get('accuracy')
        # Grab current learning rate
        current_lr = self.model.optimizer._decayed_lr(tf.float32).numpy()
        
        if batch_acc is not None:  # Avoid None values breaking your list
            acc_list.append(batch_acc)
            lr_list.append(current_lr)

# Initialize Cyclic LR (make sure this is set up correctly!)
cyclic_lr = tf.keras.callbacks.CyclicLR(
    base_lr=1e-5,
    max_lr=1e-2,
    step_size_up=200,  # Adjust based on your training batch count
    mode='triangular',
    verbose=1  # Enable this to see LR updates in the console
)

# Train with both callbacks
model.fit(
    train_dataset,
    epochs=epochs,
    callbacks=[LRAccLogger(), cyclic_lr]
)

2. Check Your Cyclic LR Configuration #

If your lists are still empty or have duplicate values, your Cyclic LR setup might be broken:

• Verify you’re using the right Cyclic LR implementation: TensorFlow doesn’t have a native CyclicLR callback in all versions. If you’re using tensorflow-addons, import it properly:

  import tensorflow_addons as tfa
  # As an optimizer wrapper (common approach)
  optimizer = tfa.optimizers.CyclicLR(
      base_lr=1e-5,
      max_lr=1e-2,
      step_size=200,
      optimizer='adam'
  )
  

• Set verbose=1: This will print learning rate updates during training—if you don’t see these, the Cyclic LR isn’t active.
• Adjust step_size_up: This should roughly be 2-8x the number of batches per epoch. For example, if you have 100 batches per epoch, set step_size_up=200 to complete one full LR cycle every 2 epochs.

3. Debug Before Plotting #

Before you even try to plot, confirm your lists have data:

print(f"Accuracy list length: {len(acc_list)}")
print(f"LR list length: {len(lr_list)}")

• If lengths are 0: Your collection code isn’t executing (e.g., training loop didn’t run, callback wasn’t registered).
• If lengths are mismatched: You’re appending one value without the other (e.g., skipping accuracy when it’s None).

4. Fix Your Plotting Code #

If your lists have data but the plot is empty, you’re probably missing a log scale for the x-axis (learning rates span multiple orders of magnitude!):

import matplotlib.pyplot as plt

plt.figure(figsize=(10, 6))
plt.plot(lr_list, acc_list, marker='.', linestyle='-', color='blue')
plt.xscale('log')  # Critical for seeing LR changes clearly
plt.xlabel('Learning Rate (Log Scale)')
plt.ylabel('Batch Accuracy')
plt.title('Cyclic LR: Learning Rate vs Accuracy')
plt.grid(True, alpha=0.3)
plt.show()

Without the log scale, your LR values might cluster so tightly that the plot looks empty.

Quick TensorFlow Syntax Notes #

• Getting the current LR: Avoid using optimizer.lr.numpy() directly if you’re using any learning rate scheduling (like Cyclic LR). Use optimizer._decayed_lr(tf.float32).numpy() to get the actual LR being applied to weights.
• Avoid None values: Always check that logs.get('accuracy') isn’t None before appending—some steps might not calculate accuracy by default.

内容的提问来源于stack exchange，提问作者DvdV