如何在TensorFlow中实现循环学习率（替代AdamOptimizer或SGD）

Alright, let's walk through implementing Cyclic Learning Rate (CLR) exactly as you requested—no relying on AdamOptimizer or other pre-built SGD variants, using the core triangular LR mechanism centered around the get_triangular_lr function you referenced.

Step 1: Complete the Triangular LR Calculation Function #

First, let's finish the get_triangular_lr function you started—this computes the learning rate for each iteration based on the standard triangular cycle pattern:

import math

def get_triangular_lr(iteration, stepsize, base_lr, max_lr):
    """根据输入计算当前迭代应适用的学习率"""
    scale_fn = lambda x: 1/(2.**(x-1))
    cycle = math.floor(1 + iteration/(2 * stepsize))
    x = abs(iteration/stepsize - 2*cycle + 1)
    # Calculate the final triangular learning rate
    return base_lr + (max_lr - base_lr) * max(0, 1 - x) * scale_fn(cycle)

Quick breakdown of the logic:

• stepsize: Number of iterations for half a cycle (LR climbs from base to max, then falls back to base)
• scale_fn: Reduces the amplitude of each subsequent cycle (you can tweak this if you want a different decay pattern)
• cycle: Tracks which full cycle we're currently in
• x: Computes our position within the current cycle to determine the exact LR value

Step 2: Integrate CLR into TensorFlow Training #

Next, we'll wire this function into a TensorFlow training loop using the basic GradientDescentOptimizer (no Adam or other SGD variants). The key is to dynamically calculate the LR for each iteration using a global step counter.

Here's a complete example with a simple linear regression model to demonstrate:

import tensorflow as tf
import numpy as np

# ----------------------
# 1. Define Model & Loss
# ----------------------
x = tf.placeholder(tf.float32, shape=[None, 1])
y_true = tf.placeholder(tf.float32, shape=[None, 1])

# Simple linear model parameters
W = tf.Variable(tf.random_normal([1,1]))
b = tf.Variable(tf.random_normal([1]))
y_pred = tf.matmul(x, W) + b

# MSE loss function
loss = tf.reduce_mean(tf.square(y_pred - y_true))

# ----------------------
# 2. Set Up CLR Logic
# ----------------------
# Track current iteration count (non-trainable variable)
global_step = tf.Variable(0, trainable=False)

# CLR hyperparameters
base_lr = 0.001    # Minimum learning rate
max_lr = 0.01      # Maximum learning rate
stepsize = 200     # Iterations per half-cycle (LR goes up then down)

# Wrap Python LR function into TensorFlow graph node
current_lr = tf.py_function(
    func=lambda step: get_triangular_lr(step, stepsize, base_lr, max_lr),
    inp=[global_step],
    Tout=tf.float32
)

# ----------------------
# 3. Training Loop
# ----------------------
# Use basic Gradient Descent (no Adam or other SGD variants)
optimizer = tf.train.GradientDescentOptimizer(learning_rate=current_lr)
train_op = optimizer.minimize(loss, global_step=global_step)

# Launch training session
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    
    # Generate synthetic training data
    train_x = np.random.rand(1000, 1) * 10
    train_y = 3 * train_x + 2 + np.random.randn(1000, 1) * 0.5
    
    # Run training iterations
    for i in range(1000):
        batch_x = train_x[i:i+32]
        batch_y = train_y[i:i+32]
        _, current_lr_val, current_step = sess.run(
            [train_op, current_lr, global_step],
            feed_dict={x: batch_x, y_true: batch_y}
        )
        
        # Print LR every 50 iterations to verify the cycle
        if i % 50 == 0:
            print(f"Iteration {current_step}, Current LR: {current_lr_val:.6f}")

Key Notes #

• tf.py_function: This bridges our Python-based LR calculation into TensorFlow's computation graph, enabling dynamic LR updates for every iteration.
• Global Step: The global_step variable automatically increments with each minimize call, so we don't have to manually track iteration counts.
• Strictly Basic Optimizer: We're using GradientDescentOptimizer exclusively, which aligns with your requirement to avoid Adam or other specialized SGD variants.

内容的提问来源于stack exchange，提问作者DvdV