TensorFlow强化学习报错：ValueError: 未提供任何变量的梯度

运行探索游戏智能体代码时触发梯度缺失报错：

ValueError: No gradients provided for any variable: (['conv2d/kernel:0', 'conv2d/bias:0', 'conv2d_1/kernel:0', 'conv2d_1/bias:0', 'conv2d_2/kernel:0', 'conv2d_2/bias:0', 'dense/kernel:0', 'dense/bias:0', 'dense_1/kernel:0', 'dense_1/bias:0', 'dense_2/kernel:0', 'dense_2/bias:0', 'dense_3/kernel:0', 'dense_3/bias:0'],). Provided grads_and_vars is ((None, <tf.Variable 'conv2d/kernel:0' shape=(3, 3, 3, 16) dtype=float32, numpy= ...

问题核心：损失计算环节中，get_novelty和translate函数完全脱离TensorFlow计算图，导致损失与模型参数间无梯度关联，无法完成梯度反向传播。

修复方案 #

关键修改方向 #

1. 将所有新奇度计算逻辑迁移到TensorFlow计算图内，使用TF原生操作替代PIL/numpy代码
2. 用TF函数重写损失映射逻辑，保证梯度可传导
3. 调整训练流程，让模型预测操作被梯度磁带追踪，建立损失与模型参数的关联

修改后完整代码 #

import vgamepad as vg
import time
import numpy as np
import tensorflow as tf
import pyautogui

gamepad = vg.VX360Gamepad()
images = []

def updatecontrols(inputs):
    cam = (inputs[0], inputs[1])
    walk = (inputs[2], inputs[3])
    gamepad.right_joystick_float(x_value_float=cam[0], y_value_float=cam[1])
    gamepad.left_joystick_float(x_value_float=walk[0], y_value_float=walk[1])
    gamepad.update()

size = 256
model = tf.keras.models.Sequential([
    tf.keras.layers.Rescaling(1./255, input_shape=(size, size, 3)),
    tf.keras.layers.Conv2D(16, 3, padding='same', activation='relu'),
    tf.keras.layers.MaxPooling2D(),
    tf.keras.layers.Conv2D(32, 3, padding='same', activation='relu'),
    tf.keras.layers.MaxPooling2D(),
    tf.keras.layers.Conv2D(64, 3, padding='same', activation='relu'),
    tf.keras.layers.MaxPooling2D(),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(128),
    tf.keras.layers.Dense(128),
    tf.keras.layers.Dense(128),
    tf.keras.layers.Dense(4),
])

dmax = tf.constant(10000000, dtype=tf.float32)

@tf.function
def translate(value, leftMin, leftMax, rightMin, rightMax):
    leftSpan = leftMax - leftMin
    rightSpan = rightMax - rightMin
    valueScaled = (value - leftMin) / leftSpan
    return rightMin + (valueScaled * rightSpan)

@tf.function
def get_novelty(image1_tensor, images_tensor):
    if tf.shape(images_tensor)[0] == 0:
        return tf.constant(0.0)
    # TF原生灰度转换
    image1_gray = tf.image.rgb_to_grayscale(image1_tensor)
    images_gray = tf.image.rgb_to_grayscale(images_tensor)
    # 计算批量图像差异总和的均值
    diffs = tf.reduce_sum(tf.abs(image1_gray - images_gray), axis=[1,2,3])
    return tf.reduce_mean(diffs)

optimizer = tf.keras.optimizers.SGD()

epochs = 2000
for epoch in range(epochs):
    time.sleep(0.1)
    # 截图并转换为TF张量
    image = pyautogui.screenshot().resize((size, size))
    image_array = np.expand_dims(np.asarray(image), axis=0)
    image_tensor = tf.convert_to_tensor(image_array, dtype=tf.float32)
    
    # 更新历史图像列表（存储为TF张量）
    images.append(image_tensor)
    images = images[-50:]
    images_batch = tf.concat(images, axis=0)
    
    with tf.GradientTape() as tape:
        # 模型预测放入磁带上下文，开启训练模式追踪梯度
        outputs = model(image_tensor, training=True)
        updatecontrols(outputs[0].numpy())
        # 计算新奇度与损失
        nov = get_novelty(image_tensor, images_batch)
        loss = translate(nov, 0.0, dmax, 1.0, 0.0)
    
    # 计算梯度并更新模型参数
    grads = tape.gradient(loss, model.trainable_weights)
    optimizer.apply_gradients(zip(grads, model.trainable_weights))
    
    # 可选：打印训练状态
    if epoch % 50 == 0:
        print(f"Epoch {epoch}, Novelty: {nov.numpy():.2f}, Loss: {loss.numpy():.4f}")

修改说明 #

• 计算图绑定：用@tf.function装饰get_novelty和translate，所有操作改用TF API，确保梯度可被磁带追踪
• 张量化存储：历史图像以TF张量形式存储，避免跨框架类型转换中断梯度链
• 预测入磁带：将模型预测操作放在梯度磁带上下文内，让模型输出与损失建立关联，梯度能够反向传播到模型参数
• 性能优化：TF原生批量操作替代循环遍历，提升计算效率

内容的提问来源于stack exchange，提问作者Takeraparterer