多输入迁移学习模型生成GRAD-CAM可视化图报错求助

基于DenseNet121构建双输入迁移学习模型，输入为两个2D图像（由3D图像切片堆叠得到），模型结构如下：

base_model = load_model(model_dir) #pre-trained model is in model_dir
base_model.trainable = False

input1 = Input(shape=(image_size, image_size, 3), name='inp1')
x = data_augmentation(input1)
x = base_model(x, training=False)
x = GlobalAveragePooling2D()(x)
x1 = Dropout(0.5)(x)

input2 = Input(shape=(image_size, image_size, 3), name='inp2')
x = data_augmentation(input2)
x = base_model(x, training=False)
x = GlobalAveragePooling2D()(x)
x2 = Dropout(0.5)(x)

concatenated = layers.concatenate([x1, x2], axis=-1)
output = layers.Dense(1, activation='sigmoid')(concatenated)
model = Model([input1, input2], output)

尝试参考Keras官方示例绘制GRAD-CAM时，出现以下错误：

ValueError: Graph disconnected: cannot obtain value for tensor KerasTensor(type_spec=TensorSpec(shape=(None, 224, 224, 3), dtype=tf.float32, name='input_2'), name='input_2', description="created by layer 'input_2'") at layer "zero_padding2d_2". The following previous layers were accessed without issue: []

错误源于构建Grad-CAM模型的代码：

grad_model = keras.models.Model(
        model.inputs, [model.get_layer(last_conv_layer_name).output, model.output]
    )

修改为以下代码后仍报错：

grad_model = tf.keras.models.Model(
        [model.get_layer('inp1').input, model.get_layer('inp2').input], [model.get_layer('densenet121').get_layer('conv5_block16_2_conv').output, model.output]) 

问题核心在于同一个预训练DenseNet121被两个输入分支共享调用，在TensorFlow的计算图中，两个分支的base_model实例属于独立的计算节点。直接获取base_model的最后卷积层输出时，只会关联第一个输入分支的计算路径，导致第二个输入分支的计算图断开，无法追溯梯度。

需要为每个输入分支单独跟踪对应的最后卷积层输出，明确每个输入到卷积层的计算路径，再构建Grad-CAM模型。以下是两种可行的实现方式：

方式一：构建主模型时保留分支卷积层输出 #

修改主模型代码，在每个分支中显式保存最后卷积层的输出，后续直接基于这些输出构建Grad-CAM模型：

base_model = load_model(model_dir)
base_model.trainable = False
# 定位DenseNet121的最后卷积层
last_conv_layer = base_model.get_layer('conv5_block16_2_conv')
# 创建仅输出最后卷积层的base_model子模型
base_conv_model = Model(inputs=base_model.input, outputs=last_conv_layer.output)

# 第一个输入分支
input1 = Input(shape=(image_size, image_size, 3), name='inp1')
x1_aug = data_augmentation(input1)
conv_out1 = base_conv_model(x1_aug, training=False)
x1_gap = GlobalAveragePooling2D()(conv_out1)
x1_drop = Dropout(0.5)(x1_gap)

# 第二个输入分支
input2 = Input(shape=(image_size, image_size, 3), name='inp2')
x2_aug = data_augmentation(input2)
conv_out2 = base_conv_model(x2_aug, training=False)
x2_gap = GlobalAveragePooling2D()(conv_out2)
x2_drop = Dropout(0.5)(x2_gap)

# 融合分支并输出
concatenated = layers.concatenate([x1_drop, x2_drop], axis=-1)
output = layers.Dense(1, activation='sigmoid')(concatenated)

# 主模型包含最终预测和两个分支的卷积层输出
model = Model([input1, input2], [output, conv_out1, conv_out2])

# 构建Grad-CAM模型
grad_model = tf.keras.models.Model(
    inputs=[input1, input2],
    outputs=[conv_out1, conv_out2, output]
)

方式二：重新构建计算图跟踪分支路径 #

如果不想修改已训练好的主模型，可以通过函数式API重新复现每个分支的计算路径，明确关联输入与卷积层输出：

# 获取主模型的输入节点
inp1 = model.get_layer('inp1').input
inp2 = model.get_layer('inp2').input

# 复现第一个分支的计算，保留卷积层输出
x1 = data_augmentation(inp1)
base_out1 = base_model(x1, training=False)
conv_out1 = base_model.get_layer('conv5_block16_2_conv').output
x1_gap = GlobalAveragePooling2D()(base_out1)
x1_drop = Dropout(0.5)(x1_gap)

# 复现第二个分支的计算，保留卷积层输出
x2 = data_augmentation(inp2)
base_out2 = base_model(x2, training=False)
conv_out2 = base_model.get_layer('conv5_block16_2_conv').output
x2_gap = GlobalAveragePooling2D()(base_out2)
x2_drop = Dropout(0.5)(x2_gap)

# 复现融合和输出层
concatenated = layers.concatenate([x1_drop, x2_drop], axis=-1)
output = layers.Dense(1, activation='sigmoid')(concatenated)

# 构建Grad-CAM模型
grad_model = tf.keras.models.Model(
    inputs=[inp1, inp2],
    outputs=[conv_out1, conv_out2, output]
)

计算Grad-CAM热力图 #

构建好grad_model后，即可分别计算两个输入对应的热力图：

# 示例输入数据（形状需匹配模型输入）
img1 = ...  # (1, image_size, image_size, 3)
img2 = ...  # (1, image_size, image_size, 3)

with tf.GradientTape(persistent=True) as tape:
    tape.watch([inp1, inp2])
    conv_out1_val, conv_out2_val, preds = grad_model([img1, img2])
    # 以二分类sigmoid输出为例，取预测值作为损失
    loss = preds[:, 0]

# 计算每个分支的梯度
grads1 = tape.gradient(loss, conv_out1_val)
grads2 = tape.gradient(loss, conv_out2_val)
del tape

# 生成第一个输入的Grad-CAM
pooled_grads1 = tf.reduce_mean(grads1, axis=(0, 1, 2))
cam1 = tf.reduce_sum(tf.multiply(pooled_grads1, conv_out1_val[0]), axis=-1)
cam1 = tf.maximum(cam1, 0)  # 应用ReLU去除负权重
cam1 = tf.image.resize(cam1, (image_size, image_size)) / tf.reduce_max(cam1)

# 生成第二个输入的Grad-CAM
pooled_grads2 = tf.reduce_mean(grads2, axis=(0, 1, 2))
cam2 = tf.reduce_sum(tf.multiply(pooled_grads2, conv_out2_val[0]), axis=-1)
cam2 = tf.maximum(cam2, 0)
cam2 = tf.image.resize(cam2, (image_size, image_size)) / tf.reduce_max(cam2)

内容的提问来源于stack exchange，提问作者Dushi Fdz