融合双图像输入特征时遇形状不兼容错误（TensorFlow）

尝试将分别对应(299, 299, 3)和(224, 224, 3)尺寸的两个图像输入特征进行融合，训练时触发形状不兼容错误，以下是相关信息及解决方案：

模型代码 #

from tensorFlow.keras.applications.inception_v3 import InceptionV3
from tensorflow.keras.applications.vgg16 import VGG16
import tensorflow as tf
from tensorflow.keras import layers, Input

inp_pre_trained_model = InceptionV3( include_top=False)
inp_pre_trained_model.trainable=False
inp_input=tf.keras.Input(shape=(299,299,3),name="input_layer_inception_V3")
inp_x=inp_pre_trained_model (inp_input)
inp_x=layers.GlobalAveragePooling2D(name="global_average_pooling_layer_inception_v3")(inp_x)
vgg_pre_trained_model = VGG16( include_top=False)
vgg_pre_trained_model.trainable=False
vgg_input=tf.keras.Input(shape=(224,224,3),name="input_layer_VGG_16")

vgg_x=vgg_pre_trained_model(vgg_input)
vgg_x=layers.GlobalAveragePooling2D(name="global_average_pooling_layer_vgg_16")(vgg_x)
x=tf.keras.layers.concatenate([inp_x,vgg_x],axis=-1)
x = tf.keras.layers.Flatten()(x)
outputs=tf.keras.layers.Dense(5,activation="softmax", name= "output_layer") (x)
model=tf.keras.Model(inputs=[inp_input,vgg_input],outputs=outputs)

model.summary()

模型摘要 #

Model: "model_9"
__________________________________________________________________________________________________
 Layer (type)                   Output Shape         Param #     Connected to                      
===================================================================================================
 input_layer_inception_V3 (InputLayer)  [(None, 224, 224, 3)]  0          []                               
                                                                                                    
 input_layer_VGG_16 (InputLayer)        [(None, 299, 299, 3)]  0          []                               
                                                                                                    
 inception_v3 (Functional)      (None, None, None, 2048)   21802784    ['input_layer_inception_V3[0][0]']                                
                                                                                                    
 vgg16 (Functional)             (None, None, None, 512)    14714688    ['input_layer_VGG_16[0][0]']      
                                                                                                    
 global_average_pooling_inception (GlobalAveragePooling2D)  (None, 2048)        0           ['inception_v3[0][0]']            
                                                                                                    
 global_average_pooling_vgg (GlobalAveragePooling2D)        (None, 512)         0           ['vgg16[0][0]']                  
                                                                                                    
 concatenate_71 (Concatenate)   (None, 2560)         0           ['global_average_pooling_inception[0][0]',                        
                                                                  'global_average_pooling_vgg[0][0]']                                
                                                                                                    
 output_layer (Dense)           (None, 5)            12805       ['concatenate_71[0][0]']          
                                                                                                    
===================================================================================================
Total params: 36,530,277
Trainable params: 12,805
Non-trainable params: 36,517,472

编译配置与数据集信息 #

model.compile(loss="sparse_categorical_crossentropy",optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),metrics=["accuracy"])

train = tf.data.Dataset.zip((cache_train_data, ceced_train_data))  
test = tf.data.Dataset.zip((cache_test_data, ceced_test_data))  
train_dataset = train.prefetch(tf.data.AUTOTUNE)  
test_dataset = test.prefetch(tf.data.AUTOTUNE)

train_dataset, test_dataset

数据集element_spec：

(<PrefetchDataset element_spec=((TensorSpec(shape=(None, 224, 224, 3), dtype=tf.float32, name=None), TensorSpec(shape=(None, 5), dtype=tf.float32, name=None)), (TensorSpec(shape=(None, 224, 224, 3), dtype=tf.float32, name=None), TensorSpec(shape=(None, 5), dtype=tf.float32, name=None)))>,
<PrefetchDataset element_spec=((TensorSpec(shape=(None, 224, 224, 3), dtype=tf.float32, name=None), TensorSpec(shape=(None, 5), dtype=tf.float32, name=None)), (TensorSpec(shape=(None, 224, 224, 3), dtype=tf.float32, name=None), TensorSpec(shape=(None, 5), dtype=tf.float32, name=None)))>)

训练代码 #

model_history = model.fit(train_dataset, 
                              steps_per_epoch=len(train_dataset),
                              epochs=3,
                          validation_data=test_dataset,
                           validation_steps=len(test_dataset))

报错信息 #

ValueError: 用户代码中出现错误：
File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 1051, in train_function  *
    return step_function(self, iterator)
File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 1040, in step_function  **
    outputs = model.distribute_strategy.run(run_step, args=(data,))
File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 1030, in run_step  **
    outputs = model.train_step(data)
File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 889, in train_step
    y_pred = self(x, training=True)
File "/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
File "/usr/local/lib/python3.7/dist-packages/keras/engine/input_spec.py", line 264, in assert_input_compatibility
    raise ValueError(f'Input {input_index} of layer "{layer_name}" is '

ValueError: 层"model_9"的输入1与该层不兼容：期望形状=(None, 299, 299, 3)，实际形状=(None, 5)

问题原因与解决方案 #

问题根源 #

模型定义时期望接收两个图像输入：InceptionV3对应(299,299,3)的图像，VGG16对应(224,224,3)的图像；但当前数据集的结构是((图像1, 标签1), (图像2, 标签2))，训练时Keras会将这个结构拆解为[(图像1,标签1), (图像2,标签2)]作为模型输入，导致第二个输入被错误地传入了标签数据（形状为(None,5)），与模型期望的(None,299,299,3)不匹配。

同时模型摘要中存在输入层形状标注错误，InceptionV3的输入层实际应为(299,299,3)，VGG16输入层应为(224,224,3)，这是代码执行时的显示bug，但不影响逻辑。

解决方案 #

1. 调整数据集结构：将两个图像合并为输入列表，标签统一取其中一组（假设两个数据集的标签一致）：

# 重构训练数据集：输入为[inception用图像, vgg用图像]，标签取第一个数据集的标签
def format_dataset(data1, data2):
    # data1: (inception_img, label), data2: (vgg_img, label)
    return [data1[0], data2[0]], data1[1]

train_dataset = train.map(format_dataset).prefetch(tf.data.AUTOTUNE)
test_dataset = test.map(format_dataset).prefetch(tf.data.AUTOTUNE)

2. 匹配图像尺寸：确保cache_train_data中的图像是(299,299,3)尺寸（对应InceptionV3的输入要求），ceced_train_data中的图像是(224,224,3)尺寸（对应VGG16的输入要求），如果当前数据集图像尺寸不符，添加resize预处理：

# 对InceptionV3的输入图像进行resize
def resize_inception(img, label):
    img = tf.image.resize(img, (299, 299))
    return img, label

# 对VGG16的输入图像进行resize
def resize_vgg(img, label):
    img = tf.image.resize(img, (224, 224))
    return img, label

# 预处理数据集
cache_train_data = cache_train_data.map(resize_inception)
ceced_train_data = ceced_train_data.map(resize_vgg)

cache_test_data = cache_test_data.map(resize_inception)
ceced_test_data = ceced_test_data.map(resize_vgg)

调整后再执行训练代码即可解决形状不兼容问题。

内容的提问来源于stack exchange，提问作者Tamilselvi S