如何解决Keras中两个CNN模型拼接时的形状不匹配问题？

问题原因 #

膨胀卷积（dilation_rate）会扩大卷积核的有效感受野，导致model2的特征图尺寸逐步缩小的幅度和model1不一致，最终输出(None,4,4,60)，而model1输出(None,5,5,60)，无法直接拼接。

可行解决方案 #

方案1：通过填充/上采样对齐特征图尺寸 #

在model2的最后添加一层，将4x4的特征图调整为5x5，匹配model1的尺寸，同时保留膨胀卷积的特性：

方法A：使用ZeroPadding2D填充

input_layer = Input(shape=(227,227,3))
model1 = Sequential([
    Conv2D(20, kernel_size=(5,5), activation='relu'),
    MaxPooling2D((2,2)),
    Conv2D(30, kernel_size=(3,3), activation='relu'),
    MaxPooling2D((2,2)),
    Conv2D(40, kernel_size=(3,3), activation='relu'),
    MaxPooling2D((2,2)),
    Conv2D(50, kernel_size=(3,3), activation='relu'),
    MaxPooling2D((2,2)),
    Conv2D(60, kernel_size=(3,3), activation='relu'),
    MaxPooling2D((2,2)),
])(input_layer)

model2 = Sequential([
    Conv2D(20, kernel_size=(5,5), activation='relu', dilation_rate=(3)),
    MaxPooling2D((2,2)),
    Conv2D(30, kernel_size=(3,3), activation='relu', dilation_rate=(2)),
    MaxPooling2D((2,2)),
    Conv2D(40, kernel_size=(3,3), activation='relu', dilation_rate=(2)),
    MaxPooling2D((2,2)),
    Conv2D(50, kernel_size=(3,3), activation='relu', dilation_rate=(1)),
    MaxPooling2D((2,2)),
    Conv2D(60, kernel_size=(3,3), activation='relu', dilation_rate=(1)),
    MaxPooling2D((2,2)),
    # 在底部和右侧各填充1个像素，将4x4转为5x5
    ZeroPadding2D(padding=((0,1), (0,1)))
])(input_layer)

# 拼接并构建完整模型
merged_model = Concatenate(axis=-1)([model1, model2])
merged_model = Flatten()(merged_model)
merged_model = Dense(1024, activation='relu')(merged_model)
merged_model = Dense(4, activation='softmax')(merged_model)

final_model = Model(inputs=input_layer, outputs=merged_model)
final_model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

方法B：使用Conv2DTranspose上采样

如果希望通过学习的方式调整尺寸，而非固定填充，可使用转置卷积：

# model1结构不变，修改model2的最后一层
model2 = Sequential([
    Conv2D(20, kernel_size=(5,5), activation='relu', dilation_rate=(3)),
    MaxPooling2D((2,2)),
    Conv2D(30, kernel_size=(3,3), activation='relu', dilation_rate=(2)),
    MaxPooling2D((2,2)),
    Conv2D(40, kernel_size=(3,3), activation='relu', dilation_rate=(2)),
    MaxPooling2D((2,2)),
    Conv2D(50, kernel_size=(3,3), activation='relu', dilation_rate=(1)),
    MaxPooling2D((2,2)),
    Conv2D(60, kernel_size=(3,3), activation='relu', dilation_rate=(1)),
    MaxPooling2D((2,2)),
    # 转置卷积将4x4转为5x5，保持通道数不变
    Conv2DTranspose(60, kernel_size=(2,2), strides=(1,1), padding='valid')
])(input_layer)

方案2：调整dilation_rate和padding，使输出尺寸原生匹配 #

通过计算特征图尺寸公式output_size = ((input_size - kernel_size - (kernel_size-1)*(dilation_rate-1)) // stride) + 1，反向推导调整model2的参数，让每一步输出尺寸和model1同步：

input_layer = Input(shape=(227,227,3))
model1 = Sequential([
    Conv2D(20, kernel_size=(5,5), activation='relu'),
    MaxPooling2D((2,2)),
    Conv2D(30, kernel_size=(3,3), activation='relu'),
    MaxPooling2D((2,2)),
    Conv2D(40, kernel_size=(3,3), activation='relu'),
    MaxPooling2D((2,2)),
    Conv2D(50, kernel_size=(3,3), activation='relu'),
    MaxPooling2D((2,2)),
    Conv2D(60, kernel_size=(3,3), activation='relu'),
    MaxPooling2D((2,2)),
])(input_layer)

# 修改dilation_rate并添加padding='same'，确保每一步尺寸和model1一致
model2 = Sequential([
    Conv2D(20, kernel_size=(5,5), activation='relu', dilation_rate=(3), padding='same'),
    MaxPooling2D((2,2)),
    Conv2D(30, kernel_size=(3,3), activation='relu', dilation_rate=(2), padding='same'),
    MaxPooling2D((2,2)),
    Conv2D(40, kernel_size=(3,3), activation='relu', dilation_rate=(2), padding='same'),
    MaxPooling2D((2,2)),
    Conv2D(50, kernel_size=(3,3), activation='relu', dilation_rate=(1), padding='same'),
    MaxPooling2D((2,2)),
    Conv2D(60, kernel_size=(3,3), activation='relu', dilation_rate=(1)),
    MaxPooling2D((2,2)),
])(input_layer)

# 后续拼接和编译逻辑同方案1
merged_model = Concatenate(axis=-1)([model1, model2])
merged_model = Flatten()(merged_model)
merged_model = Dense(1024, activation='relu')(merged_model)
merged_model = Dense(4, activation='softmax')(merged_model)

final_model = Model(inputs=input_layer, outputs=merged_model)
final_model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

方案3：将两个分支的特征图先Flatten再拼接 #

如果不想调整空间尺寸，可分别将两个分支的特征图展平后再拼接，这种方式无需对齐空间维度：

input_layer = Input(shape=(227,227,3))
model1 = Sequential([
    Conv2D(20, kernel_size=(5,5), activation='relu'),
    MaxPooling2D((2,2)),
    Conv2D(30, kernel_size=(3,3), activation='relu'),
    MaxPooling2D((2,2)),
    Conv2D(40, kernel_size=(3,3), activation='relu'),
    MaxPooling2D((2,2)),
    Conv2D(50, kernel_size=(3,3), activation='relu'),
    MaxPooling2D((2,2)),
    Conv2D(60, kernel_size=(3,3), activation='relu'),
    MaxPooling2D((2,2)),
    Flatten()  # 先展平model1的输出
])(input_layer)

model2 = Sequential([
    Conv2D(20, kernel_size=(5,5), activation='relu', dilation_rate=(3)),
    MaxPooling2D((2,2)),
    Conv2D(30, kernel_size=(3,3), activation='relu', dilation_rate=(2)),
    MaxPooling2D((2,2)),
    Conv2D(40, kernel_size=(3,3), activation='relu', dilation_rate=(2)),
    MaxPooling2D((2,2)),
    Conv2D(50, kernel_size=(3,3), activation='relu', dilation_rate=(1)),
    MaxPooling2D((2,2)),
    Conv2D(60, kernel_size=(3,3), activation='relu', dilation_rate=(1)),
    MaxPooling2D((2,2)),
    Flatten()  # 先展平model2的输出
])(input_layer)

# 直接拼接展平后的向量
merged_model = Concatenate(axis=-1)([model1, model2])
merged_model = Dense(1024, activation='relu')(merged_model)
merged_model = Dense(4, activation='softmax')(merged_model)

final_model = Model(inputs=input_layer, outputs=merged_model)
final_model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

内容的提问来源于stack exchange，提问作者Shivendra Pratap Singh