CNN+LSTM图像字幕模型训练报错：输入张量数量不匹配

构建基于CNN和LSTM的图像字幕生成模型，采用生成器批量生成训练数据，但调用model.fit传入生成器对象时出现报错，以下是问题代码及报错信息：

生成器与模型代码 #

def data_generator(train_descriptions,encoding_train,word_to_idx,max_len,num_photos_per_batch):
  X1,X2,y = [],[],[]
  n = 0
  while True:
    for key, desc_list in train_descriptions.items():
      n+=1
      photo = encoding_train[key+".jpg"]
      for desc in desc_list:
        seq = [word_to_idx[word] for word in desc.split() if word in word_to_idx]
        for i in range(1,len(seq)):
          in_seq = seq[0:i]
          out_seq = seq[i]
          in_seq = pad_sequences([in_seq],maxlen=max_len,value=0,padding='post')[0]
          out_seq = to_categorical([out_seq],num_classes=vocab_size)[0]
          X1.append(photo)
          X2.append(in_seq)
          y.append(out_seq)
      if n==num_photos_per_batch:
        yield [[np.array(X1),np.array(X2),np.array(y)]]
        X1,X2,y = [],[],[]
        n=0 
# image feature extractor model, inputs image feature vector

input_img_fea = Input(shape=(2048,))
inp_img1 = Dropout(0.3)(input_img_fea)
inp_img2 = Dense(256,activation='relu')(inp_img1)

#partial caption sequence model, inputs captions
input_cap = Input(shape=(max_len,))
inp_cap1 = Embedding(input_dim=vocab_size,output_dim=50,mask_zero=True)(input_cap)
inp_cap2 = Dropout(0.3)(inp_cap1)
inp_cap3 = LSTM(256)(inp_cap2)

decoder1 = add([inp_img2,inp_cap3])
decoder2 = Dense(256,activation='relu')(decoder1)
outputs = Dense(vocab_size,activation='softmax')(decoder2)

#Merge 2 networks
model = Model(inputs=[input_img_fea,input_cap],outputs=outputs)
model.summary()

model.layers[2].set_weights([embedding_output])
model.layers[2].trainable = False
model.compile(loss="categorical_crossentropy",optimizer="adam")

epochs = 20
number_pics_per_bath = 3
steps = len(train_descriptions)//number_pics_per_bath

for i in range(epochs):
  generator = data_generator(train_descriptions,encoding_train,word_to_idx,max_len,number_pics_per_bath)
  model.fit(generator,epochs=1,steps_per_epoch=steps)
  model.save('./model_'+str(i)+'.h5')

报错信息 #

ValueError: 用户代码中出现错误：
File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 1021, in train_function  *
    return step_function(self, iterator)
File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 1010, in step_function  **
    outputs = model.distribute_strategy.run(run_step, args=(data,))
File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 1000, in run_step  **
    outputs = model.train_step(data)
File "/usr/local/lib/python3.7/dist-packages/keras/engine/training.py", line 859, in train_step
    y_pred = self(x, training=True)
File "/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
File "/usr/local/lib/python3.7/dist-packages/keras/engine/input_spec.py", line 200, in assert_input_compatibility
    raise ValueError(f'Layer "{layer_name}" expects {len(input_spec)} input(s),'

ValueError: 层"model_2"期望2个输入，但接收到3个输入张量。接收到的输入：[<tf.Tensor 'IteratorGetNext:0' shape=(None, None) dtype=float32>, <tf.Tensor 'IteratorGetNext:1' shape=(None, None) dtype=int32>, <tf.Tensor 'IteratorGetNext:2' shape=(None, None) dtype=float32>]

问题原因与修复方案 #

报错核心原因是生成器返回的数据格式与模型输入要求不匹配：

• 模型定义了2个输入（图像特征、字幕序列）和1个输出（分类标签）
• 原生成器的yield语句将X1、X2、y都包裹在同一个列表中，导致Keras把三者都识别为模型输入，而模型只期望2个输入，因此触发错误

修改生成器中的yield语句即可解决：
将：

yield [[np.array(X1),np.array(X2),np.array(y)]]

替换为：

yield ([np.array(X1), np.array(X2)], np.array(y))

该格式符合Keras多输入模型对生成器的要求：返回**(输入列表, 输出张量)**的结构，其中输入列表对应模型定义的多个输入，输出张量对应训练标签。

内容的提问来源于stack exchange，提问作者Vedant Sharma