Keras多输入模型训练报错：矩阵尺寸不兼容

问题描述 #

构建从DataFrame读取图像和文本的多输入Keras模型，前期操作正常，但训练阶段出现以下错误：

节点: 'model_1/dense_2/MatMul'
矩阵尺寸不兼容: In[0]: [32,229], In[1]: [10000,64]
[[{{node model_1/dense_2/MatMul}}]] [Op:__inference_train_function_1955]

错误原因 #

核心问题是文本输入维度与模型定义不匹配：

1. 模型中定义文本输入维度为(10000,)，但生成器每次生成batch时都重新初始化CountVectorizer并调用fit_transform，导致每个batch的文本特征维度是当前batch的实际词汇数（示例中为229），而非预先设定的10000。
2. 训练集和验证集使用各自独立拟合的CountVectorizer，词汇表不一致，进一步加剧维度不匹配问题。

修复步骤 #

1. 在MultiGen初始化阶段，用整个数据集的文本提前拟合CountVectorizer，固定词汇表大小为max_features。
2. 生成器中仅调用transform方法转换文本，不再重新拟合词汇表，确保每个batch的文本特征维度统一为10000。
3. 确保模型输入维度与生成器输出的文本特征维度一致。

完整修正代码 #

class MultiGen:
    def __init__(self, data_path, batch_size):
        self.data_path = data_path
        self.batch_size = batch_size
        self.df = pd.read_csv(data_path)
        self.train_df = self.df[:int(0.8 * len(self.df))]
        self.val_df = self.df[int(0.8 * len(self.df)):]
        self.tokenizer = Tokenizer(num_words=10000)
        self.max_features = 10000
        
        # 提前读取所有文本并拟合CountVectorizer
        all_texts = []
        for txt_path in self.df['text']:
            txt = open(txt_path).read()
            all_texts.append(txt)
        self.vectorizer = CountVectorizer(max_features=self.max_features)
        self.vectorizer.fit(all_texts)

    def multi_input_generator(self, data_df):
        while True:
            for i in range(0, len(data_df), self.batch_size):
                batch_df = data_df[i:i+self.batch_size].reset_index(drop=True)
                images = []
                text = []
                for img_path in batch_df['image']:
                    img = load_img(img_path, target_size=(300, 300))
                    img = img_to_array(img) / 255.0
                    images.append(img)
                for txt_path in batch_df['text']:
                    txt = open(txt_path).read()
                    text.append(txt)
                    
                # 仅使用transform转换文本，不再重新拟合
                texts = self.vectorizer.transform(text)
                texts = texts.toarray()
                
                labels = batch_df['label']
                yield [np.array(images), np.array(texts)], np.array(labels)

    def train_generator(self):
        return self.multi_input_generator(self.train_df)
    
    def val_generator(self):
        return self.multi_input_generator(self.val_df)
    
    def length(self):
        return len(self.df)

    def train_length(self):
        return len(self.train_df)
    
    def val_length(self):
        return len(self.val_df)

    
    
batch_size = 32
# 初始化生成器
gen = MultiGen(data_path, batch_size=batch_size)

# 获取训练和验证生成器
train_gen = gen.train_generator()
val_gen = gen.val_generator()



input_img = Input(shape=(300, 300, 3))
input_text = Input(shape=(gen.max_features,))

x = Conv2D(32, (3, 3), activation='relu')(input_img)
x = MaxPooling2D((2, 2))(x)
x = Flatten()(x)

y = Dense(64, activation='relu')(input_text)

z = keras.layers.concatenate([x, y],axis=-1)

output = Dense(337, activation='softmax')(z)

model = Model(inputs=[input_img, input_text], outputs=output)
model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

model.summary()
# 训练模型
model.fit(train_gen, validation_data=val_gen, steps_per_epoch=gen.train_length()//batch_size, validation_steps=gen.val_length()//batch_size, epochs=10)

内容的提问来源于stack exchange，提问作者Ibrahima S Wade