在TensorFlow中为图像分类模型加入表格数据时出现报错

问题现象 #

运行模型训练代码时触发如下错误：

ValueError: Failed to find data adapter that can handle input: (<class 'list'> containing values of types {"<class 'keras.preprocessing.image.DataFrameIterator'>", "<class 'numpy.ndarray'>"}), <class 'numpy.ndarray'>

数据处理代码 #

images_generator = ImageDataGenerator()
X_train_images = images_generator.flow_from_dataframe(
    dataframe=dataframe_train,
    directory=None,
    x_col='image_path',
    y_col='target',
    class_mode='raw'
)

dataframe_train.drop("image_path", axis=1, inplace=True)
X_train_tabular = dataframe_train.iloc[:, :-1].values
# scaling etc..
y_train = dataframe_train.iloc[:, -1].values

模型构建代码 #

# Load the VGG16 model
vgg16 = keras.applications.VGG16(include_top=False, input_shape=(224, 224, 3))
for layer in vgg16.layers:
    layer.trainable = False

image_input = keras.Input(shape=(224, 224, 3), name='image')
tabular_input = keras.Input(shape=(NUM_TABULAR_COLS,), name='tabular')

vgg16_output = vgg16(image_input)
vgg16_output_flat = keras.layers.Flatten()(vgg16_output)

# Combine the flattened VGG16 output and tabular data
combined_inputs = tf.keras.layers.concatenate([vgg16_output_flat, tabular_input])

x = keras.layers.Dense(64, activation='relu')(combined_inputs)
output = keras.layers.Dense(2, activation='softmax')(x)

# Create a model using the inputs and outputs
model = keras.Model(inputs=[image_input, tabular_input], outputs=output)
model.compile(loss='categorical_crossentropy', optimizer='adam')

model.fit(
    [X_train_images, X_train_tabular],
    y_test,
    epochs=2
)

核心问题分析 #

1. 输入类型不兼容：flow_from_dataframe返回的是DataFrameIterator（生成器），而X_train_tabular是numpy数组，Keras的数据适配器无法同时处理这两种不同类型的输入组合。
2. 标签混用：训练时错误传入了测试集标签y_test，应该用训练集标签y_train。
3. 损失函数与标签不匹配：使用categorical_crossentropy时，标签需要是独热编码格式，但当前y_train是原始数值型（class_mode='raw'），二者不匹配。

修复方案 #

方案一：自定义生成器（适合大数据集） #

通过自定义生成器同步返回图像批次和对应表格数据，解决类型不兼容问题：

def custom_generator(image_gen, tabular_data, labels, batch_size):
    while True:
        # 获取图像批次（忽略生成器自带的标签，用我们自己的labels）
        img_batch, _ = next(image_gen)
        # 获取当前批次的索引范围
        current_idx = image_gen.batch_index
        if current_idx == 0:
            # 每轮结束后同步打乱索引
            np.random.shuffle(image_gen.index_array)
        start_idx = (current_idx - 1) * batch_size
        end_idx = current_idx * batch_size
        # 取出对应批次的表格数据和标签
        tab_batch = tabular_data[image_gen.index_array[start_idx:end_idx]]
        label_batch = labels[image_gen.index_array[start_idx:end_idx]]
        yield [img_batch, tab_batch], label_batch

# 重新初始化图像生成器，指定批次大小并开启打乱
batch_size = 32
X_train_images = images_generator.flow_from_dataframe(
    dataframe=dataframe_train,
    directory=None,
    x_col='image_path',
    y_col='target',
    class_mode='raw',
    batch_size=batch_size,
    shuffle=True
)

# 创建自定义训练生成器
train_generator = custom_generator(X_train_images, X_train_tabular, y_train, batch_size)

# 调整损失函数：因为y_train是数值型，用sparse_categorical_crossentropy无需转独热
model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

# 计算每轮训练步数
steps_per_epoch = len(dataframe_train) // batch_size

# 启动训练
model.fit(
    train_generator,
    steps_per_epoch=steps_per_epoch,
    epochs=2
)

方案二：加载所有图像到内存（适合小数据集） #

如果数据集规模不大，直接把所有图像加载成numpy数组，和表格数据一起传入训练：

# 批量加载图像并转为numpy数组
def load_all_images(image_paths):
    image_list = []
    for path in image_paths:
        img = keras.preprocessing.image.load_img(path, target_size=(224, 224))
        img_array = keras.preprocessing.image.img_to_array(img)
        image_list.append(img_array)
    return np.array(image_list)

# 加载训练图像
X_train_images_np = load_all_images(dataframe_train['image_path'].values)

# 转换标签为独热编码（适配categorical_crossentropy）
from tensorflow.keras.utils import to_categorical
y_train_onehot = to_categorical(y_train, num_classes=2)

# 编译模型
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

# 启动训练
model.fit(
    [X_train_images_np, X_train_tabular],
    y_train_onehot,
    batch_size=32,
    epochs=2
)

内容的提问来源于stack exchange，提问作者Newbie