基于图像生成HTML/CSS的ML模型训练输入不匹配报错解决

正在构建一个根据输入图像生成HTML和CSS代码的机器学习模型，训练时使用包含image_path、html_path、css_path的CSV数据集，但训练过程中触发报错：

ValueError: Layer "model_20" expects 3 input(s), but it received 1 input tensors. Inputs received: []

训练代码如下：

from sklearn.model_selection import train_test_split
import pandas as pd
import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

# Load data from CSV
csv_file = 'data.csv'
data = pd.read_csv(csv_file)

# Define data directories (assuming images are in the same directory as the CSV file)
image_dir = os.path.dirname(csv_file)

data_train=data[data.html_path.isnull()==False]
# data_test=data[data.html_path.isnull()==True]

vocab_size = 1000

# Split the data into training and testing sets
train_data, test_data = train_test_split(data_train, test_size=0.2, random_state=42)

# Extract data for training
train_image_paths = train_data['image_path'].tolist()
train_html_texts = train_data['html_path'].tolist()
train_css_texts = train_data['css_path'].tolist()

# Extract data for testing
test_image_paths = test_data['image_path'].tolist()
test_html_texts = test_data['html_path'].tolist()
test_css_texts = test_data['css_path'].tolist()

# Load and preprocess image data for training
train_images = []
for image_path in train_image_paths:
    img_path = os.path.join(image_dir, image_path)
    img = tf.keras.preprocessing.image.load_img(img_path, target_size=image_size)
    img = tf.keras.preprocessing.image.img_to_array(img)
    img = tf.keras.applications.vgg16.preprocess_input(img)
    train_images.append(img)

# Load and preprocess image data for testing
test_images = []
for image_path in test_image_paths:
    img_path = os.path.join(image_dir, image_path)
    img = tf.keras.preprocessing.image.load_img(img_path, target_size=image_size)
    img = tf.keras.preprocessing.image.img_to_array(img)
    img = tf.keras.applications.vgg16.preprocess_input(img)
    test_images.append(img)

# Tokenize HTML and CSS code for training
train_html_sequences = tokenizer(train_html_texts)
train_css_sequences = tokenizer(train_css_texts)

# Tokenize HTML and CSS code for testing
test_html_sequences = tokenizer(test_html_texts)
test_css_sequences = tokenizer(test_css_texts)

# Define input layers
image_input = keras.layers.Input(shape=(224, 224, 3), name='image_input')
html_input = keras.layers.Input(shape=(max_seq_length,), name='html_input')
css_input = keras.layers.Input(shape=(max_seq_length,), name='css_input')

# CNN for image feature extraction
cnn_model = keras.applications.VGG16(weights='imagenet', include_top=False, input_tensor=image_input)
flatten_layer = layers.Flatten()(cnn_model.output)
image_features = layers.Dense(256, activation='relu', name='image_features')(flatten_layer)

# RNN for HTML code generation
html_embedding = layers.Embedding(vocab_size, 128)(html_input)
html_rnn = layers.LSTM(256, name='html_rnn')(html_embedding)
html_output = layers.Dense(vocab_size, activation='softmax', name='html_output')(html_rnn)

# RNN for CSS code generation (similar to HTML)
css_embedding = layers.Embedding(vocab_size, 128)(css_input)
css_rnn = layers.LSTM(256, name='css_rnn')(css_embedding)
css_output = layers.Dense(vocab_size, activation='softmax', name='css_output')(css_rnn)

# Combine image and code features
concatenated_features = layers.Concatenate(axis=-1)([image_features, html_output, css_output])

# Final output layers for HTML and CSS
html_output_final = layers.Dense(vocab_size, activation='softmax', name='html_output_final')(concatenated_features)
css_output_final = layers.Dense(vocab_size, activation='softmax', name='css_output_final')(concatenated_features)

# Create the model
model = keras.Model(inputs=[image_input, html_input, css_input], outputs=[html_output_final, css_output_final])

# Compile the model
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])


# Define the batch size and number of epochs
batch_size = 32
epochs = 10

# Define separate datasets for each input
train_dataset = tf.data.Dataset.from_tensor_slices((train_images, train_html_sequences, train_css_sequences))
train_dataset = train_dataset.batch(batch_size)

test_dataset = tf.data.Dataset.from_tensor_slices((test_images, test_html_sequences, test_css_sequences))
test_dataset = test_dataset.batch(batch_size)

# Train the model
model.fit(train_dataset, validation_data=test_dataset, epochs=epochs)

CSV文件结构：

核心问题 #

报错源于tf.data.Dataset的输入格式与模型多输入要求不匹配：模型定义了3个独立输入，但当前Dataset将三个数据打包成单个元组，Keras无法识别为独立的输入张量；同时代码存在未定义变量、标签格式不匹配等问题。

1. 修正Dataset输入格式 #

将Dataset改为字典格式（与模型输入层的name一一对应），确保Keras能正确解析每个输入：

import numpy as np

# 训练集转换为字典格式
train_dataset = tf.data.Dataset.from_tensor_slices({
    'image_input': np.array(train_images),
    'html_input': np.array(train_html_sequences),
    'css_input': np.array(train_css_sequences)
})
train_dataset = train_dataset.batch(batch_size)

# 测试集同理
test_dataset = tf.data.Dataset.from_tensor_slices({
    'image_input': np.array(test_images),
    'html_input': np.array(test_html_sequences),
    'css_input': np.array(test_css_sequences)
})
test_dataset = test_dataset.batch(batch_size)

如果坚持使用元组格式，必须明确输入与输出的对应关系（模型有两个输出，需传入标签元组）：

# 假设训练标签为HTML和CSS序列（根据任务调整）
train_dataset = tf.data.Dataset.from_tensor_slices(
    (
        (np.array(train_images), np.array(train_html_sequences), np.array(train_css_sequences)),
        (np.array(train_html_sequences), np.array(train_css_sequences))
    )
)
train_dataset = train_dataset.batch(batch_size)

2. 补全未定义变量 #

代码中多个关键变量未定义，需补充：

# 定义图片尺寸（与输入层shape一致）
image_size = (224, 224)
# 定义序列最大长度
max_seq_length = 200

# 实例化Tokenizer并处理文本
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences

tokenizer = Tokenizer(num_words=vocab_size)
tokenizer.fit_on_texts(train_html_texts + train_css_texts)

# 生成序列并填充到固定长度
train_html_sequences = pad_sequences(tokenizer.texts_to_sequences(train_html_texts), maxlen=max_seq_length)
train_css_sequences = pad_sequences(tokenizer.texts_to_sequences(train_css_texts), maxlen=max_seq_length)

test_html_sequences = pad_sequences(tokenizer.texts_to_sequences(test_html_texts), maxlen=max_seq_length)
test_css_sequences = pad_sequences(tokenizer.texts_to_sequences(test_css_texts), maxlen=max_seq_length)

3. 适配损失函数 #

当前使用categorical_crossentropy但未将标签转为one-hot编码，建议改用sparse_categorical_crossentropy（直接支持整数序列标签）：

model.compile(optimizer='adam', 
              loss={
                  'html_output_final': 'sparse_categorical_crossentropy', 
                  'css_output_final': 'sparse_categorical_crossentropy'
              },
              metrics=['accuracy'])

4. 验证数据一致性 #

确保train_images、train_html_sequences、train_css_sequences的样本数量完全一致，避免Dataset加载时出现维度不匹配错误。

内容的提问来源于stack exchange，提问作者Shuvam