为提升模型精度尝试TabNet时，如何解决TensorFlow输入形状不匹配的ValueError？

问题背景 #

为提升模型精度尝试使用TabNet，训练集与测试集数据已上传至Google Drive。以下是原始代码：

import tensorflow as tf
import pandas as pd

# Load the train and test data into pandas dataframes
train_df = pd.read_csv("train.csv")
#train_df1 = pd.read_csv("train.csv")

test_df = pd.read_csv("test.csv")
# Split the target variable and the features
train_labels = train_df[[f'F_{i}' for i in range(40)]]
#train_labels=trai
test_labels = train_df.target

# Convert the dataframes to tensors
train_dataset = tf.data.Dataset.from_tensor_slices((train_df.values, train_labels.values))
test_dataset = tf.data.Dataset.from_tensor_slices((test_df.values, test_labels.values))

# Define the model using the TabNet architecture
model = tf.keras.models.Sequential([
    tf.keras.layers.Input(shape=(train_df.shape[1],)),
    tf.keras.layers.Dense(32, activation="relu"),
    tf.keras.layers.Dense(64, activation="relu"),
    tf.keras.layers.Dense(128, activation="relu"),
    tf.keras.layers.Dense(1)
])

# Compile the model with a mean squared error loss function and the Adam optimizer
model.compile(loss="mean_squared_error", optimizer="adam")

# Train the model on the training data
model.fit(train_dataset.batch(32), epochs=5)

# Make predictions on the test data
predictions = model.predict(test_dataset.batch(32))
#predictions = model.predict(test_dataset)

# Evaluate the model on the test data
mse = tf.keras.losses.mean_squared_error(test_labels, predictions)
print("Mean Squared Error:", mse.numpy().mean())

错误信息 #

ValueError                                Traceback (most recent call last)
<ipython-input-40-87712e1604a9> in <module>
     24 
     25 # Make predictions on the test data
---> 26 predictions = model.predict(test_dataset.batch(32))
     27 #predictions = model.predict(test_dataset)
     28 

1 frames
/usr/local/lib/python3.8/dist-packages/keras/engine/training.py in tf__predict_function(iterator)
     13                 try:
     14                     do_return = True
---> 15                     retval_ = ag__.converted_call(ag__.ld(step_function), (ag__.ld(self), ag__.ld(iterator)), None, fscope)
     16                 except:
     17                     do_return = False

ValueError: in user code:

    File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1845, in predict_function  *
        return step_function(self, iterator)
    File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1834, in step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1823, in run_step  **
        outputs = model.predict_step(data)
    File "/usr/local/lib/python3.8/dist-packages/keras/engine/training.py", line 1791, in predict_step
        return self(x, training=False)
    File "/usr/local/lib/python3.8/dist-packages/keras/utils/traceback_utils.py", line 67, in error_handler
        raise e.with_traceback(filtered_tb) from None
    File "/usr/local/lib/python3.8/dist-packages/keras/engine/input_spec.py", line 264, in assert_input_compatibility
        raise ValueError(f'Input {input_index} of layer "{layer_name}" is '

    ValueError: Input 0 of layer "sequential_6" is incompatible with the layer: expected shape=(None, 42), found shape=(None, 41)

错误原因 #

1. 特征与标签划分完全错误：将训练集中的40个F_*列当作标签，但训练时又把整个train_df（包含特征、这些F_*列和target列）作为模型输入；而测试集test_df没有这些F_*列，导致输入维度比训练时少1，引发维度不匹配。
2. 测试标签误用训练集数据：test_labels = train_df.target完全错误，应该使用测试集自身的目标列（如果存在）。
3. 未真正实现TabNet：当前代码是普通全连接网络，没有用到TabNet架构。

解决方案 #

步骤1：正确划分训练集特征与标签 #

假设任务是回归，目标列是target，训练集特征为除target外的所有列：

import tensorflow as tf
import pandas as pd

# 加载数据
train_df = pd.read_csv("train.csv")
test_df = pd.read_csv("test.csv")

# 划分训练集特征与标签
train_features = train_df.drop(columns=['target'])
train_labels = train_df['target']

步骤2：对齐测试集特征维度 #

确保测试集只保留和训练集一致的特征列：

# 测试集特征与训练集保持一致
test_features = test_df[train_features.columns]
# 如果测试集有目标列，单独提取
test_labels = test_df['target'] if 'target' in test_df.columns else None

步骤3：修正数据集构建 #

# 构建训练数据集（特征+标签）
train_dataset = tf.data.Dataset.from_tensor_slices((train_features.values, train_labels.values)).batch(32)
# 构建测试数据集：如果有标签则传入，否则只传特征
if test_labels is not None:
    test_dataset = tf.data.Dataset.from_tensor_slices((test_features.values, test_labels.values)).batch(32)
else:
    test_dataset = tf.data.Dataset.from_tensor_slices(test_features.values).batch(32)

步骤4：修正模型输入维度 #

用训练集特征的维度作为模型输入：

model = tf.keras.models.Sequential([
    tf.keras.layers.Input(shape=(train_features.shape[1],)),
    tf.keras.layers.Dense(32, activation="relu"),
    tf.keras.layers.Dense(64, activation="relu"),
    tf.keras.layers.Dense(128, activation="relu"),
    tf.keras.layers.Dense(1)
])

model.compile(loss="mean_squared_error", optimizer="adam")

步骤5：修正训练、预测与评估逻辑 #

# 训练模型
model.fit(train_dataset, epochs=5)

# 预测
predictions = model.predict(test_dataset)

# 评估（仅当测试集有标签时）
if test_labels is not None:
    mse = tf.keras.losses.mean_squared_error(test_labels, predictions)
    print("Mean Squared Error:", mse.numpy().mean())

步骤6：实现真正的TabNet #

如果要使用TabNet提升精度，推荐使用pytorch-tabnet库（更易用），示例代码：

from pytorch_tabnet.tab_model import TabNetRegressor
from sklearn.metrics import mean_squared_error

# 初始化TabNet回归器
tabnet_model = TabNetRegressor()

# 训练模型
tabnet_model.fit(
    X_train=train_features.values, y_train=train_labels.values,
    eval_set=[(test_features.values, test_labels.values)] if test_labels is not None else None,
    eval_name=['test'] if test_labels is not None else None,
    eval_metric=['mse'],
    max_epochs=50,
    batch_size=32
)

# 预测
predictions = tabnet_model.predict(test_features.values)

# 评估
if test_labels is not None:
    mse = mean_squared_error(test_labels.values, predictions)
    print("Mean Squared Error:", mse)

内容的提问来源于stack exchange，提问作者Priyangshu Hazowary