Keras模型预测因独热编码维度不匹配报错，求替代预测方法

Hey there! Let's break down how to fix this shape mismatch issue and explore alternative prediction approaches that avoid this problem in the first place.

1. Reuse the Exact OneHotEncoder from Training (The Best Fix) #

This is the most straightforward and reliable solution. You probably forgot to save the OneHotEncoder instance you used during training—when you fit it on your full training data, it learns all the unique categories for each feature and locks the output dimension to 133. For prediction, you need to use this same encoder to transform new data: it will automatically fill 0s for any categories present in the training set but missing from the new dataset, ensuring the shape matches.

If you're using scikit-learn's OneHotEncoder, here's how to implement this:

# During training
from sklearn.preprocessing import OneHotEncoder
import joblib

# Initialize and fit the encoder on your training data
encoder = OneHotEncoder(sparse_output=False)
train_encoded = encoder.fit_transform(train_data)

# Save the encoder to reuse later
joblib.dump(encoder, 'onehot_encoder.pkl')

# During prediction
# Load the saved encoder
encoder = joblib.load('onehot_encoder.pkl')
new_data_encoded = encoder.transform(new_data)

# Now new_data_encoded will have shape (10, 133), matching your model's input

If your new data has categories that weren't present in the training set, add handle_unknown='ignore' when initializing the encoder—it will skip those unseen categories instead of throwing an error.

2. Manually Align Feature Dimensions (Last Resort) #

If you didn't save the encoder, you can manually match the input shape by comparing unique categories between your training data and new data. Identify which categories exist in the training set but are missing from the new data, then add those columns filled with 0s to your encoded new data until the total dimension hits 133.

Note: This is error-prone and not recommended—you have to carefully track every category across all features, which gets messy fast. Only use this if you can't retrieve the original encoder.

3. Replace One-Hot Encoding with an Embedding Layer #

One-hot encoding often leads to shape mismatches when dealing with categorical features that have dynamic categories. Switching to an embedding layer eliminates this problem entirely:

• First, convert your categorical features to integer indices (instead of one-hot vectors) using LabelEncoder for each feature.
• Then add an Embedding layer at the start of your Keras model. This layer learns a dense, low-dimensional representation for each category, and it can handle unseen categories by mapping them to a default "out-of-vocabulary" (OOV) token.

Example code snippet for the model:

import tensorflow as tf
from tensorflow.keras.layers import Embedding, LSTM, Dense

# Assume your integer-encoded input has shape (10, num_features)
model = tf.keras.Sequential([
    Embedding(input_dim=max_category_index + 2,  # +2 for 0 padding and OOV token
              output_dim=32,
              input_length=10),
    LSTM(64),
    Dense(1, activation='sigmoid')  # Adjust based on your classification/regression task
])

This approach is cleaner for categorical data with many categories and removes the need to manage one-hot dimension alignment.

4. Use Feature Hashing #

Feature hashing (the hashing trick) maps arbitrary categorical features to a fixed-dimensional space, regardless of how many unique categories exist. This means your encoded data will always have the same shape, even if new data has unseen categories.

Using scikit-learn's FeatureHasher:

from sklearn.feature_extraction import FeatureHasher

# Use the same n_features value for both training and prediction
hasher = FeatureHasher(n_features=133, input_type='string')
train_hashed = hasher.transform(train_data.astype(str))
new_data_hashed = hasher.transform(new_data.astype(str))

Keep in mind that feature hashing can cause hash collisions (different categories mapped to the same index), but it's a lightweight solution for cases where you don't want to track all categories.

内容的提问来源于stack exchange，提问作者Velmoorthi