Keras自编码器形状不兼容问题：分类数据集异常检测排查

你的错误确实是模型输出与训练目标的形状不匹配导致的，核心问题有两个：

1. 输入与目标结构完全不匹配 #

你构建的模型输入是10个独立的Input层（对应10个分类列），但模型输出是一个合并后的张量（形状为(batch_size, 1, 总嵌入维度)）。而训练时传入的目标X_train_enc是10个一维数组（每个数组形状为(1000,)），两者结构完全不兼容，Keras无法计算损失。

2. 自编码器重构目标错位 #

自编码器的核心是“重构输入”，但你当前的模型输出是重构嵌入后的特征向量，却试图用原始的标签编码值作为目标——这两类数据维度、含义完全不同，无法匹配。

以下提供两种贴合异常检测场景的可行修正思路：

思路一：重构原始分类标签（推荐） #

这种思路下，模型会将编码后的特征解码回每个分类列的标签概率分布，通过交叉熵损失衡量与原始标签的差异，更贴合异常检测中“识别偏离正常模式样本”的需求。

修改后的完整代码：

import numpy as np
import pandas as pd
import tensorflow as tf
from sklearn.preprocessing import LabelEncoder
from sklearn.utils import column_or_1d
from tensorflow.keras.layers import Input, Embedding, concatenate, Dense, Lambda, Flatten
from tensorflow.keras.models import Model, Sequential

# 数据生成
num_rows = 1000
categories = ['A', 'B', 'C', 'D', 'E']
data = {}
for i in range(10):
    column_name = f'Column_{i+1}'
    data[column_name] = np.random.choice(categories, size=num_rows)
df = pd.DataFrame(data)

# 编码分类列并保存编码器
X_train_enc = []
label_encoders = []
for column in df.select_dtypes(include=['object']).columns:
    label_encoder = LabelEncoder()
    train_enc = label_encoder.fit_transform(column_or_1d(df[column]))
    X_train_enc.append(train_enc)
    label_encoders.append(label_encoder)

# 构建输入与嵌入层（关键：展平嵌入输出）
in_layers = []
em_layers = []
n_label_list = []
output_dims = []

for i in range(len(X_train_enc)):
    n_labels = len(np.unique(X_train_enc[i]))
    output_dim = n_labels // 2 if n_labels // 2 > 0 else 1  # 避免嵌入维度为0
    in_layer = Input(shape=(1,))
    em_layer = Embedding(n_labels, output_dim)(in_layer)
    em_layer = Flatten()(em_layer)  # 将(None,1,output_dim)转为(None,output_dim)
    in_layers.append(in_layer)
    em_layers.append(em_layer)
    n_label_list.append(n_labels)
    output_dims.append(output_dim)

# 合并嵌入特征
merge = concatenate(em_layers)

# 编码器
encoder = Sequential([
    Dense(10, activation='relu'),
    Dense(5, activation='relu'),
])(merge)

# 解码器+输出层：拆分出每个列的概率分布
decoder = Sequential([
    Dense(10, activation='relu'),
    Dense(sum(output_dims), activation='relu'),
])(encoder)

output_layers = []
start_idx = 0
for n_labels, output_dim in zip(n_label_list, output_dims):
    col_decoded = Lambda(lambda x: x[:, start_idx:start_idx+output_dim])(decoder)
    start_idx += output_dim
    col_output = Dense(n_labels, activation='softmax')(col_decoded)
    output_layers.append(col_output)

# 构建多输入多输出模型
autoencoder = Model(inputs=in_layers, outputs=output_layers)
# 用稀疏分类交叉熵（目标是整数标签）
autoencoder.compile(optimizer='adam', loss='sparse_categorical_crossentropy')

# 训练：输入与目标都是10个编码后的数组
autoencoder.fit(X_train_enc, X_train_enc, epochs=50, batch_size=64, verbose=2)

思路二：重构嵌入后的特征向量 #

如果仅需要验证自编码器的编码能力，不需要还原到原始标签，可以调整训练目标为合并后的嵌入特征：

import numpy as np
import pandas as pd
import tensorflow as tf
from sklearn.preprocessing import LabelEncoder
from tensorflow.keras.layers import Input, Embedding, concatenate, Dense, Lambda, Flatten
from tensorflow.keras.models import Model, Sequential

# 数据生成与编码部分同思路一
num_rows = 1000
categories = ['A', 'B', 'C', 'D', 'E']
data = {}
for i in range(10):
    column_name = f'Column_{i+1}'
    data[column_name] = np.random.choice(categories, size=num_rows)
df = pd.DataFrame(data)

X_train_enc = []
for column in df.select_dtypes(include=['object']).columns:
    label_encoder = LabelEncoder()
    train_enc = label_encoder.fit_transform(df[column])
    X_train_enc.append(train_enc)

# 将编码后的列合并为单输入张量
X_train_combined = np.stack(X_train_enc, axis=1)

# 构建单输入模型
input_layer = Input(shape=(10,))
embeddings = []
for col_idx in range(10):
    n_labels = len(np.unique(X_train_enc[col_idx]))
    output_dim = n_labels//2 if n_labels//2>0 else 1
    col_input = Lambda(lambda x: x[:, col_idx:col_idx+1])(input_layer)
    em_layer = Embedding(n_labels, output_dim)(col_input)
    em_layer = Flatten()(em_layer)
    embeddings.append(em_layer)
merge = concatenate(embeddings)

encoder = Sequential([
    Dense(10, activation='relu'),
    Dense(5, activation='relu'),
])(merge)

decoder = Sequential([
    Dense(10, activation='relu'),
    Dense(merge.shape[1], activation='linear'),
])(encoder)

autoencoder = Model(inputs=input_layer, outputs=decoder)
autoencoder.compile(optimizer='adam', loss='mse')

# 生成嵌入特征作为训练目标
embed_model = Model(inputs=input_layer, outputs=merge)
X_target = embed_model.predict(X_train_combined)

# 训练
autoencoder.fit(X_train_combined, X_target, epochs=50, batch_size=64, verbose=2)

• 多输入模型的输出结构必须与目标结构对应：如果输入是多个张量，输出也要是相同数量的张量，或者将目标合并为对应形状的单一张量。
• 必须展平Embedding层输出：Embedding默认输出形状为(batch_size, sequence_length, output_dim)，这里sequence_length=1，用Flatten()转为(batch_size, output_dim)可避免维度混乱。
• 明确重构目标：分类数据的自编码器需明确是还原标签还是还原嵌入特征，对应不同的损失函数和输出层设计。

内容的提问来源于stack exchange，提问作者clueless