俄语文本分类输出全为零列表？如何获取正确分类结果

针对你的模型返回全零长列表、无法得到正确分类结果的问题，以下是分步排查和解决方案：

一、先解决数据处理的核心问题 #

1. 消除PAD标记对LSTM输出的干扰 #

你使用pad_sequence补全序列后，直接取lstm_out[:, -1, :]（序列最后一个位置输出），但该位置大概率是PAD标记（youtokentome默认PAD ID为0），导致模型学到无效信息。正确做法是基于序列实际长度取有效输出：

# 先计算每条评论的实际序列长度
df['seq_len'] = df['tokens'].apply(len)
seq_lens = torch.tensor(df['seq_len'].values)

# 修改模型forward方法，处理变长序列
def forward(self, input_seq, seq_lens):
    embedded = self.embedding(input_seq)
    # 打包序列，跳过PAD部分的计算
    packed_embedded = nn.utils.rnn.pack_padded_sequence(embedded, seq_lens, batch_first=True, enforce_sorted=False)
    packed_lstm_out, _ = self.lstm(packed_embedded)
    lstm_out, _ = nn.utils.rnn.pad_packed_sequence(packed_lstm_out, batch_first=True)
    
    # 取每条序列最后一个有效位置的输出
    idx = (seq_lens - 1).view(-1, 1).expand(input_seq.size(0), self.hidden_size).unsqueeze(1)
    lstm_out = lstm_out.gather(1, idx).squeeze(1)
    
    x = self.linear(lstm_out)
    return F.log_softmax(x, dim=1)

2. 验证分词ID的有效性 #

检查df['tokens']中的ID是否全部落在0~4999范围内（对应你设置的vocab_size=5000），若存在超出范围的ID，说明BPE分词器训练不充分，需重新训练：

# 检查是否有OOV（超出词表范围）的ID
max_id = df['tokens'].apply(lambda x: max(x)).max()
if max_id >= 5000:
    print("存在OOV，需重新训练tokenizer")
    # 重新训练时确保覆盖所有文本
    yttm.BPE.train(data='text.txt', model=model_path, vocab_size=5000)

同时，分词时关闭默认的首尾特殊标记，避免引入未处理的额外ID：

df['tokens'] = df['textOriginal'].apply(lambda x: tokenizer.encode(x, output_type=yttm.OutputType.ID, bos=False, eos=False))

二、补全完整的模型训练流程 #

你只定义了模型结构，但未执行训练——随机初始化的模型输出全零（或无意义结果）是正常的，必须完成以下训练步骤：

1. 划分数据集并准备张量 #

from sklearn.model_selection import train_test_split
from torch.utils.data import TensorDataset, DataLoader

# 假设你的标签列是df['label']，已转换为0/1/2的整数
train_df, val_df = train_test_split(df, test_size=0.2, random_state=42)

def prepare_dataset(data_df):
    tokens = [torch.tensor(t) for t in data_df['tokens']]
    tokens_tensor = nn.utils.rnn.pad_sequence(tokens, batch_first=True)
    seq_lens = torch.tensor(data_df['seq_len'].values)
    labels = torch.tensor(data_df['label'].values)
    return TensorDataset(tokens_tensor, seq_lens, labels)

train_loader = DataLoader(prepare_dataset(train_df), batch_size=16, shuffle=True)
val_loader = DataLoader(prepare_dataset(val_df), batch_size=16)

2. 设置训练组件 #

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = MyModel(vocab_size=5000, embedding_dim=300, hidden_size=512, output_dim=3).to(device)

# 对应log_softmax输出，使用NLLLoss损失函数
criterion = nn.NLLLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

3. 执行训练循环 #

epochs = 10
for epoch in range(epochs):
    # 训练阶段
    model.train()
    train_loss = 0.0
    for batch_x, batch_len, batch_y in train_loader:
        batch_x, batch_len, batch_y = batch_x.to(device), batch_len.to(device), batch_y.to(device)
        
        optimizer.zero_grad()
        outputs = model(batch_x, batch_len)
        loss = criterion(outputs, batch_y)
        loss.backward()
        optimizer.step()
        
        train_loss += loss.item() * batch_x.size(0)
    
    # 验证阶段
    model.eval()
    val_loss = 0.0
    correct = 0
    total = 0
    with torch.no_grad():
        for batch_x, batch_len, batch_y in val_loader:
            batch_x, batch_len, batch_y = batch_x.to(device), batch_len.to(device), batch_y.to(device)
            outputs = model(batch_x, batch_len)
            loss = criterion(outputs, batch_y)
            val_loss += loss.item() * batch_x.size(0)
            
            _, predicted = torch.max(outputs.data, 1)
            total += batch_y.size(0)
            correct += (predicted == batch_y).sum().item()
    
    print(f"Epoch {epoch+1}/{epochs} | 训练损失: {train_loss/len(train_loader.dataset):.4f} | 验证损失: {val_loss/len(val_loader.dataset):.4f} | 验证准确率: {correct/total:.4f}")

三、模型结构优化建议 #

1. 启用Dropout防止过拟合 #

原模型定义了dropout_rate但未使用，添加Dropout层：

def __init__(self, vocab_size, embedding_dim, hidden_size, output_dim, dropout_rate=0.5):
    super(MyModel, self).__init__()
    self.embedding = nn.Embedding(vocab_size, embedding_dim)
    self.dropout = nn.Dropout(dropout_rate)
    self.lstm = nn.LSTM(input_size=embedding_dim, hidden_size=hidden_size, batch_first=True, dropout=dropout_rate)
    self.linear = nn.Linear(hidden_size, output_dim)

def forward(self, input_seq, seq_lens):
    embedded = self.embedding(input_seq)
    embedded = self.dropout(embedded)
    # 后续打包序列等操作不变
    lstm_out = self.dropout(lstm_out)
    x = self.linear(lstm_out)
    return F.log_softmax(x, dim=1)

2. 使用预训练俄文词嵌入 #

俄文数据用随机初始化的Embedding效果有限，可加载FastText等预训练俄文词向量替换初始权重：

# 假设已加载预训练词向量到word2vec字典（键为子词，值为向量）
embedding_matrix = torch.zeros(5000, 300)
for idx in range(5000):
    subword = tokenizer.id_to_subword(idx)
    if subword in word2vec:
        embedding_matrix[idx] = torch.tensor(word2vec[subword])

# 替换Embedding层权重
model.embedding.weight.data.copy_(embedding_matrix)
model.embedding.weight.requires_grad = True  # 允许微调

四、输出全零的额外排查点 #

• 检查标签分布：若某类标签占比极高（如99%），模型会倾向于预测多数类，需调整数据分布；
• 验证设备一致性：确保模型和输入张量在同一设备（CPU/GPU）；
• 查看真实概率：对log_softmax输出执行torch.exp()，确认是否为数值极小的近似零，而非真正的全零。

内容的提问来源于stack exchange，提问作者Ulyana Glebova