如何在划分数据集时将验证集独有单词级数据追加至训练集？

样本输入（df） #

query           word                    label               tag
polish          ['polish']              ['other']           [10]
angle grinder   ['angle', 'grinder']    ['other', 'other']  [10, 10]
vaccum cleaner  ['vaccum', 'cleaner']   ['other', 'other']  [10, 10]

划分后数据集 #

训练集 #

query           word                    label               tag
polish          ['polish']              ['other']           [10]
angle grinder   ['angle', 'grinder']    ['other', 'other']  [10, 10]

验证集 #

query           word                    label               tag
vaccum cleaner  ['vaccum', 'cleaner']   ['other', 'other']  [10, 10]

期望输出 #

query           word                    label               tag
polish          ['polish']              ['other']           [10]
angle grinder   ['angle', 'grinder']    ['other', 'other']  [10, 10]
vaccum          ['vaccum']              ['other']           [10]
cleaner         ['cleaner']             ['other']           [10]

尝试的代码及问题 #

尝试了以下代码，但仅能追加第一个单词，标签也直接原样追加：

train_data = df.sample(frac=1 - 0.15, random_state=20)
val_data = df.drop(index=train_data.index)

val_words = set(word for words in val_data['word'] for word in words)
train_words = set(word for words in train_data['word'] for word in words)
new_words = val_words - train_words
new_rows = []
for index, row in val_data.iterrows():
    words = row['word']
    if any(word in new_words for word in words):
        for word, label, tag in zip(words, row['label'], row['tag']):
            new_rows.append((word, [word], [label], [tag]))
train_data = train_data.append(pd.DataFrame(new_rows, columns=train_data.columns), 
ignore_index=True)

解决方案 #

问题出在判断逻辑和处理方式上：原代码中any(word in new_words for word in words)只要行内有一个单词属于训练集未收录的，就会把整行所有单词都追加，且没有针对每个单词单独判断是否需要追加。另外append方法在新版pandas中已被弃用，建议使用pd.concat。

修改后的代码如下：

import pandas as pd

# 划分数据集
train_data = df.sample(frac=1 - 0.15, random_state=20)
val_data = df.drop(index=train_data.index)

# 提取训练集所有单词存入集合，用于快速判断
train_words = set(word for words in train_data['word'] for word in words)
new_rows = []

# 遍历验证集每一行，逐个处理单词
for _, row in val_data.iterrows():
    for word, label, tag in zip(row['word'], row['label'], row['tag']):
        # 仅处理训练集未收录的单词
        if word not in train_words:
            new_rows.append({
                'query': word,
                'word': [word],
                'label': [label],
                'tag': [tag]
            })
            # 将已追加的单词加入集合，避免重复处理
            train_words.add(word)

# 合并原训练集与新生成的记录
train_data = pd.concat([train_data, pd.DataFrame(new_rows)], ignore_index=True)

代码说明 #

• 直接针对每个单词判断是否属于训练集未收录的，避免无效追加整行内容
• 使用pd.concat替代已弃用的append方法，符合pandas新版本规范
• 每追加一个单词就更新训练集单词集合，防止验证集其他行出现相同单词时重复追加

内容的提问来源于stack exchange，提问作者Prateek Singh