如何用sklearn CountVectorizer提取unigram时保留对应词性标签

问题出在CountVectorizer的分词/过滤规则（小写转换、停用词过滤、标点移除）和你原句的token拆分规则不一致，导致无法直接匹配词性标签。你可以通过以下步骤准确提取带词性标签的unigram：

步骤1：利用CountVectorizer的内置属性对齐匹配 #

CountVectorizer训练后会生成vocabulary_（保留的词表）和get_stop_words()（使用的停用词集合），我们可以用这两个属性来过滤原句的token-tag对，确保和提取的unigram完全匹配：

from sklearn.feature_extraction.text import CountVectorizer

# 初始化并训练CountVectorizer
vectorizer_unigram = CountVectorizer(analyzer='word', ngram_range=(1, 1), stop_words='english')
doc = {'sent': ['Two women are embracing while holding to go packages .'], 'tags': ['NUM NOUN AUX VERB SCONJ VERB PART VERB NOUN PUNCT']}
vectorizer_unigram.fit(doc['sent'])

# 拆分原句的token和对应标签
sent_tokens = doc['sent'][0].split()
tag_tokens = doc['tags'][0].split()
token_tag_pairs = list(zip(sent_tokens, tag_tokens))

# 按照CountVectorizer的规则过滤，保留匹配的unigram和标签
filtered_pairs = []
stop_words = set(vectorizer_unigram.get_stop_words())
for token, tag in token_tag_pairs:
    lower_token = token.lower()
    # 检查该词是否被CountVectorizer保留（已自动过滤停用词、标点等）
    if lower_token in vectorizer_unigram.vocabulary_:
        filtered_pairs.append((lower_token, tag))

# 输出结果
unigram_with_tags = dict(filtered_pairs)
print(unigram_with_tags)
# 输出: {'women': 'NOUN', 'embracing': 'VERB', 'holding': 'VERB', 'packages': 'NOUN'}

步骤2：扩展到DataFrame场景 #

如果你的数据是DataFrame格式，可以定义一个处理函数，批量应用到每一行：

import pandas as pd

# 示例DataFrame
df = pd.DataFrame({
    'sentence': ['Two women are embracing while holding to go packages .', 'A man is walking his dog in the park .'],
    'pos_tags': ['NUM NOUN AUX VERB SCONJ VERB PART VERB NOUN PUNCT', 'DET NOUN AUX VERB DET NOUN ADP DET NOUN PUNCT']
})

# 先训练CountVectorizer覆盖所有句子
vectorizer_unigram.fit(df['sentence'])
stop_words = set(vectorizer_unigram.get_stop_words())

# 定义处理单条数据的函数
def extract_unigram_tags(sentence, pos_tags):
    sent_tokens = sentence.split()
    tag_tokens = pos_tags.split()
    token_tag_pairs = list(zip(sent_tokens, tag_tokens))
    
    filtered = []
    for token, tag in token_tag_pairs:
        lower_token = token.lower()
        if lower_token in vectorizer_unigram.vocabulary_:
            filtered.append((lower_token, tag))
    return dict(filtered)

# 批量处理
df['unigram_tags'] = df.apply(lambda row: extract_unigram_tags(row['sentence'], row['pos_tags']), axis=1)

print(df['unigram_tags'])

备选方案：手动对齐过滤规则 #

如果你不想依赖CountVectorizer的vocabulary_，也可以手动实现和它一致的过滤逻辑（转小写、移除标点、过滤停用词），确保token匹配：

import string
from nltk.corpus import stopwords

# 手动实现和CountVectorizer一致的过滤规则
def filter_token_tags(tokens, tags):
    stop_words = set(stopwords.words('english'))
    punctuation = set(string.punctuation)
    filtered = []
    for token, tag in zip(tokens, tags):
        lower_token = token.lower()
        if lower_token not in stop_words and lower_token not in punctuation:
            filtered.append((lower_token, tag))
    return dict(filtered)

# 调用示例
sent_tokens = doc['sent'][0].split()
tag_tokens = doc['tags'][0].split()
print(filter_token_tags(sent_tokens, tag_tokens))

注意：这种方法需要确保停用词表和CountVectorizer使用的一致（CountVectorizer默认的停用词表和nltk的略有差异），所以优先推荐第一种方法，避免匹配误差。

内容的提问来源于stack exchange，提问作者OLGJ