如何在FreqDist中将语义相同的'pakistan'与'pak'合并统计

问题背景 #

处理推文Excel数据做文本分析时，发现词频分布中pakistan（第2高频）和pak（第4高频）语义一致，需要将二者合并为同一词汇统计。

原代码 #

import pandas as pd
from nltk.tokenize import RegexpTokenizer
import nltk
from nltk.probability import FreqDist

db=pd.read_excel(r'hello world.xlsx')
db['Sentence'] = db['Sentence'].astype(str).str.lower() # 转小写

regexp = RegexpTokenizer('\w+')
db['Sentence_token']=db['Sentence'].apply(regexp.tokenize)

stopwords = nltk.corpus.stopwords.words("english")
my_stopwords = []
stopwords.extend(my_stopwords)

db['Sentence_token'] = db['Sentence_token'].apply(lambda x: [item for item in x if item not in stopwords])   
db['Sentence_string'] = db['Sentence_token'].apply(lambda x: ' '.join([item for item in x if len(item)>0])) 

# 原代码此处变量名错误：file应为db
all_words = ' '.join([word for word in db['Sentence_string']])

tokenized_words = nltk.tokenize.word_tokenize(all_words)
fdist = FreqDist(tokenized_words)

db['Sentence_string_fdist'] = db['Sentence_token'].apply(lambda x: ' '.join([item for item in x if fdist[item] >= 2])) # 过滤低频词
db[['Sentence', 'Sentence_token', 'Sentence_string', 'Sentence_string_fdist']]

fdist

原输出 #

FreqDist({'xxx': 870, 'pakistan': 466, 'xxx': 268, 'pak': 253, 'xxx': 253, 'xxx': 251, 'xxx': 237, ...})

解决方案 #

核心思路是在去停用词之后、生成词频统计之前，添加同义词汇映射替换逻辑，将pak统一替换为pakistan（可根据需求调整映射方向）。

修改后完整代码 #

import pandas as pd
from nltk.tokenize import RegexpTokenizer
import nltk
from nltk.probability import FreqDist

db=pd.read_excel(r'hello world.xlsx')
db['Sentence'] = db['Sentence'].astype(str).str.lower() # 转小写

regexp = RegexpTokenizer('\w+')
db['Sentence_token']=db['Sentence'].apply(regexp.tokenize)

stopwords = nltk.corpus.stopwords.words("english")
my_stopwords = []
stopwords.extend(my_stopwords)

# 1. 去停用词
db['Sentence_token'] = db['Sentence_token'].apply(lambda x: [item for item in x if item not in stopwords])   

# 2. 同义词汇替换：可扩展字典处理更多词汇
word_mapping = {'pak': 'pakistan'}
db['Sentence_token'] = db['Sentence_token'].apply(
    lambda x: [word_mapping[item] if item in word_mapping else item for item in x]
)

# 3. 生成拼接字符串
db['Sentence_string'] = db['Sentence_token'].apply(lambda x: ' '.join([item for item in x if len(item)>0])) 

# 4. 统计词频
all_words = ' '.join([word for word in db['Sentence_string']])
tokenized_words = nltk.tokenize.word_tokenize(all_words)
fdist = FreqDist(tokenized_words)

# 5. 过滤低频词
db['Sentence_string_fdist'] = db['Sentence_token'].apply(lambda x: ' '.join([item for item in x if fdist[item] >= 2])) 

# 查看结果
db[['Sentence', 'Sentence_token', 'Sentence_string', 'Sentence_string_fdist']]
print(fdist)

关键修改说明 #

• 新增word_mapping字典，可灵活添加更多同义词汇映射（如{'usa': 'united states'}）
• 替换逻辑放在去停用词之后，确保只替换有效词汇，避免干扰停用词处理
• 修正原代码中file变量名错误为db，避免运行报错

修改后，词频统计中pakistan的频次将变为466+253=719，不再单独显示pak。

内容的提问来源于stack exchange，提问作者M Haider Bin Amir