Quanteda中词频与文档词频统计的去重叠问题技术问询

背景说明 #

这是包含4份文档的测试语料库，目标是用词典识别语料中词汇/短语的词频，以及它们出现的文档数量。目前存在以下问题：

• 词汇“Australians”同时出现在peep和indig两个词典键中，但按设计各键内容应互斥
• 类似地，“Australia”（oz和Australia Post）、“foreign”（foreign和multinat）、“farm/farmers”（dairy和farmers）也分别出现在两个键中，但应仅被计数一次
• 预期总词频统计结果可从kwic表的“pattern”列提取（如下方x2输出）；另外注意“industry”虽出现，但未被分配到industry键（原描述提及该词因关联indig键导致此问题）
• dairy是出现频率最高的键，出现在3份文档中，可通过kwic表“doc names”列的唯一行统计得出

技术问询 #

1. 该方法是否存在可能影响输出准确性的问题？
2. 是否有更简洁高效的方法实现此目标？
3. 如何从kwic表中提取等效于textstat的词频统计数据？

原测试代码 #

library(quanteda)
library(quanteda.textstats)

txt <- c(doc1 = "A significant percent of all farms in Australia, are dairy. 
         Although there are a lot of dairy farms in this country, 
         it is not the biggest farm industry. The life of a farmer is not easy, a dairy 
        farmer has to be an early riser. ",
         doc2 = "Australian people like milk so a healthy dairy industry is important in 
         our country",
         doc3 = "Dairy and sheep farms developed at the expense of Indigenous 
         Australians. Further many companies  are now foreign-owned",
         doc4 = "Some farmers are lucky to receive a service from Australia Post. Mail is 
         sent to many foreign countries and received more quickly than 
         delivered in some locations in Australia.")

x <- x %>%
  tokens_compound(phrase("dairy farmers"), concatenator = " ") %>%
  tokens_compound(phrase("dairy farms"), concatenator = " ") %>%
  tokens_compound(phrase("dairy farm"), concatenator = " ") %>%
  tokens_compound(phrase("dairy farming"), concatenator = " ") %>%
  tokens_compound(phrase("dairy industry"), concatenator = " ") %>%
  tokens_compound(phrase("indigenous australians"), concatenator = " ") %>%
  tokens_compound(phrase("australia post"), concatenator = " ") %>%
  tokens_compound(phrase("dairy farmer"), concatenator = " ")
x

dict <- dictionary(list(multinat = c("offshore petroleum companies", "foreign- 
         owned", "foreign owned", "foreign companies", "multinational", "multinational 
         oil companies", "multinationals", "transnational"),
         dairy = c("dairy farmers", "dairy farms","dairy farm","dairy farming","dairy 
         industry", "dairy farmer","dairy", "milk"),
         auspost = "australia post",
         oz = c("australia", "this country", "our country"),
         farmers = c("farmers", "farmer", "farm", "farms"),
         foreign = c("foreign", "foreigner", "foreigners"), 
         business =c("small business", "business", "businesses", "company", "companies"),
         indig = c("aboriginal", "aboriginals", "indigenous australians", "torres 
         strait"),
         peep = c("australians", "people of australia", "australian people", "people of 
         this nation", "people of this country"),
         industry = c("industry", "industries")))

kwicdict <- kwic(x, pattern = dict, window = 4)
write.csv(kwicdict, "D:/Output/TEST.csv")

DF <- read.csv("D://Output/TEST.csv",header=T)

## 统计KWIC表中pattern列的词频
x2 <- DF[,8]
table(x2)
# x2
# auspost business    dairy  farmers  foreign    indig industry multinat  oz  peep    
#        1        1        6        5        1        1        1        1     5    2 

问题解答 #

1. 现有方法的准确性问题 #

• 重复计数：词典内大量词汇/短语重叠（如dairy包含"dairy farmer"，farmers也包含"farmer"），kwic()会匹配所有符合条件的键，导致同一文本片段被多次计数，违背互斥设计
• 匹配优先级缺失：没有设置键的匹配优先级，出现词汇被错误分配的情况（如“industry”未匹配到对应键）
• 冗余操作风险：多次手动调用tokens_compound()易遗漏短语，且写入再读取CSV可能引入编码、格式误差，影响后续统计

2. 更简洁高效的实现方法 #

直接使用tokens_lookup()的互斥匹配功能，结合批量复合短语处理，避免冗余操作：

library(quanteda)
library(quanteda.textstats)

# 定义语料库
txt <- c(
  doc1 = "A significant percent of all farms in Australia, are dairy. 
         Although there are a lot of dairy farms in this country, 
         it is not the biggest farm industry. The life of a farmer is not easy, a dairy 
        farmer has to be an early riser. ",
  doc2 = "Australian people like milk so a healthy dairy industry is important in 
         our country",
  doc3 = "Dairy and sheep farms developed at the expense of Indigenous 
         Australians. Further many companies  are now foreign-owned",
  doc4 = "Some farmers are lucky to receive a service from Australia Post. Mail is 
         sent to many foreign countries and received more quickly than 
         delivered in some locations in Australia."
)

# 批量定义复合短语
compound_phrases <- phrase(c(
  "dairy farmers", "dairy farms", "dairy farm", "dairy farming", 
  "dairy industry", "indigenous australians", "australia post", "dairy farmer"
))

# 分词并一次性处理复合短语
toks <- tokens(txt) %>%
  tokens_compound(compound_phrases, concatenator = " ")

# 定义词典（保留原内容）
dict <- dictionary(list(
  multinat = c("offshore petroleum companies", "foreign-owned", "foreign owned", "foreign companies", "multinational", "multinational oil companies", "multinationals", "transnational"),
  dairy = c("dairy farmers", "dairy farms","dairy farm","dairy farming","dairy industry", "dairy farmer","dairy", "milk"),
  auspost = "australia post",
  oz = c("australia", "this country", "our country"),
  farmers = c("farmers", "farmer", "farm", "farms"),
  foreign = c("foreign", "foreigner", "foreigners"), 
  business =c("small business", "business", "businesses", "company", "companies"),
  indig = c("aboriginal", "aboriginals", "indigenous australians", "torres strait"),
  peep = c("australians", "people of australia", "australian people", "people of this nation", "people of this country"),
  industry = c("industry", "industries")
))

# 互斥匹配词典（按键顺序优先匹配，避免重复计数）
toks_lookup <- tokens_lookup(toks, dict, exclusive = TRUE)

# 统计词频
freq_stats <- textstat_frequency(dfm(toks_lookup))
print(freq_stats)

# 统计各键出现的文档数
doc_count <- docfreq(dfm(toks_lookup))
print(doc_count)

优势：

• exclusive = TRUE确保每个token仅匹配一个优先级最高的键（按词典键顺序）
• 批量处理复合短语，减少冗余代码
• 直接调用quanteda内置统计函数，无需读写文件，避免格式误差

3. 从kwic表提取等效textstat的词频数据 #

无需读写CSV，直接从kwic对象提取并格式化数据：

# 从kwic对象提取pattern列并统计词频
kwic_freq <- table(kwicdict$pattern)

# 转换为与textstat_frequency对齐的格式
kwic_freq_df <- data.frame(
  feature = names(kwic_freq),
  frequency = as.integer(kwic_freq),
  rank = rank(-as.integer(kwic_freq), ties.method = "min"),
  docfreq = sapply(names(kwic_freq), function(x) length(unique(kwicdict$docname[kwicdict$pattern == x]))),
  group = "all"
)
print(kwic_freq_df)

注意：若要实现textstat的互斥计数效果，需先通过tokens_lookup(exclusive = TRUE)处理分词，再生成kwic对象，否则仍会存在重复计数问题。

内容的提问来源于stack exchange，提问作者bgreen