使用Quanteda的KWIC统计否定词后目标关键词出现次数（R语言）

问题背景 #

需要统计评论中否定词（not、never、don't、seldom、won't）后目标关键词的出现次数，用于情感分析。使用Quanteda的KWIC工具时，生成的结果数据集小于原数据集，无法统计单条评论内的多次匹配（如测试数据中第2、3条评论应返回2，但现有代码仅能返回0或1）。

测试数据与初始代码 #

library(dplyr)
library(quanteda)

text_column <- c("not safe","not safe and not listening","not safe never patient", "safe","not welcoming","nice people","corporate culture school tacos","successful words words coding","not scary")
test.df <- as.data.frame(text_column)

notwords <- c("not", "never", "don't", "seldom", "won't")

# 目标关键词字典（补全初始代码的闭合括号）
safety_dict <- dictionary(list(possafety = c("open","open-minded", "listen*", "safe*", "patien*", "underst*", "willing to help", "helpful", "tight-knit", "hear*", "engage*", "support*", "comfortable", "belong*", "welcom*", "inclu*", "value", "respect*", "always someone you can go to for questions", "accept*")))

解决方案 #

方法1：基于KWIC的分组计数法 #

通过给原数据添加唯一文档ID，将KWIC结果按文档分组计数后合并回原数据，未匹配的评论计数设为0：

# 给原数据添加文档ID
test.df <- test.df %>% mutate(docid = row_number())

# 创建语料库
corp <- corpus(test.df, text_field = "text_column", docid_field = "docid")

# 提取否定词及其后1个词的KWIC结果，匹配目标字典
kwic_matches <- kwic(corp, pattern = notwords, window = c(0, 1)) %>%
  mutate(
    # 检查否定词后窗口内的词是否属于目标字典
    is_target = as.integer(token_lookup(tokens(post), dictionary = safety_dict) == 1)
  ) %>%
  filter(is_target == 1) %>%
  group_by(docname) %>%
  summarise(neg_target_count = n()) %>%
  mutate(docname = as.integer(docname))

# 合并回原数据，填充未匹配的计数为0
result_df <- test.df %>%
  left_join(kwic_matches, by = c("docid" = "docname")) %>%
  mutate(neg_target_count = replace_na(neg_target_count, 0))

print(result_df)

方法2：基于Tokens的直接统计法 #

直接在tokens层面遍历每个词，检查其前一个词是否是否定词且当前词属于目标字典，效率更高，适合大数据集：

# 创建语料库与tokens
corp <- corpus(test.df, text_field = "text_column")
toks <- tokens(corp)

# 统计每条评论的匹配次数
neg_target_counts <- sapply(toks, function(tok) {
  neg_positions <- which(tok %in% notwords)
  # 遍历每个否定词位置，检查后一个词是否匹配目标字典
  sum(sapply(neg_positions, function(pos) {
    if (pos < length(tok)) {
      token_lookup(tokens(tok[pos+1]), dictionary = safety_dict) == 1
    } else {
      FALSE
    }
  }))
})

# 添加计数到原数据
test.df$neg_target_count <- neg_target_counts

print(test.df)

两种方法都能得到预期结果：第2、3条评论的neg_target_count为2，其余评论按实际匹配情况返回0或1。

内容的提问来源于stack exchange，提问作者JT Gardner