使用quanteda的dfm_lookup()时如何获取匹配到的原始词条

dfm_lookup()默认仅返回词典类目的匹配计数，要同时获取匹配的原始token，可结合tokens_lookup()的过滤功能实现，完整实现代码如下：

library(quanteda)
# 示例词典
dict_ex <- dictionary(list(
  christmas = c("Christmas", "Santa", "holiday"),
  opposition = c("Opposition", "reject", "notincorpus"),
  taxglob = "tax*",
  taxregex = "tax.+$",
  country = c("United_States", "Sweden")
))
# 先生成tokens对象（保留原始token信息，不要直接转dfm）
toks <- tokens(
  c(
    "My Christmas was ruined by your opposition tax plan.",
    "Does the United_States or Sweden have more progressive taxation?"
  ),
  remove = stopwords("english")
)
# 原始dfm对象
dfmat_ex <- dfm(toks)
# 遍历所有词典类目生成匹配结果
result_list <- lapply(names(dict_ex), function(key) {
  # 过滤当前类目匹配到的token，若有正则规则需添加valuetype = "regex"参数
  match_toks <- tokens_lookup(
    toks, 
    dictionary(setNames(list(dict_ex[[key]]), key)), 
    output = "filter", 
    nomatch = ""
  )
  # 整理每个文档的匹配词列表，无匹配返回NA
  match_str <- sapply(match_toks, function(x) {
    x <- x[x != ""]
    ifelse(length(x) == 0, NA_character_, paste(x, collapse = ", "))
  })
  # 获取匹配计数，若有正则规则需添加valuetype = "regex"参数
  count <- as.integer(dfm_lookup(dfmat_ex, dictionary(setNames(list(dict_ex[[key]]), key))))
  # 返回计数+匹配词的两列结果
  data.frame(count, match_str, row.names = docnames(dfmat_ex))
})
# 合并所有类目结果并调整列名
names(result_list) <- names(dict_ex)
final_res <- do.call(cbind, result_list)
colnames(final_res) <- paste0(rep(names(dict_ex), each = 2), c("", ".match"))

运行后输出的final_res即为你需要的扩展结果：

print(final_res)
#       christmas christmas.match opposition opposition.match taxglob taxglob.match taxregex taxregex.match country      country.match
# text1         1       Christmas          1       Opposition       1            tax        0           <NA>       0               <NA>
# text2         0            <NA>          0             <NA>       1       taxation        0           <NA>       2 United_States, Sweden

注意事项 #

• dfm属于稀疏数值矩阵，仅支持存储数值类型数据，你需要的带匹配字符串的扩展结果只能以数据框格式存储，原始计数dfm可单独通过dfm_lookup(dfmat_ex, dict_ex)生成使用。
• 若词典类目包含正则表达式匹配规则，需要在tokens_lookup和dfm_lookup中添加valuetype = "regex"参数，否则默认按glob通配规则匹配。

内容的提问来源于stack exchange，提问作者Ivo