使用quanteda按日期统计词频及共现词占比并可视化

样例数据集 #

用户提供的分析数据集：

df <- data.frame (text  = c("The investors and their supporters shall invest and do something mostly invest",
         " Shall we tell the investors to invest ?",  "Investors shall invest.",
         "Investors may sometimes invest","spend what Investor Do"),
                  date = c("10/12/2022", "10/12/2022", "10/12/2022","11/12/2022","12/12/2022"))

需求1：按日期统计词频并可视化 #

实现代码 #

library(quanteda)
library(ggplot2)

# 1. 构建语料库并绑定日期变量
corp <- corpus(df, text_field = "text")
docvars(corp, "date") <- df$date

# 2. 分词并生成按日期聚合的文档-词项矩阵
toks <- tokens(corp, remove_punct = TRUE, remove_numbers = TRUE) %>%
  tokens_tolower()
dfm_grouped <- dfm(toks) %>%
  dfm_group(groups = date)

# 转换为长格式数据框用于可视化
freq_df <- convert(dfm_grouped, to = "data.frame") %>%
  tidyr::pivot_longer(-date, names_to = "token", values_to = "frequency")

# 3. 绘制指定词（以invest为例）的日期频次图
ggplot(freq_df[freq_df$token == "invest", ], aes(x = date, y = frequency)) +
  geom_col(fill = "#2c3e50") +
  labs(title = "invest词频随日期变化", x = "日期", y = "频次") +
  theme_minimal()

说明 #

通过dfm_group()按日期聚合词频，转换为长数据框后可灵活筛选目标词进行可视化；若需查看所有词的频次分布，去掉freq_df[freq_df$token == "invest", ]的筛选条件即可。

需求2：按日期统计investor与invest的共现占比并可视化 #

实现代码 #

# 1. 按日期分组分词结果，统计每组共现占比
doc_group <- groups(toks, groups = date)
group_stats <- lapply(doc_group, function(g) {
  has_investor <- sapply(g, function(t) "investor" %in% t)
  has_invest <- sapply(g, function(t) "invest" %in% t)
  co_occur_count <- sum(has_investor & has_invest)
  total_docs <- length(g)
  data.frame(date = unique(docvars(corp, "date")[names(g)]),
             co_occur_ratio = co_occur_count / total_docs)
})
co_occur_df <- do.call(rbind, group_stats)

# 2. 绘制共现占比图
ggplot(co_occur_df, aes(x = date, y = co_occur_ratio)) +
  geom_col(fill = "#3498db") +
  scale_y_continuous(labels = scales::percent_format()) +
  labs(title = "investor与invest的共现占比随日期变化", x = "日期", y = "共现占比") +
  theme_minimal()

说明 #

共现占比定义为「同时包含两个词的文档数 / 当日总文档数」，通过遍历日期分组的分词结果，统计每组的共现文档数与总文档数后计算占比，最终可视化展示变化趋势。

需求3：找出invest出现时的前25%高频词（替代tm::findAssocs()） #

quanteda中可通过textstat_simil()实现类似词关联分析，结合词频筛选目标词：

实现代码 #

# 1. 生成完整DFM，筛选包含invest的文档
dfm_full <- dfm(toks)
invest_docs <- which(dfm_full[, "invest"] > 0)
dfm_invest <- dfm_full[invest_docs, ]

# 2. 计算词项与invest的关联度（采用点互信息PMI）
assocs <- textstat_simil(dfm_invest, dfm_invest[, "invest"], method = "pmi")

# 3. 结合词频排序，取前25%高频词
freq_invest <- colSums(dfm_invest)
assoc_df <- data.frame(
  token = names(freq_invest),
  frequency = freq_invest,
  pmi = as.vector(assocs)
) %>%
  filter(token != "invest") %>%
  arrange(desc(frequency))

# 提取前25%的词
top_25_pct <- assoc_df[1:ceiling(nrow(assoc_df)*0.25), ]
print(top_25_pct)

说明 #

先筛选出包含invest的文档构建DFM，再用textstat_simil()计算词与invest的PMI关联度，最后按词频排序取前25%的词；若更侧重关联强度，可改为按pmi字段排序。

内容的提问来源于stack exchange，提问作者R_Student