如何在Python/R中获取多n值（1-6）n-gram及高频短语与单词

嗨，我来帮你搞定这个问题！不管是用Python还是R，都能轻松提取从1到6的n-gram，找出最常见的单词/短语，甚至能识别出最长的合理短语。下面我给你一步步演示具体实现：

我们可以用nltk库生成n-gram，结合collections.Counter统计频率，再通过**点互信息（PMI）**判断短语的合理性（值越高说明越可能是一个连贯的短语）。

步骤1：安装并导入依赖库 #

from nltk.util import ngrams
from collections import Counter
import string
from nltk.tokenize import word_tokenize
import nltk
from math import log2

# 下载分词所需的资源
nltk.download('punkt')

步骤2：预处理文本 #

先把文本转小写、分词，去掉标点符号：

# 示例文本
text = "fri evening commute can be long. some people avoid fri evening commute by choosing off-peak hours. there are much less traffic during off-peak."

# 预处理流程
tokens = word_tokenize(text.lower())
tokens = [token for token in tokens if token not in string.punctuation]

步骤3：提取1-6的n-gram并统计频率 #

# 定义n的范围（1到6）
n_range = range(1, 7)
all_ngrams = {}

for n in n_range:
    # 生成n-gram并转为字符串格式
    n_grams = ngrams(tokens, n)
    n_grams_str = [' '.join(gram) for gram in n_grams]
    # 统计频率，取前5个最常见的
    counter = Counter(n_grams_str)
    all_ngrams[f"{n}-gram"] = counter.most_common(5)

# 打印结果
print("各n-gram最常见的内容：")
for n_type, counts in all_ngrams.items():
    print(f"\n*{n_type}*")
    for phrase, count in counts:
        print(f"- `{phrase}`: 出现{count}次")

步骤4：提取最长合理短语 #

用PMI判断短语的连贯性，筛选出最长的高PMI短语：

# 先统计单个单词的频率
unigram_counts = Counter(tokens)
total_tokens = len(tokens)

# 计算PMI的函数
def calculate_pmi(gram, n, counter):
    p_gram = counter[gram] / total_tokens
    words = gram.split()
    # 计算每个单词概率的乘积
    p_words_product = 1.0
    for word in words:
        p_words_product *= unigram_counts[word] / total_tokens
    # 避免除以0的情况
    return log2(p_gram / p_words_product) if p_words_product != 0 else 0

# 收集所有高PMI的短语
max_pmi_phrases = []
for n in range(2, 7):  # 从2-gram开始，因为1-gram是单个单词
    n_grams_str = [' '.join(gram) for gram in ngrams(tokens, n)]
    counter = Counter(n_grams_str)
    # 取前10个高频短语计算PMI
    for phrase, count in counter.most_common(10):
        pmi = calculate_pmi(phrase, n, counter)
        # 设定PMI阈值（比如>1，说明比随机组合更连贯）
        if pmi > 1:
            max_pmi_phrases.append((phrase, n, count, pmi))

# 按短语长度从长到短、PMI从高到低排序
max_pmi_phrases.sort(key=lambda x: (-x[1], -x[3]))

print("\n*最长合理短语（基于PMI）*：")
for phrase, n, count, pmi in max_pmi_phrases[:3]:
    print(f"- `{phrase}`（{n}-gram）: 出现{count}次，PMI={pmi:.2f}")

Python示例输出片段 #

各n-gram最常见的内容：

*1-gram*
- `fri`: 出现2次
- `evening`: 出现2次
- `commute`: 出现2次
- `off-peak`: 出现2次
- `be`: 出现1次

*3-gram*
- `fri evening commute`: 出现2次
- `off-peak hours`: 出现1次
- `much less traffic`: 出现1次
...

最长合理短语（基于PMI）：
- `fri evening commute`（3-gram）: 出现2次，PMI=2.58

R中可以用tidytext包快速生成n-gram，结合dplyr做数据处理，同样用PMI识别合理短语。

步骤1：安装并导入依赖库 #

install.packages(c("tidytext", "dplyr", "stringr", "tidyr"))
library(tidytext)
library(dplyr)
library(stringr)
library(tidyr)

步骤2：预处理文本 #

# 示例文本
text <- "fri evening commute can be long. some people avoid fri evening commute by choosing off-peak hours. there are much less traffic during off-peak."

# 预处理：分词、小写、去标点
text_df <- tibble(text = text) %>%
  unnest_tokens(word, text, token = "words", to_lower = TRUE) %>%
  filter(!str_detect(word, paste0("[", str_escape(paste(punctuation, collapse = "")), "]")))

步骤3：提取1-6的n-gram并统计频率 #

# 定义n的范围
n_range <- 1:6
all_ngrams <- list()

for(n in n_range) {
  ngram_df <- text_df %>%
    unnest_tokens(ngram, word, token = "ngrams", n = n) %>%
    count(ngram, sort = TRUE) %>%
    head(5) # 取前5个最常见的
  all_ngrams[[paste0(n, "-gram")]] <- ngram_df
}

# 打印结果
cat("各n-gram最常见的内容：\n")
for(n_type in names(all_ngrams)) {
  cat("\n*", n_type, "*\n")
  print(all_ngrams[[n_type]], row.names = FALSE)
}

步骤4：提取最长合理短语 #

# 统计单个单词的频率和概率
unigram_counts <- text_df %>%
  count(word, sort = TRUE) %>%
  mutate(p = n / sum(n))

# 计算PMI的函数
calculate_pmi <- function(ngram_df, n) {
  # 拆分n-gram为单个单词
  split_df <- ngram_df %>%
    separate(ngram, into = paste0("word", 1:n), sep = " ")
  
  # 关联每个单词的概率
  pmi_df <- split_df
  for(i in 1:n) {
    pmi_df <- pmi_df %>%
      left_join(unigram_counts, by = setNames("word", paste0("word", i))) %>%
      rename(!!paste0("p", i) := p)
  }
  
  # 计算PMI
  pmi_df %>%
    mutate(p_gram = n / nrow(text_df),
           # 计算单词概率的乘积
           p_product = Reduce(`*`, select(., starts_with("p"))),
           pmi = log2(p_gram / p_product)) %>%
    # 合并回n-gram字符串
    unite(ngram, starts_with("word"), sep = " ") %>%
    select(ngram, count = n, pmi) %>%
    filter(!is.infinite(pmi)) # 过滤无效值
}

# 收集所有高PMI短语
max_pmi_phrases <- list()
for(n in 2:6) {
  ngram_df <- text_df %>%
    unnest_tokens(ngram, word, token = "ngrams", n = n) %>%
    count(ngram, sort = TRUE) %>%
    head(10)
  max_pmi_phrases[[n]] <- calculate_pmi(ngram_df, n) %>% filter(pmi > 1)
}

# 合并并排序
max_pmi_phrases <- bind_rows(max_pmi_phrases) %>%
  mutate(n = str_count(ngram, " ") + 1) %>% # 计算短语长度
  arrange(desc(n), desc(pmi))

cat("\n*最长合理短语（基于PMI）*：\n")
print(max_pmi_phrases %>% head(3), row.names = FALSE)

R示例输出片段 #

各n-gram最常见的内容：

* 1-gram *
    n
fri  2
evening  2
commute  2
off-peak  2
be  1

* 3-gram *
                 n
fri evening commute  2
off-peak hours  1
much less traffic  1
...

最长合理短语（基于PMI）：
                 count      pmi n
fri evening commute     2 2.58496 3

内容的提问来源于stack exchange，提问作者santoku