大语料/DFM/DTM下计算特定术语余弦相似度的R技术问题

一、只算指定术语与其他术语的相似度，解决R崩溃问题 #

不用计算全量术语相似度矩阵，直接用textstat_simil的selection参数就能精准生成你需要的结果，内存占用会大幅降低，不会再崩溃。

用你的数据子集举个例子：

library(quanteda)

# 加载你的数据
twitter_df <- structure(list(`twitterdata[1:50, ]` = c("matter time", "beatl", 
"craze left wing hippi weirdo freak rant observ", " officialmonstax", 
"bienvenido club fan dedicado informar apoyar hermosa talentosa estrella mexicana angelicaval sigueno", 
"boa lagoa", "forget thing hurt lesson learn make mistak can never never regret thing made smile", 
"vynox come soon", "offici th parkad downtown sandiego locat next petco park histor gaslamp quarter", 
"offici salvat armi chicago metropolitan divis largest direct provid social servic state illinoi", 
"jackson ude journalist skill practition field polit communic media public polici polit manag", 
"encourag motiv inspir lead execut", " negat world", "seattl bremerton elliot thorsen ethorsen", 
"laxxxxx", "west geauga high school hockey", "baltimor born rais dundalk", 
"eight thirti thirteen", "talk show produc newstalk humber colleg radio broadcast graduat dolphin magic blue jay aggi mapl leaf hotspur fan", 
"sdsu ", "read book host literari lair fix can also found gomer product now", 
"va dancer yrs old basketbal player volleybal setter yes may loser best damn loser ever meet", 
"can star magic workshopp", "offici page strongsvill ladi mustang varsiti soccer team", 
"queen hous wife mom sis aunt garden chef caregiv teacher counselor pro life liberti happi godfear christfollow biblebeliev john ", 
"sport star war hous music", "alexandria atletico jenni tcw ", 
"fun guitar", "jesus dont give fail everyth mean noth realiz good bro aredhel nargothrond elf name", 
"life give lemon return ask zayn malik pleas ", "collector thing beauti past present find rebelmous vintagedressparlour", 
"streamer aspir musician fit health", "artist illustr design d model busi commiss open charact simpl background", 
"musico poeta loco artista naturaleza", "totalment fascinado afeccion emocional biologica cuerpo humano eterno enamorado letra lavida dio guia hoy manana siempr", 
"girl mani dream shawnmend girlfriend dream", "alway look delici bigup friend", 
"sassi sexi wild lover music writer blogger product junki pierc tattoo enthusiast hopeless romant makeup artist", 
"stop useless start pizza", "keep upto date fixtur result latest news updat across gfa leagu", 
"ez lab onlin portal various nabl iso certifi diagnost lab avail provid qualiti assur healthcar consum", 
" that flick tho", " pinch dinosaurio risueno guey music drug physiotherapi campus puebla", 
"look sharp cut edg design get", "proud eph gopher track alum bs kin umn mba sp mgt cuc ski racer climber around athlet ao", 
"professor nerdi abound warn fond book turn brain", "gotta risk get biscuit mdp presleyy aspir sing avocado ladi", 
"dragonapothek ist onlin apothek allgemein dieser bieten manner sexuell gesundheit medizin kamagra", 
"help compani individu discov fit clariti therapist connect agent outgo introvert flaw believ husband dad bbq er", 
"keep negat aliv babi termin hate spread posit cudfam cudlif"
)), row.names = c(NA, -50L), class = "data.frame")

# 构建语料、DFM并计算TF-IDF
my_corpus <- corpus(twitter_df, text_field = "twitterdata[1:50, ]")
my_dfm <- dfm(my_corpus)
myTFIDF <- dfm_tfidf(my_dfm)

# 指定你关心的术语，比如这里用数据里的"left""life"，替换成你的"conservative""liberal"
target_terms <- c("left", "life")
# 只计算这些术语和其他所有术语的余弦相似度
test_cosine <- textstat_simil(myTFIDF, margin = "terms", method = "cosine", selection = target_terms)

# 转成矩阵查看结果
as.matrix(test_cosine)

二、搞清楚stringdist的余弦相似度到底算的是什么 #

stringdist的余弦相似度和你需要的语境相似度完全是两回事：

• 它不需要语料，直接对比两个字符串的字符组成，把字符串拆成字符n-gram集合，计算集合的余弦相似度，本质是看两个词的拼写像不像，不是语义或语境上的相似。
• 而textstat_simil的余弦相似度是基于术语在文档中的TF-IDF向量，衡量的是术语在语料里的语境分布相似性——也就是你要的修辞/语义层面的相似度。

所以如果你的需求是修辞相似度，别用stringdist，还是得基于语料的向量计算。

三、其他备选方案 #

要是上面的方法还不行，试试这些：

• 手动计算子矩阵相似度：先提取目标术语的TF-IDF向量，再和所有术语的向量计算余弦值，代码示例：

target_vec <- myTFIDF[target_terms, ]
all_terms_vec <- myTFIDF
# 手动计算余弦相似度
cos_sim <- t(target_vec) %*% t(all_terms_vec) / (sqrt(rowSums(target_vec^2)) %*% t(sqrt(rowSums(all_terms_vec^2))))

• 用最新版quanteda：新版本对稀疏矩阵的计算优化更好，能处理更大规模的DFM。
• 分块计算：把DFM分成若干小块，分别计算目标术语和每块术语的相似度，最后合并结果，适合超大规模语料。

内容的提问来源于stack exchange，提问作者lwe