Fasttext、Longformer与Doc2vec余弦相似度结果不一致的原因排查

我用Doc2vec模型计算网站文本数据集中样本间的余弦相似度，原本期望换成基于自有数据训练的Fasttext或预训练Longformer后，相似度结果能大致一致（已知不会完全相同）。但实际结果显示：Doc2vec与Longformer、Doc2vec与Fasttext的成对余弦相似度呈强负相关，而Longformer与Fasttext呈正相关。想请教这种现象是存在合理原因，还是我的代码有错误？

# PREPARE DATA
website_df = pd.read_csv(data_path+'cleaned_docdf_may2023.csv')
website_df[['documents_cleaned','website']]=website_df[['documents_cleaned','website']].astype(str)
website_df['documents_cleaned']=website_df['documents_cleaned'].str.lower()
website_df['documents_cleaned']=website_df['documents_cleaned'].str.strip()

####################### 
# Train Doc2vec model
#######################

# Clean data for model input (trim long docs, lower case, tokenize):
counter = 0
all_docs = []
all_docs_simple = []
for train_doc in website_df.documents_cleaned:
    doc = train_doc[:150000] if len(train_doc) > 150000 else train_doc
    # clean using simple_preprocess for Fasttext model input
    simple_pre = gensim.utils.simple_preprocess(train_doc)
    doc = remove_stopwords(doc)
    doc_tokens =nltk.word_tokenize(doc.lower())
    all_docs.append(doc_tokens)
    all_docs_simple.append(simple_pre)
    if (counter%100) == 0:
        print("{0} .. len: {1}".format(counter,len(doc)))
    counter += 1

# Creating all tagged documents
documents_websites = [TaggedDocument(doc, [i]) for i, doc in enumerate(all_docs)]
documents_simplepre_websites = [TaggedDocument(doc, [i]) for i, doc in enumerate(all_docs_simple)]
print("	. Run model")
doc2vec_model_websites = Doc2Vec(documents = documents_websites,
                vector_size=700,
                window=7,
                min_count =3)
print("	. Done")
doc2vec_model_websites.save(data_path + "doc2vec_websites.model")

# Grab document level vectors
vectors_d2v_websites = doc2vec_model_websites.dv.get_normed_vectors()

######################### 
# FASTTEXT MODEL
#########################

# create and save Fasttext input
sent_df_websites=pd.Series(documents_simplepre_websites)
with open(data_path + 'sentences_websites', 'a') as f:
    df_string = sent_df_websites.to_string(header=False, index=False)
    f.write(df_string)

# Skipgram model (use comparable model parameters to doc2vec model) :
ft_model_sg_websites = fasttext.train_unsupervised(input=sent_df_websites, model='skipgram', ws=7, epoch=10, minCount=3)
ft_model_sg_websites.save_model(data_path + "ft_websites_sg.bin")
# cbow model (use comparable model parameters to doc2vec model) :
ft_model_cbow_websites = fasttext.train_unsupervised(input=data_path + 'sentences_websites', model='cbow', ws=7, epoch=10, minCount=3)
ft_model_cbow_websites.save_model(data_path + "ft_websites_cbow.bin")

def generateVector(sentence):
    return ft.get_sentence_vector(sentence)

ft = ft_model_sg_websites
website_df['embeddings_sg'] = website_df['documents_cleaned'].apply(generateVector)
embeddings_sg_website=website_df['embeddings_sg']
ft = ft_model_cbow_websites
website_df['embeddings_cbow'] = website_df['documents_cleaned'].apply(generateVector)
embeddings_cbow_website=website_df['embeddings_sg']  # 此处存在赋值错误

#########################
# LONGFORMER 
#########################

model_name = 'allenai/longformer-base-4096'
tokenizer = LongformerTokenizer.from_pretrained(model_name)
model = LongformerModel.from_pretrained(model_name)

def get_longformer_embeddings(text):
    encoded_input = tokenizer(text, return_tensors="pt", max_length=4096, truncation=True)
    output = model(**encoded_input, output_hidden_states=True)
    embeddings = output.last_hidden_state
    avg_emb = embeddings.mean(dim=1)
    return avg_emb.cpu().detach().numpy()

def get_cosine_sim(a,b):
    value = dot(a, b)/(norm(a)*norm(b))
    return value

# subset data for speed during pilot
website_subset = website_df[:100]
website_subset['embeddings'] = website_subset['documents_cleaned'].apply(get_longformer_embeddings)

#########################
# EVALUATE CONSISTENCY BETWEEN MODELS
#########################

# create dataframe of random pairwise combinations
rand = np.random.randint(1,100,size=(500,2))
df = pd.DataFrame(rand, columns=['rand1', 'rand2'])
df['sim_lf']=0
df['sim_dv']=0
df['sim_ft_sg']=0
df['sim_ft_cbow']=0

for ind in df.index:
    a_loc = df['rand1'][ind]
    b_loc = df['rand2'][ind]
    a_vec_dv = vectors_d2v_websites[a_loc]
    b_vec_dv = vectors_d2v_websites[b_loc]
    a_vec_ft_sg = embeddings_sg_website[a_loc]
    b_vec_ft_sg = embeddings_sg_website[b_loc]
    a_vec_ft_cbow = embeddings_cbow_website[a_loc]
    b_vec_ft_cbow = embeddings_cbow_website[b_loc]
    a_vec_lf = website_subset['embeddings'][a_loc]
    b_vec_lf = website_subset['embeddings'][b_loc].T  # 此处转置可能冗余且引发维度问题
    cos_sim_lf = get_cosine_sim(a_vec_lf, b_vec_lf)
    cos_sim_dv = get_cosine_sim(a_vec_dv, b_vec_dv)
    cos_sim_ft_sg = get_cosine_sim(a_vec_ft_sg,b_vec_ft_sg)
    cos_sim_ft_cbow = get_cosine_sim(a_vec_ft_cbow,b_vec_ft_cbow)
    df['sim_lf'][ind]=cos_sim_lf
    df['sim_dv'][ind]=cos_sim_dv
    df['sim_ft_sg'][ind]=cos_sim_ft_sg
    df['sim_ft_cbow'][ind]=cos_sim_ft_cbow

print('MY WEBSITE DATA SIMILARITY')
print('corr(Longformer, Fasttext (skipgram)) = ',df['sim_lf'].corr(df['sim_ft_sg']))
print('corr(Longformer, Fasttext (cbow)) = ',df['sim_lf'].corr(df['sim_ft_cbow']))
print('corr(Longformer, d2v) = ',df['sim_lf'].corr(df['sim_dv']))
print('corr(Fasttext (skipgram), d2v) = ',df['sim_ft_sg'].corr(df['sim_dv']))
print('corr(Fasttext (cbow), d2v) = ',df['sim_ft_cbow'].corr(df['sim_dv']))

一、代码中的明显错误 #

1. Fasttext CBOW向量赋值错误
   代码中embeddings_cbow_website=website_df['embeddings_sg']这一行明显写错了，应该改为embeddings_cbow_website=website_df['embeddings_cbow']。当前逻辑导致CBOW的相似度计算实际用的是Skipgram的向量，虽然这可能不是Doc2vec与其他模型负相关的核心原因，但会导致CBOW的结果完全不可信，必须先修正。

2. Longformer向量维度处理不当
   get_longformer_embeddings返回的是形状为(1, 768)的二维数组，而代码中对b_vec_lf做了转置操作b_vec_lf = website_subset['embeddings'][b_loc].T，得到(768, 1)的数组。虽然numpy的dot函数能处理这种维度的矩阵乘法，但计算余弦相似度时，应该将向量统一为一维数组（比如用flatten()），否则可能出现意外的计算结果，甚至导致相关性异常。

3. Doc2vec与Fasttext的预处理不一致

   • Doc2vec使用的是remove_stopwords后再用nltk分词的文本；
   • Fasttext使用的是gensim.utils.simple_preprocess处理后的文本。
     两者的清洗规则不同（比如simple_preprocess会自动过滤短词、处理标点，而nltk分词+手动去停用词的逻辑可能保留不同的词汇），导致模型学习的文本基础单元差异极大，这很可能是相似度相关性异常的核心原因之一。建议统一两者的预处理流程，比如都用simple_preprocess或者都用nltk分词+去停用词。

二、模型本身的潜在差异（非代码错误） #

即使修正了代码，三个模型的相似度结果也不会完全一致，但强负相关是不符合预期的，更多是代码问题导致。不过模型架构本身的差异也会带来语义空间的不同：

• Doc2vec是基于词袋的分布式表示，依赖局部窗口和文档标签，更偏向于捕捉文档的主题词分布；
• Fasttext通过子词建模，能更好地处理OOV词汇，但本质是词向量的平均，语义表达能力有限；
• Longformer是预训练Transformer模型，能捕捉长文本的上下文语义，理解能力远强于前两者。
  正常情况下，三者的相似度结果应该是弱正相关或无显著相关，强负相关大概率是代码错误导致的。

内容的提问来源于stack exchange，提问作者John