如何在Python情感分析代码中融入主题建模或NER

先修正你现有代码里的一个关键问题：你生成了预处理后的corpus，但训练逻辑回归时直接用了原始文本X_train——模型无法直接处理文本，必须先做向量化，后续示例会补上这部分。

一、集成主题建模（以LDA为例） #

主题建模能挖掘评论中的核心话题，把每个评论的主题分布作为额外特征，和文本特征结合后喂给情感分析模型，可提升模型对情感倾向的判断精度。

实现步骤 #

1. 对预处理后的corpus做词袋向量化（LDA模型的输入要求）
2. 训练LDA模型，生成每个评论的主题概率分布
3. 将主题分布特征与TF-IDF文本特征拼接，作为最终特征训练情感分析模型

完整代码示例 #

import re
import pandas as pd
import numpy as np
from nltk.corpus import stopwords
from nltk.stem.porter import PorterStemmer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, accuracy_score, classification_report
# 新增主题建模依赖库
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.decomposition import LatentDirichletAllocation

# 读取数据集
dataset = pd.read_csv('full_db.csv') 
X = dataset.iloc[:,0].values
y = dataset.iloc[:, 1].values

# 原有文本预处理逻辑保留
corpus = []
ps = PorterStemmer()
all_stopwords = stopwords.words('english')
no_stopwords = ["not","don't",'aren','don','ain',"aren't", 'couldn', "couldn't", "wasn't"]
for Nostopword in no_stopwords:
    all_stopwords.remove(Nostopword)

for i in range(0, len(X)):
    review = re.sub('[^a-zA-Z]', ' ', dataset['Review'][i])
    review = review.lower()
    review = review.split()
    review = [ps.stem(word) for word in review if not word in set(all_stopwords)] 
    review = ' '.join(review)
    corpus.append(review)

# --- 主题建模模块 ---
# 1. 生成词袋矩阵（LDA输入）
cv = CountVectorizer(max_features=1000)
bow_matrix = cv.fit_transform(corpus).toarray()

# 2. 训练LDA模型，假设提取5个主题（可根据业务调整数量）
lda = LatentDirichletAllocation(n_components=5, random_state=42)
topic_distributions = lda.fit_transform(bow_matrix)  # 每条评论对应5个主题的概率分布

# 3. 生成文本TF-IDF特征（情感分析常用）
tfidf = TfidfVectorizer(max_features=2000)
tfidf_features = tfidf.fit_transform(corpus).toarray()

# 拼接文本特征与主题分布特征
combined_features = np.hstack((tfidf_features, topic_distributions))

# --- 模型训练与评估 ---
X_train, X_test, y_train, y_test = train_test_split(combined_features, y, test_size=0.2, random_state=42)

logistic = LogisticRegression(random_state=42, solver='lbfgs', multi_class='multinomial')
logistic.fit(X_train, y_train)
y_pred = logistic.predict(X_test)

# 输出评估结果
print(confusion_matrix(y_test, y_pred))
print(f"准确率: {accuracy_score(y_test, y_pred):.2f}")
print(classification_report(y_test, y_pred))

额外功能：查看主题关键词 #

训练完LDA后，可以打印每个主题的核心关键词，直观了解评论中的话题方向：

feature_names = cv.get_feature_names_out()
for topic_idx, topic in enumerate(lda.components_):
    top_features_ind = topic.argsort()[-10:]
    top_features = [feature_names[i] for i in top_features_ind]
    print(f"主题 {topic_idx+1}: {', '.join(top_features)}")

二、集成命名实体识别（NER，以spaCy为例） #

NER能提取评论中的实体（如产品名、品牌、地点等），这些实体往往和情感倾向强相关（比如提到某品牌的负面评论），将实体特征加入模型可提升情感判断的精准度。

实现步骤 #

1. 加载spaCy英文模型（首次使用需执行python -m spacy download en_core_web_sm下载）
2. 对原始评论提取实体，转化为可用于模型的特征（如实体类型出现次数）
3. 将实体特征与TF-IDF文本特征拼接，训练情感分析模型

完整代码示例 #

import re
import pandas as pd
import numpy as np
from nltk.corpus import stopwords
from nltk.stem.porter import PorterStemmer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, accuracy_score, classification_report
from sklearn.feature_extraction.text import TfidfVectorizer
# 新增NER依赖库
import spacy

# 加载spaCy英文模型
nlp = spacy.load('en_core_web_sm')

# 读取数据集
dataset = pd.read_csv('full_db.csv') 
X = dataset.iloc[:,0].values
y = dataset.iloc[:, 1].values

# 原有文本预处理逻辑保留
corpus = []
ps = PorterStemmer()
all_stopwords = stopwords.words('english')
no_stopwords = ["not","don't",'aren','don','ain',"aren't", 'couldn', "couldn't", "wasn't"]
for Nostopword in no_stopwords:
    all_stopwords.remove(Nostopword)

for i in range(0, len(X)):
    review = re.sub('[^a-zA-Z]', ' ', dataset['Review'][i])
    review = review.lower()
    review = review.split()
    review = [ps.stem(word) for word in review if not word in set(all_stopwords)] 
    review = ' '.join(review)
    corpus.append(review)

# --- NER特征提取模块 ---
# 定义需要关注的实体类型（可根据业务调整，比如电商评论重点关注产品、品牌）
target_entities = ['PRODUCT', 'ORG', 'PERSON']

# 提取每条评论的实体特征：各目标类型的出现次数
ner_features = []
for text in dataset['Review']:  # 用原始文本提取实体，避免预处理丢失实体信息
    doc = nlp(text)
    entity_counts = {ent_type:0 for ent_type in target_entities}
    for ent in doc.ents:
        if ent.label_ in target_entities:
            entity_counts[ent.label_] +=1
    ner_features.append(list(entity_counts.values()))

# 转换为数组格式
ner_features = np.array(ner_features)

# --- 文本TF-IDF特征 ---
tfidf = TfidfVectorizer(max_features=2000)
tfidf_features = tfidf.fit_transform(corpus).toarray()

# 拼接文本特征与NER特征
combined_features = np.hstack((tfidf_features, ner_features))

# --- 模型训练与评估 ---
X_train, X_test, y_train, y_test = train_test_split(combined_features, y, test_size=0.2, random_state=42)

logistic = LogisticRegression(random_state=42, solver='lbfgs', multi_class='multinomial')
logistic.fit(X_train, y_train)
y_pred = logistic.predict(X_test)

# 输出评估结果
print(confusion_matrix(y_test, y_pred))
print(f"准确率: {accuracy_score(y_test, y_pred):.2f}")
print(classification_report(y_test, y_pred))

扩展思路 #

• 可以将实体本身转化为词袋特征（比如某特定品牌是否在评论中出现）
• 统计不同实体与正负情感的关联度，生成更针对性的特征

内容的提问来源于stack exchange，提问作者Sanne112