社交媒体评论情感分析中emoji转文本情感失败问题求助
问题背景
采集了某产品广告对应的Facebook、Twitter评论数据,拟对评论开展情感分析工作。文本清洗环节需将emoji转换为对应的文本情感,最大化捕获评论中的全部情感信息。已尝试emoji.demojize(text)及各类公开解决方案,均无法实现转换效果,排查一周未定位问题。
已尝试的代码方案
方案1:自定义emoji映射转换
import io import json import re def handleEmojis(text, keep_emoticons = False): global emoji_sentiment_matching if not 'emoji_sentiment_matching' in globals(): with io.open('emoji.json', 'r', encoding = "UTF-8") as outfile: emoji_sentiment_matching = json.load(outfile) HASHTAG_PATTERN = re.compile(r'#\w*') EMOJIS_PATTERN_PLAIN_TEXT = re.compile(r"(?:X|:|;|=)(?:-)?(?:\)|\(|O|D|P|S){1,}", re.IGNORECASE) EMOJIS_PATTERN_SYMBOLS = re.compile(u'[\U00002600-\U000027BF]|[\U0001f300-\U0001f64F]|[\U0001f680-\U0001f6FF]') if keep_emoticons: # Replace emoji with sentiment for emoji in emoji_sentiment_matching: if emoji["emoji"] in text: ## Adding space if text follows right away / is right before the emoticon idx = text.find(emoji["emoji"]) (space1,space2) = ("","") if (idx-1) >= 0 and text[idx-1] != " ": space1 = " " if (idx+1) <= len(text) and text[idx+1] != " ": space2 = " " ## replace emoticon with its sentiment text = text.replace(emoji["emoji"], "{}emoji%%{}{}".format(space1, emoji["subgroup"], space2)) ## TO IMPLEMENT: Sentiment of other emoticons like :), :-), :-/ else: for r in re.findall(EMOJIS_PATTERN_SYMBOLS,text): text = text.replace(r, "") for r in re.findall(EMOJIS_PATTERN_PLAIN_TEXT,text): text = text.replace(r, "") return text.strip() FB_df['demojified']=FB_df['Text'] for i in range(len(FB_df)): text = FB_df.loc[i,"demojified"] handleEmojis(text, keep_emoticons = False) print(FB_df)
运行后demojified列无预期转换效果。
方案2:调用emot库转换
import re from emot.emo_unicode import UNICODE_EMO, EMOTICONS from emoji import demojize def convert_emojis(text): for emot in UNICODE_EMO: text = re.sub(r'('+emot+')', "_".join(UNICODE_EMO[emot].replace(",","").replace(":","").split()), text) return text def convert_emoticons(text): for emot in EMOTICONS: text = re.sub(u'('+emot+')', "_".join(EMOTICONS[emot].replace(",","").split()), text) return text FB_df['demojified']=FB_df['Text'] for row in FB_df['demojified']: for text in row: text=text convert_emojis(text) FB_df.loc[:,'demojified']
方案3:结合demojize转换
import re from emot.emo_unicode import UNICODE_EMO, EMOTICONS import emoji def convert_emojis(text): for emot in UNICODE_EMO: text = re.sub(r'('+emot+')',"_".join(UNICODE_EMO[emot].replace(",","").replace(":","").split()), text) return text def convert_emoticons(text): for emot in EMOTICONS: text = re.sub(u'('+emot+')', "_".join(EMOTICONS[emot].replace(",","").split()), text) return text FB_df['demojified']=FB_df['Text'] for row in FB_df['demojified']: for text in row: text=str(text) text = emoji.demojize(text)
问题根因与修复方案
核心错误点
- 临时变量未回写:所有遍历过程中修改的
text都是临时变量,转换完成后没有重新赋值回DataFrame对应单元格,修改结果直接被丢弃 - 参数设置错误:方案1调用时传入
keep_emoticons = False,该分支逻辑是直接删除所有emoji而非转换为情感文本,和需求完全相反 - 循环逻辑错误:方案2、3中遍历
FB_df['demojified']时嵌套遍历row的每个字符,实际是对单个字符做转换而非整行文本 - 语法错误:部分版本的
convert_emoticons函数return语句写在循环内部,第一次循环就直接返回,只处理了第一个匹配到的表情 - 依赖兼容问题:低版本
emoji、emot库对新emoji支持不足,部分库旧版本中UNICODE_EMO已改名为UNICODE_EMOJI
修复后可运行代码
import re import emoji import pandas as pd from emot.emo_unicode import UNICODE_EMOJI, EMOTICONS # 转换unicode emoji为情感文本 def convert_emojis(text): # 先做基础转换,再统一处理格式 text = emoji.demojize(text, delimiters=(" ", " ")) for emot, desc in UNICODE_EMOJI.items(): text = text.replace(emot, desc.replace(",", "").replace(":", "").replace(" ", "_")) return text # 转换颜文字为情感文本 def convert_emoticons(text): for emot, desc in EMOTICONS.items(): text = re.sub(re.escape(emot), desc.replace(",", "").replace(" ", "_"), text) return text # 统一处理入口 def process_emoji(text): text = str(text) text = convert_emojis(text) text = convert_emoticons(text) return text # 用pandas apply批量处理,避免手写循环出错 FB_df['demojified'] = FB_df['Text'].apply(process_emoji)
内容的提问来源于stack exchange,提问作者Sivan
相关产品推荐
相关产品推荐

