Gibbs LDA主题建模遇UnicodeEncodeError错误，求修复方案

问题背景 #

执行Gibbs LDA主题建模时出现编码错误，相关代码及报错信息如下：

运行代码 #

In[1]  import numpy as np
       import pandas as pd
       import gibbslda

In[2]  df = pd.read_csv(f, sep=',',encoding='utf-8')
       print(df.shape)
       list(df)
       
       ...词形还原处理过程

In[44] texts = [[text for text in doc.split()] for doc in Y_data]
       gibbs = gibbslda.CGibbsLDA()
       gibbs.SetInputDocs(texts)
       print("Calculations...")

In[45] texts[0] 

In[46] print("Calculations...")
       # 参数ldamethod可选值：
       # "lda" - 标准Gibbs采样流程
       # "granulate" - GLDA模型，需指定granwnd参数
       # "embeddings" - 带嵌入的模型
       # out_dir="outcsv" - 存储phi、theta矩阵的文件夹

       gibbs.SetParameters(alpha=0.1, beta=0.1, topics=75, niters=200, nitersave=200, ldamethod="lda", fixtopics = 0, granwnd = 0, out_dir="tang_outcsv")
       # 若ldamethod="embeddings"，需调用SetEmbeddings()指定嵌入文件路径和所需词数
       gibbs.StartCalculation()

       print("OK")

报错信息 #

Calculations...
Iterations done:  200
Python WNDPROC handler failed
---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)
C:\Topic_modeling\gibbslda.py in OnDataReady(self, hwnd, message, wparam, lparam)
     73             ArrayType = ctypes.c_void_p*self._doccnt
     74             dbl_ar = np.frombuffer(ArrayType.from_address(lparam), dtype=ctypes.c_void_p)
---> 75             self.saveTheta(dbl_ar, niter)
     76 
     77         if datatype==LDA_PHI:

C:\Topic_modeling\gibbslda.py in saveTheta(self, dbl_ar, niter)
    116                 # calc perplexity
    117                 if sng_ar[t]>doc_prob: sum_prob += 1
--> 118             writer.writerow(csvrow)
    119 
    120         ugof.close()

C:\ProgramData\Anaconda3\lib\encodings\cp1251.py in encode(self, input, final)
     17 class IncrementalEncoder(codecs.IncrementalEncoder):
     18     def encode(self, input, final=False):
---> 19         return codecs.charmap_encode(input,self.errors,encoding_table)[0]
     20 
     21 class IncrementalDecoder(codecs.IncrementalDecoder):

UnicodeEncodeError: 'charmap' codec can't encode character '\xe1' in position 33: character maps to <undefined>

Python WNDPROC handler failed
---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)
C:\Topic_modeling\gibbslda.py in OnDataReady(self, hwnd, message, wparam, lparam)
     78             ArrayType = ctypes.c_void_p*self._topics
     79             dbl_ar = np.frombuffer(ArrayType.from_address(lparam), dtype=ctypes.c_void_p)
---> 80             self.savePhi(dbl_ar, niter)
     81 
     82         if datatype==LDA_NW:

C:\Topic_modeling\gibbslda.py in savePhi(self, dbl_ar, niter)
    143                 # calc perplexity
    144                 if sng_ar[wrdid]>cur_prob: sum_prob += 1
--> 145             writer.writerow(csvrow)
    146 
    147         ugof.close()

C:\ProgramData\Anaconda3\lib\encodings\cp1251.py in encode(self, input, final)
     17 class IncrementalEncoder(codecs.IncrementalEncoder):
     18     def encode(self, input, final=False):
---> 19         return codecs.charmap_encode(input,self.errors,encoding_table)[0]
     20 
     21 class IncrementalDecoder(codecs.IncrementalDecoder):

UnicodeEncodeError: 'charmap' codec can't encode character '\xef' in position 3: character maps to <undefined>

OK

已完成词形还原得到干净文本，尝试过常规论坛方案未解决。

修复步骤 #

错误根源是gibbslda库保存CSV文件时默认使用系统编码（此处为cp1251），无法兼容文本中的特殊Unicode字符。需修改gibbslda.py文件的编码设置：

1. 打开本地的C:\Topic_modeling\gibbslda.py文件
2. 找到saveTheta和savePhi两个函数
3. 修改文件打开语句，指定UTF-8编码并处理换行符：
   • 原代码（以saveTheta为例）：

     ugof = open(outfn, "w")
     writer = csv.writer(ugof)
     

   修改为：

   ugof = open(outfn, "w", encoding='utf-8', newline='')
   writer = csv.writer(ugof)
   

4. 对savePhi函数执行同样的修改，确保两个CSV保存逻辑都使用UTF-8编码

修改后重新运行代码，UTF-8编码支持所有Unicode字符，可解决特殊字符的编码失败问题。

内容的提问来源于stack exchange，提问作者poma