LatentDirichletAllocation执行时出现UnicodeEncodeError问题求助

问题场景 #

在本地Jupyter Notebook中对数组执行LatentDirichletAllocation.fit()时，持续触发UnicodeEncodeError，错误提示为'ascii' codec can't encode characters in position 18-19: ordinal not in range(128)。更换数据集、调整CSV读取编码均无效，但代码在Google Colab中可正常运行，推测为本地Windows环境的编码或多进程资源追踪问题。

错误栈信息 #

UnicodeEncodeError                        Traceback (most recent call last)
<timed exec> in <module>

D:\Anacondo\lib\site-packages\sklearn\decomposition\_lda.py in fit(self, X, y)
    624         last_bound = None
    625         n_jobs = effective_n_jobs(self.n_jobs)
--> 626         with Parallel(n_jobs=n_jobs, verbose=max(0, self.verbose - 1)) as parallel:
    627             for i in range(max_iter):
    628                 if learning_method == "online":

D:\Anacondo\lib\site-packages\joblib\parallel.py in __enter__(self)
    723     def __enter__(self):
    724         self._managed_backend = True
--> 725         self._initialize_backend()
    726         return self
    727 

D:\Anacondo\lib\site-packages\joblib\parallel.py in _initialize_backend(self)
    733         """Build a process or thread pool and return the number of workers"""
    734         try:
--> 735             n_jobs = self._backend.configure(n_jobs=self.n_jobs, parallel=self,
    736                                              **self._backend_args)
    737             if self.timeout is not None and not self._backend.supports_timeout:

D:\Anacondo\lib\site-packages\joblib\_parallel_backends.py in configure(self, n_jobs, parallel, prefer, require, idle_worker_timeout, **memmappingexecutor_args)
    492                 SequentialBackend(nesting_level=self.nesting_level))
    493 
--> 494         self._workers = get_memmapping_executor(
    495             n_jobs, timeout=idle_worker_timeout,
    496             env=self._prepare_worker_env(n_jobs=n_jobs),

D:\Anacondo\lib\site-packages\joblib\executor.py in get_memmapping_executor(n_jobs, **kwargs)
     18 
     19 def get_memmapping_executor(n_jobs, **kwargs):
--> 20     return MemmappingExecutor.get_memmapping_executor(n_jobs, **kwargs)
     21 
     22 

D:\Anacondo\lib\site-packages\joblib\executor.py in get_memmapping_executor(cls, n_jobs, timeout, initializer, initargs, env, temp_folder, context_id, **backend_args)
     40         _executor_args = executor_args
     41 
--> 42         manager = TemporaryResourcesManager(temp_folder)
     43 
     44         # reducers access the temporary folder in which to store temporary

D:\Anacondo\lib\site-packages\joblib\_memmapping_reducer.py in __init__(self, temp_folder_root, context_id)
    529             # exposes exposes too many low-level details.
    530             context_id = uuid4().hex
--> 531         self.set_current_context(context_id)
    532 
    533     def set_current_context(self, context_id):

D:\Anacondo\lib\site-packages\joblib\_memmapping_reducer.py in set_current_context(self, context_id)
    533     def set_current_context(self, context_id):
    534         self._current_context_id = context_id
--> 535         self.register_new_context(context_id)
    536 
    537     def register_new_context(self, context_id):

D:\Anacondo\lib\site-packages\joblib\_memmapping_reducer.py in register_new_context(self, context_id)
    558                 new_folder_name, self._temp_folder_root
    559             )
--> 560             self.register_folder_finalizer(new_folder_path, context_id)
    561             self._cached_temp_folders[context_id] = new_folder_path
    562 

D:\Anacondo\lib\site-packages\joblib\_memmapping_reducer.py in register_folder_finalizer(self, pool_subfolder, context_id)
    588         # semaphores and pipes
    589         pool_module_name = whichmodule(delete_folder, 'delete_folder')
--> 590         resource_tracker.register(pool_subfolder, "folder")
    591 
    592         def _cleanup():

D:\Anacondo\lib\site-packages\joblib\externals\loky\backend\resource_tracker.py in register(self, name, rtype)
    189         '''Register a named resource, and increment its refcount.'''
    190         self.ensure_running()
--> 191         self._send('REGISTER', name, rtype)
    192 
    193     def unregister(self, name, rtype):

D:\Anacondo\lib\site-packages\joblib\externals\loky\backend\resource_tracker.py in _send(self, cmd, name, rtype)
    202 
    203     def _send(self, cmd, name, rtype):
--> 204         msg = '{0}:{1}:{2}\n'.format(cmd, name, rtype).encode('ascii')
    205         if len(name) > 512:
    206             # posix guarantees that writes to a pipe of less than PIPE_BUF

UnicodeEncodeError: 'ascii' codec can't encode characters in position 18-19: ordinal not in range(128)

用到的代码 #

数据处理代码

import pandas as pd
import numpy as np

df = pd.read_csv("android.csv", sep=",",  thousands=",")

df["Number of ratings"] = df["Number of ratings"].astype(int) # fix data type
df = df.drop_duplicates(subset=["App"]).reset_index(drop=True)

permission_columns = list(df.columns[10:])
app_names = list(df["App"])
app_ratings = np.array(df["Number of ratings"])
df_perms = df[permission_columns]
X = df_perms.values

TF-IDF与LDA拟合代码

from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.decomposition import LatentDirichletAllocation

transformer = TfidfTransformer()
X_tfidf = transformer.fit_transform(X)

# convert sparse matrix to numpy array
X_tfidf = X_tfidf.toarray()

n_topics = 10
lda = LatentDirichletAllocation(n_components=n_topics, max_iter=10,
                                learning_method='online',
                                n_jobs=-1, random_state=3)
lda.fit(X_tfidf)

解决方法 #

1. 禁用多进程快速验证 #

问题根源是多进程模式下，joblib依赖的loky库在处理含非ASCII字符的临时文件夹路径时，强制用ASCII编码导致报错。先将n_jobs设为1禁用多进程，验证是否解决：

lda = LatentDirichletAllocation(n_components=n_topics, max_iter=10,
                                learning_method='online',
                                n_jobs=1, random_state=3)

2. 指定纯英文临时文件夹路径 #

手动设置joblib的临时目录为纯英文路径，避免路径中出现中文等非ASCII字符：

import joblib
import os

# 创建纯英文临时目录（如果不存在）
temp_dir = "D:/temp/joblib_temp"
os.makedirs(temp_dir, exist_ok=True)
joblib.memory.location = temp_dir

3. 修改loky资源追踪器的编码（永久修复） #

找到报错的resource_tracker.py文件（路径：D:\Anacondo\lib\site-packages\joblib\externals\loky\backend\resource_tracker.py），定位到第204行：

msg = '{0}:{1}:{2}\n'.format(cmd, name, rtype).encode('ascii')

将其修改为UTF-8编码：

msg = '{0}:{1}:{2}\n'.format(cmd, name, rtype).encode('utf-8')

4. 设置全局UTF-8环境变量 #

在代码开头添加环境变量设置，强制Python使用UTF-8编码：

import os
os.environ['PYTHONUTF8'] = '1'

或者在启动Jupyter Notebook前，在命令行执行：

set PYTHONUTF8=1 && jupyter notebook

内容的提问来源于stack exchange，提问作者Alex