Python虚拟环境导入datasets时遭遇dill相关导入及属性错误

在Python虚拟环境中安装datasets包后，执行from datasets import load_dataset语句时触发以下关键错误：

• ImportError: cannot import name '_is_imported_module' from 'dill._dill'
• AttributeError: module 'dill' has no attribute '_dill'

完整报错信息：

[autoreload of dill.session failed: Traceback (most recent call last):
File "C:\Users\joshu\anaconda3\envs\sentence_similarity\Lib\site-packages\IPython\extensions\autoreload.py", line 276, in check
superreload(m, reload, self.old_objects)
File "C:\Users\joshu\anaconda3\envs\sentence_similarity\Lib\site-packages\IPython\extensions\autoreload.py", line 475, in superreload
module = reload(module)
^^^^^^^^^^^^^^
File "C:\Users\joshu\anaconda3\envs\sentence_similarity\Lib\importlib\__init__.py", line 131, in reload
_bootstrap._exec(spec, module)
File "<frozen importlib._bootstrap>", line 866, in _exec
File "<frozen importlib._bootstrap_external>", line 995, in exec_module
File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
File "C:\Users\joshu\anaconda3\envs\sentence_similarity\Lib\site-packages\dill\session.py", line 24, in <module>
from ._dill import (
ImportError: cannot import name '_is_imported_module' from 'dill._dill' (C:\Users\joshu\anaconda3\envs\sentence_similarity\Lib\site-packages\dill\_dill.py)
]
[autoreload of dill._shims failed: Traceback (most recent call last):
File "C:\Users\joshu\anaconda3\envs\sentence_similarity\Lib\site-packages\IPython\extensions\autoreload.py", line 276, in check
superreload(m, reload, self.old_objects)
File "C:\Users\joshu\anaconda3\envs\sentence_similarity\Lib\site-packages\IPython\extensions\autoreload.py", line 500, in superreload
update_generic(old_obj, new_obj)
File "C:\Users\joshu\anaconda3\envs\sentence_similarity\Lib\site-packages\IPython\extensions\autoreload.py", line 397, in update_generic
update(a, b)
File "C:\Users\joshu\anaconda3\envs\sentence_similarity\Lib\site-packages\IPython\extensions\autoreload.py", line 365, in update_class
update_instances(old, new)
File "C:\Users\joshu\anaconda3\envs\sentence_similarity\Lib\site-packages\IPython\extensions\autoreload.py", line 323, in update_instances
object.__setattr__(ref, "__class__", new)
TypeError: __class__ assignment: '_CallableReduce' object layout differs from '_CallableReduce'
]
Traceback (most recent call last):

File ~\anaconda3\envs\sentence_similarity\Lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec
exec(code, globals, locals)

File c:\users\joshu.spyder-py3\temp.py:14
from datasets import load_dataset

File ~\anaconda3\envs\sentence_similarity\Lib\site-packages\datasets\__init__.py:18
from .arrow_dataset import Dataset

File ~\anaconda3\envs\sentence_similarity\Lib\site-packages\datasets\arrow_dataset.py:68
from .arrow_writer import ArrowWriter, OptimizedTypedSequence

File ~\anaconda3\envs\sentence_similarity\Lib\site-packages\datasets\arrow_writer.py:29
from .features import Features, Image, Value

File ~\anaconda3\envs\sentence_similarity\Lib\site-packages\datasets\features\__init__.py:17
from .audio import Audio

File ~\anaconda3\envs\sentence_similarity\Lib\site-packages\datasets\features\audio.py:11
from ..download.streaming_download_manager import xopen, xsplitext

File ~\anaconda3\envs\sentence_similarity\Lib\site-packages\datasets\download\__init__.py:9
from .download_manager import DownloadManager, DownloadMode

File ~\anaconda3\envs\sentence_similarity\Lib\site-packages\datasets\download\download_manager.py:43
from ..utils.py_utils import NestedDataStructure, map_nested, size_str

File ~\anaconda3\envs\sentence_similarity\Lib\site-packages\datasets\utils\py_utils.py:44
from ._dill import (  # noqa: F401 # imported for backward compatibility. TODO: remove in 3.0.0

File ~\anaconda3\envs\sentence_similarity\Lib\site-packages\datasets\utils\_dill.py:27
class Pickler(dill.Pickler):

File ~\anaconda3\envs\sentence_similarity\Lib\site-packages\datasets\utils\_dill.py:28 in Pickler
dispatch = dill._dill.MetaCatchingDict(dill.Pickler.dispatch.copy())

AttributeError: module 'dill' has no attribute '_dill'

已尝试重装dill、安装旧版本dill，均无法解决问题；但相同版本的datasets和dill在Jupyter Notebook中可正常运行。

1. 禁用Spyder的自动重载功能 #

报错根源是Spyder默认开启的IPython自动重载（autoreload）功能在重载dill模块时触发冲突，关闭该功能即可：

• 打开Spyder，依次点击工具 -> 偏好设置 -> IPython控制台 -> 高级设置
• 取消勾选启用自动重载，重启Spyder后重新执行代码

2. 重建虚拟环境 #

虚拟环境依赖缓存可能存在损坏，完全重建可解决：

1. 删除现有虚拟环境：

   conda remove -n sentence_similarity --all
   

2. 创建新的虚拟环境（推荐指定稳定Python版本，如3.10）：

   conda create -n sentence_similarity python=3.10
   

3. 激活环境后，先安装兼容版本的dill，再安装datasets：

   pip install dill==0.3.6
   pip install datasets
   

3. 临时 workaround：手动重载dill #

在导入datasets前先手动重载dill模块，可临时绕过自动重载的问题：

import dill
import importlib
importlib.reload(dill)
from datasets import load_dataset

内容的提问来源于stack exchange，提问作者Ju Chen