Colab中使用Huggingface Dataset的map/select方法报错：dill._dill无log属性

在Colab环境下，使用Huggingface Datasets库调用dataset.map()或dataset.select(range(10))时，触发错误：AttributeError: module 'dill._dill' has no attribute 'log'。尝试更换不同版本的dill库（包括旧版本）均无法解决。

错误堆栈 #

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-39-dd9b972a8f3f> in <module>
      1 test_data = load_dataset("scientific_papers", "arxiv", ignore_verifications=True, split="test")
      2 print(test_data)
----> 3 data = test_data.select(range(10))

16 frames
/usr/local/lib/python3.9/dist-packages/datasets/arrow_dataset.py in wrapper(*args, **kwargs)
    155         }
    156         # apply actual function
--> 157         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    158         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    159         # re-apply format to the output

/usr/local/lib/python3.9/dist-packages/datasets/fingerprint.py in wrapper(*args, **kwargs)
    155                     if kwargs.get(fingerprint_name) is None:
    156                         kwargs_for_fingerprint["fingerprint_name"] = fingerprint_name
--> 157                         kwargs[fingerprint_name] = update_fingerprint(
    158                             self._fingerprint, transform, kwargs_for_fingerprint
    159                         )

/usr/local/lib/python3.9/dist-packages/datasets/fingerprint.py in update_fingerprint(fingerprint, transform, transform_args)
    103     for key in sorted(transform_args):
    104         hasher.update(key)
--> 105         hasher.update(transform_args[key])
    106     return hasher.hexdigest()
    107 

/usr/local/lib/python3.9/dist-packages/datasets/fingerprint.py in update(self, value)
     55     def update(self, value):
     56         self.m.update(f"=={type(value)}==".encode("utf8"))
--> 57         self.m.update(self.hash(value).encode("utf-8"))
     58 
     59     def hexdigest(self):

/usr/local/lib/python3.9/dist-packages/datasets/fingerprint.py in hash(cls, value)
     51             return cls.dispatch[type(value)](cls, value)
     52         else:
--> 53             return cls.hash_default(value)
     54 
     55     def update(self, value):

/usr/local/lib/python3.9/dist-packages/datasets/fingerprint.py in hash_default(cls, value)
     44     @classmethod
     45     def hash_default(cls, value):
--> 46         return cls.hash_bytes(dumps(value))
     47 
     48     @classmethod

/usr/local/lib/python3.9/dist-packages/datasets/utils/py_utils.py in dumps(obj)
    387     file = StringIO()
    388     with _no_cache_fields(obj):
--> 389         dump(obj, file)
    390     return file.getvalue()
    391 

/usr/local/lib/python3.9/dist-packages/datasets/utils/py_utils.py in dump(obj, file)
    359 def dump(obj, file):
    360     """pickle an object to a file"""
--> 361     Pickler(file, recurse=True).dump(obj)
    362     return
    363 

/usr/local/lib/python3.9/dist-packages/dill/_dill.py in dump(self, obj)
    392         f = filename
    393     else:
--> 394         f = open(filename, 'wb')
    395     try:
    396         if byref:

/usr/lib/python3.9/pickle.py in dump(self, obj)
    485         if self.proto >= 4:
    486             self.framer.start_framing()
--> 487         self.save(obj)
    488         self.write(STOP)
    489         self.framer.end_framing()

/usr/local/lib/python3.9/dist-packages/dill/_dill.py in save(self, obj, save_persistent_id)
    386 def dump_session(filename='/tmp/session.pkl', main=None, byref=False, **kwds):
    387     """pickle the current state of __main__ to a file"""
--> 388     from .settings import settings
    389     protocol = settings['protocol']
    390     if main is None: main = _main_module

/usr/lib/python3.9/pickle.py in save(self, obj, save_persistent_id)
    558             f = self.dispatch.get(t)
    559             if f is not None:
--> 560                 f(self, obj)  # Call unbound method with explicit self
    561                 return
    562 

/usr/local/lib/python3.9/dist-packages/dill/_dill.py in save_singleton(pickler, obj)
   1524 def pickles(obj,exact=False,safe=False,**kwds):
   1525     """
-> 1526     Quick check if object pickles with dill.
   1527 
   1528     If *exact=True* then an equality test is done to check if the reconstructed

/usr/lib/python3.9/pickle.py in save_reduce(self, func, args, state, listitems, dictitems, state_setter, obj)
    689             write(NEWOBJ)
    690         else:
--> 691             save(func)
    692             save(args)
    693             write(REDUCE)

/usr/local/lib/python3.9/dist-packages/dill/_dill.py in save(self, obj, save_persistent_id)
    386 def dump_session(filename='/tmp/session.pkl', main=None, byref=False, **kwds):
    387     """pickle the current state of __main__ to a file"""
--> 388     from .settings import settings
    389     protocol = settings['protocol']
    390     if main is None: main = _main_module

/usr/lib/python3.9/pickle.py in save(self, obj, save_persistent_id)
    558             f = self.dispatch.get(t)
    559             if f is not None:
--> 560                 f(self, obj)  # Call unbound method with explicit self
    561                 return
    562 

/usr/local/lib/python3.9/dist-packages/datasets/utils/py_utils.py in save_function(pickler, obj)
    583         dill._dill.log.info("# F1")
    584     else:
--> 585         dill._dill.log.info("F2: %s" % obj)
    586         name = getattr(obj, "__qualname__", getattr(obj, "__name__", None))
    587         dill._dill.StockPickler.save_global(pickler, obj, name=name)

AttributeError: module 'dill._dill' has no attribute 'log'

修复方法与替代方案 #

方法1：禁用Dataset的指纹功能 #

错误根源是Datasets库在计算指纹时调用了dill内部已移除的log属性。可以通过在调用map或select时添加fingerprint=False参数跳过指纹计算：

# 使用select时
data = test_data.select(range(10), fingerprint=False)

# 使用map时
data = test_data.map(your_processing_func, fingerprint=False)

方法2：锁定dill与Datasets的兼容版本组合 #

某些版本的Datasets依赖特定版本的dill，尝试安装以下兼容组合：

# 卸载现有版本
pip uninstall -y dill datasets
# 安装指定兼容版本
pip install datasets==2.12.0 dill==0.3.6

方法3：修改Datasets库的源码临时修复 #

如果上述方法无效，可以直接修改datasets/utils/py_utils.py中的错误代码：

1. 在Colab中执行以下命令打开文件：

%edit /usr/local/lib/python3.9/dist-packages/datasets/utils/py_utils.py

2. 找到save_function函数中调用dill._dill.log.info的两行（第583、585行），直接注释掉或删除这两行代码。
3. 保存修改后重启Colab运行时，重新执行代码。

方法4：使用Dataset的切片替代select #

如果只是需要取前N条数据，可以直接用Python切片语法替代select，跳过指纹计算流程：

# 替代select(range(10))
data = test_data[:10]

内容的提问来源于stack exchange，提问作者user21401461