如何解决无法序列化get_tokenizer本地lambda对象的AttributeError？

DevOps 小满

2026-7-5

问题与报错

编写get_tokenizer函数后，在另一个.py文件调用时触发如下报错：

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
d:\abie\Coding\Tugas Akhir brow\VisualGPT-main\VisualGPT-main\VisualGPT.ipynb Cell 1 line 3
    296 dict_dataloader_test = DataLoader(dict_dataset_test, batch_size=args.batch_size // 5)
    299 if not use_rl:
--&gt; 300     train_loss = train_xe(model, dataloader_train, text_field,gpt_optimizer,dataloader_val,args)

d:\abie\Coding\Tugas Akhir brow\VisualGPT-main\VisualGPT-main\VisualGPT.ipynb Cell 1 line 8
     84 running_loss = .0
     85 with tqdm(desc='Epoch %d - train' % e, unit='it', total=len(dataloader)) as pbar:
---&gt; 86     for it, (detections, captions) in enumerate(dataloader):
     88         detections, captions = detections.to(device), captions.to(device)
     91         out,past= model(detections, captions)

File c:\Users\Axioo Pongo\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\utils\data\dataloader.py:438, in DataLoader.__iter__(self)
    436     return self._iterator
    437 else:
--&gt; 438     return self._get_iterator()

File c:\Users\Axioo Pongo\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\utils\data\dataloader.py:386, in DataLoader._get_iterator(self)
    384 else:
    385     self.check_worker_number_rationality()
--&gt; 386     return _MultiProcessingDataLoaderIter(self)

File c:\Users\Axioo Pongo\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\utils\data\dataloader.py:1039, in _MultiProcessingDataLoaderIter.__init__(self, loader)
...
     58 def dump(obj, file, protocol=None):
     59     '''Replacement for pickle.dump() using ForkingPickler.'''
---&gt; 60     ForkingPickler(file, protocol).dump(obj)

AttributeError: Can't pickle local object 'get_tokenizer.&lt;locals&gt;.&lt;lambda&gt;'
Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings

解决方法

报错根源是PyTorch DataLoader启用多进程时，需要序列化（pickle）数据集相关对象，但get_tokenizer函数内部定义的lambda是局部匿名函数，无法被pickle序列化。可通过以下方案解决：

方案1：替换lambda为全局函数
把原本在get_tokenizer里的lambda逻辑抽离出来，定义成模块级别的全局函数：

# 模块全局范围内定义函数
def tokenize_process(text):
    # 写入原本lambda的处理逻辑
    return text.split()

def get_tokenizer():
    tokenizer = YourTokenizerClass()
    tokenizer.tokenize = tokenize_process  # 替换lambda为全局函数
    return tokenizer

方案2：用自定义可序列化类替代lambda
若处理逻辑复杂，可实现一个包含__call__方法的类，确保类在全局作用域可访问：

class TokenizeProcessor:
    def __call__(self, text):
        # 写入原本lambda的处理逻辑
        return text.split()

def get_tokenizer():
    tokenizer = YourTokenizerClass()
    tokenizer.tokenize = TokenizeProcessor()
    return tokenizer

方案3：临时关闭DataLoader多进程
调试阶段可修改DataLoader参数，强制单进程运行以规避pickle操作：
```
dataloader_train = DataLoader(dataset_train, batch_size=args.batch_size, num_workers=0)
```

内容的提问来源于stack exchange，提问作者Aabie Nugraha