使用自定义词汇表的BertTokenizer编码值列表触发ValueError问题排查

问题场景 #

手动为推荐任务构建的词汇表text_vocab.txt内容如下：

3456
4567
5678

通过以下代码初始化BertTokenizer：

tokenizer = BertTokenizer(vocab_file="./test_vocab.txt")

尝试编码值列表['3456', '4567', '5678']时，触发如下ValueError：

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/kakao/.pyenv/versions/danke-feature-seq/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 2561, in __call__
    encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
  File "/Users/kakao/.pyenv/versions/danke-feature-seq/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 2667, in _call_one
    return self.encode_plus(
  File "/Users/kakao/.pyenv/versions/danke-feature-seq/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 2740, in encode_plus
    return self._encode_plus(
  File "/Users/kakao/.pyenv/versions/danke-feature-seq/lib/python3.9/site-packages/transformers/tokenization_utils.py", line 652, in _encode_plus
    return self.prepare_for_model(
  File "/Users/kakao/.pyenv/versions/danke-feature-seq/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 3219, in prepare_for_model
    encoded_inputs = self.pad(
  File "/Users/kakao/.pyenv/versions/danke-feature-seq/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 3009, in pad
    raise ValueError(
ValueError: type of None unknown: <class 'NoneType'>. Should be one of a python, numpy, pytorch or tensorflow object.

错误原因 #

1. 缺少BERT必需的特殊标记：BertTokenizer依赖[PAD]、[CLS]、[SEP]等特殊标记实现padding、句子边界处理等核心功能，你构建的词汇表仅包含业务相关的数字ID，没有这些特殊标记，导致tokenizer在执行padding操作时因无法识别默认填充标记，出现NoneType相关的类型错误。
2. 词汇表路径不匹配：初始化代码中指定的词汇表路径是./test_vocab.txt，但实际文件名是text_vocab.txt，路径错误会导致加载的词汇表异常，进一步触发问题。

内容的提问来源于stack exchange，提问作者dawn_hyeonseung