保存GPT2 Tokenizer时遇TypeError：property类型无法JSON序列化的解决方法

问题重现 #

用户尝试保存GPT2 Tokenizer的代码如下：

from transformers import GPT2Tokenizer, GPT2LMHeadModel
import pandas as pd

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
tokenizer.pad_token = GPT2Tokenizer.eos_token
dataset_file = "x.csv"
df = pd.read_csv(dataset_file, sep=",")
input_ids = tokenizer.batch_encode_plus(list(df["x"]), max_length=1024,padding='max_length',truncation=True)["input_ids"]

# saving the tokenizer
tokenizer.save_pretrained("tokenfile")

运行后触发错误：

TypeError: Object of type property is not JSON serializable

完整错误回溯：

TypeError                                 Traceback (most recent call last)
Cell In[x], line 3
      1 # Save the fine-tuned model
----> 3 tokenizer.save_pretrained("tokenfile")

File /3tb/share/anaconda3/envs/ak_env/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:2130, in PreTrainedTokenizerBase.save_pretrained(self, save_directory, legacy_format, filename_prefix, push_to_hub, **kwargs)
   2128 write_dict = convert_added_tokens(self.special_tokens_map_extended, add_type_field=False)
   2129 with open(special_tokens_map_file, "w", encoding="utf-8") as f:
-> 2130     out_str = json.dumps(write_dict, indent=2, sort_keys=True, ensure_ascii=False) + "\n"
   2131     f.write(out_str)
   2132 logger.info(f"Special tokens file saved in {special_tokens_map_file}")

File /3tb/share/anaconda3/envs/ak_env/lib/python3.10/json/__init__.py:238, in dumps(obj, skipkeys, ensure_ascii, check_circular, allow_nan, cls, indent, separators, default, sort_keys, **kw)
    232 if cls is None:
    233     cls = JSONEncoder
    234 return cls(
    235     skipkeys=skipkeys, ensure_ascii=ensure_ascii,
    236     check_circular=check_circular, allow_nan=allow_nan, indent=indent,
    237     separators=separators, default=default, sort_keys=sort_keys,
-> 238     **kw).encode(obj)

File /3tb/share/anaconda3/envs/ak_env/lib/python3.10/json/encoder.py:201, in JSONEncoder.encode(self, o)
    199 chunks = self.iterencode(o, _one_shot=True)
...
    178     """
-> 179     raise TypeError(f'Object of type {o.__class__.__name__} '
    180                     f'is not JSON serializable')

TypeError: Object of type property is not JSON serializable

错误原因 #

代码中tokenizer.pad_token = GPT2Tokenizer.eos_token这一行存在核心问题：

• GPT2Tokenizer.eos_token是类的property属性，直接访问它得到的是一个property对象，而非实际的特殊标记字符串（比如GPT2默认的<|endoftext|>）。
• 当调用tokenizer.save_pretrained()时，会将tokenizer的特殊标记映射序列化为JSON文件，property对象无法被JSON序列化，因此抛出错误。

解决方法 #

将赋值语句改为使用tokenizer实例的eos_token属性，这样获取到的是实际的字符串值，可正常序列化：

修正后的完整代码 #

from transformers import GPT2Tokenizer, GPT2LMHeadModel
import pandas as pd

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
# 改为使用实例的eos_token属性
tokenizer.pad_token = tokenizer.eos_token
dataset_file = "x.csv"
df = pd.read_csv(dataset_file, sep=",")
input_ids = tokenizer.batch_encode_plus(list(df["x"]), max_length=1024,padding='max_length',truncation=True)["input_ids"]

# 保存tokenizer
tokenizer.save_pretrained("tokenfile")

额外说明 #

如果需要直接指定pad token的字符串，也可以写成：

tokenizer.pad_token = "<|endoftext|>"

效果和使用tokenizer.eos_token一致，因为GPT2的默认eos token就是<|endoftext|>。

内容的提问来源于stack exchange，提问作者AKMalkadi