自定义Normalizer无法序列化，如何解决Tokenizer保存异常？

问题场景 #

为自定义Tokenizer实现了Python版自定义Normalizer，训练完成后调用tokenizer.save()时抛出异常：

Exception: Custom Normalizer cannot be serialized

自定义Normalizer代码：

class CustomNormalizer:
    def normalize(self, normalized: NormalizedString):
        # Most of these can be replaced by a `Sequence` combining some provided Normalizer,
        # (ie Sequence([ NFKC(), Replace(Regex("\\s+"), " "), Lowercase() ])
        # and it should be the prefered way. That being said, here is an example of the kind
        # of things that can be done here:
        try:
            if normalized is None:
                noramlized = NormalizedString("")
            else:
                normalized.nfkc()
                normalized.filter(lambda char: not char.isnumeric())
                normalized.replace(Regex("\\s+"), " ")
                normalized.lowercase()
        except TypeError as te:
            print("Custom Normalizer TypeError:", te)
            print(normalized)

配套的Tokenizer训练代码：

model = models.WordPiece(unk_token="[UNK]")
tokenizer = Tokenizer(model)
tokenizer.normalizer = Normalizer.custom(CustomNormalizer())
trainer = trainers.WordPieceTrainer(
    vocab_size=2500, 
    special_tokens=special_tokens,
    show_progress=True
)
tokenizer.train_from_iterator(get_training_corpus(), trainer=trainer, length=len(dataset))

# Save the Tokenizer result
tokenizer.save('saved.json') # 抛出异常的行

解决方案 #

核心原因 #

tokenizers库底层基于Rust实现，保存Tokenizer时会将所有组件序列化为JSON格式，但自定义Python类无法被序列化为JSON，因此触发序列化失败异常。官方文档也明确推荐使用内置Normalizer组件组合替代自定义Python类。

具体实现 #

将自定义Normalizer的逻辑替换为官方提供的内置Normalizer组件组合，这些组件原生支持序列化：

from tokenizers import normalizers
from tokenizers.normalizers import NFKC, Lowercase, Replace, Regex, Filter

# 用内置组件组合实现与自定义Normalizer完全相同的逻辑
tokenizer.normalizer = normalizers.Sequence([
    NFKC(),  # 对应原代码的normalized.nfkc()
    Filter(lambda char: not char.isnumeric()),  # 对应filter逻辑
    Replace(Regex("\\s+"), " "),  # 对应替换空格逻辑
    Lowercase()  # 对应lowercase()
])

替换后重新训练Tokenizer，调用tokenizer.save('saved.json')即可正常保存。

额外提示 #

原自定义Normalizer代码中存在拼写错误：noramlized = NormalizedString("")应为normalized = NormalizedString("")，使用内置组件组合可避免此类代码错误。

内容的提问来源于stack exchange，提问作者Raptor