TensorFlow 2.17.0-dev中无法导入Keras Tokenizer，无法创建tokenizer.pkl

1. 正确导入Tokenizer（适配TensorFlow 2.17.0-dev） #

TensorFlow 2.16及以上版本默认使用Keras 3.x，原有的Tokenizer导入路径已变更，使用以下任一方式导入：

# 方式1：TensorFlow内置Keras路径
from tensorflow.keras.preprocessing.text import Tokenizer

# 方式2：单独安装Keras 3.x后的路径（需先pip install keras）
from keras.preprocessing.text import Tokenizer

如果环境中存在多版本Keras冲突，优先选择TensorFlow内置路径。

2. 替代keras_export实现自定义Tokenizer #

keras_export是Keras内部的API导出装饰器，自定义Tokenizer时完全不需要依赖它。只需实现核心功能即可，以下是极简版实现示例：

import numpy as np
from collections import defaultdict

class CustomTokenizer:
    def __init__(self, num_words=None, oov_token=None):
        self.num_words = num_words
        self.oov_token = oov_token
        self.word_index = {}
        self.index_word = {}
        self.word_counts = defaultdict(int)

    def fit_on_texts(self, texts):
        # 统计词频
        for text in texts:
            for word in text.split():
                self.word_counts[word] += 1
        # 按词频排序生成词索引
        sorted_words = sorted(self.word_counts.items(), key=lambda x: (-x[1], x[0]))
        # 处理OOV token
        current_idx = 1
        if self.oov_token is not None:
            self.word_index[self.oov_token] = current_idx
            current_idx += 1
        # 填充词-索引映射
        for word, _ in sorted_words:
            if self.num_words and current_idx >= self.num_words:
                break
            self.word_index[word] = current_idx
            self.index_word[current_idx] = word
            current_idx += 1

    def texts_to_sequences(self, texts):
        sequences = []
        for text in texts:
            seq = []
            for word in text.split():
                if word in self.word_index:
                    seq.append(self.word_index[word])
                elif self.oov_token:
                    seq.append(self.word_index[self.oov_token])
            sequences.append(seq)
        return sequences

    def sequences_to_texts(self, sequences):
        texts = []
        for seq in sequences:
            text = ' '.join([self.index_word.get(idx, '') for idx in seq if idx in self.index_word])
            texts.append(text)
        return texts

该实现覆盖了图像字幕生成所需的核心功能。

3. 创建并保存tokenizer.pkl文件 #

使用pickle即可完成Tokenizer的保存与加载：

import pickle

# 初始化并拟合Tokenizer
tokenizer = Tokenizer(num_words=10000, oov_token='<OOV>')
tokenizer.fit_on_texts(your_caption_dataset)  # 替换为你的字幕文本列表

# 保存为pkl文件
with open('tokenizer.pkl', 'wb') as f:
    pickle.dump(tokenizer, f)

# 后续加载使用
with open('tokenizer.pkl', 'rb') as f:
    loaded_tokenizer = pickle.load(f)

内容的提问来源于stack exchange，提问作者Bolofo