LangChain Hugging Face分词文本拆分器存在token溢出与丢词问题

问题1：拆分破坏词结构导致token超量 #

拆分器按token拆分文本时，可能破坏原词的分词结构，导致拆分后的片段重新分词时token数量增加，最终超出模型最大长度限制。

以all-MiniLM-L6-v2模型为例：它会将“8 trillions”分词为['[CLS]', '8', 'trillion', '##s', '[SEP]']，若拆分点落在##s处，单独的“##s”重新分词会变成['[CLS]', '#', '#', 's', '[SEP]']，token数量从1个变为3个，直接导致片段长度超标。

复现代码 #

from transformers import (
    AutoTokenizer,
    PreTrainedTokenizer
)

model_name: str = "all-MiniLM-L6-v2"
tokenizer: PreTrainedTokenizer = AutoTokenizer.from_pretrained(
     f'sentence-transformers/{model_name}'
)
text = "8 trillions"
# 输出：['[CLS]', '8', 'trillion', '##s', '[SEP]']
print(tokenizer.batch_decode(tokenizer(text)['input_ids']))

# 单独处理拆分出的"##s"
# 输出：['[CLS]', '#', '#', 's', '[SEP]'] —— 原后缀token变为3个，数量增加
print(tokenizer.batch_decode(tokenizer("##s")['input_ids']))

问题2：一刀切截断首尾token引发内容丢失 #

拆分器默认所有模型的分词结构都是[start_token][tokens][end_token]，因此硬编码使用[1:-1]截断首尾token。但部分模型（如gtr-t5-large）的分词结果并非始终包含起始token，这种一刀切的操作会直接丢失文本内容。

比如分词“Madame”时，gtr-t5-large的结果无起始token，[1:-1]切片会直接去掉首词，导致拆分后的片段缺失关键内容。

复现代码 #

from transformers import (
    AutoTokenizer,
    PreTrainedTokenizer
)

model_name: str = "gtr-t5-large"
tokenizer: PreTrainedTokenizer = AutoTokenizer.from_pretrained(
     f'sentence-transformers/{model_name}'
)

text = "a"
# 输出：[3, 9, 1]
print(tokenizer(text)['input_ids'])
# 输出：['', 'a', '</s>']
print(tokenizer.batch_decode(tokenizer(text)['input_ids']))

text = "Madame"
# 输出：[27328, 1] —— 无起始token
print(tokenizer(text)['input_ids'])
# 输出：['Madame', '</s>']
print(tokenizer.batch_decode(tokenizer(text)['input_ids']))

from langchain.text_splitter import SentenceTransformersTokenTextSplitter

splitter = SentenceTransformersTokenTextSplitter(
    model_name="gtr-t5-large",
    chunk_overlap=0
)

text: str = "Madame Speaker, Vice President Biden, members of Congress, distinguished guests, and fellow Americans:"
# 输出：['Speaker, Vice President Biden, members of Congress, distinguished guests, and fellow Americans:'] —— "Madame"丢失
print(splitter.split_text(text))

核心逻辑缺陷代码 #

问题根源在于拆分器的编码函数硬编码截断首尾token，未适配不同模型的分词结构：

from typing import List

def split_text(self, text: str) -> List[str]:
    def encode_strip_start_and_stop_token_ids(text: str) -> List[int]:
        return self._encode(text)[1:-1]     # <----- 一刀切截断，未考虑模型分词结构差异

    tokenizer = Tokenizer(
        chunk_overlap=self._chunk_overlap,
        tokens_per_chunk=self.tokens_per_chunk,
        decode=self.tokenizer.decode,
        encode=encode_strip_start_and_stop_token_ids,
    )

    return split_text_on_tokens(text=text, tokenizer=tokenizer)

内容的提问来源于stack exchange，提问作者mon