如何在LangChain中避免特定文本块被RecursiveCharacterTextSplitter拆分

我在使用LangChain的RecursiveCharacterTextSplitter拆分字符串时，需要让一段被<nosplit>标记的子串保持完整，不被拆分（可以单独成块，也可以和前后内容合并）。

示例场景 #

现有代码：

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(chunk_size=5, chunk_overlap=2, separators=[' '], keep_separator=False)
nosplit = '<nosplit>Keep all this together, very important! Seriously though it is...<nosplit>'
text = 'Giggity! ' + nosplit + 'Ahh yeah...\nI just buy a jetski.'
chunks = splitter.split_text(text)
print(chunks)

当前输出：

['Giggity!', 'Keep', 'all', 'this', 'together,', 'very', 'important!', 'Seriously', 'though', 'it', 'is...Ahh', 'yeah...\nI', 'just', 'buy a', 'jetski.']

期望输出：

['Giggity!', 'Keep all this together, very important! Seriously though it is...', 'Ahh', 'yeah...\nI', 'just', 'buy a', 'jetski.']

现有方案的局限 #

我尝试过将<nosplit>设为优先分隔符，临时替换子串内的其他分隔符为占位符，拆分后再还原：

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(chunk_size=5, chunk_overlap=2, separators=['<nosplit>', ' '], keep_separator=False)
nosplit = '<nosplit>Keep all this together, very important! Seriously though it is...<nosplit>'
space_word = 'x179lp'
nosplit = nosplit.replace(' ', space_word)
text = 'Giggity!' + nosplit + 'Ahh yeah...\nI just buy a jetski.'
chunks = splitter.split_text(text)
for i, chunk in enumerate(chunks):
    chunks[i] = chunk.replace(space_word, ' ')
print(chunks)

但这种方法无法支持包含空字符串''的默认分隔符列表（我需要使用["<nosplit>", "\n\n", "\n", " ", ""]来实现字符级拆分）。

可行解决方案 #

可以通过占位符替换+映射还原的方式实现，步骤如下：

1. 提取文本中所有被<nosplit>包裹的内容，用唯一占位符替换它们，并建立占位符与原内容的映射。
2. 使用RecursiveCharacterTextSplitter处理替换后的文本，占位符会被视为不可拆分的整体。
3. 将拆分结果中的占位符还原为原内容。

示例代码：

import re
from uuid import uuid4
from langchain.text_splitter import RecursiveCharacterTextSplitter

# 定义处理逻辑
def split_with_keep_nosplit(text, chunk_size=5, chunk_overlap=2):
    # 1. 提取<nosplit>块并替换为唯一占位符
    nosplit_pattern = re.compile(r'<nosplit>(.*?)<nosplit>', re.DOTALL)
    placeholder_map = {}
    
    def replace_nosplit(match):
        content = match.group(1)
        placeholder = f"__NOSPLIT_{uuid4().hex[:8]}__"
        placeholder_map[placeholder] = content
        return placeholder
    
    processed_text = nosplit_pattern.sub(replace_nosplit, text)
    
    # 2. 使用默认分隔符列表（包含''）拆分文本
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=chunk_size,
        chunk_overlap=chunk_overlap,
        separators=["\n\n", "\n", " ", ""]
    )
    chunks = splitter.split_text(processed_text)
    
    # 3. 还原占位符为原内容
    final_chunks = []
    for chunk in chunks:
        for placeholder, content in placeholder_map.items():
            chunk = chunk.replace(placeholder, content)
        final_chunks.append(chunk)
    
    return final_chunks

# 测试
nosplit = '<nosplit>Keep all this together, very important! Seriously though it is...<nosplit>'
text = 'Giggity! ' + nosplit + 'Ahh yeah...\nI just buy a jetski.'
chunks = split_with_keep_nosplit(text, chunk_size=5, chunk_overlap=2)
print(chunks)

运行输出：

['Giggity!', 'Keep all this together, very important! Seriously though it is...', 'Ahh', 'yeah...\nI', 'just', 'buy a', 'jetski.']

方案说明 #

• 使用正则表达式精准匹配<nosplit>包裹的内容，避免误替换。
• 用UUID生成唯一占位符，确保不会与文本原有内容冲突。
• 兼容包含空字符串的默认分隔符列表，支持字符级拆分，同时保证标记的子串始终完整。

内容的提问来源于stack exchange，提问作者DMcC