传递异步函数作为RecursiveCharacterTextSplitter参数时触发BigInt转换错误

我尝试将第三方tokenizer作为lengthFunction搭配RecursiveCharacterTextSplitter使用，根据文档说明RecursiveCharacterTextSplitter支持Promise类型的lengthFunction，但运行代码时触发TypeError: Cannot convert undefined to a BigInt错误，仅在报错前能看到console.log(tok === undefined);的输出。若用带虚拟延迟的Promise包裹数值，代码可正常运行。

代码示例 #

import { RecursiveCharacterTextSplitter } from "langchain/text_splitter";
import { BartTokenizer } from "@xenova/transformers";

// interface TextSplitterParams {
//     chunkSize: number;
//     chunkOverlap: number;
//     keepSeparator: boolean;
//     lengthFunction?: ((text: string) => number) | ((text: string) => Promise<number>);
// }

async function tokenizer_len(x) {
  return BartTokenizer.from_pretrained("facebook/bart-large").then(
    (tok) => tok(x)["input_ids"].size
  );
}

const splitter = new RecursiveCharacterTextSplitter({
  chunkSize: 1024,
  chunkOverlap: 50,
  lengthFunction: tokenizer_len,
});

async function chunkify(text) {
  let output = await splitter.createDocuments([text]);
  return output.map((key) => key.pageContent);
}

const data1 = fs.readFileSync("text.txt", "utf8").toString();
chunkify(data1).then((op) => {
  console.log(op);
});

错误信息 #

TypeError: Cannot convert undefined to a BigInt
    at BigInt (<anonymous>)
    at Array.map (<anonymous>)
    at Function._call (file:///media/instantinopaul/data/Code/ML/js-summarize/node_modules/@xenova/transformers/src/tokenizers.js:2325:50)
    at closure (file:///media/instantinopaul/data/Code/ML/js-summarize/node_modules/@xenova/transformers/src/utils/core.js:62:28)
    at RecursiveCharacterTextSplitter.tokenizer_len [as lengthFunction] (file:///media/instantinopaul/data/Code/ML/js-summarize/main.mjs:14:18)
    at async RecursiveCharacterTextSplitter._splitText (file:///media/instantinopaul/data/Code/ML/js-summarize/node_modules/langchain/dist/text_splitter.js:236:18)
    at async RecursiveCharacterTextSplitter.createDocuments (file:///media/instantinopaul/data/Code/ML/js-summarize/node_modules/langchain/dist/text_splitter.js:76:33)
    at async chunkify (file:///media/instantinopaul/data/Code/ML/js-summarize/main.mjs:27:16)

解决方案 #

问题根源 #

每次调用tokenizer_len都会重新加载BartTokenizer，而Tokenizer的初始化是异步操作，在RecursiveCharacterTextSplitter并发调用lengthFunction时，可能出现tokenizer还未完全初始化就被调用的情况，导致tok(x)返回的input_ids为undefined，进而触发BigInt转换错误。

修复代码 #

提前初始化并复用tokenizer，避免重复加载：

import { RecursiveCharacterTextSplitter } from "langchain/text_splitter";
import { BartTokenizer } from "@xenova/transformers";
import fs from 'fs'; // 补充缺失的fs模块导入

// 顶级await初始化tokenizer（需环境支持），或放在async入口函数中
const tok = await BartTokenizer.from_pretrained("facebook/bart-large");

async function tokenizer_len(x) {
  // 直接使用已初始化完成的tokenizer
  return tok(x)["input_ids"].size;
}

const splitter = new RecursiveCharacterTextSplitter({
  chunkSize: 1024,
  chunkOverlap: 50,
  lengthFunction: tokenizer_len,
});

async function chunkify(text) {
  let output = await splitter.createDocuments([text]);
  return output.map((key) => key.pageContent);
}

const data1 = fs.readFileSync("text.txt", "utf8").toString();
chunkify(data1).then((op) => {
  console.log(op);
});

说明 #

修改后tokenizer只会被加载一次，确保每次调用tokenizer_len时tokenizer已经就绪。如果环境不支持顶级await，可以将初始化逻辑放在一个async入口函数中执行。

内容的提问来源于stack exchange，提问作者Sayan Dey