使用Langchain GenericLoader加载Java文件遇MIME类型不支持错误及解决

问题场景 #

需要将大量Java文件生成嵌入向量供LLM使用并存储到Chroma DB，此前处理PDF文件正常，但使用GenericLoader加载Java文件时触发错误：ValueError: Unsupported mime type: text/x-java-source

原错误代码 #

from langchain_community.document_loaders.generic import GenericLoader

def load_documents():
    loader = GenericLoader.from_filesystem(
        "./path/to/java/files",
        glob="**/*",
        suffixes=[".java"],
        parser=LanguageParser()
    )
    return loader.load()

错误栈信息 #

main()
  File "/Volumes/SamsungT5/langchain/populate_database.py", line 33, in main
    documents = load_documents()
                ^^^^^^^^^^^^^^^^
  File "/path/populate_database.py", line 50, in load_documents
    return loader.load()
           ^^^^^^^^^^^^^
  File "/opt/anaconda3/lib/python3.11/site-packages/langchain_core/document_loaders/base.py", line 29, in load
    return list(self.lazy_load())
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/lib/python3.11/site-packages/langchain_community/document_loaders/generic.py", line 116, in lazy_load
    yield from self.blob_parser.lazy_parse(blob)
  File "/opt/anaconda3/lib/python3.11/site-packages/langchain_community/document_loaders/parsers/generic.py", line 70, in lazy_parse
    raise ValueError(f"Unsupported mime type: {mimetype}")
ValueError: Unsupported mime type: text/x-java-source

解决方案 #

改用DirectoryLoader结合TextLoader加载Java文件，直接将Java文件作为纯文本处理，同时开启多线程提升加载效率：

from langchain_community.document_loaders import DirectoryLoader, TextLoader

def load_documents():
    DATA_PATH = "./path/to/java/files"
    loader = DirectoryLoader(DATA_PATH, glob="**/*.java", loader_cls=TextLoader, use_multithreading=True)
    return loader.load()

原因说明 #

GenericLoader搭配LanguageParser时，无法识别Java文件对应的text/x-java-source MIME类型，导致抛出不支持的错误。而TextLoader无需依赖MIME类型识别，直接读取文件文本内容，适配Java文件的加载需求。

内容的提问来源于stack exchange，提问作者chhil