M1 Mac下Langchain加载PDF遇Unicode问题：修复与文件定位

问题描述 #

在M1 Mac的Anaconda环境中用Langchain实现多PDF问答功能，部分100K-3MB的PDF加载失败。执行代码index = VectorstoreIndexCreator().from_loaders(loaders)时，出现错误：

"Ignoring (part of) ToUnicode map because the PDF data does not conform to the format. This could result in (cid) values in the output. The start and end byte have different lengths."

另一条提示"detectron2 is not installed. Cannot use the hi_res partitioning strategy. Falling back to partitioning with the fast strategy."无影响。自身Python基础较弱，原本希望错误信息能直接指出出错的PDF文件。

需求 #

1. 是否有Python或M1 Mac平台的Unix命令可修复PDF中的Unicode格式问题？
2. 如何获取触发上述错误的PDF文件报告，定位具体出错的PDF？

一、修复PDF的Unicode格式问题 #

1. Unix命令（M1 Mac可用）

借助pdftocairo（属于poppler工具集）重新生成PDF，重建正确的ToUnicode映射：

# 先通过Homebrew安装poppler
brew install poppler
# 转换单个PDF
pdftocairo -pdf input.pdf output_fixed.pdf
# 批量处理文件夹内所有PDF
for file in /path/to/your/pdf_folder/*.pdf; do
    pdftocairo -pdf "$file" "${file%.pdf}_fixed.pdf"
done

2. Python方案

使用PyPDF2重新保存PDF，修复编码兼容问题：

from PyPDF2 import PdfReader, PdfWriter
import os

def fix_pdf_unicode(input_path, output_path):
    reader = PdfReader(input_path)
    writer = PdfWriter()
    
    for page in reader.pages:
        writer.add_page(page)
    
    with open(output_path, "wb") as f:
        writer.write(f)

# 批量处理目标文件夹
pdf_folder_path = "/Users/.../Documents/.../testpdfs"
for fn in os.listdir(pdf_folder_path):
    if fn.endswith(".pdf"):
        input_path = os.path.join(pdf_folder_path, fn)
        output_path = os.path.join(pdf_folder_path, f"{os.path.splitext(fn)[0]}_fixed.pdf")
        fix_pdf_unicode(input_path, output_path)

二、定位出错的PDF文件 #

修改原代码，逐个加载PDF并捕获错误，直接输出问题文件名：

#load required packages
from langchain.document_loaders import UnstructuredPDFLoader
from langchain.indexes import VectorstoreIndexCreator
import os
os.environ["OPENAI_API_KEY"] = "你的API密钥"

pdf_folder_path = "/Users/.../Documents/.../testpdfs"

loaders = []
error_files = []

# 逐个加载PDF并捕获错误
for fn in os.listdir(pdf_folder_path):
    if fn.endswith(".pdf"):
        file_path = os.path.join(pdf_folder_path, fn)
        try:
            loader = UnstructuredPDFLoader(file_path)
            # 主动触发加载，检测错误
            loader.load()
            loaders.append(loader)
            print(f"成功加载: {fn}")
        except Exception as e:
            print(f"加载失败: {fn}，错误信息: {str(e)}")
            error_files.append(fn)

# 输出所有加载失败的文件
print("\n加载失败的PDF文件列表:")
for fn in error_files:
    print(fn)

# 仅用正常加载的文件创建索引
if loaders:
    index = VectorstoreIndexCreator().from_loaders(loaders)
    # 执行查询
    %%time
    index.query_with_sources('Tell me about xyz')
else:
    print("没有可用的PDF文件创建索引")

内容的提问来源于stack exchange，提问作者RyanNiccolls.com