使用Langchain UnstructuredURLLoader时遇libmagic不可用问题求助

问题描述 #

使用UnstructuredURLLoader加载网页时出现两类核心错误：

• 提示“libmagic is unavailable but assists in filetype detection on file-like objects”
• 加载URL时抛出“Invalid file. The FileType.UNK file type is not supported in partition”

已尝试操作 #

• 安装langchain
• 安装unstructured、libmagic、python-magic、python-magic-bin
• 指定安装python-magic-bin==0.4.13
• 尝试不同版本的python-magic wheel包（运行环境为AMD64架构Windows）
• 卸载重装相关依赖
• 检索技术社区寻找解决方案

解决方案 #

方案1：修复libmagic依赖（推荐） #

Windows下python-magic-bin包内置预编译的libmagic库，需确保正确安装：

1. 彻底卸载现有magic相关包：

   pip uninstall -y python-magic python-magic-bin libmagic
   

2. 安装兼容Windows的指定版本：

   pip install python-magic-bin==0.4.14
   

3. 安装unstructured的HTML处理专用依赖：

   pip install unstructured[html]
   

4. 重新运行原代码，此时libmagic应能正常工作，Unstructured可识别网页MIME类型。

方案2：绕过文件类型检测（依赖安装无效时） #

手动获取网页内容并指定文件类型为HTML，避免libmagic的依赖问题：

from langchain.document_loaders import UnstructuredFileLoader
from io import BytesIO
import requests

urls = [
    "https://www.moneycontrol.com/news/business/banks/hdfc-bank-re-appoints-sanmoy-chakrabarti-as-chief-risk-officer-11259771.html",
    "https://www.moneycontrol.com/news/business/markets/market-corrects-post-rbi-ups-inflation-forecast-icrr-bet-on-these-top-10-rate-sensitive-stocks-ideas-11142611.html"
]

# 模拟浏览器请求头，规避网站反爬限制
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36"
}

data = []
for url in urls:
    resp = requests.get(url, headers=headers)
    # 将网页内容转为文件对象，手动指定类型为HTML
    file_obj = BytesIO(resp.content)
    loader = UnstructuredFileLoader(file_obj, file_type="text/html")
    data.extend(loader.load())

print(len(data))

方案3：使用无依赖替代加载方式 #

如果上述方案仍无效，可直接使用langchain的URLLoader（基于BeautifulSoup）替代UnstructuredURLLoader，无需依赖libmagic：

from langchain.document_loaders import URLLoader

loader = URLLoader(
    urls = [
        "https://www.moneycontrol.com/news/business/banks/hdfc-bank-re-appoints-sanmoy-chakrabarti-as-chief-risk-officer-11259771.html",
        "https://www.moneycontrol.com/news/business/markets/market-corrects-post-rbi-ups-inflation-forecast-icrr-bet-on-these-top-10-rate-sensitive-stocks-ideas-11142611.html"
    ],
    headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36"}
)
data = loader.load()
print(len(data))

内容的提问来源于stack exchange，提问作者Gregory Morris