使用LangChain的UnstructuredURLLoader加载数据时遇TP_NUM_C_BUFS过小错误

尝试复现LangChain文档中的代码，通过UnstructuredURLLoader从URL列表加载HTML文件为文档格式，供NLP模型执行下游任务。执行data = loader.load()时卡顿超半小时，未加载任何文件，还抛出TP_NUM_C_BUFS too small: 50错误及堆栈跟踪。该问题曾在LangChain仓库标记已解决，原提问者通过Windows命令提示符执行解决，但本人操作后仍无效。

from langchain.document_loaders import UnstructuredURLLoader
import session_info

session_info.show()

urls = [
    "https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-february-8-2023",
    "https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-february-9-2023",
]

print(urls)

loader = UnstructuredURLLoader(urls=urls)

print(loader)

data = loader.load()

print(data)

D:\path>C:/Python310/python.exe d:/path/src/langchain-url-mwe.py
-----
langchain           0.0.157
session_info        1.0.0
-----
Python 3.10.8 (tags/v3.10.8:aaaf517, Oct 11 2022, 16:50:30) [MSC v.1933 64 bit (AMD64)]
Windows-10-10.0.19045-SP0
-----
Session information updated at 2023-05-14 21:22
['https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-february-8-2023', 'https://www.understandingwar.org/backgrounder/russian-offensive-campaign-assessment-february-9-2023']
<langchain.document_loaders.url.UnstructuredURLLoader object at 0x0000022A495A7C40>
      0 [main] python (12524) C:\Python310\python.exe: *** fatal error - Internal error: TP_NUM_C_BUFS too small: 50
Stack trace:
Frame        Function    Args
045573E7140  001800629AE (001802AEB6B, 00180274E41, 00180318880, 045573E4A70)
045573E7140  0018004846A (00000000020, 045573E5C94, 00000000001, 00040400000)
045573E7140  001800484A2 (00000000032, 000000010A1, 00180318880, 045573E5B70)
045573E7140  00180168194 (00061948FE1, 00000000000, 045573E5CA0, 045571C3000)
045573E7140  00180106358 (00000000000, 008000010A1, 00180318880, 22A4BD5DBC0)
045573E7140  0018005ADC3 (045573E7140, 00000000000, 0018024EBA0, 008000003E0)
045573E7140  0018014E0B4 (7FFD54886255, 00000000016, 008000003E0, 004B394F3A6)
045573E7140  00180198EEB (7FFD54886255, 00000000016, 008000003E0, 004B394F3A6)
045573E7140  004B393212C (00000000020, 22A487BBAF8, 00000000000, 22A6C7F4120)
045573E7140  004B3936867 (00000000000, 00000000001, 00000000000, 045573E7170)
045573E7140  7FFD1B0F4461 (0000000000A, 045573E73A0, 22A6C7F3D60, D6D941F662B1)
045573E7170  7FFD1B0F418D (00000000000, 045573E7390, 00000000000, 00000000000)
045573E73E0  7FFD1B0F4042 (22A487D3B10, 045573E7278, 7FFD1BD4B4D8, 045573E7380)
045573E73E0  7FFD1B1032B5 (004B39319A0, 045573E7390, 045573E73A0, 22A00000000)
045573E73E0  7FFD1B102EC8 (22A6DC70940, 22A6DB2F030, 00000000000, 04500001101)
00000000000  7FFD1B102A8C (7FFD1B102940, 00000000000, 00000000000, 00000000000)
045573E7610  7FFD1BD33457 (00000000001, 22A487D3B10, 22A6DBB9BA8, 00000000001)
045573E7970  7FFD1BD2F616 (00000000000, 045573E7C40, 00000000000, 00000000000)
22A6DC85D10  7FFD1BD49BAD (22A6DC7FC00, 90DC8CB43D62805D, 00000000000, 00000000000)
22A6DC85D10  7FFD1BDDCDB6 (7FFD1C1017F8, 22A6DC85D10, 22A6DBCF0E0, 00000000000)
7FFD1C1017F8  7FFD1BDDCC3F (22A6DC70918, 7FFD1C0F32A0, 00000000000, 22A6DC70918)
22A6DC7FC00  7FFD1BDDCB3F (00000000002, 22A49634BD0, 00000000000, 22A6DC70900)
00000000002  7FFD1BDA0FDD (22A487D3B10, 00000000002, 172B7BEF138823D3, 00000000001)
22A487D3B10  7FFD1BD593C8 (22A487D3B10, 22A6DC7FE00, 00000000000, 045573E7FE0)
22A6DC7FE00  7FFD1BD5919F (22A496350D0, 22A6DC70900, 22A487D3B10, 22A6DBAD7E0)
22A496350D0  7FFD1BD592E3 (045573E7F50, 00000000001, 22A48828EA0, 22A6DC7FE00)
045573E7F50  7FFD1BD345CE (00000000002, 22A496015B0, 22A6C709708, 00000000003)
045573E8300  7FFD1BD2D787 (22A6C709700, 8000000000000003, 22A6C709700, 00000000002)
045573E8300  7FFD1BD30018 (00000000002, 22A487D3B10, 22A6C70AE30, 00000000002)
045573E8660  7FFD1BD2F616 (00000000001, 22A49602180, 22A6C70A470, 00000000001)
045573E8A10  7FFD1BD2D787 (22A6C70A468, 8000000000000001, 22A6DC83EB0, 22A4962C7F0)
045573E8A10  7FFD1BD2ECE4 (00000000002, 22A49603380, 22A6DBE81D0, 00000000002)
End of stack trace (more stack frames may be present)
  23855 [main] python (12524) C:\Python310\python.exe: *** fatal error - Internal error: TP_NUM_C_BUFS too small: 50

TP_NUM_C_BUFS too small:50是Cygwin/MinGW运行时的内存缓冲区不足错误，UnstructuredURLLoader依赖的unstructured库底层使用这类环境工具处理HTML，在Windows系统下易触发该问题。虽有用户通过CMD执行解决，但Python版本、依赖库版本、系统权限等环境差异可能导致方案失效。

• 升级依赖库：将langchain和unstructured库升级至最新版本，新版本可能修复Windows下的兼容性问题：

  pip install --upgrade langchain unstructured
  

• 替换加载器：改用LangChain的WebBaseLoader或BeautifulSoupLoader替代UnstructuredURLLoader，规避底层工具的兼容性问题：

  from langchain.document_loaders import WebBaseLoader
  
  loader = WebBaseLoader(urls)
  data = loader.load()
  

• 禁用多进程加载：UnstructuredURLLoader默认启用多进程，在Windows下易引发冲突，手动切换为单进程模式：

  loader = UnstructuredURLLoader(urls=urls, mode="single")
  

• 使用WSL环境执行：在Windows子系统Linux（WSL）中运行代码，避开Windows下的Cygwin/MinGW兼容性限制。

内容的提问来源于stack exchange，提问作者Carlos Luis Rivera