WandB代理方案与OpenAI API不兼容引发ProxySchemeUnknown错误求助

环境信息 #

• WandB版本：0.15.3
• Langchain版本：0.0.172
• 操作系统：WSL Ubuntu 22.04.1 LTS
• Python版本：3.11.0

问题现象 #

使用以下代理配置时，OpenAI API和pip可正常工作：

export http_proxy="192.168.10.166:11111"
export https_proxy="192.168.10.166:11111"
export ALL_PROXY="192.168.10.166:11111"
export no_proxy="localhost,127.0.0.1,::1,192.168.*.*,10.0.*.*"

但为代理地址添加http://协议头（如export HTTPS_PROXY="http://192.168.10.166:11111/"）后，OpenAI API和pip无法正常工作；而不添加协议头时，运行含WandbCallbackHandler的代码（如下）会触发ProxySchemeUnknown错误，命令行执行wandb命令也会出现相同错误。

测试代码：

from langchain.callbacks import WandbCallbackHandler, StdOutCallbackHandler
from datetime import datetime

session_group = datetime.now().strftime("%m.%d.%Y_%H.%M.%S")
wandb_callback = WandbCallbackHandler(
    job_type="inference",
    project="langchain_callback_demo",
    group=f"minimal_{session_group}",
    name="llm",
    tags=["test"],
)
callbacks = [StdOutCallbackHandler(), wandb_callback]

错误详情 #

运行代码时的错误：

File ~/miniconda3/envs/lang/lib/python3.11/site-packages/urllib3/poolmanager.py:481, in ProxyManager.__init__(self, proxy_url, num_pools, headers, proxy_headers, proxy_ssl_context, use_forwarding_for_https, **connection_pool_kw)
    478 proxy = parse_url(proxy_url)
    480 if proxy.scheme not in ("http", "https"):
--> 481     raise ProxySchemeUnknown(proxy.scheme)
    483 if not proxy.port:
    484     port = port_by_scheme.get(proxy.scheme, 80)
ProxySchemeUnknown: Proxy URL had no scheme, should start with http:// or https://

命令行执行wandb命令的错误：

Traceback (most recent call last):
  File "/home/ocean/miniconda3/envs/lang/bin/wandb", line 5, in <module>
    from wandb.cli.cli import cli
  File "/home/ocean/miniconda3/envs/lang/lib/python3.11/site-packages/wandb/__init__.py", line 200, in <module>
    _sentry.setup()
  File "/home/ocean/miniconda3/envs/lang/lib/python3.11/site-packages/wandb/analytics/sentry.py", line 40, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ocean/miniconda3/envs/lang/lib/python3.11/site-packages/wandb/analytics/sentry.py", line 76, in setup
    client = sentry_sdk.Client(
             ^^^^^^^^^^^^^^^^^^
  File "/home/ocean/miniconda3/envs/lang/lib/python3.11/site-packages/sentry_sdk/client.py", line 143, in __init__
    self._init_impl()
  File "/home/ocean/miniconda3/envs/lang/lib/python3.11/site-packages/sentry_sdk/client.py", line 211, in _init_impl
    self.transport = make_transport(self.options)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ocean/miniconda3/envs/lang/lib/python3.11/site-packages/sentry_sdk/transport.py", line 534, in make_transport
    return transport_cls(options)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ocean/miniconda3/envs/lang/lib/python3.11/site-packages/sentry_sdk/transport.py", line 154, in __init__
    self._pool = self._make_pool(
                 ^^^^^^^^^^^^^^^^
  File "/home/ocean/miniconda3/envs/lang/lib/python3.11/site-packages/sentry_sdk/transport.py", line 444, in _make_pool
    return urllib3.ProxyManager(proxy, **opts)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ocean/miniconda3/envs/lang/lib/python3.11/site-packages/urllib3/poolmanager.py", line 481, in __init__
    raise ProxySchemeUnknown(proxy.scheme)
urllib3.exceptions.ProxySchemeUnknown: Proxy URL had no scheme, should start with http:// or https://
Exception ignored in atexit callback: <bound method Sentry.end_session of <wandb.analytics.sentry.Sentry object at 0x7ff642be4a10>>
Traceback (most recent call last):
  File "/home/ocean/miniconda3/envs/lang/lib/python3.11/site-packages/wandb/analytics/sentry.py", line 40, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ocean/miniconda3/envs/lang/lib/python3.11/site-packages/wandb/analytics/sentry.py", line 163, in end_session
    assert self.hub is not None
AssertionError:

方案一：统一代理变量的协议头配置 #

部分工具（如pip、OpenAI SDK）支持不带协议头的代理地址，但WandB依赖的urllib3严格要求代理地址带协议头。可以通过同时设置大小写的代理变量来兼容：

# 带协议头的变量供WandB使用
export HTTP_PROXY="http://192.168.10.166:11111"
export HTTPS_PROXY="http://192.168.10.166:11111"
export ALL_PROXY="http://192.168.10.166:11111"
# 不带协议头的变量供OpenAI和pip使用
export http_proxy="192.168.10.166:11111"
export https_proxy="192.168.10.166:11111"
export all_proxy="192.168.10.166:11111"
export no_proxy="localhost,127.0.0.1,::1,192.168.*.*,10.0.*.*"

注：大部分工具会优先读取大写的代理变量，若读取不到才会用小写；部分旧工具仅支持小写变量，因此同时设置可覆盖所有场景。

方案二：代码层面单独配置WandB代理 #

如果不想修改系统环境变量，可以在代码中单独为WandB设置带协议头的代理，不影响其他工具的配置：

import os
# 先为WandB单独设置带协议头的代理
os.environ['HTTP_PROXY'] = 'http://192.168.10.166:11111'
os.environ['HTTPS_PROXY'] = 'http://192.168.10.166:11111'

# 再导入并初始化WandbCallbackHandler
from langchain.callbacks import WandbCallbackHandler, StdOutCallbackHandler
from datetime import datetime

session_group = datetime.now().strftime("%m.%d.%Y_%H.%M.%S")
wandb_callback = WandbCallbackHandler(
    job_type="inference",
    project="langchain_callback_demo",
    group=f"minimal_{session_group}",
    name="llm",
    tags=["test"],
)
callbacks = [StdOutCallbackHandler(), wandb_callback]

方案三：修改urllib3的代理解析逻辑（不推荐） #

可以通过猴子补丁的方式修改urllib3的parse_url方法，自动为不带协议头的代理地址添加http://：

from urllib3.util import parse_url

original_parse_url = parse_url

def patched_parse_url(url):
    if url and '://' not in url:
        # 假设代理是http协议，若实际是https则替换成https://
        url = f'http://{url}'
    return original_parse_url(url)

# 应用补丁
import urllib3.util
urllib3.util.parse_url = patched_parse_url

# 之后再导入并使用WandB相关代码

注：该方法属于hack手段，可能会引发其他未知问题，仅在紧急情况下使用。

内容的提问来源于stack exchange，提问作者ocean11