Pandas read_csv访问有效URL触发HTTPError: Not Found如何解决

数据老陈

2026-9-1

问题背景

调用pandas的read_csv方法读取HTTP协议的远程CSV文件时，抛出HTTPError: Not Found异常
目标URL直接在浏览器中访问可正常获取CSV文件，确认文件存在、URL地址有效
相同代码在旧Python版本下可正常运行，更换为Python 3.9.12、Pandas 1.2.5环境后出现异常
复现代码及完整报错堆栈如下：

import pandas as pd

url= r'http://fc3eqrp01:8000/data/SimulationOutputFilesNewModel_ShortTerm/s2outs.csv'
df = pd.read_csv(url)

Traceback (most recent call last):
  File E:\PythontoSQL\Save S2Outs\Save_S2Outs.py:13 in <module>
    df = pd.read_csv(url)
  File ~\AppData\Roaming\Python\Python39\site-packages\pandas\io\parsers.py:610 in read_csv
    return _read(filepath_or_buffer, kwds)
  File ~\AppData\Roaming\Python\Python39\site-packages\pandas\io\parsers.py:462 in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File ~\AppData\Roaming\Python\Python39\site-packages\pandas\io\parsers.py:819 in __init__
    self._engine = self._make_engine(self.engine)
  File ~\AppData\Roaming\Python\Python39\site-packages\pandas\io\parsers.py:1050 in _make_engine
    return mapping[engine](self.f, **self.options)
  File ~\AppData\Roaming\Python\Python39\site-packages\pandas\io\parsers.py:1867 in __init__
    self._open_handles(src, kwds)
  File ~\AppData\Roaming\Python\Python39\site-packages\pandas\io\parsers.py:1362 in _open_handles
    self.handles = get_handle(
  File ~\AppData\Roaming\Python\Python39\site-packages\pandas\io\common.py:558 in get_handle
    ioargs = _get_filepath_or_buffer(
  File ~\AppData\Roaming\Python\Python39\site-packages\pandas\io\common.py:289 in _get_filepath_or_buffer
    req = urlopen(filepath_or_buffer)
  File ~\AppData\Roaming\Python\Python39\site-packages\pandas\io\common.py:195 in urlopen
    return urllib.request.urlopen(*args, **kwargs)
  File C:\ProgramData\Anaconda3\Lib\urllib\request.py:214 in urlopen
    return opener.open(url, data, timeout)
  File C:\ProgramData\Anaconda3\Lib\urllib\request.py:523 in open
    response = meth(req, response)
  File C:\ProgramData\Anaconda3\Lib\urllib\request.py:632 in http_response
    response = self.parent.error(
  File C:\ProgramData\Anaconda3\Lib\urllib\request.py:561 in error
    return self._call_chain(*args)
  File C:\ProgramData\Anaconda3\Lib\urllib\request.py:494 in _call_chain
    result = func(*args)
  File C:\ProgramData\Anaconda3\Lib\urllib\request.py:641 in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

HTTPError: Not Found

问题根因

从报错堆栈可以确认，pandas 1.2.5读取远程URL时，直接调用Python标准库urllib.request.urlopen发起请求，未自定义任何请求头。
Python 3.9版本的urllib默认携带的User-Agent标识为Python-urllib/3.9，大量HTTP文件服务、反向代理规则会拦截非浏览器标识的客户端请求，直接返回404状态码，这就是浏览器访问正常、代码请求报错的核心原因。旧版本Python的urllib默认请求特征未命中服务端拦截规则，因此代码可正常运行。

解决方法

两种方案都可解决该问题，优先推荐第一种：

方案1：使用requests库手动发起请求，自定义请求头模拟浏览器访问，将返回的内容传给pandas解析，绕开pandas内置的请求逻辑，不受版本默认行为变动影响

import pandas as pd
import requests
from io import StringIO

url = r'http://fc3eqrp01:8000/data/SimulationOutputFilesNewModel_ShortTerm/s2outs.csv'
# 模拟浏览器请求头，避免被服务端拦截
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
}
resp = requests.get(url, headers=headers)
resp.raise_for_status()  # 请求非200状态时直接抛出异常
df = pd.read_csv(StringIO(resp.text))

方案2：不引入第三方requests库，基于标准库urllib手动构造带合法UA的请求，再传给pandas读取

import pandas as pd
from urllib.request import Request, urlopen

url = r'http://fc3eqrp01:8000/data/SimulationOutputFilesNewModel_ShortTerm/s2outs.csv'
req = Request(
    url,
    headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"}
)
with urlopen(req) as resp:
    df = pd.read_csv(resp)

内容的提问来源于stack exchange，提问作者Bobby Python