使用Python的requests库下载URL文件时遇重定向错误页的解决方法

问题描述 #

尝试用Python下载Dallas DCAD 2024评估数据文件，浏览器中可正常下载，但Python请求会被重定向到错误页面，返回Errors.aspx的HTML代码而非ZIP二进制数据。

尝试的代码 #

import requests

url = 'https://www.dallascad.org/ViewPDFs.aspx?type=3&id=\\DCAD.ORG\WEB\WEBDATA\WEBFORMS\DATA%20PRODUCTS\DCAD2024_CURRENT.ZIP'
headers = {
    "User-Agent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.93 Safari/537.36'
    }
r = requests.get(url, allow_redirects=True, headers=headers, timeout=None)
print(f"URL: {r.url}")
print(f"Status Code: {r.status_code}")
for i,h in enumerate(r.history):
    print(f"History[{i}] URL: {h.url}")
    print(f"History[{i}] Status: {h.status_code}")
    print(f"History[{i}] Headers: {h.headers}")

输出结果 #

URL: https://www.dallascad.org/Errors/ErrorPage.aspx?aspxerrorpath=/ViewPDFs.aspx
Status Code: 200
History[0] URL: https://www.dallascad.org/ViewPDFs.aspx?type=3&id=%5CDCAD.ORG%5CWEB%5CWEBDATA%5CWEBFORMS%5CDATA%20PRODUCTS%5CDCAD2024_CURRENT.ZIP
History[0] Status: 302
History[0] Headers: {'Cache-Control': 'private', 'Content-Type': 'text/html; charset=utf-8', 'Location': '/Errors/ErrorPage.aspx?aspxerrorpath=/ViewPDFs.aspx', 'Server': 'Microsoft-IIS/8.5', 'Content-Disposition': 'attachment;filename=DCAD2024_CURRENT.ZIP', 'X-AspNet-Version': '4.0.30319', 'X-Powered-By': 'ASP.NET', 'Date': 'Tue, 26 Mar 2024 14:35:36 GMT', 'Content-Length': '168'}

解决方案及修正代码 #

问题根源在于ASP.NET站点的会话验证和URL转义错误，修正方案如下：

1. 修正URL转义：Python字符串中的反斜杠会被解析为转义字符，导致请求参数与浏览器不一致，直接使用URL编码后的完整链接或原始字符串。
2. 维持会话Cookie：ASP.NET站点依赖Session ID验证请求合法性，用requests.Session()模拟浏览器会话。
3. 补充Referer头：告诉服务器请求来自站点内部，符合正常访问流程。

修正后的代码：

import requests

# 使用URL编码后的完整请求链接
url = 'https://www.dallascad.org/ViewPDFs.aspx?type=3&id=%5CDCAD.ORG%5CWEB%5CWEBDATA%5CWEBFORMS%5CDATA%20PRODUCTS%5CDCAD2024_CURRENT.ZIP'
headers = {
    "User-Agent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.93 Safari/537.36',
    "Referer": 'https://www.dallascad.org/'
}

# 初始化会话，维持Cookie状态
session = requests.Session()
# 先访问主页获取必要的Session Cookie
session.get('https://www.dallascad.org/', headers=headers)

# 发起下载请求
response = session.get(url, headers=headers, allow_redirects=True, timeout=None)

# 验证响应并保存文件
if 'attachment' in response.headers.get('Content-Disposition', '') or response.headers.get('Content-Type') == 'application/zip':
    with open('DCAD2024_CURRENT.ZIP', 'wb') as f:
        f.write(response.content)
    print("文件下载成功")
else:
    print("未获取到有效ZIP文件，响应状态：", response.status_code)
    print("响应内容预览：", response.text[:500])

内容的提问来源于stack exchange，提问作者user3311719