使用Python Requests库爬取站点XML文件却得到空文件的问题

我尝试用Python从https://media.waec.wa.gov.au/下载目标XML文件，以下两种实现方式最终都生成空的XML文件。第一种方法通过指定本地保存的站点证书PEM文件路径消除了InsecureRequestWarning警告，但两种方法结果一致。

原代码示例 #

代码1 #

r = requests.get('https://media.waec.wa.gov.au/2022%20North%20West%20Central%20By-Election%20-%20LA%20VERBOSE%20RESULTS.xml', verify='~ file path for locally saved site certificate PEM file ~')
r.raw.decode_content = True
with open('~ file path for saved file ~', 'wb') as f:
        shutil.copyfileobj(r.raw, f)

代码2 #

r = requests.get('https://media.waec.wa.gov.au/2022%20North%20West%20Central%20By-Election%20-%20LA%20VERBOSE%20RESULTS.xml', verify=False)
r.raw.decode_content = True
with open('~ file path for saved file ~', 'wb') as f:
        shutil.copyfileobj(r.raw, f)

排查与解决方法 #

• 先验证请求状态：在requests.get后立即打印r.status_code，确认请求是否成功（200为正常）。若返回404，说明URL路径有误；若返回5xx系列码，大概率是目标服务器临时故障。
• 替换文件写入方式：放弃r.raw+shutil.copyfileobj的组合，直接使用r.content写入文件，这种方式更稳定：

import requests

url = 'https://media.waec.wa.gov.au/2022%20North%20West%20Central%20By-Election%20-%20LA%20VERBOSE%20RESULTS.xml'
r = requests.get(url, verify=False)

if r.status_code == 200:
    with open('本地保存路径.xml', 'wb') as f:
        f.write(r.content)
else:
    print(f"请求失败，状态码：{r.status_code}")

• 添加请求头模拟浏览器：部分服务器会拦截无User-Agent的请求，添加浏览器UA可规避这类限制：

import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'
}
url = 'https://media.waec.wa.gov.au/2022%20North%20West%20Central%20By-Election%20-%20LA%20VERBOSE%20RESULTS.xml'
r = requests.get(url, headers=headers, verify=False)

if r.status_code == 200:
    with open('本地保存路径.xml', 'wb') as f:
        f.write(r.content)

• 检查响应内容：打印r.text或r.content，确认返回的是预期XML内容还是错误页面（如403权限提示），以此定位问题根源。

内容的提问来源于stack exchange，提问作者Sergei Walankov