如何确保Python requests库获取完整的CSV请求响应内容？

我用Python的requests库获取一份537KB的天气数据CSV文件，每次运行代码时，响应长度都有细微差异，导致转换后的pandas DataFrame行数不一致。

代码示例 #

import requests
import pandas as pd
import io

url = "http://data-prod-cocorahs-org.azurewebsites.net/export/exportreports.aspx?ReportType=Daily&dtf=1&Format=CSV&ReportDateType=reportdate&Date=10/12/2023&TimesInGMT=False"
response = requests.get(url, timeout=2.50)
print(f"response length: {len(response.content)}")
content = response.content.decode('utf-8')
df = pd.read_csv(io.StringIO(content), index_col=False)
print(len(df))

两次运行结果示例 #

response length: 1551255
11257

response length: 1556748
11297

1. 先确认请求是否成功 #

首先检查响应状态码，排除请求失败导致的不完整内容：

response = requests.get(url, timeout=2.50)
response.raise_for_status()  # 状态码非200时直接抛出异常

2. 流式读取确保完整接收 #

改用stream=True分块读取响应内容，避免网络波动导致的截断：

response = requests.get(url, timeout=2.50, stream=True)
response.raise_for_status()

# 逐块读取全部内容
content = b''
for chunk in response.iter_content(chunk_size=8192):
    content += chunk

# 后续解析逻辑不变
content_str = content.decode('utf-8')
df = pd.read_csv(io.StringIO(content_str), index_col=False)

3. 验证内容长度（如果服务器支持） #

对比响应头中的Content-Length和实际接收的字节数，确认内容完整：

response = requests.get(url, timeout=2.50)
response.raise_for_status()

expected_length = int(response.headers.get('Content-Length', 0))
actual_length = len(response.content)

if expected_length != 0 and actual_length != expected_length:
    print(f"内容不完整：预期{expected_length}字节，实际{actual_length}字节")
    # 此处可添加重试逻辑

注意：若服务器未返回Content-Length头（动态生成内容常见），此方法不适用。

4. 添加自动重试机制 #

针对网络不稳定的情况，用重试库自动重新请求：
先安装依赖：

pip install tenacity

再修改代码：

from tenacity import retry, stop_after_attempt, wait_exponential

@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10))
def get_full_csv(url):
    response = requests.get(url, timeout=2.50)
    response.raise_for_status()
    # 可选：验证内容长度（若有Content-Length）
    expected_length = int(response.headers.get('Content-Length', 0))
    if expected_length != 0 and len(response.content) != expected_length:
        raise ValueError("内容不完整，触发重试")
    return response.content

content = get_full_csv(url)
content_str = content.decode('utf-8')
df = pd.read_csv(io.StringIO(content_str), index_col=False)

5. 直接用pandas读取URL（最简方案） #

pandas内置了HTTP请求处理逻辑，直接读取URL可能更稳定：

import pandas as pd

url = "http://data-prod-cocorahs-org.azurewebsites.net/export/exportreports.aspx?ReportType=Daily&dtf=1&Format=CSV&ReportDateType=reportdate&Date=10/12/2023&TimesInGMT=False"
df = pd.read_csv(url, index_col=False)
print(len(df))

内容的提问来源于stack exchange，提问作者p1unge