使用Pandas加载Google Drive Parquet文件时偶发ArrowInvalid错误求助

尝试用以下代码从Google Drive读取Parquet文件，文件ID已确认正确，但代码有时成功有时失败，失败时抛出错误：
ArrowInvalid: Could not open Parquet input source '<Buffer>': Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.

代码如下：

import pandas as pd 
import numpy as np
import requests
from io import BytesIO

def load_google(code):
    url = f"https://drive.google.com/uc?export=download&id={code}"
    file = requests.get(url)
    bytesio = BytesIO(file.content)
    return pd.read_parquet(bytesio, dtype_backend="pyarrow")

def load_data():
    list_fonds = load_google('161Bg01QO6sqBzgQbeuDrvhApPSxQqOGz')
    list_sdg   = load_google('163OEuakI8pEVx2t3Myy7KrGlM3YgD5zr')
    list_aum = load_google('1679R9CLRCvtzlDutH98BJ03eLWq2O2Av')
    categorie_fonds = load_google('16XGciQdAasNLVNPA-FRChInhOkyEN20O')
    iso_pays = load_google('166Kcz-KS2p8ypbilSOdshAx-nlSbkAll')
    return list_sdg, list_fonds, list_aum, iso_pays, categorie_fonds

# import of data 
list_sdg, list_fonds, list_aum, iso_pays, categorie_fonds = load_data()

• Google Drive反爬/流量限制：频繁请求或被判定为可疑请求时，返回的不是Parquet文件，而是HTML验证页面/警告页面，内容不符合Parquet格式，触发magic bytes错误。
• 大文件未处理验证：文件体积较大时，Google Drive会要求额外确认步骤，直接用requests.get获取到的是确认页面内容，而非实际文件。
• 网络不稳定：请求过程中断连或内容丢失，导致获取的文件不完整，Parquet格式损坏。

方法1：处理验证页面与重定向 #

修改load_google函数，适配Google Drive的大文件验证机制：

import pandas as pd 
import requests
from io import BytesIO

def load_google(code):
    url = f"https://drive.google.com/uc?export=download&id={code}"
    session = requests.Session()
    # 添加浏览器UA降低被拦截概率
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'
    }
    response = session.get(url, stream=True, headers=headers)
    
    # 处理大文件的下载验证cookie
    for key, value in response.cookies.items():
        if key.startswith('download_warning'):
            params = {'id': code, 'confirm': value}
            response = session.get(url, params=params, stream=True, headers=headers)
            break
    
    bytesio = BytesIO(response.content)
    return pd.read_parquet(bytesio, dtype_backend="pyarrow")

方法2：添加重试机制 #

针对网络不稳定情况，用重试逻辑提高成功率（需先安装tenacity：pip install tenacity）：

from tenacity import retry, stop_after_attempt, wait_exponential

@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10))
def load_google(code):
    # 放入上述修改后的请求代码
    url = f"https://drive.google.com/uc?export=download&id={code}"
    session = requests.Session()
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'
    }
    response = session.get(url, stream=True, headers=headers)
    
    for key, value in response.cookies.items():
        if key.startswith('download_warning'):
            params = {'id': code, 'confirm': value}
            response = session.get(url, params=params, stream=True, headers=headers)
            break
    
    bytesio = BytesIO(response.content)
    return pd.read_parquet(bytesio, dtype_backend="pyarrow")

方法3：检查文件权限 #

确保Google Drive文件权限设置为**"任何有链接的人都可以查看/下载"**，权限不足会返回HTML权限提示页面，导致格式错误。

内容的提问来源于stack exchange，提问作者jacques