R httr包POST认证下载交互模式正常，函数中执行失败求助

I’ve run into this exact kind of session-based authentication issue before—interactive mode keeps your authenticated state alive across requests, but wrapping code into a function often breaks this because you’re accidentally starting fresh unauthenticated sessions for each call. Here’s how to fix it, tailored to ICPSR’s updated site:

The Core Problem #

In interactive mode, every requests call uses a hidden default session that automatically retains cookies between requests. But when you move code into a function, if you don’t explicitly reuse a single session object, each POST or download request will spin up a brand new, unauthenticated session. That’s why your auth works interactively but fails when wrapped.

Step-by-Step Solution #

1. Explicitly Use a requests.Session() Object #

This is the critical fix. A session persists cookies across all requests made with it, so your authentication state stays intact from the first POST to the final download.

2. Mirror Your Interactive Request Exactness #

ICPSR’s updated site might require you to first fetch the login page to grab hidden form fields (like CSRF tokens) before submitting credentials. Make sure every parameter, header, and request order matches what works in your interactive tests.

Example Working Function #

import requests

def icpsr_authenticate_and_download(username, password, download_url):
    # Create a single session to persist auth cookies across all requests
    with requests.Session() as sess:
        # First: Fetch the login page to get necessary cookies/hidden form fields
        login_page = sess.get("https://www.icpsr.umich.edu/rpxlogin?path=ICPSR")
        login_page.raise_for_status()  # Fail fast if the login page can't load

        # Build the first authentication payload (match what worked in interactive mode)
        first_auth_payload = {
            "username": username,
            "password": password,
            "submit": "Login",
            # Add any hidden fields you spotted in your interactive POST (check browser dev tools)
        }

        # Send first POST authentication request
        first_auth_resp = sess.post(
            "https://www.icpsr.umich.edu/rpxlogin?path=ICPSR",
            data=first_auth_payload
        )
        first_auth_resp.raise_for_status()

        # Send second POST authentication request (adjust URL/payload to match your working code)
        second_auth_payload = {
            # Fill in the exact parameters that worked in your interactive tests
        }
        second_auth_resp = sess.post(
            "https://www.icpsr.umich.edu/[your-second-auth-endpoint]",
            data=second_auth_payload
        )
        second_auth_resp.raise_for_status()

        # Download the data using the authenticated session
        download_resp = sess.get(download_url, stream=True)
        download_resp.raise_for_status()

        # Save the downloaded file (adjust filename as needed)
        with open("icpsr_downloaded_data.zip", "wb") as f:
            for chunk in download_resp.iter_content(chunk_size=8192):
                f.write(chunk)
        
        print(f"Success! Data saved to icpsr_downloaded_data.zip")

# Call the function with your credentials and target download URL
icpsr_authenticate_and_download(
    "your-icpsr-username",
    "your-icpsr-password",
    "https://www.icpsr.umich.edu/your-target-download-url"
)

Debugging Tips If It Still Fails #

• Compare Request Details: In interactive mode, print first_auth_resp.request.body and sess.headers, then do the same in the function. Ensure they’re identical—even a missing hidden field can break authentication.
• Inspect Response Content: Add print(first_auth_resp.text) after each auth request to see if ICPSR returns an error message (like invalid credentials or a missing CSRF token).
• Verify Redirects: ICPSR might redirect after auth—requests.Session() handles this by default, but you can check first_auth_resp.history to confirm redirects occurred as expected.

内容的提问来源于stack exchange，提问作者Anthony Damico