提交两次POST表单后调用维多利亚州能源比价API获取报价列表返回重定向提示的技术问询

Let's break down what might be causing the {'status': 'redirect', 'message': 'no data'} error and fix your scraping flow step by step:

Key Issues Identified #

1. Incorrect Request Body Format: You're sending form data as URL-encoded (data=form_data), but the API likely expects JSON payloads—especially since your form_data1 includes a nested distributor object. URL-encoding can't properly serialize nested structures, so the server isn't saving your data correctly.
2. Stale serverCacheId: The serverCacheId might update after each form submission. Using the initial ID from the first GET request could lead to invalid state.
3. Missing Request Headers: Many modern APIs block non-browser requests. Adding basic browser-like headers can help avoid being flagged.

Step-by-Step Fix #

1. Update Request Headers #

First, configure your session to mimic a browser request. Add these headers to your requests.Session():

import requests as rs
from form_data import form_data1, form_data2

base_url = "https://compare.energy.vic.gov.au/api"

with rs.Session() as s:
    # Mimic browser headers to avoid being blocked
    s.headers.update({
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
        "Accept": "application/json, text/plain, */*",
        "Content-Type": "application/json"
    })

2. Fetch Initial serverCacheId #

Keep your initial GET request, but add error handling to confirm it works:

url_ = f"{base_url}/get-psb-details?serverCacheId=null"
    r = s.get(url_)
    r.raise_for_status()  # Throw error if request fails
    serverCacheId = r.json()["serverCacheId"]
    print(f"Initial serverCacheId: {serverCacheId}")

3. Submit Forms with JSON Payloads #

Replace data=form_data with json=form_data to send properly serialized JSON. Also, update serverCacheId from each POST response (it might change after form submissions):

# Submit first form
    r = s.post(f"{base_url}/save-form-data", json=form_data1)
    r.raise_for_status()
    first_post_response = r.json()
    print("First form submission response:", first_post_response)
    # Update serverCacheId if returned in response
    if "serverCacheId" in first_post_response:
        serverCacheId = first_post_response["serverCacheId"]

    # Submit second form
    r = s.post(f"{base_url}/save-form-data", json=form_data2)
    r.raise_for_status()
    second_post_response = r.json()
    print("Second form submission response:", second_post_response)
    if "serverCacheId" in second_post_response:
        serverCacheId = second_post_response["serverCacheId"]

4. Fetch Offers with Updated serverCacheId #

Use the latest serverCacheId to request offers, and verify the request parameters:

url_offers = f"{base_url}/get-offers"
    params = {
        "serverCacheId": str(serverCacheId),
        "loopBack": "false",
        "selectedEnergy": "/offer"
    }
    r = s.get(url_offers, params=params)
    r.raise_for_status()
    
    offer_data = r.json()
    print("Offer Data:", offer_data)

Additional Checks #

• Verify Form Submissions: Check the JSON responses from each save-form-data POST. Look for success indicators (like status: "success") to confirm your data was saved properly.
• Check for Required Fields: Ensure your form_data1 and form_data2 include all mandatory fields. The server might reject incomplete submissions even if it returns a 200 status.
• Try POST for Offers: If the GET request still fails, try sending the parameters as a JSON payload via POST (some APIs prefer this for data retrieval):

  r = s.post(url_offers, json=params)
  

Expected Result #

If all steps work correctly, you should receive the JSON structure you described, with the offers key containing the list of electricity quotes.

内容的提问来源于stack exchange，提问作者agstudy