如何利用开发者工具与Python从网站提取图表数据集？及mcl_crop_status_data.php接口请求报错排查

Let's break down what's going wrong and how to fix this issue step by step:

1. Understand the Root Error #

The Notice: Undefined index: field error tells us exactly what's missing: the PHP endpoint mcl_crop_status_data.php expects a POST parameter named field to generate the crop status data, but your current request isn't sending it. This required parameter is what triggers the script to return the correct chart data.

2. Find the Exact field Parameter Value #

To get the valid value for field, use your browser's DevTools to inspect the actual request the site makes:

• Re-load the original chart page in Chrome/Firefox
• Open the Network tab in DevTools, filter requests by XHR
• Look for the mcl_crop_status_data.php POST request that loads the chart
• Check the Form Data section of that request—you'll see the exact field value the browser sends (it’s likely something like crop_status or a specific identifier tied to the chart type)

3. Updated Python Code with Fixes #

Here’s the revised script that addresses the missing parameter and other minor issues:

import requests

actionURL = "https://sasri.sasa.org.za/agronomy/mycanesimlite/mcl_crop_status_data.php"
referer_url = "https://sasri.sasa.org.za/agronomy/mycanesimlite/mcl_single_run_get_input.php?start_date=2021-06-15&harvest_date=2022-06-15&ratoon=R&residue=0&tam=150&irrigation=Rainfed&weather_station=14&forecast=Normal"

# Get valid session cookies dynamically (avoid hardcoding expired ones)
session = requests.Session()
session.get(referer_url)

# Add the missing 'field' parameter (replace with the value you found in DevTools)
post_data = {
    "method": "POST",
    "field": "crop_status"  # Use the actual value from your browser's request!
}

headers = {
    "Host": "sasri.sasa.org.za",
    "Connection": "keep-alive",
    "Accept": "*/*",
    "Origin": "https://sasri.sasa.org.za",
    "Sec-Fetch-Dest": "empty",
    "Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
    "X-Requested-With": "XMLHttpRequest",
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0",
    "sec-ch-ua-platform": "Linux",
    "Referer": referer_url,
    "Accept-Encoding": "gzip, deflate, br",
    "Accept-Language": "es-ES,es;q=0.8,en-US;q=0.5,en;q=0.3",
    "Sec-Fetch-Site": "same-origin",
    "Sec-Fetch-Mode": "cors"
}

# Send the POST request (requests handles Content-Length automatically)
response = session.post(actionURL, data=post_data, headers=headers)

# Handle response properly
if response.status_code == 200:
    try:
        crop_data = response.json()
        print(crop_data)
    except ValueError:
        print("Response is not valid JSON:", response.text)
else:
    print(f"Request failed with status code {response.status_code}:")
    print(response.text)

Key Improvements: #

• Added the missing field parameter: This fixes the core PHP error.
• Dynamic session cookies: Using a Session object ensures you always have a valid, non-expired session (hardcoded cookies will break eventually).
• Removed duplicate headers: The original code had two Origin entries, which can cause server-side validation issues.
• Removed manual Content-Length: Requests calculates this automatically, so manual values often lead to mismatches that break the request.
• Basic error handling: Helps debug if the response isn’t JSON or the request fails for other reasons.

Why Firefox Behaved Differently? #

Firefox likely handles the missing parameter more gracefully (or the server returns a less explicit error), but the underlying issue—missing the required field parameter—was still present. Fixing that parameter will resolve the problem across all browsers and scripts.

内容的提问来源于stack exchange，提问作者user19580074