爬取含多select字段的天气网站：提交参数后返回非预期日期数据

Hey there! Let's get you the December 5th, 2018 weather data you're after instead of December 1st. Here's what's going wrong and how to fix it:

1. Your POST Parameters Don't Match What the Site Expects #

When you submit that select dropdown on the website, the browser sends specific form fields that your current code isn't replicating correctly. Guessing parameter names/values often leads to the default response (which is why you keep getting Dec 1st).

2. How to Find the Correct Parameters #

First, let's uncover exactly what the site wants:

• Open the target page in your browser, then hit F12 to open Developer Tools.
• Go to the Network tab, then use the dropdown on the page to select December 5th, 2018 and click the submit button (usually labeled something like "View History").
• Look for the POST request in the Network log (it'll be to the same URL you're using). Click on it, then check the Form Data section—this shows every key-value pair the browser sends.

3. Corrected Code Example #

Based on how timeanddate.com's form typically works, here's a revised version of your code that should work. We'll use the actual parameters the site expects, plus maintain a session to handle cookies (which some sites require):

import requests

# Create a session to persist cookies (important for validating requests)
session = requests.Session()

# First, load the initial page to set up session cookies
initial_url = "https://www.timeanddate.com/weather/usa/new-york/historic?month=12&year=2018"
session.get(initial_url)

# Send the POST request with the correct form data
post_url = "https://www.timeanddate.com/weather/usa/new-york/historic"
data = {
    "date": "2018-12-05",  # This is the standard key for the selected date
    "month": "12",
    "year": "2018",
    "wt-his-select": "20181205",  # Keep this if it appears in the Form Data
    # Add any other hidden fields you find in the Form Data here
}

response = session.post(post_url, data=data)

# Verify you got the right page (check the first 500 characters)
print(response.text[:500])

4. Key Tips to Remember #

• Always use real Form Data: Don't guess parameters—sites often use hidden inputs or specific naming conventions that aren't obvious from the HTML alone.
• Session persistence: Using requests.Session() ensures cookies from the initial page load are carried over to the POST request, which many sites require to confirm you're a legitimate visitor.
• Simplify the POST URL: Notice we're posting to the base historic URL instead of the one with month/year parameters—those values are handled via the form data, not the URL.

内容的提问来源于stack exchange，提问作者Isomorphism