遍历URL爬取数据时，循环调用日期字段遇阻求助

Hey David, let's get that loop working properly—you're already close, just missing a key step to pull year/month/day from your date elements!

The Core Issue #

Right now, your loop references year, month, and day like they're standalone variables, but they don't exist in the loop's context. Even if those fields are in a DataFrame somewhere, your loop is iterating over date_list, so you need to extract the year/month/day directly from each date item in that list.

Solutions Based on Your date_list Type #

First, figure out what kind of data is in date_list—here are the two most common scenarios:

1. If date_list contains datetime objects

If each date is a Python datetime object, you can pull the year/month/day directly with built-in attributes. Just make sure to format months/days as two-digit strings (most weather URLs expect this, like 05 instead of 5):

for date in date_list: # iterate through dates
    # Extract and format year/month/day
    year = str(date.year)
    month = f"{date.month:02d}"  # Adds leading zero for single-digit months
    day = f"{date.day:02d}"     # Same for days
    
    # Build the full URL and fetch data
    full_url = f"{base_url}KOWD/{year}/{month}/{day}{end_url}"
    # Fetch the table(s) — read_html returns a list of DataFrames
    fetched_tables = pd.read_html(full_url)
    
    # Add the relevant table to your weather_list (usually the first one)
    weather_list.append(fetched_tables[0])

2. If date_list contains date strings (e.g., "2024-05-20")

If your dates are strings, convert them to datetime objects first to easily extract components:

from datetime import datetime

for date_str in date_list: # iterate through date strings
    # Convert string to datetime object
    date = datetime.strptime(date_str, "%Y-%m-%d")  # Adjust format if your strings are different
    
    # Extract and format components
    year = str(date.year)
    month = f"{date.month:02d}"
    day = f"{date.day:02d}"
    
    # Build URL and fetch data
    full_url = f"{base_url}KOWD/{year}/{month}/{day}{end_url}"
    fetched_tables = pd.read_html(full_url)
    
    # Append to your master list
    weather_list.append(fetched_tables[0])

Quick Extra Tip #

Notice I changed weather_list = pd.read_html(...) to weather_list.append(...)—your original code would overwrite the list every loop, so you'd only end up with the last day's data. Using append keeps all your collected data in one place.

内容的提问来源于stack exchange，提问作者David Black