Selenium爬取Nifty Indices历史数据时日期选择超时错误调试求助

Hey there, let's figure out why your date selection is timing out! The core issue here is iframe context confusion and some small missteps in how you're handling page context when interacting with date pickers. Let's break down the fixes step by step:

1. Root Cause Analysis #

Your script switches to the correct iframe when loading the page, but the select_date function tries to switch to an iframe again—this throws your driver into the wrong context, making it unable to find the date picker elements. Additionally, switching back to default_content right after selecting a date cuts off access to other form elements (like the second date picker and submit button) that live inside the iframe.

2. Step-by-Step Fixes & Modified Code #

Here's the revised script with key fixes highlighted:

import os
import time
import traceback
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import Select, WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# ✅ ChromeDriver Path
chrome_driver_path = r"C:\Users\abcd\Documents\chromedriver-win64\chromedriver.exe"

# ✅ Download directory
download_path = r"C:\Users\abcd\Downloads\Indices"

# ✅ Chrome Options
options = Options()
options.add_argument("--window-size=1920x1080")
options.add_argument("--disable-popup-blocking")
options.add_argument("--disable-blink-features=AutomationControlled")
# 🛠️ Auto-save downloads to your target folder (no need to move files later)
prefs = {"download.default_directory": download_path}
options.add_experimental_option("prefs", prefs)
# 🛠️ Mimic a real user agent to avoid anti-bot detection
options.add_argument("--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36")

# ✅ Simplified WebDriver Initialization (no need for Remote for local use)
service = Service(chrome_driver_path)
driver = webdriver.Chrome(service=service, options=options)

driver.set_page_load_timeout(300)
wait = WebDriverWait(driver, 60)

try:
    # ✅ Open the webpage
    url = "https://www.niftyindices.com/reports/historical-data"
    print(" Opening webpage...")
    driver.get(url)

    # ✅ Wait for page to fully load
    time.sleep(5)

    # ✅ Switch to the correct iframe ONCE (stay here for all form interactions)
    iframes = driver.find_elements(By.TAG_NAME, "iframe")
    if iframes:
        print(" Switching to iframe...")
        driver.switch_to.frame(iframes[0])

    # ✅ Select "P/E, P/B & Div. Yield values"
    print(" Clicking Historical Menu...")
    menu_button = wait.until(EC.element_to_be_clickable((By.ID, "HistoricalMenu")))
    menu_button.click()
    time.sleep(2)

    print(" Selecting 'P/E, P/B & Div.Yield values'...")
    div_yield_option = wait.until(
        EC.element_to_be_clickable((By.XPATH, "//li[contains(text(), 'P/E, P/B & Div.Yield values')]"))
    )
    div_yield_option.click()
    time.sleep(2)

    # ✅ Select "Equity" from dropdown
    print(" Selecting 'Equity' dropdown...")
    equity_dropdown = wait.until(EC.element_to_be_clickable((By.ID, "ddlHistoricaldivtypee")))
    Select(equity_dropdown).select_by_value("Equity")
    time.sleep(2)

    # ✅ Select "NIFTY 500" from dropdown
    print(" Selecting 'NIFTY 500' dropdown...")
    index_dropdown = wait.until(EC.element_to_be_clickable((By.ID, "ddlHistoricaldivtypeeindex")))
    Select(index_dropdown).select_by_value("NIFTY 500")
    time.sleep(2)

    # ✅ Revised Date Selection Function (no iframe switching!)
    def select_date(date_field_id, target_day, target_month, target_year):
        print(f" Selecting date: {target_day}-{target_month}-{target_year}")
    
        try:
            # Wait for date field to be interactable
            date_field = wait.until(EC.element_to_be_clickable((By.ID, date_field_id)))
            
            # Scroll to the field to ensure it's visible
            driver.execute_script("arguments[0].scrollIntoView({block: 'center'});", date_field)
            time.sleep(1)

            # Remove readonly attribute to allow direct value input
            driver.execute_script("arguments[0].removeAttribute('readonly');", date_field)
            
            # 🛠️ Directly set date via JavaScript (more reliable than UI date picker)
            driver.execute_script(f"arguments[0].value = '{target_day}/{target_month}/{target_year}';", date_field)
            time.sleep(2)

            print(f"✅ Successfully selected {target_day}-{target_month}-{target_year}")
    
        except Exception as e:
            print(f"❌ Error selecting date: {target_day}-{target_month}-{target_year}")
            print(traceback.format_exc())

    # ✅ Select FROM and TO dates
    select_date("datepickerFromDivYield", target_day=1, target_month=1, target_year=2024)
    select_date("datepickerToDivYield", target_day=31, target_month=1, target_year=2025)

    # ✅ Click Submit Button
    print(" Clicking Submit button...")
    submit_button = wait.until(EC.element_to_be_clickable((By.ID, "submit_buttonDivdata")))
    driver.execute_script("arguments[0].scrollIntoView({block: 'center'});", submit_button)
    time.sleep(1)
    submit_button.click()

    # ✅ Wait for CSV Download Link
    print(" Waiting for CSV download link...")
    csv_link = wait.until(EC.element_to_be_clickable((By.ID, "exporthistoricaldiv")))
    driver.execute_script("arguments[0].scrollIntoView({block: 'center'});", csv_link)
    time.sleep(1)
    csv_link.click()

    # ✅ Wait for Download to Complete
    print(" Waiting for file download...")
    time.sleep(15)

    # ✅ Verify Downloaded File
    print(" Checking downloaded files...")
    files = os.listdir(download_path)
    # Filter for relevant Nifty CSV files
    csv_files = [f for f in files if f.endswith(".csv") and "Nifty" in f]

    if csv_files:
        # Sort to get the latest downloaded file
        csv_files.sort(key=lambda x: os.path.getmtime(os.path.join(download_path, x)), reverse=True)
        print(f"✅ CSV file downloaded successfully: {csv_files[0]}")
    else:
        print("❌ No CSV file downloaded.")

except Exception as e:
    print(f"⚠️ An error occurred: {e}")
    print(traceback.format_exc())

finally:
    # Good practice: Switch back to default content before quitting
    driver.switch_to.default_content()
    driver.quit()

3. Key Improvements Explained #

• Fixed iframe context: We only switch to the iframe once at the start, and stay in that context for all form interactions (no more switching back and forth).
• Reliable date input: Replaced the fragile date picker UI clicks with direct JavaScript value injection—this avoids issues with date picker dropdowns timing out or being unresponsive.
• Anti-bot evasion: Added a real user agent string to mimic human traffic, and disabled automation detection flags.
• Simplified WebDriver setup: Removed unnecessary webdriver.Remote usage for local ChromeDriver.
• Better download handling: Configured Chrome to save files directly to your target folder, and added logic to verify the latest downloaded CSV.

4. Extra Troubleshooting Tips #

1. Check ChromeDriver version: Ensure your ChromeDriver version exactly matches your installed Chrome browser version.
2. Increase wait times: If you still get timeouts, bump up the WebDriverWait duration (e.g., from 60 to 90 seconds).
3. Inspect elements again: Double-check that the element IDs (datepickerFromDivYield, submit_buttonDivdata) haven't changed on the website.

备注：内容来源于stack exchange，提问作者Nithish