Selenium Python遍历非Select下拉框：获取全部选项及循环方案

Hey Raymond, great question! Custom dropdowns (non-native <select> elements) can be tricky because they often lazy-load options as you scroll—this is exactly why you only pulled 6 options instead of the full 50. Let's break down how to fix this and loop through all options efficiently:

Why You're Only Seeing 6 Options #

That custom dropdown doesn't load all options upfront. It only renders the ones visible in the viewport, and loads more as you scroll down the dropdown container. Your initial code grabbed whatever was already loaded, hence the small count.

Step 1: Load All Options by Scrolling the Dropdown #

First, we'll open the dropdown, then repeatedly scroll its container until all options are loaded (we'll know when the scroll height stops changing):

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

# 1. Open the dropdown (using explicit wait instead of sleep for reliability)
dropdown_trigger = WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.XPATH, '//span[@id="ctl00_ContentPlaceHolder1_ReportViewer1_ctl09_ctl21_ctl01"]'))
)
dropdown_trigger.click()

# 2. Wait for the dropdown container to be visible
dropdown_container = WebDriverWait(driver, 10).until(
    EC.visibility_of_element_located((By.ID, "ctl00_ContentPlaceHolder1_ReportViewer1_ctl09_ctl21_divDropDown"))
)

# 3. Scroll to load all options
last_scroll_height = driver.execute_script("return arguments[0].scrollHeight", dropdown_container)
while True:
    # Scroll to the bottom of the container
    driver.execute_script("arguments[0].scrollTop = arguments[0].scrollHeight", dropdown_container)
    time.sleep(0.5)  # Give time for new options to load
    new_scroll_height = driver.execute_script("return arguments[0].scrollHeight", dropdown_container)
    
    # If scroll height doesn't change, we've loaded all options
    if new_scroll_height == last_scroll_height:
        break
    last_scroll_height = new_scroll_height

Step 2: Grab All Options & Loop Through Them #

Now that every option is loaded, we can collect them, then loop through each one. Note that after selecting an option, the dropdown closes—so we'll need to re-open it for each selection:

# Get all option elements (target the actual option tags, not all child elements)
# Adjust the XPath to match the actual option elements (e.g., li, div with specific class)
all_options = dropdown_container.find_elements(By.XPATH, ".//div[starts-with(@id, 'ctl00_ContentPlaceHolder1_ReportViewer1_ctl09_ctl21_ctl03')]")
print(f"Total options loaded: {len(all_options)}")

# Collect option texts to use for re-locating later (in case the DOM refreshes)
option_texts = [opt.text.strip() for opt in all_options if opt.text.strip()]

# Loop through each option
for option_text in option_texts:
    # Re-open the dropdown
    dropdown_trigger = WebDriverWait(driver, 10).until(
        EC.element_to_be_clickable((By.XPATH, '//span[@id="ctl00_ContentPlaceHolder1_ReportViewer1_ctl09_ctl21_ctl01"]'))
    )
    dropdown_trigger.click()
    
    # Wait for the option to be clickable, then select it
    target_option = WebDriverWait(driver, 10).until(
        EC.element_to_be_clickable((By.XPATH, f"//div[@id='ctl00_ContentPlaceHolder1_ReportViewer1_ctl09_ctl21_divDropDown']//*[text()='{option_text}']"))
    )
    target_option.click()
    
    # Wait for the report to load (adjust timeout based on how long the report takes)
    WebDriverWait(driver, 20).until(
        EC.presence_of_element_located((By.ID, "ctl00_ContentPlaceHolder1_ReportViewer1_ctl09_ctl07"))  # Replace with a report-specific element
    )
    
    # Add your logic here: extract data, take screenshots, etc.
    print(f"Selected option: {option_text}")

Pro Tips for Stability #

• Avoid time.sleep where possible: Use explicit waits (WebDriverWait) to wait for elements to be clickable/visible—this makes your script more resilient to slow page loads.
• Double-check XPath selectors: Inspect the dropdown options to confirm their tag names, classes, or ID patterns. If the options use a consistent class (e.g., dropdown-option), use that instead of ID starts-with for better reliability.
• Handle stale elements: If the page refreshes after selecting an option, you'll need to re-locate elements each time (which the code above already does by re-finding the dropdown trigger).

内容的提问来源于stack exchange，提问作者Raymond van zonneveld