Selenium多下拉框加载等待问题及URL获取方案求助

需求为访问指定页面，遍历国家、州/省、城市三级地区下拉框，生成所有可能的最终URL（例如 https://www.xpel.com/clearbra-installers/united-states/california/bakersfield）。

尝试过点击选项、遍历下拉框、收集选项生成DataFrame后构造URL访问等多种方法，但始终遇到等待时间相关问题，以及频繁出现的「stale element not found」错误。

目标国家选择下拉框的HTML代码：

<select class="dealer-locator-select" id="dealer-locator-country-select" name="dealer-locator-country-select" data-action="country-select" data-filter="">
<option value=""> Region, Country or Area </option>
<option value="albania"> Albania </option>
<option value="algeria"> Algeria </option>
<option value="argentina"> Argentina </option>
<option value="armenia"> Armenia </option>
<option value="australia"> Australia </option>
<option value="austria"> Austria </option>
<option value="bahrain"> Bahrain </option>
<option value="belgium"> Belgium </option>
<option value="bosnia-and-herzegovina"> Bosnia and Herzegovina </option>
...
<option value="vietnam"> Vietnam </option>
</select>

尝试的国家检索代码：

# 设置等待时间
delay = 10 # 秒
wait = WebDriverWait(driver, delay)

# 遍历国家选项
for i in range(2,96):
    wait.until(EC.presence_of_element_located((By.XPATH, "//select[@id='dealer-locator-country-select']/option[" + str(i) + "]")))
    country_name = driver.find_element(By.XPATH, "//select[@id='dealer-locator-country-select']/option[" + str(i) + "]").text.lower().replace(' ', '-')
    country = Select(driver.find_element(By.XPATH, "//select[@id='dealer-locator-country-select']")).select_by_value(country_name)

print("done")

仅获取国家名称打印时运行正常，但执行选择操作就出错。也尝试过try/except/finally语句，仍未解决：

# 设置等待时间
delay = 10 # 秒
wait = WebDriverWait(driver, delay)

# 遍历每个国家选项
for i in range(2,96):
    wait.until(EC.presence_of_element_located((By.XPATH, "//select[@id='dealer-locator-country-select']/option[" + str(i) + "]")))
    country_name = driver.find_element(By.XPATH, "//select[@id='dealer-locator-country-select']/option[" + str(i) + "]").text.lower().replace(' ', '-')
    try:
        country = Select(driver.find_element(By.XPATH, "//select[@id='dealer-locator-country-select']"))
    except:
        sleep(5)
        country = Select(driver.find_element(By.XPATH, "//select[@id='dealer-locator-country-select']"))
    finally:
        country.select_by_value(country_name)

print("done")

问题本质 #

「stale element not found」是因为选中国家后页面DOM会刷新——比如加载州/省选项时，国家下拉框组件可能被重新渲染，导致之前定位的元素引用失效。另外硬编码range(2,96)遍历选项也不灵活，一旦选项数量变化就会报错。

修复思路 #

1. 提前缓存所有选项值：在初始页面一次性提取所有国家的value值存到列表，避免反复定位DOM元素。
2. 用更可靠的显式等待：不要只等元素存在，改用EC.element_to_be_clickable或EC.visibility_of_element_located，确保元素可交互后再操作。
3. 避免复用元素引用：每次循环都重新定位下拉框，防止DOM刷新后元素失效。
4. 动态获取选项：通过Select对象直接获取所有选项，排除空选项后遍历，不用固定索引范围。

优化后代码示例 #

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait, Select
from selenium.webdriver.support import expected_conditions as EC
import time

driver = webdriver.Chrome()
driver.get("https://www.xpel.com/clearbra-installers/")
wait = WebDriverWait(driver, 15)

# 第一步：缓存所有国家的value值
country_select = wait.until(EC.visibility_of_element_located((By.ID, "dealer-locator-country-select")))
country_options = Select(country_select).options
country_values = [opt.get_attribute("value") for opt in country_options if opt.get_attribute("value") != ""]

# 遍历每个国家
for country_val in country_values:
    try:
        # 每次循环重新定位国家下拉框
        current_country_select = wait.until(EC.element_to_be_clickable((By.ID, "dealer-locator-country-select")))
        Select(current_country_select).select_by_value(country_val)
        
        # 等待州/省下拉框加载完成
        state_select = wait.until(EC.visibility_of_element_located((By.ID, "dealer-locator-state-select")))
        state_options = Select(state_select).options
        state_values = [opt.get_attribute("value") for opt in state_options if opt.get_attribute("value") != ""]
        
        # 遍历每个州/省
        for state_val in state_values:
            current_state_select = wait.until(EC.element_to_be_clickable((By.ID, "dealer-locator-state-select")))
            Select(current_state_select).select_by_value(state_val)
            
            # 等待城市下拉框加载完成
            city_select = wait.until(EC.visibility_of_element_located((By.ID, "dealer-locator-city-select")))
            city_options = Select(city_select).options
            city_values = [opt.get_attribute("value") for opt in city_options if opt.get_attribute("value") != ""]
            
            # 构造并输出最终URL
            for city_val in city_values:
                final_url = f"https://www.xpel.com/clearbra-installers/{country_val}/{state_val}/{city_val}"
                print(final_url)
                # 可将URL写入文件或数据库
                
    except Exception as e:
        print(f"处理{country_val}时出错: {str(e)}")
        # 出错后刷新页面，重新开始当前循环
        driver.get("https://www.xpel.com/clearbra-installers/")
        time.sleep(1)
        continue

driver.quit()

额外提示 #

• 若页面有加载动画，可在选择操作后加0.5-1秒的短暂等待，但优先用显式等待。
• 可增加重试机制，比如某个选项选择失败时重试2-3次再跳过。
• 直接构造URL的方式更高效：如果能确认URL规则固定，可先爬取三级选项的所有value值（注意层级依赖），批量生成URL，无需每次点击页面，彻底规避DOM刷新问题。

内容的提问来源于stack exchange，提问作者VSherr