使用Selenium爬取Investing.com时Firefox驱动点击下拉项后崩溃

可能的原因及修复方案 #

1. 元素定位不准确或动态加载问题 #

原XPATH可能因页面动态渲染导致定位失效，改用基于选项文本的稳定定位方式：

from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver

driver = webdriver.Firefox()
url = "https://tr.investing.com/equities/turkey"
driver.get(url)

# 点击下拉菜单
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, "index-select"))).click()
# 通过文本定位目标选项
target_option = WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.XPATH, '//div[contains(@class, "js-index-select-dropdown")]//div[text()="BİST Ulusal Tüm"]'))
)
target_option.click()

2. 反爬机制触发页面退出 #

模拟真实用户操作节奏，添加交互延迟：

import time
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver

driver = webdriver.Firefox()
url = "https://tr.investing.com/equities/turkey"
driver.get(url)

# 等待页面核心元素加载完成
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, "index-select")))
time.sleep(2)  # 模拟用户停顿
# 点击下拉菜单
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, "index-select"))).click()
time.sleep(1)  # 等待下拉选项展开
# 点击目标选项
target_option = WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.XPATH, '//div[text()="BİST Ulusal Tüm"]'))
)
target_option.click()

3. 浏览器与驱动版本不兼容 #

检查Firefox浏览器版本和geckodriver版本是否匹配，确保两者版本一致，下载对应版本的geckodriver即可。

4. 禁用驱动自动关闭行为 #

初始化驱动时添加配置，防止浏览器意外关闭：

from selenium.webdriver.firefox.options import Options
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver

options = Options()
options.add_argument("--no-sandbox")
options.add_argument("--disable-dev-shm-usage")
# 保持浏览器打开（调试场景可用）
options.add_experimental_option("detach", True)

driver = webdriver.Firefox(options=options)
url = "https://tr.investing.com/equities/turkey"
driver.get(url)

WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.ID, "index-select"))).click()
target_option = WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.XPATH, '//div[text()="BİST Ulusal Tüm"]'))
)
target_option.click()

内容的提问来源于stack exchange，提问作者Gokita