使用Selenium和Python无法访问Yahoo财经HTML元素的问题

我希望通过Selenium访问Yahoo财经页面（https://finance.yahoo.com/quote/ABBV/analysis）中「研究分析」→「分析师目标价」里的**平均值（Average）**元素。以下是简化后的代码：

url = 'https://finance.yahoo.com/quote/ABBV/analysis'
print('############# Kursziel (via Yahoo)')

val_price_target = '999999.'

print("URL (yahoo):",url)

# grab price target from webpage
driver.implicitly_wait(10)
driver.get(url)
average_price = (By.CSS_SELECTOR, "div.label.average > span.price")
val_price_target = WebDriverWait(driver, 10).until(EC.visibility_of_element_located(average_price)).text
print("Value: ", val_price_target)

完整代码会遍历Excel表格（Pandas DataFrame），当数据缺失时调用这段代码获取数据。

遇到的错误 #

第一次运行正常，现在出现TimeoutException，在开发者工具控制台中也无法找到目标元素：

############# Kursziel (via Yahoo)
URL (yahoo): https://finance.yahoo.com/quote/ABBV/analysis
Traceback (most recent call last): File "c:\Projects\Python_Projekt_ACN_intern\Code\Yahoo-Problem_fullcode.py", line 109, in get_price_target
result_string = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.XPATH, xpath_string))).get_attribute("innerHTML")
                
File "C:\Users\eduard.szilaghi\AppData\Local\Programs\Python\Python312\Lib\site-packages\selenium\webdriver\support\wait.py", line 105, in until raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
Stacktrace: GetHandleVerifier [0x00007FF61EB31572+60802]
(No symbol) [0x00007FF61EAAAC72]
(No symbol) [0x00007FF61E967CE4]
(No symbol) [0x00007FF61E9B6D4D]
(No symbol) [0x00007FF61E9B6E1C]
(No symbol) [0x00007FF61E9FCE37]
(No symbol) [0x00007FF61E9DABBF]
(No symbol) [0x00007FF61E9FA224]
(No symbol) [0x00007FF61E9DA923]
(No symbol) [0x00007FF61E9A8FEC]
(No symbol) [0x00007FF61E9A9C21]
GetHandleVerifier [0x00007FF61EE3420D+3217949]
GetHandleVerifier [0x00007FF61EE761A7+3488183]
GetHandleVerifier [0x00007FF61EE6F12F+3459391]
GetHandleVerifier [0x00007FF61EBEB936+823622]
(No symbol) [0x00007FF61EAB600F]
(No symbol) [0x00007FF61EAB0F34]
(No symbol) [0x00007FF61EAB10C2]
(No symbol) [0x00007FF61EAA1914]
BaseThreadInitThunk [0x00007FFA475F257D+29]
RtlUserThreadStart [0x00007FFA48FAAA48+40]


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "c:\Projects\Python_Projekt_ACN_intern\Code\Yahoo-Problem_fullcode.py", line 386, in <module>
pt_val, pt_cur = get_price_target(isin)
                 ^^^^^^^^^^^^^^^^^^^^^^
File "c:\Projects\Python_Projekt_ACN_intern\Code\Yahoo-Problem_fullcode.py", line 124, in get_price_target
val_price_target = get_price_target_yahoo(uscode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "c:\Projects\Python_Projekt_ACN_intern\Code\Yahoo-Problem_fullcode.py", line 84, in get_price_target_yahoo
val_price_target = WebDriverWait(driver, 10).until(EC.visibility_of_element_located(average_price)).text
                   
File "C:\Users\eduard.szilaghi\AppData\Local\Programs\Python\Python312\Lib\site-packages\selenium\webdriver\support\wait.py", line 105, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

解决思路 #

• 更新元素选择器：Yahoo财经的页面结构可能已变更，旧的CSS选择器失效。打开目标页面，用开发者工具重新定位「分析师目标价」平均值的元素，改用更稳定的选择器。例如当前页面可尝试：

  # 示例XPath，需根据实际页面调整
  average_price = (By.XPATH, '//div[text()="Average"]/following-sibling::span')
  

• 绕过反爬检测：Yahoo可能识别出Selenium自动化请求，可通过以下方式规避：
  • 添加真实浏览器的User-Agent：

    from selenium.webdriver.chrome.options import Options
    options = Options()
    options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36")
    driver = webdriver.Chrome(options=options)
    

  • 增加随机请求间隔，避免频繁访问；
  • 使用undetected-chromedriver替代常规ChromeDriver，绕过自动化检测。
• 优化等待策略：不要同时混用implicitly_wait和WebDriverWait，两者会互相干扰，建议只使用显式等待（WebDriverWait）。
• 检查iframe嵌套：如果目标元素在iframe内，需先切换到iframe再查找元素：

  iframe = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.TAG_NAME, 'iframe')))
  driver.switch_to.frame(iframe)
  # 之后再查找目标元素
  

• 直接抓取接口数据：目标数据可能通过AJAX加载，可通过浏览器开发者工具的网络面板找到对应API接口，直接请求接口获取数据，比解析页面更稳定。

内容的提问来源于stack exchange，提问作者Eduard Szilaghi