Python Selenium动态页下载文件报NoSuchElementException如何解决

我在Python环境下使用ChromeDriver + Selenium实现文件自动下载功能，目标文件所在页面可点击此处访问。

获取目标文件的操作逻辑是：

• 先点击页面上的最新日期选项（例如2022年6月22日就点击对应日期项）
• 之后点击「Baixar Arquivo」链接触发下载

我用搭配ChromeWebDriver的Selenium编写自动化脚本实现上述流程，但所有元素定位方法都抛出异常。观察到页面存在大量嵌套元素，多个div标签属性重名，先后尝试绝对XPath、CSS_SELECTOR、LINK_TEXT三种定位方式全部失败，测试代码如下：

from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from selenium.webdriver.common.by import By
from webdriver_manager.chrome import ChromeDriverManager
import traceback

service = ChromeService(executable_path=ChromeDriverManager().install())
op = webdriver.ChromeOptions()
op.add_argument('headless')

try:
    driver = webdriver.Chrome(service=service, options=op)
    driver.get('https://www.b3.com.br/pt_br/market-data-e-indices/servicos-de-dados/market-data/consultas/boletim-diario/dados-publicos-de-produtos-listados-e-de-balcao/')
    container = driver.find_element(by=By.XPATH, value='/html/body/div/div/div/div/div[2]/div[1]/div/div[1]/a/div/div/div') # 定位失败
    container = driver.find_element(by=By.CSS_SELECTOR, value='html body div#wrapper div div div.container div.row div.col-lg-8.list div.accordion div.card div#collapse.collapse.show div.card-block.col-12 div.list-avatar.two-line div.list-avatar-row.tamanho div.content p.fonte.secondary-text a') #定位失败
    container = driver.find_element(by=By.LINK_TEXT , value='Baixar Arquivo') # 定位失败
    print(container)
except Exception:
     print(traceback.format_exc())
finally:
    driver.close()

每次运行都返回no such element（元素不存在）错误，完整错误栈如下：

Traceback (most recent call last):
  File "/tmp/ipykernel_182073/1979137013.py", line 8, in <cell line: 5>
    container = driver.find_element(by=By.LINK_TEXT , value='Baixar Arquivo')
  File "/home/guilherme/miniconda3/envs/investment/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 1251, in find_element
    return self.execute(Command.FIND_ELEMENT, {
  File "/home/guilherme/miniconda3/envs/investment/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 430, in execute
    self.error_handler.check_response(response)
  File "/home/guilherme/miniconda3/envs/investment/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 247, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"link text","selector":"Baixar Arquivo"}
  (Session info: headless chrome=103.0.5060.53)
Stacktrace:
#0 0x561b67a2db13 <unknown>
#1 0x561b67834688 <unknown>
#2 0x561b6786bcc7 <unknown>
#3 0x561b6786be91 <unknown>
#4 0x561b6789ee34 <unknown>
#5 0x561b678898dd <unknown>
#6 0x561b6789cb94 <unknown>
#7 0x561b678897a3 <unknown>
#8 0x561b6785f0ea <unknown>
#9 0x561b67860225 <unknown>
#10 0x561b67a752dd <unknown>
#11 0x561b67a792c7 <unknown>
#12 0x561b67a5f22e <unknown>
#13 0x561b67a7a0a8 <unknown>
#14 0x561b67a53bc0 <unknown>
#15 0x561b67a966c8 <unknown>
#16 0x561b67a96848 <unknown>
#17 0x561b67ab0c0d <unknown>
#18 0x7f9208c4c609 <unknown>

补充说明：上述元素定位策略在其他网站测试均可正常运行，推测问题和页面多层嵌套元素有关。

定位失败和普通元素嵌套没有直接关系，核心问题有三个：

1. 页面内容为动态加载，页面刚打开时目标DOM还未渲染完成，直接执行定位必然找不到元素
2. 日期列表、下载链接的实际内容放在独立的iframe框架内，Selenium默认只会检索主页面DOM，不会自动穿透iframe查找内容，所有未切换上下文的定位方式都会失效
3. 旧版headless无头模式存在明显的自动化特征，站点会对这类请求做拦截，返回和正常浏览器不一致的页面结构，进一步导致定位失败

• 页面加载后使用显式等待替代直接定位，给DOM渲染留出缓冲时间
• 定位到承载业务内容的iframe元素，调用driver.switch_to.frame()切换到iframe内部上下文再做元素检索
• 升级无头模式配置，添加反自动化检测参数，伪装成普通用户浏览器请求
• 操作完成后记得切回主页面上下文，避免后续页面操作异常

from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from webdriver_manager.chrome import ChromeDriverManager
import time

service = ChromeService(executable_path=ChromeDriverManager().install())
op = webdriver.ChromeOptions()
# 新版无头模式+反检测参数
op.add_argument('--headless=new')
op.add_argument('--disable-blink-features=AutomationControlled')
op.add_argument('user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36')
op.add_experimental_option("excludeSwitches", ["enable-automation"])
op.add_experimental_option('useAutomationExtension', False)
# 如需配置自动下载路径、关闭下载弹窗，可在options里追加对应下载配置

driver = None
try:
    driver = webdriver.Chrome(service=service, options=op)
    # 移除navigator.webdriver自动化特征
    driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
    driver.get('https://www.b3.com.br/pt_br/market-data-e-indices/servicos-de-dados/market-data/consultas/boletim-diario/dados-publicos-de-produtos-listados-e-de-balcao/')
    wait = WebDriverWait(driver, 15)

    # 等待iframe加载完成并切换上下文
    iframe = wait.until(EC.presence_of_element_located((By.TAG_NAME, 'iframe')))
    driver.switch_to.frame(iframe)

    # 点击第一条（最新日期）条目
    latest_date_item = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, '.list-avatar-row a')))
    latest_date_item.click()

    # 等待下载链接渲染完成后点击
    download_link = wait.until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, 'Baixar Arquivo')))
    print("获取到下载链接：", download_link.get_attribute('href'))
    download_link.click()

    # 等待下载触发，可结合本地下载目录做文件存在性校验
    time.sleep(5)

except Exception as e:
    import traceback
    print(traceback.format_exc())
finally:
    if driver:
        # 切回主页面上下文后关闭驱动
        driver.switch_to.default_content()
        driver.quit()

内容的提问来源于stack exchange，提问作者Guilherme Noronha