Selenium中Load More按钮加载失败/无法点击的问题排查与解决

问题描述 #

我尝试通过循环点击“Load more”按钮滚动至页面底部提取所有数据，但遇到两个核心问题：

1. 即便使用Selenium显式等待（WebDriverWait），仍频繁触发TimeoutException，按钮要么加载失败，要么执行点击操作后无响应（已通过打印“Load more clicked”确认代码执行了点击逻辑）
2. 初始加载的“Overview”分类下，Load more按钮完全不显示，但中途停止进程后刷新同一Selenium Chrome窗口，按钮又能正常显示并工作

使用环境：Selenium 4.1.5 + Chrome Driver 104，原代码如下：

import pandas as pd
import time
import csv
import selenium.webdriver as webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException, ElementClickInterceptedException, NoSuchElementException, ElementNotInteractableException
from selenium.webdriver.chrome.service import Service as ChromeService
from selenium.webdriver.chrome.options import Options as ChromeOptions
from selenium.webdriver.support.ui import WebDriverWait, Select
from webdriver_manager.chrome import ChromeDriverManager

def ChromeDriver():
    options = ChromeOptions()
    options.add_argument('--disable-gpu')
    options.add_argument("force-device-scale-factor=1.00")
    options.headless = False
    service = ChromeService("C:\Python39\chromedriver.exe")
    driver = webdriver.Chrome(options=options, service=service)
    #driver.maximize_window()
    return driver

def toggle_button():
    toggle_button = driver.find_element(By.XPATH, "//input[@type='checkbox']")
    toggle_button.click()

def set_currency(currency):
    active_currency = driver.find_element(By.XPATH, "//span[contains(@class, 'modeTitleActive-AXg2YMF0')]").text
    if active_currency!= currency:
        toggle_button()
    else:
        pass

def accept_all():
    try:
        promtpw = driver.find_element(By.XPATH, "//div[@class='item-x59qvYIc']")
        promtpw = promtpw.find_element(By.XPATH, "//button[@class='acceptAll-ofWyKNKk button-KXqj4Lf5 size-xsmall-KXqj4Lf5 color-brand-KXqj4Lf5 variant-primary-KXqj4Lf5']").click()
    except NoSuchElementException:
        pass
    except ElementNotInteractableException:
        pass

def save_as_csv(data_list):
    for idx, data in enumerate(data_list[0:4]):
        file_title = data_list[4][idx]
        file = f"csv/tv_crypto_prices_{file_title}.csv"
        with open(file, "w", newline="") as csv_file:
            tv_writer = csv.writer(csv_file)
            tv_writer.writerow([file_title.upper()])
        data.to_csv(file, mode="a")
    print("csv files saved")

def quit_driver():
    driver.close()
    driver.quit()

def get_categories():
    categories = driver.find_elements(By.XPATH, "//div[starts-with(@class, 'item-x59qvYIc')]")
    return categories

def get_categories_list(categories_objs):
    categories_list = []
    for category in categories_objs:
        categories_list.append(category.text)
    return categories_list

def fetch_data():
    data = []
    categories_list = get_categories_list(categories)
    #categories_list = categories_list.append(categories_list.pop(categories_list.index("Overview")))
    accept_all()
    for category in categories:
        category.click()
        time.sleep(2)
        print(category.text)
        
        #Load data
        load_more = True
        while load_more:
            try:
                time.sleep(2)
                load_more_button = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//span[@class='tv-load-more__btn']")))
                load_more_button.click()
                print("Load more clicked")
            except NoSuchElementException:
                load_more = False
            except ElementNotInteractableException:
                load_more = False
        
        crypto = pd.read_html(driver.page_source)[1]
        crypto.columns.values[0] = "Name" #Filter column values
        data.append(crypto)
    data.append(categories_list)
    quit_driver()
    return data

def run_tvc():
    #Get driver
    driver = ChromeDriver()
    driver.get("https://www.tradingview.com/markets/cryptocurrencies/prices-all/")
    
    #Set currency
    currency = "BTC"
    set_currency(currency)
    
    #Get data
    d = fetch_data()
    save_as_csv(d)
run_tvc()

解决方案 #

问题1：解决WebDriverWait触发TimeoutException的问题 #

• 优化元素定位：原XPATH仅依赖class属性，容易受页面动态渲染影响，换成结合文本和父元素的精准定位：

  # 替换原定位表达式
  load_more_xpath = "//div[@class='tv-load-more']//span[text()='Load more']"
  

• 先滚动到按钮再点击：按钮可能处于视口外导致无法交互，先通过JS滚动到元素位置，再等待可点击：

  load_more_button = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, load_more_xpath)))
  # 滚动到按钮位置，确保完全可见
  driver.execute_script("arguments[0].scrollIntoView({block: 'center'});", load_more_button)
  # 等待按钮可点击后执行点击
  WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, load_more_xpath))).click()
  

• 处理点击拦截异常：如果有弹窗或浮层覆盖按钮，改用JS直接点击绕过拦截：

  try:
      load_more_button.click()
  except ElementClickInterceptedException:
      driver.execute_script("arguments[0].click();", load_more_button)
  

• 等待数据加载完成：每次点击后，等待列表元素数量变化，确保数据加载完成再进入下一次循环：

  # 点击前记录当前列表行数
  before_row_count = len(driver.find_elements(By.XPATH, "//tr[contains(@class, 'tv-data-table__row')]"))
  load_more_button.click()
  # 等待行数增加，确认数据加载完成
  WebDriverWait(driver, 15).until(lambda d: len(d.find_elements(By.XPATH, "//tr[contains(@class, 'tv-data-table__row')]")) > before_row_count)
  

问题2：修复“Overview”分类Load more按钮不显示的问题 #

• 调整操作顺序，先处理Cookie弹窗：原代码先获取分类再处理Cookie，弹窗可能阻塞页面渲染，改为进入页面后先处理Cookie：

  def run_tvc():
      driver = ChromeDriver()
      driver.get("https://www.tradingview.com/markets/cryptocurrencies/prices-all/")
      # 先等待页面加载并处理Cookie
      WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, "//button[@class='acceptAll-ofWyKNKk']")))
      accept_all()
      # 再设置货币和获取数据
      set_currency("BTC")
      d = fetch_data()
      save_as_csv(d)
  

• 切换分类后等待内容加载：点击“Overview”分类后，等待核心数据表格加载完成，再触发加载更多：

  def fetch_data():
      data = []
      accept_all()
      categories = get_categories()
      categories_list = get_categories_list(categories)
      for category in categories:
          category.click()
          # 等待分类对应的表格加载完成
          WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.XPATH, "//table[contains(@class, 'tv-data-table')]")))
          print(category.text)
          
          # 针对Overview分类强制触发滚动，唤起加载按钮
          if category.text == "Overview":
              driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
              time.sleep(1)
          
          # 加载更多逻辑...
  

• 避免全局变量依赖：原代码中categories是全局变量，可能导致元素状态失效，改为在fetch_data内部获取最新的分类元素：

  def fetch_data():
      data = []
      accept_all()
      # 每次获取最新的分类元素，避免 stale element 异常
      categories = get_categories()
      categories_list = get_categories_list(categories)
      # 后续循环逻辑...
  

内容的提问来源于stack exchange，提问作者Andrea