使用Selenium爬取佳能网页时Load More按钮定位失败求助

DevOps 小满

2026-6-27

问题

爬取佳能美国官网无反相机分类页面（https://www.usa.canon.com/shop/cameras/mirrorless-cameras）时，页面带有「Load More」按钮，点击可加载更多相机内容。编写了滚动到按钮并点击的函数，但运行后持续出现「Load more button not found」和「Timeout waiting for the load more button」错误，相关代码如下：

def find_load_more(driver, xpath):
    try:
        return WebDriverWait(driver, 10).until(
            EC.element_to_be_clickable((By.XPATH, xpath))
        )
    except TimeoutException:
        print("Timeout waiting for the load more button.")
        return None


def scroll_to_load_more(driver, xpath):
    print(xpath)
    try:
        load_more_button = find_load_more(driver, xpath)
        if load_more_button:
            driver.execute_script("arguments[0].scrollIntoView();", load_more_button)
            load_more_button.click()
        else:
            print("Load more button not found.")
    except NoSuchElementException:
        print("The 'Load More' button does not exist.")


def scrape_canon_preview(category, driver):
    url = f"https://www.usa.canon.com/shop/cameras/{category}"
    driver.get(url)
    wait_for_page_load(driver)
    # Click Load more to load all the cameras
    scroll_to_load_more(driver, "//button[@class='primary amscroll-load-button']")

解决方案

以下是针对问题的具体修复建议：

验证元素定位准确性
打开浏览器开发者工具（F12），检查「Load More」按钮的实际class属性是否为primary amscroll-load-button。页面动态加载可能导致class名变化，可改用结合文本的更精确xpath：//button[contains(text(), 'Load More') and @class='primary amscroll-load-button']。
优化滚动与等待逻辑
1. 延长等待时间：将WebDriverWait(driver, 10)中的10秒调整为20秒，给页面足够的动态加载时间；
2. 优化滚动方式：使用scrollIntoView({block: 'center'})确保按钮完全进入视口，避免被页面元素遮挡：
```
driver.execute_script("arguments[0].scrollIntoView({block: 'center'});", load_more_button)
```
3. 增加滚动到页面底部的步骤：先滚动到底部触发按钮加载，再定位按钮：
```
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
```
替换点击方式
直接调用元素的click()方法可能因页面交互限制失效，改用JavaScript强制点击：
```
driver.execute_script("arguments[0].click();", load_more_button)
```

循环点击直到按钮消失
原代码仅点击一次，无法加载全部内容。修改函数实现循环点击，直到按钮不再出现：

def scroll_to_load_more(driver, xpath):
    while True:
        try:
            load_more_button = WebDriverWait(driver, 20).until(
                EC.element_to_be_clickable((By.XPATH, xpath))
            )
            driver.execute_script("arguments[0].scrollIntoView({block: 'center'});", load_more_button)
            driver.execute_script("arguments[0].click();", load_more_button)
            # 等待新内容加载完成
            WebDriverWait(driver, 10).until(
                EC.staleness_of(load_more_button)
            )
        except (TimeoutException, NoSuchElementException):
            print("No more content to load or button not found.")
            break

完善页面加载等待逻辑
确保wait_for_page_load函数正确等待页面完全加载，比如等待document.readyState变为complete：

def wait_for_page_load(driver):
    WebDriverWait(driver, 20).until(
        lambda d: d.execute_script('return document.readyState') == 'complete'
    )

内容的提问来源于stack exchange，提问作者Chavle