Selenium点击无URL变更的‘Load More’按钮时无法持续加载页面数据的问题咨询

Hey there, let's figure out why your loop is bailing early when clicking that "Load More" button. The issue here isn't about the URL not changing—it's about how Selenium interacts with the dynamic DOM after each click. Here's what's going wrong and how to fix it:

What's Causing the Early Exit? #

Your current code checks if the "Load More" button is clickable, but after the first click, a few things might happen:

• The button stays in the DOM temporarily (even if it's disabled while loading new content)
• The DOM updates after loading, making the previously found button element "stale" (invalid)
• You're not waiting for new content to finish loading before trying to click again

This leads the loop to either click a non-functional button or throw an error prematurely.

Fixes to Try #

1. Wait for New Content to Load (Most Reliable)

Instead of immediately checking for the button again, track the number of result items and wait until that count increases after clicking. This ensures you only proceed once new content is actually on the page:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException

def startWebDriver():
    global driver
    chrome_options = webdriver.ChromeOptions()
    chrome_options.add_argument("--incognito")
    chrome_options.add_argument("--window-size=1920x1080")
    driver = webdriver.Chrome(options=chrome_options)

startWebDriver()
driver.get("https://together.bunq.com/all")
# Initial wait for page to load
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, "[title='Load More']")))

# Replace this with the actual CSS selector for individual result items on the page
RESULT_ITEM_SELECTOR = ".discussion-item"  # Example selector—adjust as needed
previous_item_count = len(driver.find_elements(By.CSS_SELECTOR, RESULT_ITEM_SELECTOR))

while True:
    try:
        # Find the Load More button fresh each time
        load_more_btn = WebDriverWait(driver, 10).until(
            EC.element_to_be_clickable((By.CSS_SELECTOR, "[title='Load More']"))
        )
        load_more_btn.click()
        print("Loading more page results")
        
        # Wait until new items are loaded (item count increases)
        WebDriverWait(driver, 15).until(
            lambda d: len(d.find_elements(By.CSS_SELECTOR, RESULT_ITEM_SELECTOR)) > previous_item_count
        )
        # Update the count for the next iteration
        previous_item_count = len(driver.find_elements(By.CSS_SELECTOR, RESULT_ITEM_SELECTOR))
        
    except TimeoutException:
        print("All page results displayed")
        break

Important: You need to replace RESULT_ITEM_SELECTOR with the actual selector that targets each result on the page. Inspect the page to find this (e.g., if results are in <div class="discussion-item">, use .discussion-item).

2. Handle Stale Element Exceptions

After clicking, the DOM might refresh, making the button element you found earlier invalid. Fix this by re-finding the button in every loop iteration and catching stale element errors:

from selenium.common.exceptions import TimeoutException, StaleElementReferenceException

# ... (keep your startWebDriver function and initial setup)

while True:
    try:
        # Re-locate the button every time to avoid stale elements
        load_more_btn = WebDriverWait(driver, 10).until(
            EC.element_to_be_clickable((By.CSS_SELECTOR, "[title='Load More']"))
        )
        load_more_btn.click()
        print("Loading more page results")
        # Wait for the button to become stale (indicates DOM update)
        WebDriverWait(driver, 5).until(EC.staleness_of(load_more_btn))
    except (TimeoutException, StaleElementReferenceException):
        print("All page results displayed")
        break

3. Check if the Button is Disabled/Hidden

Sometimes, when there are no more results, the "Load More" button is disabled or hidden instead of removed. You can check its state before clicking:

import time
from selenium.common.exceptions import TimeoutException

# ... (initial setup)

while True:
    try:
        load_more_btn = WebDriverWait(driver, 10).until(
            EC.visibility_of_element_located((By.CSS_SELECTOR, "[title='Load More']"))
        )
        if load_more_btn.is_enabled():
            load_more_btn.click()
            print("Loading more page results")
            time.sleep(2)  # Short wait for loading to start
        else:
            # Button exists but can't be clicked—no more results
            print("All page results displayed")
            break
    except TimeoutException:
        # Button is no longer visible
        print("All page results displayed")
        break

Pro Tips #

• Avoid using time.sleep() unless necessary—use Selenium's explicit waits (like WebDriverWait) instead, as they're more reliable.
• Don't use a broad except: clause—catch specific exceptions (like TimeoutException) so you can debug issues easier.
• Always inspect the page to understand how the "Load More" button behaves when loading content (does it disappear? Disable? Stay visible but unresponsive?).

内容的提问来源于stack exchange，提问作者Fabian Kraus