Got it, your current fixed-loop approach works for a set number of pages, but we can make it smarter to handle any number of pages automatically. Here are two reliable ways to do this:

Method 1: Use Try-Except Block (Simple & Effective) #

We’ll run an infinite loop, try to find and click the "Next" button each time, and exit the loop when the button can’t be found (or isn’t interactable). This is straightforward because Selenium throws an exception when it can’t locate the element.

First, make sure you import the necessary exceptions:

from selenium.common.exceptions import NoSuchElementException, ElementNotInteractableException

Then replace your for loop with this:

count = 0  # Track page number for your token_holders cleanup
while True:
    try:
        # Locate and click the Next button
        next_btn = driver.find_element_by_link_text('Next')
        next_btn.click()
        
        # Wait for page to load (we’ll keep your sleep for now, but see the pro tip below for a better approach)
        time.sleep(5)
        
        # Parse the new page content
        soup = BeautifulSoup(driver.page_source, 'html.parser')
        token_holders.extend(soup.find_all('tr'))
        
        # Adjust your cleanup logic since we’re no longer using a fixed loop index
        if count > 0:
            del token_holders[50 * count]
        
        count += 1
        time.sleep(5)
    except (NoSuchElementException, ElementNotInteractableException):
        # Exit loop when no more Next button exists or it can’t be clicked
        print("No more Next button to click. Exiting loop.")
        break

Method 2: Check for Button Presence First (More Explicit) #

If you prefer to avoid relying on exceptions, you can check if the button exists and is enabled before attempting to click it. Using find_elements_by_link_text (plural) returns an empty list if the button isn’t present:

count = 0
while True:
    # Get all elements matching "Next" link text (empty list if none exist)
    next_btns = driver.find_elements_by_link_text('Next')
    
    if not next_btns:
        print("No Next button found. Stopping.")
        break
    
    # Check if the button is clickable
    next_btn = next_btns[0]
    if not next_btn.is_enabled():
        print("Next button is disabled. Stopping.")
        break
    
    # Proceed with clicking and parsing
    next_btn.click()
    time.sleep(5)
    
    soup = BeautifulSoup(driver.page_source, 'html.parser')
    token_holders.extend(soup.find_all('tr'))
    
    if count > 0:
        del token_holders[50 * count]
    
    count += 1
    time.sleep(5)

Pro Tip: Replace time.sleep with WebDriverWait #

Hardcoding time.sleep is unreliable—pages might load faster or slower than expected. Using WebDriverWait makes your script more robust by waiting until the page content is ready:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

# Inside the loop, replace time.sleep(5) with this:
WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.TAG_NAME, 'tr'))  # Wait for table rows to load on the new page
)

This waits up to 10 seconds for the table rows to appear, which is more efficient than blindly waiting a fixed amount of time.

内容的提问来源于stack exchange，提问作者user9701233