Selenium切换标签页后爬虫滚动停止问题求助

Alright, let's figure out why your Selenium script stops loading more products when the tab isn't active—and how to fix it. The core issue here is that modern browsers throttle JavaScript execution and suppress user-like events (like clicks) for inactive tabs to save system resources. That's why your arrow.click() call stops working as soon as you switch away from the tab.

Here are a few reliable solutions to keep your scraper running even when the tab isn't in focus:

1. Use Headless Mode (Recommended) #

Headless mode runs the browser without a visible window, so there's no concept of an "active tab" to begin with. This is the most efficient and straightforward fix for your problem.

Here's how to set it up for Chrome:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

# Configure Chrome options for headless mode
chrome_options = Options()
# Use the updated headless mode (Chrome 112+)
chrome_options.add_argument("--headless=new")
# Disable GPU to avoid compatibility issues
chrome_options.add_argument("--disable-gpu")
# Set a window size to ensure all elements load correctly
chrome_options.add_argument("--window-size=1920,1080")

# Initialize the browser
browser = webdriver.Chrome(options=chrome_options)
browser.get("your_ecommerce_site_url")

# Loop to load more products using JavaScript clicks
while True:
    try:
        # Execute JavaScript to click the "show more" button
        browser.execute_script('''
            const loadMoreBtn = document.querySelector('#see-more-products.show-more.btn');
            if (loadMoreBtn) {
                loadMoreBtn.click();
            } else {
                throw new Error("No more load button found");
            }
        ''')
        # Wait for products to load (adjust the time based on site speed)
        browser.implicitly_wait(4)
    except Exception as e:
        print(f"Stopping load loop: {e}")
        break

For Firefox, the setup is similar—just use FirefoxOptions instead of ChromeOptions and add the --headless argument.

2. Replace Native Element Clicks with JavaScript Execution #

If you need to run the browser in visible mode (non-headless), replacing your arrow.click() with a JavaScript click will bypass the tab-focus restriction. Native Selenium clicks rely on browser event handling that's throttled in background tabs, but direct JavaScript DOM calls run independently of tab focus.

Modify your code like this:

# Instead of finding the element and calling .click()
# arrow = browser.find_element_by_xpath('//div[@id="see-more-products" and @class="show-more btn"]')
# arrow.click()

# Use JavaScript to trigger the click directly
browser.execute_script('document.querySelector("#see-more-products.show-more.btn").click();')

You can also add a scroll-to-element step to ensure the button is in the DOM's viewport (though not strictly necessary for JS clicks):

browser.execute_script('''
    const btn = document.querySelector("#see-more-products.show-more.btn");
    btn.scrollIntoView({block: "center"});
    btn.click();
''')

3. Use Undetected Chromedriver (For Anti-Scraper Sites) #

If the e-commerce site has anti-scraping measures that block regular Selenium, undetected-chromedriver can help. It mimics a real user's browser behavior better, and it often avoids tab-focus-related throttling issues automatically.

Install it first:

pip install undetected-chromedriver

Then use it in your script:

import undetected_chromedriver as uc

# Initialize the browser (can run headless or visible)
browser = uc.Chrome(headless=True)  # Set headless=False if you need visible window
browser.get("your_ecommerce_site_url")

# Load more products using JS clicks as before
while True:
    try:
        browser.execute_script('document.querySelector("#see-more-products.show-more.btn").click();')
        browser.implicitly_wait(3)
    except Exception as e:
        print(f"No more products: {e}")
        break

Key Takeaway #

The main reason your original code fails in background tabs is that native Selenium interactions depend on the browser's active tab event loop. By switching to headless mode or JavaScript-driven clicks, you bypass this restriction entirely. Headless mode is the best option for most scraping tasks since it's faster and doesn't require a visible window.

内容的提问来源于stack exchange，提问作者SK Singh