Python Selenium隐式/显式等待失效，求助网页爬虫功能实现

Hey there! Let's tackle that wait issue you're facing with Selenium—those implicit/explicit wait failures can be so frustrating, especially when you’re trying to loop through your names list and scrape consistently. Let’s break down what’s probably going wrong and fix your code step by step.

Common Reasons Your Waits Are Failing #

• Mixing implicit and explicit waits: Selenium’s docs explicitly warn against this combo because it leads to totally unpredictable wait times. Implicit waits apply globally, and layering explicit waits on top creates messy, hard-to-debug behavior.
• Unstable locators: If you’re using dynamic IDs, auto-generated classes, or brittle XPaths, the element might not be found even after waiting—locators need to be static and reliable.
• Ignoring page transitions: After clicking the search result, the new page loads, and you need to wait for elements on that new page (not the search results page) to render.
• Rushing after hitting Enter: Pressing Enter triggers the search, but the DOM often hasn’t finished updating when you try to grab the first result—you need to wait for results to fully load first.

Revised Code with Proper Waits & Flow #

Let’s clean up your code, ditch implicit waits entirely (the Selenium-recommended approach), and build in solid explicit waits for every critical step. We’ll also handle page transitions and loop through your names array smoothly.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import xlrd

# Initialize ChromeDriver (update the path to match your local setup)
driver = webdriver.Chrome(executable_path="path/to/your/chromedriver")
wait = WebDriverWait(driver, 12)  # 12-second max wait—adjust if your pages load slower

# Your preloaded names list (swap this with your actual array)
names = ["John Doe", "Jane Smith", "Bob Johnson"]

try:
    for name in names:
        # Navigate to your search page (replace with your real search URL)
        driver.get("https://your-search-page-url.com")
        
        # Wait for the search bar to be ready, then input the name
        search_bar = wait.until(EC.element_to_be_clickable((By.ID, "search-input")))  # Update this locator to match your site
        search_bar.clear()  # Clear leftover text from previous searches
        search_bar.send_keys(name)
        
        # Press Enter to run the search
        search_bar.send_keys(Keys.ENTER)
        
        # Wait for the first search result link to be clickable
        first_result = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.search-results-container a:first-child")))  # Update this locator too
        first_result.click()
        
        # Handle new tabs (common for search results!)
        # Switch to the newly opened tab
        driver.switch_to.window(driver.window_handles[-1])
        
        # Wait for your target element to be visible on the new page
        target_element = wait.until(EC.visibility_of_element_located((By.XPATH, "//div[@class='target-info-block']")))  # Update to your element's locator
        
        # Extract and print the content
        element_content = target_element.text
        print(f"Scraped content for {name}: {element_content}")
        
        # Clean up: close the tab and switch back to the search page
        driver.close()
        driver.switch_to.window(driver.window_handles[0])
        
finally:
    # Make sure the driver quits even if an error pops up mid-loop
    driver.quit()

Key Fixes & Pro Tips #

• No more implicit waits: We’re using WebDriverWait with explicit conditions exclusively—this gives you full control over when to wait and what to wait for.
• Stable locators: Replace all the placeholder locators (like By.ID, "search-input") with ones that don’t change. Stick to CSS selectors or XPaths based on static classes/text instead of dynamic attributes.
• Tab handling: Most search results open in new tabs, so we added code to switch between tabs and clean up after each scrape. If your results open in the same tab, you can remove the window switching lines.
• Error safety: The try/finally block ensures your driver doesn’t hang around if something goes wrong during the loop.

Quick Troubleshooting If Waits Still Fail #

• Bump up the wait time in WebDriverWait(driver, 12) to 15 or 20 seconds for slower-loading pages.
• Check if your target element is inside an <iframe>—if so, you’ll need to switch to the iframe first with driver.switch_to.frame(iframe_element) before locating the element.
• Swap EC.visibility_of_element_located with EC.presence_of_element_located if you just need the element to exist in the DOM (not be fully visible).
• Enable headless mode (add options.add_argument("--headless=new") when initializing ChromeDriver) to rule out visual rendering issues.

内容的提问来源于stack exchange，提问作者Michael Di Martino