使用Selenium操作DuckDuckGo时遇NoSuchElementException问题求助

Hey there, let's break down why you're hitting that NoSuchElementException and fix your script step by step. This error almost always boils down to unreliable waits, outdated syntax, or incorrect element targeting—all easy fixes once you know where to look.

Common Issues in Your Current Script #

• Flaky fixed waits: Using time.sleep() is a gamble. Pages load at different speeds, so your script might try to interact with elements before they even exist on the page.
• Outdated Selenium methods: Calls like find_element_by_id are deprecated in Selenium 4.x. The modern approach uses the By class for cleaner, supported locators.
• Wrong link targeting: find_element_by_link_text() matches the visible text of a link, not the URL. If you're trying to click a link pointing to your hotel site, this method won't work unless the visible text exactly matches your URL (which it almost never does).
• Blind "More Results" clicks: Clicking this button four times without checking if it's still present will throw an error once you reach the end of available results.

Revised Script with Fixes #

First, make sure you're running the latest Selenium (pip install --upgrade selenium), then use this updated code:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# Initialize Firefox driver (Selenium 4+ doesn't need executable_path if geckodriver is in your system PATH)
driver = webdriver.Firefox()
wait = WebDriverWait(driver, 15)  # Explicit wait with 15-second timeout

try:
    # Navigate to DuckDuckGo
    driver.get("https://duckduckgo.com")

    # Wait for search field to be ready, then submit your query
    search_field = wait.until(EC.visibility_of_element_located((By.ID, 'search_form_input_homepage')))
    search_field.clear()
    search_field.send_keys('best hotel in india')
    search_field.submit()

    # Safely click "More Results" until it's no longer available
    try:
        while True:
            more_results_btn = wait.until(EC.element_to_be_clickable((By.LINK_TEXT, "More Results")))
            more_results_btn.click()
            # Wait for new results to load before checking again
            wait.until(EC.staleness_of(more_results_btn))
    except:
        # Exit loop when the button disappears (no more results)
        pass

    # Target your hotel link using its URL (replace with your actual hotel site URL)
    your_hotel_url = "your-hotel-website-url.com"
    # Option 1: Use CSS selector to find links containing your URL
    hotel_link = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, f'a[href*="{your_hotel_url}"]')))
    # Option 2: If you know the visible link text, use partial match:
    # hotel_link = wait.until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, "Your Hotel Name")))
    
    hotel_link.click()

    # Optional: Wait for your hotel page to load (replace with your page's title snippet)
    wait.until(EC.title_contains("Your Hotel Brand Name"))
finally:
    # Ensure the driver closes even if an error occurs
    driver.quit()

Key Improvements Explained #

• Explicit waits: Instead of guessing load times, WebDriverWait with expected_conditions ensures we only interact with elements when they're fully ready—this eliminates most "element not found" errors.
• Modern locator syntax: Using the By class aligns with current Selenium best practices and avoids deprecated methods.
• Smart link targeting: The CSS selector a[href*="your-url"] finds any link that contains your hotel's URL, which is far more reliable than matching visible text.
• Safe result pagination: The try-except loop stops clicking "More Results" once the button is gone, preventing unnecessary errors.
• Error-safe cleanup: The try-finally block guarantees the browser closes properly, even if something goes wrong mid-script.

Quick Additional Tips #

• Double-check that your geckodriver version matches your Firefox browser version—mismatches can cause weird element-locator bugs.
• Always check a site's robots.txt before scraping—some sites prohibit automated access.
• If your hotel link is still hard to find, inspect DuckDuckGo's result HTML to identify consistent patterns (e.g., result containers with a specific class) you can target first.

内容的提问来源于stack exchange，提问作者Yash