Snapdeal商品评论及星级爬取问题：点击星级筛选选项触发ElementNotInteractable Exception

Looks like you're hitting a classic dynamic element issue with Snapdeal's review filters—those star options don't always load or become interactable right when you click the dropdown. Let's break down what's going wrong in your code and fix it step by step.

Key Issues in Your Current Code #

1. Outdated Selenium Methods: The find_element_by_* functions are deprecated in modern Selenium versions, which can lead to unexpected behavior.
2. Unreliable Timing: Fixed time.sleep() calls don't account for dynamic page loading—you're trying to click options before they're actually ready.
3. Fragile Element Locator: Using li[{x}] to target star options relies on fixed index positions, which can break if Snapdeal updates their page structure.
4. Missing Comment Collection: Your code only tracks star ratings, not the actual review text you mentioned needing.

Revised, Working Code #

Here's an optimized version of your script that addresses all these issues:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import (
    NoSuchElementException,
    ElementClickInterceptedException,
    TimeoutException
)
import time

# Initialize driver (use ChromeOptions for headless mode if needed)
driver = webdriver.Chrome()
url_snapdeal = 'https://www.snapdeal.com/'
driver.get(url_snapdeal)
time.sleep(2)

# Search for smartphones (using explicit wait for reliability)
search = WebDriverWait(driver, 10).until(
    EC.element_to_be_clickable((By.ID, 'inputValEnter'))
)
search.clear()
search.send_keys('smartphone')
search.send_keys(Keys.ENTER)
time.sleep(2)

# Scroll to load more products
for _ in range(3):
    driver.execute_script('window.scrollTo(0, document.body.scrollHeight)')
    time.sleep(1)

# Collect product URLs
urls = []
product_links = WebDriverWait(driver, 10).until(
    EC.presence_of_all_elements_located((By.XPATH, "//div[@class='product-desc-rating ']/a"))
)
for link in product_links:
    urls.append(link.get_attribute('href'))

snap_reviews = []
snap_ratings = []

for url in urls:
    driver.get(url)
    try:
        # Wait for reviews section to load
        WebDriverWait(driver, 15).until(
            EC.presence_of_element_located((By.CLASS_NAME, 'user-review'))
        )

        # Iterate over 5-star to 1-star filters
        for star_rating in range(5, 0, -1):
            # Open the star filter dropdown
            filter_dropdown = WebDriverWait(driver, 10).until(
                EC.element_to_be_clickable((By.CLASS_NAME, 'selectarea'))
            )
            filter_dropdown.click()

            # Wait for star option to be clickable (target by text, not index)
            star_option = WebDriverWait(driver, 10).until(
                EC.element_to_be_clickable((By.XPATH, f"//div[@class='options']/ul/li[text()='{star_rating} Star']"))
            )

            # Fallback: Click via JavaScript if element is intercepted
            try:
                star_option.click()
            except ElementClickInterceptedException:
                driver.execute_script("arguments[0].click();", star_option)

            time.sleep(1.5)  # Give time for filtered reviews to load

            # Collect ratings and review text
            reviews = WebDriverWait(driver, 10).until(
                EC.presence_of_all_elements_located((By.CLASS_NAME, 'user-review'))
            )
            for review in reviews:
                # Get star count
                active_stars = review.find_elements(By.XPATH, ".//i[@class='sd-icon sd-icon-star active']")
                snap_ratings.append(len(active_stars))
                # Get review content
                review_text = review.find_element(By.XPATH, ".//p[@class='user-review-desktop']").text
                snap_reviews.append(review_text)

    except NoSuchElementException:
        print(f"Warning: No star filter found for product: {url}")
    except TimeoutException:
        print(f"Warning: Timed out loading content for product: {url}")
    except Exception as e:
        print(f"Error processing product {url}: {str(e)}")

driver.quit()

Critical Improvements Explained #

• Explicit Waits: Replaced most time.sleep() with WebDriverWait to ensure elements are fully loaded and interactable before we act on them.
• Text-Based Locators: Targets star options by their visible text (e.g., "5 Star") instead of index positions, making the code resilient to page structure changes.
• Element Interception Handling: Added a JavaScript click fallback for cases where the star option is blocked by another UI element.
• Full Review Collection: Now captures both star ratings and review text, as you requested.
• Updated Selenium Syntax: Uses the modern By.XPATH/By.CLASS_NAME syntax instead of deprecated methods.

Additional Tips for Smooth Scraping #

• Headless Mode: Speed up scraping by running Chrome in headless mode (add options.add_argument('--headless=new') when initializing the driver).
• Rate Limiting: Add random wait times between actions (e.g., time.sleep(random.uniform(1,3))) to avoid triggering Snapdeal's anti-scraping measures.
• Pagination Support: The current code only grabs first-page reviews. Add logic to click the "Next" button if you need more reviews per star filter.

内容的提问来源于stack exchange，提问作者crushed_bug