Python爬虫问题：乐高创意项目评论区Show the reply按钮无法点击

Hey there! Let's work through that stubborn "Show the reply" button problem you're facing. From what you've described, it sounds like the issue stems from a combination of outdated tooling and missing proper dynamic element handling. Here's how to fix it step by step:

1. Ditch PhantomJS (It's Outdated!) #

PhantomJS has been deprecated for years and doesn't play well with modern JavaScript-driven sites like Lego Ideas. Swap it for ChromeDriver or GeckoDriver (for Firefox) — they're actively maintained and handle dynamic content way better.

For Chrome, you can set up headless mode (to avoid a visible browser window) like this:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--headless=new")  # Headless mode for Chrome 112+
driver = webdriver.Chrome(options=options)

2. Use Explicit Waits Instead of time.sleep() #

Directly clicking elements without waiting for them to load properly is a common pitfall. Use WebDriverWait to ensure the button is visible and clickable before attempting to interact with it:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(driver, 15)  # Wait up to 15 seconds for elements

3. Correctly Locate the "Show the reply" Buttons #

First, use your browser's developer tools (F12) to inspect the actual HTML of the button. It might not be a <button> tag — often these are <div> or <a> elements with a specific class or text.

Example Locators: #

• By exact text: Use this if the button text is exactly "Show the reply" (no extra spaces):

  reply_buttons = wait.until(EC.presence_of_all_elements_located(
      (By.XPATH, "//*[text()='Show the reply']")
  ))
  

• By partial text: Use this if the text has extra spaces or variations:

  reply_buttons = wait.until(EC.presence_of_all_elements_located(
      (By.XPATH, "//*[contains(text(), 'Show the reply')]")
  ))
  

• By class name: If the button has a unique class (e.g., show-reply-btn):

  reply_buttons = wait.until(EC.presence_of_all_elements_located(
      (By.CLASS_NAME, "show-reply-btn")
  ))
  

4. Handle Clicks Reliably #

Sometimes elements are hidden behind other UI elements, or Selenium can't click them directly. Use these tricks to ensure clicks work:

for button in reply_buttons:
    try:
        # Scroll the button into view first
        driver.execute_script("arguments[0].scrollIntoView({block: 'center'});", button)
        # Wait until the button is clickable
        wait.until(EC.element_to_be_clickable(button))
        # Use JavaScript to click if direct click fails
        driver.execute_script("arguments[0].click();", button)
        # Wait for replies to load (adjust the locator to match actual reply elements)
        wait.until(EC.presence_of_element_located((By.CLASS_NAME, "reply-content")))
    except Exception as e:
        print(f"Failed to click button: {str(e)}")
        continue

5. Verify After Clicking #

After clicking each button, make sure the reply content actually loads before moving on. Using WebDriverWait to check for a reply-specific element (like a class for reply text) is more reliable than time.sleep().

Full Modified Code Snippet #

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# Set up headless Chrome
options = Options()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)

try:
    driver.get('https://ideas.lego.com/projects/6f1d4d6e-f47a-467a-8b10-d918d04e6d5c/comments_tab')
    
    # First handle "Load more comments" (you said this is already working, but just in case)
    while True:
        try:
            load_more_btn = wait.until(EC.element_to_be_clickable(
                (By.XPATH, "//*[text()='Load more comments']")
            ))
            load_more_btn.click()
            wait.until(EC.staleness_of(load_more_btn))  # Wait for new comments to load
        except:
            break  # No more load more buttons
    
    # Now handle "Show the reply" buttons
    wait = WebDriverWait(driver, 15)
    reply_buttons = wait.until(EC.presence_of_all_elements_located(
        (By.XPATH, "//*[contains(text(), 'Show the reply')]")
    ))
    
    for button in reply_buttons:
        try:
            driver.execute_script("arguments[0].scrollIntoView({block: 'center'});", button)
            wait.until(EC.element_to_be_clickable(button))
            driver.execute_script("arguments[0].click();", button)
            # Wait for reply to appear (adjust the locator to match actual reply elements)
            wait.until(EC.presence_of_element_located((By.CLASS_NAME, "comment-reply")))
        except Exception as e:
            print(f"Error with reply button: {str(e)}")
            continue
    
    # Now scrape all comments and replies
    soup = BeautifulSoup(driver.page_source, 'html.parser')
    # Your scraping logic here

finally:
    driver.quit()

Key Notes: #

• Always use your browser's dev tools to confirm element selectors — Lego's UI might change, so hardcoding selectors without verification leads to failures.
• Avoid relying on time.sleep(); explicit waits make your script more robust and faster.
• Headless Chrome is a drop-in replacement for PhantomJS with far better compatibility.

内容的提问来源于stack exchange，提问作者Emad M