Selenium爬取网页无法获取最新渲染内容问题求助

问题背景 #

我是Python及网页爬虫新手，因本科项目需求开展网页爬取工作。目标网页存在「Alle anzeigen」和「Sterne anzeigen」按钮，点击可触发JavaScript事件加载容器内容（容器已存在但无所需内容）。能成功点击按钮，但有时Selenium返回的页面仍不包含所需的带data-score属性的span元素。Chrome浏览器和ChromeDriver版本均为119.0.6045.105，更换新版本后问题依旧。测试链接为：https://www.kununu.com/de/procar-automobile/kommentare，爬取得到的页面缺少h4元素旁带data-score属性的div元素，预期页面应包含该元素。

当前实现代码 #

def connectAndGet(url):
        url = url
        driver = webdriver.Chrome(options=chromeOptionsSettingMaster())
        driver.set_window_size(1920, 1080)
        driver.get(url)

        # Handle cookie consent
        try:
            consent_button = WebDriverWait(driver, 10).until(
                ec.presence_of_element_located((By.ID, "consent-accept-button")))
            if consent_button:
                consent_button.click()
        except Exception as e:
            print("Cookie consent button not found or not clickable" + str(e))

        actions = ActionChains(driver)
        while True:
            try:
                WebDriverWait(driver, 10).until(
                    ec.presence_of_element_located((By.CLASS_NAME, 'index__contentBlock__KbUvv')))


                # for the button "Mehr anzeigen"
                buttonMA = driver.find_elements(By.CLASS_NAME, "reviews-show-all-button")
                if buttonMA:
                    for but in buttonMA:
                        actions.move_to_element(but).perform()
                        but.click()
                        time.sleep(randomizSleepOneToThree())

                # for the button "Sterne anzeigen"
                buttonSA = driver.find_elements(By.CLASS_NAME, "reviews-show-star-ratings-button")
                if buttonSA:
                    for buts in buttonSA:
                        if buts.text == "Sterne anzeigen":
                            actions.move_to_element(buts).perform()
                            buts.click()
                            time.sleep(randomizSleepOneToThree())

                # Wait for the button to be clickable "mehr Bewertungen lesen" (adjust selector accordingly)
                button = WebDriverWait(driver, 10).until(ec.element_to_be_clickable((By.ID,   "reviews-read-more-cta")))
                if button:
                    button.click()
                    time.sleep(randomizSleepOneToThree())

            except TimeoutException:
                print("Timeout waiting for element.")
                break
            except Exception as e:
                print(f"An unexpected error occurred: {e}")

        time.sleep(20)
        html_text = driver.page_source
        soup = BeautifulSoup(html_text, 'html.parser')
        soup.prettify()
        driver.quit()
        return soup

def randomizSleepOneToThree():
    min_sleep_time = 1  # Minimum sleep time in seconds
    max_sleep_time = 3  # Maximum sleep time in seconds
    return random.uniform(min_sleep_time, max_sleep_time)

def chromeOptionsSettingMaster():
    # Set the working directory to the location of your chromedriver and chrome
    # Get the directory of the current script
    script_directory = os.path.dirname(os.path.abspath(__file__))
    # add the path to chromedriver.exe
    final_directory = os.path.join(script_directory, "chromedriver", "chromedriver.exe")
    # add the path to chrome.exe
    final_binary = os.path.join(script_directory, "chromeBrowser", "chrome.exe")
    # manipulate the chrome binary to our test chrome
    chrome_options = webdriver.ChromeOptions()
    chrome_options.binary_location = final_binary
    chrome_options.add_argument("executable_path={}".format(final_directory))
    chrome_options.add_argument('--headless')
    chrome_options.add_argument('--disable-gpu')  # important for windows
    chrome_options.page_load_strategy = 'normal'

    return chrome_options

修复建议 #

1. 替换固定等待为显式等待 #

当前依赖time.sleep()的方式无法确保元素完全加载，建议在点击按钮后，直接等待目标data-score元素出现，避免因网络延迟或渲染慢导致的元素缺失：

# 点击"Sterne anzeigen"后，等待带data-score的元素加载完成
WebDriverWait(driver, 15).until(
    ec.presence_of_element_located((By.CSS_SELECTOR, '[data-score]'))
)

2. 优化按钮点击逻辑 #

• 点击按钮前，用ec.element_to_be_clickable()确保按钮处于可交互状态，而非仅存在于DOM中；
• 避免重复点击已切换状态的按钮，比如"Sterne anzeigen"点击后会变为"Sterne verbergen"，可增加文本判断过滤：

# 仅处理未点击过的"Sterne anzeigen"按钮
buttonSA = WebDriverWait(driver, 10).until(
    ec.presence_of_all_elements_located((By.CLASS_NAME, "reviews-show-star-ratings-button"))
)
for buts in buttonSA:
    if buts.text.strip() == "Sterne anzeigen" and ec.element_to_be_clickable((By.CLASS_NAME, "reviews-show-star-ratings-button")):
        actions.move_to_element(buts).perform()
        buts.click()
        # 等待目标元素加载
        WebDriverWait(driver, 15).until(
            ec.presence_of_element_located((By.CSS_SELECTOR, '[data-score]'))
        )

3. 调整无头模式配置 #

无头模式下可能存在渲染差异，添加模拟真实浏览器的参数，减少被识别为自动化工具的概率：

chrome_options.add_argument('--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36')
chrome_options.add_argument('--disable-blink-features=AutomationControlled')
chrome_options.add_experimental_option("excludeSwitches", ["enable-automation"])
chrome_options.add_experimental_option('useAutomationExtension', False)

4. 验证页面完全加载 #

在获取页面源码前，通过JavaScript确认页面渲染完成：

# 等待页面完全加载
WebDriverWait(driver, 20).until(
    lambda d: d.execute_script('return document.readyState') == 'complete'
)

5. 优化循环逻辑 #

当前while True循环可能重复点击按钮造成冗余，可改为判断是否存在未处理的可点击按钮，而非无限循环直到超时。

内容的提问来源于stack exchange，提问作者Momo