Selenium中Show more按钮点击无效问题（Travelocity爬取场景）

我正在尝试爬取travelocity.com的酒店搜索页面，预期执行流程如下：

• 进入目标站点
• 滚动至页面底部
• 点击“Show more”按钮加载更多结果
• 重复滚动至底部并点击按钮的操作

但点击“Show more”按钮后，页面并未加载新结果，持续点击也无响应，推测可能被网站识别为机器人。该按钮位于页面底部：

我试过两种点击按钮的代码，均无效：

第一种（点击并按住释放）：

try:
    show_more_button = driver.find_element(By.XPATH, "//button[contains(text(), 'Show more')]")
    action_chains = ActionChains(driver)
    action_chains.click_and_hold(show_more_button).perform()
    time.sleep(1)
    action_chains.release().perform()
    time.sleep(5)
except NoSuchElementException:
    print("All results are loaded.")
    break

第二种（直接点击）：

show_more_button = driver.find_element(By.XPATH, "//button[contains(text(), 'Show more')]")
show_more_button.click()

完整代码如下：

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains

URL = 'https://www.travelocity.com/Hotel-Search?destination=San%20Francisco%20%28and%20vicinity%29%2C%20California%2C%20United%20States%20of%20America&regionId=178305&latLong=37.7874%2C-122.4082&flexibility=0_DAY&d1=2024-05-08&startDate=2024-05-08&d2=2024-05-22&endDate=2024-05-22&adults=2&rooms=1&theme=&userIntent=&semdtl=&useRewards=false&sort=RECOMMENDED'
driver = webdriver.Chrome()
driver.get(URL)
driver.maximize_window()
time.sleep(4)

while True:
    try:
        driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
        show_more_button = driver.find_element(By.XPATH, "//button[contains(text(), 'Show more')]")
        action_chains = ActionChains(driver)
        action_chains.click_and_hold(show_more_button).perform()
        time.sleep(0.5)
        action_chains.release().perform()
        
        time.sleep(3)
    except NoSuchElementException:
        print("All results are loaded.")
        break

driver.quit()

1. 优化反爬规避策略 #

Travelocity的反爬机制会识别自动化操作，需要模拟更真实的用户行为：

• 替换固定延迟为随机等待：用random.uniform()生成1-7秒的随机间隔，避免机械固定的等待时间
• 模拟分步滚动：不要直接跳转到页面底部，分多次滚动随机距离，模拟用户浏览动作
• 配置Chrome伪装参数：禁用自动化标识、设置真实UA，避免被网站直接识别为机器人

2. 改进按钮交互逻辑 #

• 等待元素可点击：用WebDriverWait等待按钮完全加载并处于可点击状态，避免无效点击
• 改用JS点击：如果按钮被前端事件拦截，直接通过JS触发点击，绕过部分反爬限制

优化后的完整代码 #

import time
import random
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.common.exceptions import NoSuchElementException, TimeoutException
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait

URL = 'https://www.travelocity.com/Hotel-Search?destination=San%20Francisco%20%28and%20vicinity%29%2C%20California%2C%20United%20States%20of%20America&regionId=178305&latLong=37.7874%2C-122.4082&flexibility=0_DAY&d1=2024-05-08&startDate=2024-05-08&d2=2024-05-22&endDate=2024-05-22&adults=2&rooms=1&theme=&userIntent=&semdtl=&useRewards=false&sort=RECOMMENDED'

# 配置Chrome规避反爬
options = webdriver.ChromeOptions()
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
options.add_argument("--disable-blink-features=AutomationControlled")
options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36")

driver = webdriver.Chrome(options=options)
driver.get(URL)
driver.maximize_window()
time.sleep(random.uniform(3, 5))

def scroll_to_bottom():
    """模拟真实滚动到页面底部"""
    last_height = driver.execute_script("return document.body.scrollHeight")
    while True:
        scroll_distance = random.randint(400, 600)
        driver.execute_script(f"window.scrollBy(0, {scroll_distance});")
        time.sleep(random.uniform(0.3, 0.8))
        new_height = driver.execute_script("return document.body.scrollHeight")
        if new_height == last_height:
            break
        last_height = new_height

while True:
    try:
        scroll_to_bottom()
        # 等待按钮可点击
        show_more_button = WebDriverWait(driver, 10).until(
            EC.element_to_be_clickable((By.XPATH, "//button[contains(text(), 'Show more')]"))
        )
        # JS触发点击
        driver.execute_script("arguments[0].click();", show_more_button)
        # 随机等待加载
        time.sleep(random.uniform(4, 7))
    except TimeoutException:
        print("等待按钮超时，检查是否已加载全部结果")
        break
    except NoSuchElementException:
        print("所有结果已加载完成。")
        break

driver.quit()

内容的提问来源于stack exchange，提问作者bSmall