Selenium爬取USNews大学排名时“加载更多”按钮异常求助

问题原因 #

1. 反爬机制拦截：USNews能识别裸ChromeDriver的自动化特征，直接请求会被标记，导致加载失败
2. 元素定位不稳定：原XPATH依赖页面结构，可能因页面更新失效
3. 点击逻辑问题：ActionChains的点击方式有时会触发反爬检测，且未确保元素完全可见

修复方案 #

1. 配置Chrome规避反爬 #

给ChromeDriver添加参数，模拟真实浏览器并隐藏自动化标识：

from selenium.webdriver.chrome.options import Options

chrome_options = Options()
# 模拟真实浏览器User-Agent
chrome_options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36")
# 禁用自动化检测开关
chrome_options.add_experimental_option("excludeSwitches", ["enable-automation"])
chrome_options.add_experimental_option('useAutomationExtension', False)
# 关闭Blink自动化控制特征
chrome_options.add_argument("--disable-blink-features=AutomationControlled")

# 初始化带配置的Driver
driver = webdriver.Chrome(executable_path=driver_path, options=chrome_options)

2. 优化元素定位与点击逻辑 #

• 用按钮文本定位，比固定XPATH更稳定
• 点击前滚动到元素可见位置，避免点击失效

# 等待Load More按钮可点击，用文本定位
show_more = wait.until(EC.element_to_be_clickable((By.XPATH, "//button[contains(text(), 'Load More')]")))
# 滚动到按钮居中位置
driver.execute_script("arguments[0].scrollIntoView({block: 'center'});", show_more)
time.sleep(1)
# 直接点击按钮，替代ActionChains
show_more.click()

3. 自动处理广告弹窗 #

即使手动关闭有效，自动处理能避免干扰后续操作：

try:
    # 等待广告关闭按钮出现并点击
    close_ad = wait.until(EC.element_to_be_clickable((By.XPATH, "//button[contains(@class, 'close')]")))
    close_ad.click()
    time.sleep(1)
except:
    # 无广告则跳过
    pass

完整修复代码 #

import requests
import os
from bs4 import BeautifulSoup
from selenium import webdriver
import time
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options

driver_path = 'xxx'  # 替换为你的chromedriver路径
chrome_options = Options()
chrome_options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36")
chrome_options.add_experimental_option("excludeSwitches", ["enable-automation"])
chrome_options.add_experimental_option('useAutomationExtension', False)
chrome_options.add_argument("--disable-blink-features=AutomationControlled")

driver = webdriver.Chrome(executable_path=driver_path, options=chrome_options)
url2 = 'https://www.usnews.com/education/best-global-universities/rankings'
wait = WebDriverWait(driver, 30)
driver.get(url2)
driver.maximize_window()

# 自动关闭广告
try:
    close_ad = wait.until(EC.element_to_be_clickable((By.XPATH, "//button[contains(@class, 'close')]")))
    close_ad.click()
    time.sleep(1)
except:
    pass

count = 1
while True:
    try:
        show_more = wait.until(EC.element_to_be_clickable((By.XPATH, "//button[contains(text(), 'Load More')]")))
        driver.execute_script("arguments[0].scrollIntoView({block: 'center'});", show_more)
        time.sleep(1)
        show_more.click()
        print(f"已加载第{count}次更多内容")
        count += 1
        # 等待内容加载完成（通过按钮状态变化判断）
        wait.until(EC.staleness_of(show_more))
        time.sleep(2)
        if count >= 5:  # 测试时限制加载次数，可按需调整
            break
    except Exception as e:
        print(f"加载失败: {str(e)}")
        break

driver.quit()

额外提示 #

• 不要频繁点击，适当增加等待时间，避免触发反爬阈值
• 如果仍失败，可尝试更换代理IP，分散请求来源
• 定期检查按钮定位规则，防止页面更新导致定位失效

内容的提问来源于stack exchange，提问作者Luo Ocean