Selenium网页爬取异常：翻页至第2页停止问题排查

问题描述 #

我用Selenium编写了Coursera Python课程的数据爬取脚本，需求是遍历课程元素并自动翻页，但运行时仅到第2页就停止。相关代码如下：

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC 


chromedriver_path = '/Users/charl/Downloads/chromedriver-win32/chromedriver.exe'
driver = webdriver.Chrome(service=Service(chromedriver_path))
wait = WebDriverWait(driver,9)

driver.get('https://www.coursera.org/search?query=python')

# close cookies popup
wait.until(EC.element_to_be_clickable((By.ID, 'onetrust-reject-all-handler')))
time.sleep(1)
wait.until(EC.element_to_be_clickable((By.ID, 'onetrust-reject-all-handler'))).click()

course_data = {key:[] for key in ['Título','Puntuación','Descripción', 'Institución', 'Dificultad, Tipo, Duración']}
#course_data = {key:[] for key in ['Título','Puntuación','Descripción', 'Institución']}
page_count = 10

for page in range(page_count):

    print(f'current page: {driver.find_element(By.CSS_SELECTOR, "button[class*=current]").text}', end='\r')  #elements?
    
    courses = wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, 'main ul>li')))
    for course in courses:
        course_data['Título']       += [driver.execute_script('return arguments[0].querySelector("a[class*=CommonCard-title]")?.innerText', course)]
        course_data['Descripción'] += [driver.execute_script('return arguments[0].querySelector("div[class*=ProductCard-body]")?.innerText', course)]  #div+p bueno #p:has(>span) #h2 + p
        course_data['Puntuación']      += [driver.execute_script('return arguments[0].querySelector("div[class*=CommonCard-ratings]")?.innerText.replace("\n\n","⭐")', course)]
        course_data['Institución']       += [driver.execute_script('return arguments[0].querySelector("span")?.innerText', course)]
        course_data['Dificultad, Tipo, Duración']       += [driver.execute_script('return arguments[0].querySelector("div[class*=CommonCard-metadata]")?.innerText', course)] #div+p #div:last-child p

    next_btn = driver.find_elements(By.CSS_SELECTOR, 'button[aria-label="Next Page"]')
    for next_btn in next_btn:
        next_btn.click()

import pandas as pd
df = pd.DataFrame(course_data)
df.to_csv('course_data.csv', index=False, encoding='utf-8-sig')

df.head(10)

#df 

问题原因与修复方案 #

1. 翻页按钮点击逻辑错误 #

• 原代码用find_elements获取所有匹配"Next Page"的按钮并循环点击，会导致同一页按钮被重复点击，或后续页面按钮未就绪时点击失败，直接终止翻页流程。
• 修复：
  改用WebDriverWait等待按钮可交互，用find_element精准定位单个按钮，点击前滚动到可见区域，避免遮挡：

  # 替换原翻页代码
  try:
      next_btn = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'button[aria-label="Next Page"]')))
      driver.execute_script("arguments[0].scrollIntoView(true);", next_btn)
      next_btn.click()
      # 等待页面加载完成（等待当前页码元素失效）
      wait.until(EC.staleness_of(driver.find_element(By.CSS_SELECTOR, "button[class*=current]")))
  except Exception as e:
      print(f"翻页失败: {e}")
      break
  

2. 页面元素定位不稳定 #

• 原课程选择器main ul>li可能匹配非课程元素，导致后续页面抓取时元素定位失效；打印当前页码时直接调用find_element，容易触发陈旧元素异常。
• 修复：
  • 用更精准的课程卡片选择器：div[data-test="search-result-card"]（Coursera官方测试属性，稳定性更高）；
  • 打印页码时重新等待元素可见，避免陈旧元素问题：

    current_page_elem = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "button[class*=current]")))
    print(f'current page: {current_page_elem.text}', end='\r')
    

3. 缺少页面加载等待机制 #

• 点击下一页后直接抓取数据，新页面内容未完全加载，会导致抓取到旧数据或元素未就绪，触发异常终止流程。
• 修复：
  点击翻页后，等待新的课程列表加载完成：

  wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, 'div[data-test="search-result-card"]')))
  

内容的提问来源于stack exchange，提问作者making