使用Selenium多页爬取Yahoo梦幻足球ADP数据遇分页点击异常

尝试爬取Yahoo Fantasy Football的ADP数据，但爬虫只能获取初始页面的30名球员，点击下一页时持续触发NoSuchElementException异常，求技术解决方案。

原代码： #

Player_Name = []
Position_Team = []
Draft_Position = []

url = 'https://football.fantasysports.yahoo.com/f1/draftanalysis?guccounter=1&guce_referrer=aHR0cHM6Ly93d3cuYmluZy5jb20v&guce_referrer_sig=AQAAAGyHhLSemb5ibj1nJvdsNbQg14lWsna6Y9rAs7DlZ8-d_QA5SkDx_7bjMHMhdGMkQMOLR9F8a56RJ3Xq2zeuUu7QCCivRyHB3FAKv9Oi54Nk01Aev3KFisakR7REUsv42FuWxEKthpGw-BScwgVqzUdRc0a6oATq71C7j85X1C16'

driver.get(url)
time.sleep(1)

for i in range(3):
    players = driver.find_elements(by='xpath', value='//tr[contains(@data-tst, "table-row")]')
delay = 20
      WebDriverWait(driver,delay).until(EC.presence_of_element_located((By.XPATH,'//tr[contains(@data-tst, "table-row")]')))
    for player in players:
        Player_Name.append(player.find_element(by='xpath', value ='.//div[contains(@data-tst, "player-name")]').text)
        Position_Team.append(player.find_element(by='xpath', value = './/div[contains(@class, "C($text-secondary)")]').text)
        Draft_Position.append(player.find_element(by='xpath', value ='.//div[contains(@class, "Ta(c)")]').text)

        next = driver.find_element(by = 'xpath', value = '//*[@id="yui_3_18_1_1_1724006094920_810"]')
        next.click()

问题分析与解决方案： #

• 动态ID导致下一页按钮定位失败
  原代码用//*[@id="yui_3_18_1_1_1724006094920_810"]定位下一页，这个ID是页面动态生成的，每次加载都会变，肯定抓不到元素。换成稳定属性定位，比如通过按钮的data-tst属性或文本，比如//a[contains(@data-tst, "pagination-next")]（实际要打开F12检查页面元素确认）。

• 代码结构混乱，等待逻辑错误
  原代码中delay = 20和WebDriverWait的缩进完全错误，不在循环内部，而且time.sleep(1)时间太短，页面可能还没加载完就开始抓数据。要把显式等待放到循环里，确保球员列表加载完成后再抓取。

• 翻页时机错误
  原代码在遍历每个球员时就点击下一页，会导致页面提前跳转，后续球员数据还没抓完就切换页面。必须等整页球员数据都抓取完成后，再点击下一页。

修正后的示例代码： #

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

Player_Name = []
Position_Team = []
Draft_Position = []

url = 'https://football.fantasysports.yahoo.com/f1/draftanalysis?guccounter=1&guce_referrer=aHR0cHM6Ly93d3cuYmluZy5jb20v&guce_referrer_sig=AQAAAGyHhLSemb5ibj1nJvdsNbQg14lWsna6Y9rAs7DlZ8-d_QA5SkDx_7bjMHMhdGMkQMOLR9F8a56RJ3Xq2zeuUu7QCCivRyHB3FAKv9Oi54Nk01Aev3KFisakR7REUsv42FuWxEKthpGw-BScwgVqzUdRc0a6oATq71C7j85X1C16'

# 初始化Chrome浏览器驱动，需确保已配置好驱动路径
driver = webdriver.Chrome()
driver.get(url)
time.sleep(2)  # 给页面初始化留基础时间

for i in range(3):
    # 显式等待，确保当前页所有球员行加载完成
    WebDriverWait(driver, 20).until(
        EC.presence_of_all_elements_located((By.XPATH, '//tr[contains(@data-tst, "table-row")]'))
    )
    players = driver.find_elements(By.XPATH, '//tr[contains(@data-tst, "table-row")]')
    
    # 遍历抓取当前页每个球员的数据
    for player in players:
        try:
            name = player.find_element(By.XPATH, './/div[contains(@data-tst, "player-name")]').text
            pos_team = player.find_element(By.XPATH, './/div[contains(@class, "C($text-secondary)")]').text
            draft_pos = player.find_element(By.XPATH, './/div[contains(@class, "Ta(c)")]').text
            
            Player_Name.append(name)
            Position_Team.append(pos_team)
            Draft_Position.append(draft_pos)
        except Exception as e:
            print(f"抓取单个球员数据失败: {e}")
            continue
    
    # 点击下一页，先等待按钮可点击
    try:
        next_button = WebDriverWait(driver, 10).until(
            EC.element_to_be_clickable((By.XPATH, '//a[contains(@data-tst, "pagination-next")]'))
        )
        next_button.click()
        time.sleep(1)  # 翻页后等待页面切换完成
    except Exception as e:
        print(f"翻页失败: {e}")
        break

# 关闭浏览器
driver.quit()

# 输出抓取结果统计
print(f"总共抓取到 {len(Player_Name)} 名球员的数据")

额外注意事项： #

1. 定位下一页按钮时，一定要自己打开浏览器开发者工具（F12）查看元素的稳定属性，比如data-tst、aria-label或者按钮上的文本（比如"Next"），不要用动态生成的ID。
2. 加入try-except块可以避免单个元素抓取失败导致整个爬虫崩溃，提高稳定性。
3. 尽量用WebDriverWait的显式等待替代time.sleep，前者会根据元素加载情况自动等待，更高效稳定。

内容的提问来源于stack exchange，提问作者Peter O'Boyle