使用Selenium爬取YouTube频道视频数据时下一个按钮点击报错问题

核心出错原因 #

• 下一个播放按钮的CSS选择器存在语法错误，路径错误且字符串未闭合，无法正确定位元素
• 使用Python内置函数名next作为变量名，存在命名冲突
• 未添加元素等待逻辑，页面未完全加载就执行定位、点击操作导致报错
• YouTube播放器控制栏会在播放几秒后自动隐藏，未触发显示的情况下无法定位到控制栏的按钮

修复后代码 #

import time
import pandas as pd
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains

# 此处保留你原有初始化webdriver、打开目标播放列表的逻辑
Reviews = pd.DataFrame()
# 按需修改要采集的视频数量
for i in range(10):
    # 等待视频标题加载完成，确认页面加载就绪
    WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, '//*[@id="container"]/h1/yt-formatted-string')))
    time.sleep(2)

    # 抓取视频标题
    all_posts = driver.find_elements(By.XPATH, '//*[@id="container"]/h1/yt-formatted-string')
    all_posts = [x.text for x in all_posts]
    posts = pd.Series(all_posts)

    # 抓取播放量
    all_views = driver.find_elements(By.XPATH, "//*[@class='short-view-count style-scope ytd-video-view-count-renderer']")
    all_views = [x.text for x in all_views]
    views = pd.Series(all_views)

    # 抓取点赞数
    all_likes = driver.find_elements(By.XPATH, '/html/body/ytd-app/div/ytd-page-manager/ytd-watch-flexy/div[5]/div[1]/div/div[8]/div[2]/ytd-video-primary-info-renderer/div/div/div[3]/div/ytd-menu-renderer/div/ytd-toggle-button-renderer[1]/a/yt-formatted-string')
    all_likes = [x.text for x in all_likes]
    likes = pd.Series(all_likes)

    # 注意：2021年起YouTube已公开隐藏精确点踩数，普通访问场景下无法抓取到该数据
    all_dislikes = driver.find_elements(By.XPATH, '/html/body/ytd-app/div/ytd-page-manager/ytd-watch-flexy/div[5]/div[1]/div/div[8]/div[2]/ytd-video-primary-info-renderer/div/div/div[3]/div/ytd-menu-renderer/div/ytd-toggle-button-renderer[2]/a/yt-formatted-string')
    all_dislikes = [x.text for x in all_dislikes]
    dislikes = pd.Series(all_dislikes)

    # 滚动到评论区加载评论数
    driver.execute_script("window.scrollTo(0, 600);")
    time.sleep(1)
    all_comments = driver.find_elements(By.XPATH, '/html/body/ytd-app/div/ytd-page-manager/ytd-watch-flexy/div[5]/div[1]/div/ytd-comments/ytd-item-section-renderer/div[1]/ytd-comments-header-renderer/div[1]/h2/yt-formatted-string/span[1]')
    all_comments = [x.text for x in all_comments]
    comments = pd.Series(all_comments)

    df = pd.DataFrame({"Title":posts, "Views":views, "Likes":likes, "Dislikes":dislikes, "No of Comments":comments})
    # 原有append方法已被pandas废弃，替换为concat
    Reviews = pd.concat([Reviews, df], ignore_index=True)

    # 触发播放器控制栏显示
    player = driver.find_element(By.ID, "movie_player")
    ActionChains(driver).move_to_element(player).perform()
    time.sleep(0.5)
    # 等待下一个按钮可点击后执行点击
    next_btn = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, 'a.ytp-next-button.ytp-button')))
    next_btn.click()
    # 等待下一个视频加载完成
    time.sleep(3)

print(Reviews)

额外注意事项 #

• 可适当延长等待时间，避免触发YouTube反爬机制导致IP被限制
• 如果播放过程中出现前置广告，可添加逻辑等待广告跳过按钮出现后点击跳过，避免后续逻辑报错
• 原有代码中两层无意义的嵌套循环已删除，不影响数据采集逻辑

内容的提问来源于stack exchange，提问作者user14255498