如何用Python和Selenium提取YouTube搜索结果的时长及缩略图链接

YouTube搜索结果页的时长和缩略图节点不在你当前遍历的.text-wrapper.style-scope.ytd-video-renderer容器内，所以你在该节点下查找对应元素会失效，只需调整遍历根节点、补充对应选择器即可正常提取。 #

具体修改点 #

1. 调整遍历的根节点：将原遍历.text-wrapper改为遍历最外层的视频条目节点ytd-video-renderer，可同时覆盖文本、缩略图、时长三类数据
2. 时长选择器：span.ytd-thumbnail-overlay-time-status-renderer，提取文本后需去除前后空白字符
3. 缩略图URL选择器：ytd-thumbnail img，优先提取src属性，若为空可提取延迟加载属性data-src
4. 兼容Selenium 4+的方法调用：原find_element_by_css_selector已废弃，建议替换为find_element(By.CSS_SELECTOR, 选择器)写法

调整后的完整代码 #

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import pandas as pd
import time

# 处理Chrome SSL问题
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--ignore-ssl-errors')
options.add_argument('--ignore-certificate-errors-spki-list')
options.add_argument('log-level=3') 
options.add_argument('--disable-notifications')
options.add_argument('--headless=new') # 适配新版Chrome无头模式

keyword = 'selenium python tutorial for beginners'

def get_video_results():
    youtube_data = []
    driver = webdriver.Chrome(options=options)
    driver.get(f'https://www.youtube.com/results?search_query=%22{keyword}%22')

    # 自动处理同意弹窗
    consent_button_xpath = '/html/body/ytd-app/ytd-consent-bump-v2-lightbox/tp-yt-paper-dialog/div[2]/div[2]/div[5]/div[2]/ytd-button-renderer[2]/a'
    consent = WebDriverWait(driver, 40).until(EC.element_to_be_clickable((By.XPATH, consent_button_xpath)))
    consent.click()

    # 滚动加载全部结果
    while True:
        end_result = driver.find_element(By.CSS_SELECTOR, '#message').is_displayed()
        driver.execute_script("var scrollingElement = (document.scrollingElement || document.body);scrollingElement.scrollTop = scrollingElement.scrollHeight;")
        time.sleep(1) # 等待内容加载完成
        if end_result:
            break

    # 遍历所有视频条目
    for result in driver.find_elements(By.CSS_SELECTOR, 'ytd-video-renderer.style-scope.ytd-item-section-renderer'):
        # 提取标题
        title = result.find_element(By.CSS_SELECTOR, '.title-and-badge.style-scope.ytd-video-renderer').text.strip()
        # 提取时长
        duration = result.find_element(By.CSS_SELECTOR, 'span.ytd-thumbnail-overlay-time-status-renderer').text.strip()
        # 提取缩略图URL
        thumbnail_elem = result.find_element(By.CSS_SELECTOR, 'ytd-thumbnail img')
        thumbnail_url = thumbnail_elem.get_attribute('src') or thumbnail_elem.get_attribute('data-src')

        scraped_keywords = {
          'keyword': keyword,
          'title': title,
          'duration': duration,
          'thumbnail': thumbnail_url
        }
        youtube_data.append(scraped_keywords)

    # 输出并保存结果
    print(pd.DataFrame(youtube_data))
    df = pd.DataFrame(youtube_data)
    df.to_csv('Extracted_info.csv', index=False)

    driver.quit()

get_video_results()

补充说明 #

• 若同意弹窗的XPath失效，可替换为按文本匹配的更稳定选择器，适配不同地区的语言版本
• 部分直播类内容无时长属性，可加try-except捕获异常，避免脚本中断
• 爬取频率不要过高，避免触发YouTube的反爬机制

内容的提问来源于stack exchange，提问作者MarkWP