Python+Selenium爬取MyAnimeList卡顿：无法提取动漫信息

我是Python编程新手，正在学习网页爬取技术，最终选择用Selenium开发工具从MyAnimeList提取动漫数据，但遇到了问题：爬取工具打开第一个链接后卡住，提取标题、评分或类型时触发超时错误。试过更换CSS元素定位无效，指定chromedriver.exe路径反而导致代码无法运行，目标是将数据存入CSV文件的标题、评分、类型三列中。

原代码 #

import pandas as pd
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
wait = WebDriverWait(driver, 5)

links = []
for page in range(0, 100, 50):
    driver.get('https://myanimelist.net/topanime.php?limit=' + str(page))
    link = driver.find_elements(By.CSS_SELECTOR, 'div[class="detail"] h3 a')
    for item in link:
        links.append(item.get_attribute('href'))

titles = []
ratings = []
genres = []
for item_link in links:
    driver.get(item_link)
    title = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'div[class="h1-title"]')))
    rating = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'div[class="score-label"]')))
    genre = wait.until(EC.visibility_of_element_located((By.XPATH, 'div[class="//*[contains(text(), "Genre")]/parent::div')))
    titles.append(title.text)
    ratings.append(rating.text)
    genres.append(genre.text)

my_data = {'Titles': titles, 'Ratings': ratings, 'Genres': genres}
df = pd.DataFrame(my_data)

csv_file_path = 'C:/Users/maldo/Desktop/anime/AnimeList.csv'
df.to_csv(csv_file_path, index=False)

错误信息 #

Traceback (most recent call last):
  File "C:\Users\maldo\PycharmProjects\MAL-WebScraper\main.py", line 23, in <module>
    rating = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'div[class="score-label"]')))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\maldo\PycharmProjects\MAL-WebScraper\venv\Lib\site-packages\selenium\webdriver\support\wait.py", line 95, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: 
Stacktrace:
Backtrace:
    GetHandleVerifier [0x0045A813+48355]
    (No symbol) [0x003EC4B1]
    (No symbol) [0x002F5358]
    (No symbol) [0x003209A5]
    (No symbol) [0x00320B3B]
    (No symbol) [0x0034E232]
    (No symbol) [0x0033A784]
    (No symbol) [0x0034C922]
    (No symbol) [0x0033A536]
    (No symbol) [0x003182DC]
    (No symbol) [0x003193DD]
    GetHandleVerifier [0x006BAABD+2539405]
    GetHandleVerifier [0x006FA78F+2800735]
    GetHandleVerifier [0x006F456C+2775612]
    GetHandleVerifier [0x004E51E0+616112]
    (No symbol) [0x003F5F8C]
    (No symbol) [0x003F2328]
    (No symbol) [0x003F240B]
    (No symbol) [0x003E4FF7]
    BaseThreadInitThunk [0x74F97D59+25]
    RtlInitializeExceptionChain [0x76FEB79B+107]
    RtlClearBits [0x76FEB71F+191]

问题分析与修正方案 #

代码主要问题在于定位器语法错误、等待时间不足，以及未考虑网站反爬机制，以下是修正后的代码：

import pandas as pd
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.options import Options

# 配置浏览器选项，模拟正常用户访问
chrome_options = Options()
chrome_options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36")

driver = webdriver.Chrome(options=chrome_options)
wait = WebDriverWait(driver, 10)  # 延长等待时间至10秒，适配页面加载速度

links = []
for page in range(0, 100, 50):
    driver.get(f'https://myanimelist.net/topanime.php?limit={page}')
    # 等待页面核心元素加载完成
    wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, 'div.detail h3 a')))
    link_elements = driver.find_elements(By.CSS_SELECTOR, 'div.detail h3 a')
    for item in link_elements:
        links.append(item.get_attribute('href'))
    time.sleep(2)  # 添加短延迟，避免请求频率过高

titles = []
ratings = []
genres = []

for item_link in links:
    driver.get(item_link)
    time.sleep(1)  # 页面加载缓冲
    
    # 提取标题：修正CSS选择器，h1标签才是标题容器
    title = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'h1.h1-title')))
    titles.append(title.text.strip())
    
    # 提取评分：确保元素定位准确
    rating = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'div.score-label')))
    ratings.append(rating.text.strip())
    
    # 提取类型：修正XPath，找到Genre标签后的所有类型链接并拼接
    genre_elements = wait.until(EC.presence_of_all_elements_located((By.XPATH, '//span[text()="Genre:"]/following-sibling::a')))
    genre_text = ', '.join([g.text.strip() for g in genre_elements])
    genres.append(genre_text)
    
    time.sleep(2)  # 单页爬取后延迟，降低反爬风险

# 保存数据到CSV，指定编码避免中文乱码
my_data = {'Titles': titles, 'Ratings': ratings, 'Genres': genres}
df = pd.DataFrame(my_data)
csv_file_path = 'C:/Users/maldo/Desktop/anime/AnimeList.csv'
df.to_csv(csv_file_path, index=False, encoding='utf-8-sig')

# 关闭浏览器，清理进程
driver.quit()

关键修正点 #

1. 修复定位器错误：修正了类型提取的XPath语法错误，同时调整标题的CSS选择器为更准确的h1.h1-title
2. 延长等待时间：将WebDriverWait的超时时间从5秒改为10秒，适配MyAnimeList的页面加载速度
3. 添加反爬适配：设置浏览器UA模拟正常访问，增加请求间隔延迟，避免被网站拦截
4. 优化数据处理：将类型元素文本拼接为字符串，保存CSV时指定utf-8-sig编码避免中文乱码
5. 资源清理：添加driver.quit()关闭浏览器，避免残留进程

内容的提问来源于stack exchange，提问作者lacarabela