Chrome Driver解析时崩溃：爬取AFD新闻页点击特定文章出错

尝试从AFD新闻页面爬取每篇文章内容，使用以下代码：

import requests
from bs4 import BeautifulSoup
    
from selenium.webdriver import Chrome
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait as wait 
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
from time import sleep 
from tqdm import tqdm 

browser = Chrome('AFD/chromedriver')
url = 'https://www.afd.de/presse/'
browser.get(url)

posts = browser.find_elements (By.CLASS_NAME, 'blog-shortcode-post-title')

results = []
for post in posts:
    actions = ActionChains(browser)
    actions.move_to_element(post)
    post.click()

    page_source = browser.page_source
    soup = BeautifulSoup(page_source, 'lxml')
    paragraphs = soup.findAll('p')
    paragraphs = soup.find('div', class_='post-content').findAll('p')

    for paragraph in paragraphs:
        paragraph_texted = paragraph.text
        results.append(paragraph_texted)
    sleep(3)
    browser.back()
    sleep(3)

流程正常，直到处理某篇文章时ChromeDriver崩溃，错误截图如下：

核心原因 #

1. 元素引用失效：初始获取的posts元素列表在页面导航（browser.back()）后，原DOM元素已被销毁，后续循环操作的是无效元素引用，触发驱动崩溃。
2. 版本不兼容：Chrome浏览器与ChromeDriver版本不匹配，访问包含特殊JS/资源的页面时容易触发崩溃。
3. 页面加载异常：目标文章页可能存在特殊脚本、弹窗或资源加载阻塞，导致驱动进程崩溃。

解决步骤 #

1. 修复元素引用问题

每次回到列表页后重新获取元素，避免操作失效对象：

browser = Chrome('AFD/chromedriver')
url = 'https://www.afd.de/presse/'
browser.get(url)

# 先获取总文章数，避免重复计数
total_posts = len(browser.find_elements(By.CLASS_NAME, 'blog-shortcode-post-title'))
results = []

for i in range(total_posts):
    # 每次循环重新获取元素列表
    posts = browser.find_elements(By.CLASS_NAME, 'blog-shortcode-post-title')
    post = posts[i]
    
    actions = ActionChains(browser)
    actions.move_to_element(post)
    post.click()

    # 显式等待文章内容加载完成，替代固定sleep
    wait(browser, 10).until(EC.presence_of_element_located((By.CLASS_NAME, 'post-content')))
    
    page_source = browser.page_source
    soup = BeautifulSoup(page_source, 'lxml')
    paragraphs = soup.find('div', class_='post-content').findAll('p')

    for paragraph in paragraphs:
        results.append(paragraph.text)
    
    browser.back()
    # 等待列表页重新加载完成
    wait(browser, 10).until(EC.presence_of_element_located((By.CLASS_NAME, 'blog-shortcode-post-title')))

2. 匹配Chrome与ChromeDriver版本

查看Chrome浏览器版本（设置→关于Chrome），下载对应版本的ChromeDriver替换现有驱动，确保版本完全兼容。

3. 增加异常捕获

避免单个文章出错导致整个脚本终止：

for i in range(total_posts):
    try:
        posts = browser.find_elements(By.CLASS_NAME, 'blog-shortcode-post-title')
        post = posts[i]
        actions = ActionChains(browser)
        actions.move_to_element(post)
        post.click()

        wait(browser, 10).until(EC.presence_of_element_located((By.CLASS_NAME, 'post-content')))
        
        page_source = browser.page_source
        soup = BeautifulSoup(page_source, 'lxml')
        paragraphs = soup.find('div', class_='post-content').findAll('p')

        for paragraph in paragraphs:
            results.append(paragraph.text)
        
        browser.back()
        wait(browser, 10).until(EC.presence_of_element_located((By.CLASS_NAME, 'blog-shortcode-post-title')))
    except Exception as e:
        print(f"处理第{i+1}篇文章时出错: {str(e)}")
        # 出错后强制回到列表页，继续下一篇
        browser.get(url)
        wait(browser, 10).until(EC.presence_of_element_located((By.CLASS_NAME, 'blog-shortcode-post-title')))
        continue

4. 优化浏览器配置

禁用图片加载等非必要功能，减少资源消耗：

from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument('--blink-settings=imagesEnabled=false')
# 若页面无需JS渲染可禁用，需根据实际情况调整
# options.add_argument('--disable-javascript')
browser = Chrome('AFD/chromedriver', options=options)

内容的提问来源于stack exchange，提问作者Mikhail Rotar