Python爬虫（Selenium+BeautifulSoup）爬VK时随机页面崩溃错误求助

我用Python结合Selenium与BeautifulSoup编写VK社交网络用户页面爬虫，逻辑是批量加载用户链接，循环分析每个页面提取姓名、好友数、订阅数等数据。但循环过程中会随机出现页面崩溃错误，导致无法加载下一页，同一数据集下该问题仍会随机发生。

循环代码 #

people = BSFile.find_all(attrs={"class": "fans_fan_lnk"})

for i in range(len(people)):
    print(people[i]["href"])

x = int(input())

for j in range(len(people)):
    time.sleep(0.1)
    driver.get("https://vk.com"+str(people[j]['href']))
    time.sleep(0.1)
    element = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CLASS_NAME, "AvatarRich__background")))
    main_page = driver.page_source
    time.sleep(0.1)
    BSFile = bs(main_page, "html.parser")
    MasForUnpack = []

    if BSFile.find(attrs = {"class": "ClosedProfileBlock-module__title--eb2WU vkuiTitle vkuiTitle--l-3 vkuiTitle--w-2"}) != None:
        print("Профиль закрыт")
        Numbers = BSFile.find_all(attrs={"class": "vkuiHeader__indicator vkuiCaption vkuiCaption--l-1"})
        TagOfName = BSFile.h2

        for st in TagOfName.stripped_strings:
            MasForUnpack.append(st)

        if (len(MasForUnpack) == 2):
            if (MasForUnpack[1].find('заходил') == -1) and (MasForUnpack[1].find('онлайн')== -1):
                print(MasForUnpack[0] + " " + MasForUnpack[1])
            else:
                print(MasForUnpack[0])

        if (len(MasForUnpack) == 3):
            print(MasForUnpack[0] + " " + MasForUnpack[1])

        #Names = BSFile.find_all(attrs={"class":"vkuiHeader__content-in"})
        for i in range(len(Numbers)):
            print("Друзья: "+ Numbers[i].string)
    else:
        print("Профиль открыт")
        Numbers = []
        Names = []
        Numbers = BSFile.find_all(attrs={"class": "vkuiHeader__indicator vkuiCaption vkuiCaption--l-1 vkuiCaption--w-1"})
        Names = BSFile.find_all(attrs={"class":"vkuiHeader__content-in"})

        TagOfName = BSFile.h2

        for st in TagOfName.stripped_strings:
            MasForUnpack.append(st)

        if (len(MasForUnpack) == 2):
            if (MasForUnpack[1].find('заходил') == -1) and (MasForUnpack[1].find('онлайн')== -1):
                print(MasForUnpack[0] + " " + MasForUnpack[1])
            else:
                print(MasForUnpack[0])

        if (len(MasForUnpack) == 3):
            print(MasForUnpack[0] + " " + MasForUnpack[1])

        for i in range(len(Numbers)):
            print(Names[i].string + ": "+ Numbers[i].string)
    print()

错误信息 #

Traceback (most recent call last):
  File "G:\_prog_files\Html\proba\2.py", line 51, in <module>
    driver.get("https://vk.com"+str(people[j]['href']))
  File "G:\Python\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 441, in get
    self.execute(Command.GET, {'url': url})
  File "G:\Python\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 429, in execute
    self.error_handler.check_response(response)
  File "G:\Python\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 243, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: session deleted because of page crash
from unknown error: cannot determine loading status
from tab crashed

已尝试操作 #

• 更换Chrome为Edge浏览器，问题未解决
• 给Chrome添加以下启动参数，无效：

chrome_options = Options()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=chrome_options )

解决方案 #

1. 调整Chrome启动参数（针对Windows环境） #

添加更多内存与渲染相关参数，避免页面崩溃：

from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager

chrome_options = Options()
# 禁用GPU加速，避免渲染冲突
chrome_options.add_argument('--disable-gpu')
# 关闭内存压力检测
chrome_options.add_argument('--memory-pressure-off')
# 解决渲染代码完整性检查导致的崩溃
chrome_options.add_argument('--disable-features=RendererCodeIntegrity')
# 原有参数保留
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
# 设置页面加载策略为DOM就绪即停止，减少资源占用
chrome_options.page_load_strategy = 'eager'

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=chrome_options)

2. 加入异常捕获与重试机制 #

在循环中捕获页面崩溃异常，自动重试当前页面或重启driver：

from selenium.common.exceptions import WebDriverException
import time

max_retries = 3  # 单页面最大重试次数

for j in range(len(people)):
    retry_count = 0
    success = False
    while retry_count < max_retries and not success:
        try:
            # 停止上一页的加载，释放资源
            driver.execute_script('window.stop();')
            driver.get("https://vk.com"+str(people[j]['href']))
            # 智能等待必要元素，替换固定sleep
            element = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CLASS_NAME, "AvatarRich__background")))
            main_page = driver.page_source
            BSFile = bs(main_page, "html.parser")
            
            # 后续数据提取逻辑保持不变...
            
            success = True
        except WebDriverException as e:
            if "tab crashed" in str(e):
                retry_count += 1
                print(f"页面崩溃，重试第{retry_count}次...")
                time.sleep(2)
                # 重试前可尝试重启driver
                if retry_count == max_retries - 1:
                    driver.quit()
                    driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=chrome_options)
            else:
                raise

3. 优化资源占用 #

• 每次处理完页面后，清理cookies和本地存储：

driver.delete_all_cookies()
driver.execute_script('window.localStorage.clear();')
driver.execute_script('window.sessionStorage.clear();')

• 避免过短的time.sleep，可将0.1调整为0.5-1，降低请求频率，减少服务器压力和浏览器负载

内容的提问来源于stack exchange，提问作者Limaximy