Python3.12+Selenium爬虫Ubuntu卡顿，Windows正常问题求助

问题描述 #

我编写的房产网页抓取Selenium爬虫在Windows10上运行正常，但在Ubuntu24.02服务器上启动时直接挂起。两台设备均使用Python3.12.3版本，处于同一子网，Ubuntu可ping通目标网站、curl能获取内容，UFW已关闭，排除网络问题。

按下Ctrl+C终止进程后，栈信息显示卡在driver = webdriver.Chrome(service=service, options=chrome_options)的HTTP请求阶段：

bloom@bloom:~/centris_scrap/webScrap_Selenium$ python3 main.py -tp 1
^CTraceback (most recent call last):
  File "/home/bloom/centris_scrap/webScrap_Selenium/main.py", line 210, in <module>
    driver = webdriver.Chrome(service=service, options=chrome_options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/selenium/webdriver/chrome/webdriver.py", line 45, in __init__
    super().__init__(
  File "/usr/lib/python3/dist-packages/selenium/webdriver/chromium/webdriver.py", line 61, in __init__
    super().__init__(command_executor=executor, options=options)
  File "/usr/lib/python3/dist-packages/selenium/webdriver/remote/webdriver.py", line 208, in __init__
    self.start_session(capabilities)
  File "/usr/lib/python3/dist-packages/selenium/webdriver/remote/webdriver.py", line 292, in start_session
    response = self.execute(Command.NEW_SESSION, caps)["value"]
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/selenium/webdriver/remote/webdriver.py", line 345, in execute
    response = self.command_executor.execute(driver_command, params)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/selenium/webdriver/remote/remote_connection.py", line 302, in execute
    return self._request(command_info[0], url, body=data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/selenium/webdriver/remote/remote_connection.py", line 322, in _request
    response = self._conn.request(method, url, body=body, headers=headers)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/urllib3/_request_methods.py", line 118, in request
    return self.request_encode_body(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/urllib3/_request_methods.py", line 217, in request_encode_body
    return self.urlopen(method, url, **extra_kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/urllib3/poolmanager.py", line 443, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 791, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 537, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 461, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse
    response.begin()
  File "/usr/lib/python3.12/http/client.py", line 331, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/http/client.py", line 292, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/socket.py", line 707, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt

补充信息 #

编写简化版脚本，去掉所有抓取逻辑后，问题依然存在，还是卡在浏览器实例化阶段。简化后的脚本如下：

import logging

from selenium import webdriver
from selenium.common import ElementClickInterceptedException, NoSuchElementException
import argparse
from selenium.webdriver.chrome.service import Service as ChromeService
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
import time
import pandas as pd
from datetime import datetime
import uuid
import glob
import os
import os.path
from jinja2 import Environment, FileSystemLoader


def scrap_pages(driver):
    sqft=0
    year=0
    parking=0

    listings = driver.find_elements(By.CLASS_NAME, 'description')

    if listings[-1].text.split('\n')[0] == '': del listings[-1]

    for listing in listings:
        price=12333
        mls = '12333'
        prop_type = 'test'
        addr = 'test'
        city = 'test'
        sector = 'test'
        bedrooms = 1
        bathrooms=1
        listing_item = {
                'mls': mls,
                'price': price,
                'address': addr,
                'property type': prop_type,
                'city': city,
                'bedrooms': bedrooms,
                'bathrooms': bathrooms,
                'sector': sector,
                'living sqft': sqft,
                'lot sqft': sqft,
                'year': year,
                'parking': parking
            }
        centris_list.append(listing_item)


if __name__ == '__main__':
    today=datetime.now()
    today=today.strftime("%Y%m%d")
    start_time = time.time()
    UUID = str(uuid.uuid4())[-4:]

    parser = argparse.ArgumentParser()
    parser.add_argument("-s", "--skip_scrape", type=bool, default=False, help='dont scrape the webpage')
    parser.add_argument("-tp","--total_pages", type=int, help='number of pages to scrape')
    args = parser.parse_args()

    filename = f"centris_{today}_{UUID}_app.log"

    logging.basicConfig(
        filename=filename,
        level=logging.INFO,
        datefmt="%Y-%m-%d %H:%M",
        force=True
    )

    logging.info(f"We are starting the app")
    logging.info(f"We are scraping : {args.total_pages}")

    if not args.skip_scrape:
        chrome_options = Options()
        chrome_options.add_experimental_option("detach", True)
        chrome_options.add_argument('--headless')
        user_agent_win = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.6943.53 Safari/537.36'
        user_agent_u24 = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.6943.53 Safari/537.36'

        driver_path_win = 'C:\\WebDriver\\bin\\chromedriver132\\chromedriver.exe'
        driver_path_u24 = r'/usr/lib/chromium-browser/chromedriver'

        user_agent = user_agent_win if os.path.exists(driver_path_win) else user_agent_u24
        chrome_options.add_argument(f'user-agent={user_agent}')

        service = ChromeService(executable_path=driver_path_win) if os.path.exists(driver_path_win) else ChromeService(executable_path=driver_path_u24)
        driver = webdriver.Chrome(service=service, options=chrome_options)

        centris_list = []
        url = 'https://www.centris.ca/en/properties~for-sale~brossard?view=Thumbnail'
        '''
        driver.get(url)
        time.sleep(5)
        driver.find_element(By.ID, 'didomi-notice-agree-button').click()
        total_pages = driver.find_element(By.CLASS_NAME, 'pager-current').text.split('/')[1].strip()
        total = args.total_pages if args.total_pages is not None else int(total_pages)
        for i in range(0, total):
            try:
                scrap_pages(driver)
                driver.find_element(By.CSS_SELECTOR, 'li.next> a').click()
                time.sleep(3)
            except ElementClickInterceptedException as initial_error:
                try:
                    if len(driver.find_elements(By.XPATH, ".//div[@class='DialogInsightLightBoxCloseButton']")) > 0:
                        driver.find_element(By.XPATH, ".//div[@class='DialogInsightLightBoxCloseButton']").click()
                        time.sleep(3)
                    print('pop-up closed')
                    scrap_pages(driver)
                    driver.find_element(By.CSS_SELECTOR, 'li.next> a').click()
                    time.sleep(3)
                except NoSuchElementException:
                    raise initial_error
        '''
        driver.close()

    end_time=time.time()
    elapsed_seconds =end_time-start_time
    elapsed_time=elapsed_seconds/60
    logging.info(f"excution time is {elapsed_time:.2f}")

解决方案 #

• 移除detach选项：detach=True会让浏览器在驱动退出后保持运行，无头服务器环境下容易引发进程阻塞，删除chrome_options.add_experimental_option("detach", True)。
• 完善无头模式参数：Ubuntu无GUI环境下，仅--headless不够，添加以下适配参数：

  chrome_options.add_argument('--headless=new')  # 新版无头模式，兼容性更强
  chrome_options.add_argument('--no-sandbox')    # 禁用沙箱，避免权限限制
  chrome_options.add_argument('--disable-dev-shm-usage')  # 解决/dev/shm空间不足问题
  chrome_options.add_argument('--disable-gpu')   # 无头模式无需GPU加速
  

• 校验ChromeDriver与浏览器版本匹配：执行chromium-browser --version和chromedriver --version确认版本完全一致，版本不匹配会导致启动失败或阻塞。
• 指定Chrome二进制路径：手动指定浏览器二进制文件位置，避免Selenium找不到执行文件：

  chrome_options.binary_location = '/usr/bin/chromium-browser'  # 根据实际路径调整
  

• 添加驱动启动超时：设置服务启动超时，防止无限挂起：

  service = ChromeService(executable_path=driver_path_u24)
  service.start_timeout = 10  # 设置10秒超时阈值
  

内容的提问来源于stack exchange，提问作者D.Zou