Selenium报‘Timed out receiving message’错误原因排查求助

问题 #

使用Selenium时遇到Timed out receiving message错误，想咨询该错误是否由页面超时导致？是否与implicitly_wait有关？

错误日志 #

Jul 11 09:34:35 PM  2023-07-12 01:34:35,727 loglevel=ERROR  logger=services.extract_link get_link_soup() L158  Failed to save screenshot (png_full) from URL
Jul 11 09:34:35 PM  Error message: Message: timeout: Timed out receiving message from renderer: 10.000
Jul 11 09:34:35 PM    (Session info: headless chrome=114.0.5735.198)
Jul 11 09:34:35 PM  Stacktrace:
Jul 11 09:34:35 PM  #0 0x561cabb024e3 <unknown>
Jul 11 09:34:35 PM  #1 0x561cab831c76 <unknown>
Jul 11 09:34:35 PM  #2 0x561cab81b284 <unknown>
Jul 11 09:34:35 PM  #3 0x561cab81afa0 <unknown>
Jul 11 09:34:35 PM  #4 0x561cab8199bf <unknown>
Jul 11 09:34:35 PM  #5 0x561cab81a162 <unknown>
Jul 11 09:34:35 PM  #6 0x561cab83bf9c <unknown>
Jul 11 09:34:35 PM  #7 0x561cab8b1bc2 <unknown>
Jul 11 09:34:35 PM  #8 0x561cab88d012 <unknown>
Jul 11 09:34:35 PM  #9 0x561cab8a530e <unknown>
Jul 11 09:34:35 PM  #10 0x561cab88cde3 <unknown>
Jul 11 09:34:35 PM  #11 0x561cab8622dd <unknown>
Jul 11 09:34:35 PM  #12 0x561cab86334e <unknown>
Jul 11 09:34:35 PM  #13 0x561cabac23e4 <unknown>
Jul 11 09:34:35 PM  #14 0x561cabac63d7 <unknown>
Jul 11 09:34:35 PM  #15 0x561cabad0b20 <unknown>
Jul 11 09:34:35 PM  #16 0x561cabac7023 <unknown>
Jul 11 09:34:35 PM  #17 0x561caba951aa <unknown>
Jul 11 09:34:35 PM  #18 0x561cabaeb6b8 <unknown>
Jul 11 09:34:35 PM  #19 0x561cabaeb847 <unknown>
Jul 11 09:34:35 PM  #20 0x561cabafb243 <unknown>
Jul 11 09:34:35 PM  #21 0x7fc4dd240fa3 start_thread

相关代码 #

def get_link_soup(self, url: str) -> Optional[BeautifulSoup]:
    """
    Download the content of a URL and return a BeautifulSoup object for parsing it.

    :param url: The URL to download the content of.
    :return: A BeautifulSoup object and a screenshot path if the download is successful and the status code is 200, None otherwise.
    :raises ValueError: If the URL is invalid (i.e. doesn't have a scheme or network location).
    """
    if not self.__is_valid_url(url):
        raise ValueError("Invalid URL: protocol must be http or https")

    driver = None
    soup = None
    try:
        # Set up the Selenium WebDriver
        options = webdriver.ChromeOptions()
        options.headless = True
        options.add_argument("--headless")
        options.add_argument('--no-sandbox')
        options.add_argument('--ignore-certificate-errors')
        options.add_argument('--disable-dev-shm-usage')
        options.add_argument('--disable-extensions')
        options.add_argument('--disable-infobars')
        options.add_argument('--user-agent={}'.format(helpers.get_user_agents()))

        driver = webdriver.Chrome(options=options)
        driver.set_page_load_timeout(90)

        # Load the URL and get the page source
        driver.implicitly_wait(10)
        driver.get(url)
        html = driver.page_source

        # Create a BeautifulSoup object from the HTML
        soup = BeautifulSoup(html, 'html.parser')

    except (selenium.common.exceptions.WebDriverException, Exception) as e:
        traceback.print_exc()
        logger.error(f"Error downloading content from URL: {url}\nError message: {str(e)}")
        return None

    # Set filename and created tmp dir if needed
    url_hash = UrlBackup.url_to_hash(url)
    if not os.path.exists("tmp"):
        os.mkdir("tmp")
    if not os.path.exists(f"tmp/{url_hash}"):
        os.mkdir(f"tmp/{url_hash}")
    png_full = f"tmp/{url_hash}/full.png"
    png_fold = f"tmp/{url_hash}/fold.png"

    # Take a screenshot of the `full` web page
    try:
        set_width = 1440
        set_height = driver.execute_script(
            "return Math.max(document.body.scrollHeight, document.documentElement.scrollHeight)"
        )
        driver.set_window_size(set_width, set_height)
        driver.save_screenshot(png_full)
    except Exception as e:
        logger.error(f"Failed to save screenshot (png_full) from URL: {url}\nError message: {str(e)}")

    # Take a screenshot above the `fold`
    try:
        new_width = 1440
        new_height = 900
        driver.set_window_size(new_width, new_height)
        driver.save_screenshot(png_fold)
    except Exception as e:
        logger.error(f"Failed to save screenshot (png_fold) from URL: {url}\nError message: {str(e)}")

    finally:
        if driver is not None:
            # Quit the driver to free up resources
            driver.quit()

    # Return the soup object and the screenshot path
    return soup

分析与解决 #

错误本质 #

这个报错不是页面加载超时（代码已经成功获取到page_source并生成BeautifulSoup，说明90秒的set_page_load_timeout已经完成），也和implicitly_wait无关——implicitly_wait仅作用于元素查找类操作，而报错发生在截图阶段。

错误核心是：Chrome渲染进程未在10秒内响应WebDriver的指令（执行JS获取高度、设置窗口大小或截图），导致通信超时。

解决方法 #

1. 延长渲染进程超时时间
   在ChromeOptions中添加参数，将渲染超时从默认10秒调至更长（比如30秒）：

   options.add_argument('--timeout=30000')
   options.add_argument('--renderer-timeout=30000')
   

2. 优化截图前的等待逻辑
   页面加载完成后可能仍有动态内容加载，直接截图易触发超时。可通过显式等待确保页面稳定：

   from selenium.webdriver.support.ui import WebDriverWait
   from selenium.webdriver.support import expected_conditions as EC
   
   # 等待页面完全加载完成
   WebDriverWait(driver, 10).until(
       lambda d: d.execute_script('return document.readyState') == 'complete'
   )
   

3. 调整窗口大小逻辑
   部分页面高度会在获取后继续变化，建议获取高度后加短暂等待再设置窗口：

   set_height = driver.execute_script("return Math.max(document.body.scrollHeight, document.documentElement.scrollHeight)")
   import time
   time.sleep(1)  # 等待1秒让页面稳定
   driver.set_window_size(set_width, set_height)
   

4. 避免极端窗口高度
   若页面超长，过大的窗口高度会导致渲染卡顿，可限制最大高度：

   max_height = 10000
   set_height = min(set_height, max_height)
   

内容的提问来源于stack exchange，提问作者Mark