FastAPI接口中CustomBrowser无头浏览器调用失败问题

问题场景 #

为FastAPI应用编写了CustomBrowser类作为无头浏览器工具，直接通过asyncio.run()调用时运行正常，但在FastAPI接口中调用时抛出错误：

RuntimeError: Browser not initialized. Call initialize_browser first.

原始代码 #

CustomBrowser类 #

from playwright.async_api import async_playwright
import logging


logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

class CustomBrowser:
    browser = None  # 类级属性存储Playwright浏览器实例
    playwright = None  # 类级属性存储Playwright进程
    @classmethod
    async def initialize_browser(cls):
        """
        异步初始化Playwright浏览器实例。

        示例:
            await CustomBrowser.initialize_browser()
        """
        try:
            if cls.browser is None:
                if cls.playwright is None:
                    cls.playwright = await async_playwright().__aenter__()  # 初始化Playwright进程
                cls.browser = await cls.playwright.chromium.launch(headless=True)
            logger.info('Browser successfully initialized.')
        except Exception as e:
            logger.error(f'Failed to initialize browser: {e}')

    async def get_page_content(self, url):
        """
        访问指定URL并返回页面内容。

        参数:
            url (str): 要访问的URL。

        返回:
            str: 页面HTML内容。

        示例:
            content = await instance.get_page_content("https://example.com")
        """
        if self.browser is None:
            raise RuntimeError("Browser not initialized. Call initialize_browser first.")

        context = await self.browser.new_context()
        page = await context.new_page()
        await page.goto(url)
        content = await page.content()
        await context.close()
        return content

    @classmethod
    async def setup_and_get_content(cls, url):
        """
        初始化浏览器（如果未初始化）并获取指定URL的内容。

        参数:
            url (str): 要访问的URL。

        返回:
            str: 页面HTML内容。

        示例:
            content = await CustomBrowser.setup_and_get_content("https://example.com")
        """
        await cls.initialize_browser()  # 初始化浏览器
        instance = cls()  # 创建实例
        return await instance.get_page_content(url)  # 获取并返回内容

正常运行的调用代码 #

import asyncio

async def main():
    content = await CustomBrowser.setup_and_get_content("https://example.com")
    print(content)

asyncio.run(main())

FastAPI接口代码 #

from fastapi import FastAPI

app = FastAPI()

@app.get("/parse/")
async def read_item():
    content = await CustomBrowser.setup_and_get_content('https://example.com')
    return content

报错信息 #

raise RuntimeError("Browser not initialized. Call initialize_browser first.")
RuntimeError: Browser not initialized. Call initialize_browser first.

问题原因 #

1. ASGI并发与类级属性竞态：FastAPI基于ASGI协议处理请求，多请求并发时，类级别的browser和playwright属性可能出现一个请求触发初始化但尚未完成，另一个请求就检查到browser为None，导致误判未初始化。
2. Playwright上下文管理不当：直接使用async_playwright().__aenter__()却未正确管理上下文生命周期，在ASGI环境中可能导致Playwright实例未完成初始化就被访问。
3. 多Worker进程隔离：如果FastAPI使用多Worker部署（如uvicorn的--workers参数），每个Worker是独立进程，类级属性无法跨进程共享，原代码的初始化逻辑无法保证每个Worker都完成浏览器初始化。

解决方案 #

修改后的CustomBrowser类 #

将类级属性改为实例属性，并完善资源生命周期管理：

from playwright.async_api import async_playwright
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

class CustomBrowser:
    def __init__(self):
        self.browser = None
        self.playwright = None

    async def initialize(self):
        """异步初始化浏览器实例"""
        try:
            if self.playwright is None:
                self.playwright = await async_playwright().__aenter__()
            self.browser = await self.playwright.chromium.launch(headless=True)
            logger.info('Browser successfully initialized.')
        except Exception as e:
            logger.error(f'Failed to initialize browser: {e}')
            raise

    async def get_page_content(self, url):
        """访问URL并返回页面内容"""
        if self.browser is None:
            raise RuntimeError("Browser not initialized. Call initialize first.")

        context = await self.browser.new_context()
        page = await context.new_page()
        await page.goto(url)
        content = await page.content()
        await context.close()
        return content

    async def shutdown(self):
        """清理浏览器和Playwright资源"""
        if self.browser:
            await self.browser.close()
        if self.playwright:
            await self.playwright.__aexit__(None, None, None)
        logger.info('Browser successfully shut down.')

修改后的FastAPI应用 #

使用FastAPI的生命周期事件在应用启动时初始化浏览器，关闭时清理资源：

from fastapi import FastAPI

app = FastAPI()
# 创建全局浏览器实例
browser = CustomBrowser()

@app.on_event("startup")
async def startup_event():
    """应用启动时初始化浏览器"""
    await browser.initialize()

@app.on_event("shutdown")
async def shutdown_event():
    """应用关闭时清理浏览器资源"""
    await browser.shutdown()

@app.get("/parse/")
async def read_item():
    """处理请求，返回页面内容"""
    content = await browser.get_page_content('https://example.com')
    return {"content": content}

修改说明 #

1. 实例属性替代类级属性：避免多并发请求或多Worker进程下的状态冲突，每个应用实例（或Worker进程）拥有独立的浏览器实例。
2. 生命周期事件管理：利用FastAPI的startup和shutdown事件，确保浏览器在应用启动时完成初始化，所有请求到来时浏览器已准备就绪；应用关闭时自动清理资源，避免内存泄漏。
3. 完善错误处理：初始化失败时主动抛出异常，确保应用启动时就能发现问题，而非在请求时才报错。

内容的提问来源于stack exchange，提问作者Mehdi Zare