Playwright异步Python报错：target page context or browser has been closed 求助

在异步Python环境中使用Playwright抓取网站动态内容时，反复遇到错误：target page context or browser has been closed。相关代码如下：

爬虫类代码 #

from playwright.async_api import async_playwright
from chatbot import init_logger
import datetime

logger = init_logger("DynamicCrawling")

class DynamicCrawler:
    def __init__(self):
        self.browser = None
        self.context = None

    async def get_browser(self):
        if not self.browser:
            async with async_playwright() as p:
                self.browser = await p.chromium.launch()
                self.context = await self.browser.new_context()
        return self.browser, self.context

    async def get_page(self, url):
        try:
            browser, context = await self.get_browser()
            page = await context.new_page()
            await page.goto(url)
            await page.wait_for_load_state('networkidle')
            dynamic_content = await page.content()
            return {
                'data': dynamic_content,
                'success': True
            }
        except Exception as ex:
            logger.error('Dynamic crawler exception msg ' + str(ex))
            return {
                'data': None,
                'success': False
            }

    async def close_browser(self):
        if self.browser:
            await self.browser.close()

    async def get_title(self, url):
        async with await self.get_page(url) as page:
            title = await page.title()
        return title

接口代码 #

@app.get('/chatbot/dynamic-crawl')
async def chatbot_test(request: Request, url: str = Query(alias='url')):
    return await crawler.get_page(url)

错误原因分析 #

1. Playwright实例生命周期不匹配：get_browser方法中用async with async_playwright() as p创建的Playwright实例，代码块结束后会自动销毁，导致已创建的browser和context失去底层支撑，后续操作触发"已关闭"错误。
2. get_title方法逻辑错误：get_page返回的是包含内容的字典，而非page对象，用async with包裹会直接抛出异常，且未正确处理页面资源。
3. 页面资源未释放：每次调用get_page创建新页面后未关闭，长期运行会导致上下文资源耗尽，间接引发错误。

修复方案及代码 #

关键修改点 #

• 将Playwright实例保存为类属性，与爬虫类生命周期绑定，避免提前销毁。
• 调整浏览器初始化逻辑，确保Playwright实例持续有效。
• 每次使用页面后关闭页面，释放资源。
• 修复get_title方法的逻辑，正确获取页面并处理资源。

修改后的完整代码 #

from playwright.async_api import async_playwright
from chatbot import init_logger
import datetime

logger = init_logger("DynamicCrawling")

class DynamicCrawler:
    def __init__(self):
        self.browser = None
        self.context = None
        self.playwright = None  # 保存Playwright实例，避免提前销毁

    async def init_browser(self):
        # 单独初始化Playwright和浏览器，替代原get_browser方法
        if not self.playwright:
            self.playwright = await async_playwright().start()
            self.browser = await self.playwright.chromium.launch()
            self.context = await self.browser.new_context()

    async def get_page(self, url):
        try:
            await self.init_browser()  # 确保浏览器已初始化
            page = await self.context.new_page()
            await page.goto(url)
            await page.wait_for_load_state('networkidle')
            dynamic_content = await page.content()
            await page.close()  # 使用后关闭页面，释放资源
            return {
                'data': dynamic_content,
                'success': True
            }
        except Exception as ex:
            logger.error('Dynamic crawler exception msg ' + str(ex))
            # 异常时如果页面已创建，也要关闭
            if 'page' in locals():
                await page.close()
            return {
                'data': None,
                'success': False
            }

    async def close_all(self):
        # 顺序关闭资源：上下文 -> 浏览器 -> Playwright实例
        if self.context:
            await self.context.close()
        if self.browser:
            await self.browser.close()
        if self.playwright:
            await self.playwright.stop()

    async def get_title(self, url):
        try:
            await self.init_browser()
            page = await self.context.new_page()
            await page.goto(url)
            await page.wait_for_load_state('networkidle')
            title = await page.title()
            await page.close()
            return title
        except Exception as ex:
            logger.error('Get title exception msg ' + str(ex))
            if 'page' in locals():
                await page.close()
            return None

接口代码调整（补充资源清理） #

如果使用FastAPI等框架，建议在应用关闭时触发爬虫的资源清理：

from fastapi import FastAPI, Request, Query

app = FastAPI()
crawler = DynamicCrawler()

@app.get('/chatbot/dynamic-crawl')
async def chatbot_test(request: Request, url: str = Query(alias='url')):
    return await crawler.get_page(url)

@app.on_event("shutdown")
async def shutdown_event():
    await crawler.close_all()

额外注意事项 #

• 可考虑复用页面而非每次创建新页面，提升性能，但需注意清理页面状态（如page.goto前执行page.clear_cookies()）。
• 若需多并发请求，可创建多个上下文（browser.new_context()），每个上下文对应独立会话环境。

内容的提问来源于stack exchange，提问作者James K J