Pyppeteer代码偶现Target closed异常，调用顺序影响问题求助

问题描述 #

我用Pyppeteer开发了爬取Wildberries和Ozon商品价格的脚本，运行时大概每4次启动就会出现一次异常，错误信息如下：

Error: (Future exception was never retrieved
future:<Future finished exception=NetworkError('Protocol error Target.sendMessageToTarget: Target closed.')>
pyppeteer.errors.NetworkError: Protocol error Target.sendMessageToTarget: Target closed)

但如果把main函数里的print(await get_ozon_price(main_browser))改成再次调用get_wildberries_price，程序就完全正常。搞不懂调用顺序为啥会影响异常，求技术支持。完整代码如下：

# https://bot.sannysoft.com/

from pyppeteer import *
from pyppeteer_stealth import stealth
import asyncio

# --------------------------------------------------------------

def handle_price(some_price):
    for element in some_price:
      try:
          int(element)
      except:
          some_price = some_price.replace(element, '')
    return some_price

#-----

async def get_wildberries_price(browser):
  page = await browser.newPage()
  await page.goto('https://www.wildberries.ru/catalog/98649739/detail.aspx')
  await page.waitFor('.price-block__final-price')
  raw_price = await page.evaluate('''
  () => {
    return document.querySelector('.price-block__final-price').textContent
  }
  ''')
  final_price = handle_price(raw_price)
  await page.screenshot({'path': 'wildberries.png'})
  return final_price

#-----

async def get_ozon_price(browser):
  page = await browser.newPage()
  await stealth(page)
  await page.goto('https://www.ozon.ru/product/setka-sadovaya-plastikovaya-0-61-sm-opornaya-dlya-rasteniy-rulon-5-m-yacheyka-7-10-mm-zelenaya-746399567/?advert=LMWD3jhEugrU0NaiTNSLwpAIIbCdFMNn7pzKseZ54oiYJzffMkWL5dFUOvP2QxFxTe6r-62ds5iYdNtmm39GeVmWNx7Drk2RfA-NFruujb9wW_wbefQotjVI3Nw6nmw-7bz9eBuQf1jcWO9oKG03ATRZlVPYDWDUpt-Vr6TP01QKzNpkDGeyKNFDYr0mQ6zHtTc2xumWB3-dv2Gz-N1MubGGYv1HNBGFFtjS2sjma9sLgVekajUUaWojBH262l4QrAW1M9rCl-81e2UuCuA6e9e30jQi6V5VQGajDtoh7hlSqVBeuWTgQwBh93dIZprdLVVLO8wjhyroY4zH5kCXsPosmppupEfHuVI3WyxszZDRKZaftEvaKMKR1BDr1rtJDPW_pNvM1ZhWuCyJp9hsDQsED8fQ3RFRWR8LNZ5WYr8h_sjfYw1xlrl5-rjcS2MaBuBeYL3GQ4pXFwrkRKOEFuLsDScNPdIL2rDN9aeFT_lW5QnmTG_L177BU5QyZM2DLu7pN8H3K0To7QDSzvLpfVqCHWaudAT54N6mTU8GAtwBLp6sWpQNKDwnqqFLxmdC1s-3-U2t3h6jwkvEH5nue2hZdmrdu4Z-eD2qbnblV340tMRQZ3V8vcw-QO0BiDqNQupojM-i61GULUdv8a8qa12Uir-HZ1OOaKekgidc-IKoCUbTCa8YfpV5zXASOBlxvqs84FoG8vEg2yNkN4fRSOOkfcSp29rssttskpEkp14dZ3o_XnJ93SI1SR13ffRmFMGVZiwDjOyHjovEELGadA&avtc=1&avte=2&avts=1674570447&keywords=%D0%BF%D0%BB%D0%B0%D1%81%D1%82%D0%B8%D0%BA%D0%BE%D0%B2%D0%B0%D1%8F+%D1%81%D0%B5%D1%82%D0%BA%D0%B0+%D1%81%D0%BB%D0%B0%D0%B2&sh=6RhLQaLOhA')
  await page.waitFor('[slot="content"] span')
  raw_price = await page.evaluate('''
  () => {
    return document.querySelector('[slot="content"] span').textContent
  }
  ''')
  final_price = handle_price(raw_price)
  await page.screenshot({'path': 'ozon.png'})
  return final_price

# --------------------------------------------------------------

async def main():
  main_browser = await launch({
    # 'headless': False,
  })
  
  print(await get_wildberries_price(main_browser))
  print(await get_ozon_price(main_browser))

  await main_browser.close()

main_loop = asyncio.new_event_loop()
main_loop.run_until_complete(main())

问题根源 #

这个异常的核心原因是Ozon页面使用了pyppeteer_stealth插件，插件内部的异步操作和页面生命周期没有同步处理，加上Ozon的反爬机制可能会更快终止页面连接，当在Wildberries之后调用Ozon时，浏览器上下文的资源释放或消息传递出现了竞态条件，导致Target（页面）被提前关闭，但插件还在尝试向其发送消息。

而两次调用Wildberries时，没有使用stealth插件，不存在额外的异步消息操作，所以不会触发这个竞态问题。

修复方案 #

1. 确保页面操作完成后正确关闭页面 #

在每个爬取函数末尾添加await page.close()，避免浏览器残留无效的页面上下文。

2. 给stealth和页面跳转添加等待/超时控制 #

stealth的某些操作是异步的，需要确保其完成后再进行页面跳转；同时给page.goto增加超时参数，避免页面加载超时导致的连接异常。

3. 优化浏览器启动参数 #

增加稳定性相关参数，比如禁用沙箱（Linux环境下常见问题）、禁用共享内存限制等。

修改后的完整代码 #

from pyppeteer import *
from pyppeteer_stealth import stealth
import asyncio

def handle_price(some_price):
    for element in some_price:
        try:
            int(element)
        except:
            some_price = some_price.replace(element, '')
    return some_price

async def get_wildberries_price(browser):
    page = await browser.newPage()
    try:
        await page.goto('https://www.wildberries.ru/catalog/98649739/detail.aspx', timeout=60000)
        await page.waitFor('.price-block__final-price', timeout=30000)
        raw_price = await page.evaluate('''
        () => {
            return document.querySelector('.price-block__final-price').textContent
        }
        ''')
        final_price = handle_price(raw_price)
        await page.screenshot({'path': 'wildberries.png'})
        return final_price
    finally:
        await page.close()  # 确保页面关闭

async def get_ozon_price(browser):
    page = await browser.newPage()
    try:
        await stealth(page)
        # 等待stealth操作完全完成
        await asyncio.sleep(1)
        await page.goto('https://www.ozon.ru/product/setka-sadovaya-plastikovaya-0-61-sm-opornaya-dlya-rasteniy-rulon-5-m-yacheyka-7-10-mm-zelenaya-746399567/?advert=LMWD3jhEugrU0NaiTNSLwpAIIbCdFMNn7pzKseZ54oiYJzffMkWL5dFUOvP2QxFxTe6r-62ds5iYdNtmm39GeVmWNx7Drk2RfA-NFruujb9wW_wbefQotjVI3Nw6nmw-7bz9eBuQf1jcWO9oKG03ATRZlVPYDWDUpt-Vr6TP01QKzNpkDGeyKNFDYr0mQ6zHtTc2xumWB3-dv2Gz-N1MubGGYv1HNBGFFtjS2sjma9sLgVekajUUaWojBH262l4QrAW1M9rCl-81e2UuCuA6e9e30jQi6V5VQGajDtoh7hlSqVBeuWTgQwBh93dIZprdLVVLO8wjhyroY4zH5kCXsPosmppupEfHuVI3WyxszZDRKZaftEvaKMKR1BDr1rtJDPW_pNvM1ZhWuCyJp9hsDQsED8fQ3RFRWR8LNZ5WYr8h_sjfYw1xlrl5-rjcS2MaBuBeYL3GQ4pXFwrkRKOEFuLsDScNPdIL2rDN9aeFT_lW5QnmTG_L177BU5QyZM2DLu7pN8H3K0To7QDSzvLpfVqCHWaudAT54N6mTU8GAtwBLp6sWpQNKDwnqqFLxmdC1s-3-U2t3h6jwkvEH5nue2hZdmrdu4Z-eD2qbnblV340tMRQZ3V8vcw-QO0BiDqNQupojM-i61GULUdv8a8qa12Uir-HZ1OOaKekgidc-IKoCUbTCa8YfpV5zXASOBlxvqs84FoG8vEg2yNkN4fRSOOkfcSp29rssttskpEkp14dZ3o_XnJ93SI1SR13ffRmFMGVZiwDjOyHjovEELGadA&avtc=1&avte=2&avts=1674570447&keywords=%D0%BF%D0%BB%D0%B0%D1%81%D1%82%D0%B8%D0%BA%D0%BE%D0%B2%D0%B0%D1%8F+%D1%81%D0%B5%D1%82%D0%BA%D0%B0+%D1%81%D0%BB%D0%B0%D0%B2&sh=6RhLQaLOhA', timeout=60000)
        await page.waitFor('[slot="content"] span', timeout=30000)
        raw_price = await page.evaluate('''
        () => {
            return document.querySelector('[slot="content"] span').textContent
        }
        ''')
        final_price = handle_price(raw_price)
        await page.screenshot({'path': 'ozon.png'})
        return final_price
    finally:
        await page.close()  # 确保页面关闭

async def main():
    main_browser = await launch({
        # 'headless': False,
        'args': [
            '--no-sandbox',
            '--disable-setuid-sandbox',
            '--disable-dev-shm-usage',
            '--disable-web-security'
        ]
    })
    
    try:
        print(await get_wildberries_price(main_browser))
        print(await get_ozon_price(main_browser))
    finally:
        await main_browser.close()

main_loop = asyncio.new_event_loop()
main_loop.run_until_complete(main())

额外说明 #

• 使用try/finally确保无论是否发生异常，页面都会被关闭，避免浏览器资源泄漏。
• 添加timeout参数防止页面加载卡住，同时给stealth操作后加短暂等待，确保插件完成所有注入操作。
• 浏览器启动参数中的--no-sandbox等参数是为了提升稳定性，尤其是在Linux服务器环境下。

内容的提问来源于stack exchange，提问作者samiro