Puppeteer复用页面跳转慢，新建页面CPU占用高，如何优化？

Hey there, let's break down what's happening with your Puppeteer workflow and find a middle ground that keeps navigation fast without maxing out your CPU.

Why Your Current Approaches Have Tradeoffs #

• Reusing the same tab (slow navigation): Every time you navigate to a new link in the same page, the browser doesn't fully reset the context. Leftover DOM elements, cached resources, JavaScript memory leaks, and stale CDP session state build up over 4k+ navigations. This bloat makes each subsequent goto slower and slower.
• Closing and creating new tabs (100% CPU): Spinning up a fresh browser page every time forces Chrome to initialize a new renderer process—this is inherently CPU-heavy. Doing this 4,000+ times means your CPU is constantly handling process creation and destruction, hence the maxed-out usage.

Optimized Fixes to Try #

1. Clean Up Page Context Between Navigations (Reuse Tabs Efficiently)

Instead of letting the page accumulate bloat, reset key parts of the context before each new navigation. This avoids the overhead of creating new pages while keeping performance snappy.

Adjust your code like this:

// Initialize the CDP session once for the reused page
const session1 = await currentPage.target().createCDPSession();
await session1.send('Page.enable');

// For each link in your list:
for (const link of links) {
  // Reset lifecycle state
  await session1.send('Page.setWebLifecycleState', { state: 'active' });
  
  // Clear cached data and storage
  await session1.send('Network.clearBrowserCache');
  await session1.send('Storage.clearDataForOrigin', {
    origin: '*',
    storageTypes: 'all'
  });
  
  // Wipe the DOM and reset the page to a blank state
  await currentPage.evaluate(() => {
    document.body.innerHTML = '';
    window.location.href = 'about:blank';
  });
  await currentPage.waitForNavigation({ waitUntil: 'load' });
  
  // Navigate to the new link and run your task
  await currentPage.goto(link, { timeout: 0 });
  // YOUR JOB CODE HERE
}

This approach gives you a fresh starting point for each navigation without closing the page, keeping CPU usage low while speeding up goto calls.

2. Use a Page Pool (Balance Speed & CPU Usage)

Instead of reusing one page or creating a new one every time, create a small pool of pages (5-10 works for most cases) and rotate them. This way, you avoid the overhead of 4k process creations, while each page gets reset often enough to prevent slowdown.

Example implementation:

// Initialize a pool of 5 pages
const pagePool = [];
const poolSize = 5;
for (let i = 0; i < poolSize; i++) {
  const page = await browser.newPage();
  const session = await page.target().createCDPSession();
  await session.send('Page.enable');
  pagePool.push({ page, session });
}

// Iterate over links, rotating through the pool
let currentIndex = 0;
for (const link of links) {
  const { page, session } = pagePool[currentIndex];
  
  // Reset page context (same cleanup as above)
  await session.send('Page.setWebLifecycleState', { state: 'active' });
  await session.send('Network.clearBrowserCache');
  await page.evaluate(() => {
    document.body.innerHTML = '';
    window.location.href = 'about:blank';
  });
  await page.waitForNavigation({ waitUntil: 'load' });
  
  // Run your task
  await page.goto(link, { timeout: 0 });
  // YOUR JOB CODE HERE
  
  // Rotate to the next page in the pool
  currentIndex = (currentIndex + 1) % poolSize;
}

// Clean up the pool when done
for (const { page } of pagePool) {
  await page.close();
}

Tweak the pool size based on your system: if CPU is still high, reduce to 3; if navigation is slow, bump it to 10.

3. Tweak Puppeteer Launch Options for Lower Overhead

Adjust your browser launch settings to cut down on unnecessary resource usage:

const browser = await puppeteer.launch({
  headless: 'new', // Newer, more efficient headless mode
  args: [
    '--disable-features=VizDisplayCompositor', // Reduce GPU overhead
    '--no-sandbox', // Skip sandbox (only safe in controlled environments)
    '--disable-setuid-sandbox',
    '--disable-extensions', // Disable unused extensions
    '--disable-dev-shm-usage' // Avoid shared memory limits
  ]
});

Final Tips #

• Monitor memory usage: If memory creeps up over time, occasionally close and recreate one page in the pool to free up resources.
• Test different cleanup steps: Depending on your use case, you might not need all the cleanup steps—experiment to find the minimal set that keeps navigation fast.

内容的提问来源于stack exchange，提问作者DuyTung