使用Puppeteer/Node.js爬取时如何等待JS函数执行完成？

通过Chrome DevTools Recorder录制并适配的Puppeteer脚本，在操作网站生成报告并下载的流程中，前半段正常，但生成报告时后台有JavaScript函数在执行，脚本提前尝试点击未就绪的元素导致报错。具体表现为等待选择器li.ui-state-hover超时，即使调整超时时间也无法解决。

报错前的关键代码如下：

console.log('Navigating to the target page...');
await page.goto('https://website.url', { waitUntil: 'networkidle2' });

console.log('Clicking language selector...');
await page.waitForSelector('div.active-lang', { visible: true });
await page.click('div.active-lang');

console.log('Switching to English...');
await Promise.all([
    page.waitForNavigation({ waitUntil: 'networkidle2' }),
    page.click('li.en-gb > a')
]);

console.log('Opening report...');
await Promise.all([
    page.waitForNavigation({ waitUntil: 'networkidle2' }),
    page.click('#treeMenu\:0_4_1_3 a')
]);

console.log('Selecting year...');
await page.waitForSelector('#input > table:nth-of-type(1) span', { visible: true });
await page.click('#input > table:nth-of-type(1) span');
await page.waitForSelector('#yearBeg_0', { visible: true });
await page.click('#yearBeg_0');

console.log('Generating report...');
await page.waitForSelector('td.action_c2 span', { visible: true });
await page.click('td.action_c2 span', { timeout: 30000, waitUntil: 'networkidle2' });

console.log('Exporting file...');
await page.waitForSelector('td.action_c1 span.ui-button-text', { visible: true });
await page.click('td.action_c1 span.ui-button-text', { timeout: 60000, waitUntil: 'networkidle2' });
await page.screenshot({ path: 'debug.png_exporting' });

console.log('Hovering menu...');
// Hover over the parent menu to reveal XLS option
await page.waitForSelector('li.ui-state-hover', { visible: true, timeout: 30000, waitUntil: 'networkidle2' });
await page.hover('li.ui-state-hover > a', { timeout: 30000, waitUntil: 'networkidle2' });  // Hover to reveal the submenu
await page.screenshot({ path: 'debug.png_hovering' });


console.log('Selecting XLS format...');
try {
    // Wait for the XLS option and click it by text content
    const xlsMenuItem = await page.waitForFunction(() => {
        const menuItems = Array.from(document.querySelectorAll('li.ui-state-hover span.ui-menuitem-text'));
        return menuItems.find(item => item.textContent.trim() === 'XLS');
    }, { timeout: 30000 });

    if (xlsMenuItem) {
        console.log('Clicking on XLS menu item...');
        await xlsMenuItem.click();
    } else {
        throw new Error('XLS menu item not found.');
    }
    console.log('Successfully clicked on XLS format.');
} catch (error) {
    console.error('Failed to select XLS format:', error);
    throw error;
}

1. 等待页面完全空闲 #

后台JS执行时，页面处于忙碌状态，可通过waitForFunction等待浏览器进入空闲状态：

// 在点击"生成报告"按钮后添加，等待后台JS执行完毕
console.log('Waiting for background JS to finish...');
await page.waitForFunction(() => {
    return new Promise(resolve => window.requestIdleCallback(() => resolve(true)));
}, { timeout: 60000 });

2. 等待加载指示器消失 #

如果生成报告时页面有加载遮罩或动画，优先等待这些元素隐藏：

// 替换为页面实际的加载遮罩选择器
await page.waitForSelector('.loading-mask', { hidden: true, timeout: 60000 });

3. 修正hover逻辑：先等父菜单可见再触发hover #

ui-state-hover是鼠标hover后的动态类，脚本提前等待会超时。正确逻辑是先找到父菜单元素，再执行hover：

console.log('Hovering menu...');
// 替换为导出按钮所在的父菜单选择器
await page.waitForSelector('li.ui-menuitem[data-menu="export"]', { visible: true, timeout: 60000 });
const exportParentMenu = await page.$('li.ui-menuitem[data-menu="export"]');
// 触发hover显示子菜单
await exportParentMenu.hover();

4. 直接等待XLS选项可见 #

绕过hover状态判断，直接等待XLS子菜单元素出现后点击：

console.log('Selecting XLS format...');
try {
    // 先触发父菜单hover
    const exportButton = await page.$('td.action_c1 span.ui-button-text');
    await exportButton.hover();
    // 等待XLS选项并点击
    await page.waitForSelector('span.ui-menuitem-text:has-text("XLS")', { visible: true, timeout: 60000 });
    await page.click('span.ui-menuitem-text:has-text("XLS")');
    console.log('Successfully clicked on XLS format.');
} catch (error) {
    console.error('Failed to select XLS format:', error);
    throw error;
}

5. 等待后台请求完成 #

如果生成报告有对应的API请求，用waitForResponse等待请求结束：

// 点击生成报告按钮时，同步等待对应API请求完成
console.log('Generating report...');
await Promise.all([
    page.waitForResponse(res => res.url().includes('report/generate') && res.ok(), { timeout: 60000 }),
    page.click('td.action_c2 span', { timeout: 30000 })
]);

内容的提问来源于stack exchange，提问作者David