向PlaywrightCrawler传递userData失败：session.userData为空问题求助

我在extractFromSearchPage函数中向PlaywrightCrawler的请求传入了userData，但在requestHandler里访问session.userData始终为空，日志输出显示userData是{}，相关代码如下：

原始代码 #

extractFromSearchPage函数 #

async function extractFromSearchPage(url, log) {
    searchOptions = {'somevalue'}
    likelyPaginationButtonResponse = {'somevalue'}
    await searchPageCrawler.run([{
        url: url,
        userData: {
            searchOptions: searchOptions,
            paginationButton: likelyPaginationButtonResponse
        }}])

}

PlaywrightCrawler配置 #

// Function to interact with the page using Playwright
const searchPageCrawler = new PlaywrightCrawler({
    launchContext: {
        launchOptions: {
            launcher: 'firefox',
            headless: false, // Run in head-full mode to mimic a real user
            firefoxUserPrefs: {
                'general.useragent.override': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:90.0) Gecko/20100101 Firefox/90.0', // Example user agent
            },
            args: [
                '--disable-web-security', // Disables web security to prevent CORS issues
                '--disable-features=IsolateOrigins,site-per-process', // Disables site isolation
                '--blink-settings=imagesEnabled=true', // Ensures images are loaded
            ]
        },
        // stealth: true, // Use stealth plugin to prevent detection
    },
    async requestHandler({ page, request, session, log }) {
        log.info(`Starting the crawler.`);
        log.info(`Processing search page at URL: ${request.url}`);
        log.info(`Processing search page at URL: ${request.url} with details: ${JSON.stringify(session)}`);
        log.info(`Processing search page at URL: ${request.url} with details: ${JSON.stringify(session.userData)}`);
    
        await page.goto(request.url);
        await page.waitForSelector('body'); // Ensure the page has loaded
    
});

问题原因与解决方案 #

问题核心是访问路径错误：你传入的userData是请求级别的自定义数据，它存储在request对象中，而非session对象。session.userData用于存储会话级持久化数据（如登录态），和请求传入的userData并非同一概念。

修改步骤 #

1. 修正userData的访问方式：将session.userData替换为request.userData
2. 补全变量声明：给searchOptions和likelyPaginationButtonResponse添加const，避免全局变量污染

修改后的代码 #

修正后的requestHandler

async requestHandler({ page, request, session, log }) {
    log.info(`Starting the crawler.`);
    log.info(`Processing search page at URL: ${request.url}`);
    // 正确访问请求传入的userData
    log.info(`Received userData: ${JSON.stringify(request.userData)}`);
    // 解构取出需要的参数
    const { searchOptions, paginationButton } = request.userData;
    log.info(`Search options: ${JSON.stringify(searchOptions)}`);
    
    await page.goto(request.url);
    await page.waitForSelector('body');
}

修正后的extractFromSearchPage

async function extractFromSearchPage(url, log) {
    const searchOptions = {'somevalue'};
    const likelyPaginationButtonResponse = {'somevalue'};
    await searchPageCrawler.run([{
        url: url,
        userData: {
            searchOptions: searchOptions,
            paginationButton: likelyPaginationButtonResponse
        }}])
}

内容的提问来源于stack exchange，提问作者Cauder