如何使用Puppeteer获取<div>容器内的所有<p>标签

问题场景 #

需要从指定的.c-product-review-card容器内提取所有<p>标签的文本内容，目标HTML结构如下：

<div class="c-product-review-card">
    <div class="c-product-review-card__container">
        <div class="c-product-review-card__left-column">
            <div class="c-product-review-user-info c-product-review-card__user-info">
                <h5 class="c-product-review-user-info__username u-spacer--1pt5">USERNAME</h5>
                <div class="c-product-review-user-info__details-container">
                    <div class="c-product-review-user-info__item">
                        <p class="o-text--caption c-product-review-user-info__details"><span
                                class="u-text--gray">Size:</span> ONE SIZE</p>
                        <p class="o-text--caption c-product-review-user-info__details"><span
                                class="u-text--gray">Color:</span> black mult...</p>
                        <p class="o-text--caption c-product-review-user-info__details"><span
                                class="u-text--gray">Height:</span> 5'3"</p>
                        <p class="o-text--caption c-product-review-user-info__details"><span
                                class="u-text--gray">Weight:</span> 135 lbs.</p>
                    </div>
                    <div class="c-product-review-user-info__item">
                        <p class="o-text--caption c-product-review-user-info__details"><span class="u-text--gray">Body
                                Type:</span> Pear</p>
                        <p class="o-text--caption c-product-review-user-info__details"><span class="u-text--gray">Bra
                                Size:</span> 34B</p>
                        <p class="o-text--caption c-product-review-user-info__details"><span
                                class="u-text--gray">Age:</span> 29</p>
                    </div>
                </div>
            </div>
        </div>
        <div class="c-product-review-card__details c-product-review-card__details--list"><!---->
            <div class="c-product-review-card__review-body-container">
                <div class="c-product-review-card__review-body">
                    <h4 class="u-spacer--1 c-product-review-card__review-title">Cute and breezy</h4>
                    <p class="o-text--caption">Packed this on a trip to Peru. It came in handy on those cool spring
                        nights there, perfect for strolling in Lima. It’s not too light and not too heavy. Worked well
                        with a simple outfit underneath </p>
                </div>
                <div class="c-product-review-card__review-picture-container"><!----></div>
            </div>
        </div>
    </div><!---->
</div>

原尝试代码无法正常工作：

const reviews = (await page.$$(cssSelectors.REVIEW_CARD_CONTAINER)).splice(3);

let reviewsRes = {"reviews": []};
for(const review of reviews.splice(0, 1)){
    const userName = await page.evaluate(el => el.querySelector('div > h5').textContent, review);
    console.log(userName);
    const pTags = await page.evaluate(`div > p`, (paragraphs) => {
        return paragraphs.map((p) => p.textContent);
    }, review);
    console.log(pTags);
}

错误原因 #

1. page.evaluate参数使用错误：第一个参数必须是可在浏览器上下文执行的函数，不能直接传字符串选择器。
2. 选择器范围局限：div > p只能匹配容器直接子级的<p>，无法获取嵌套层级的<p>标签。
3. 参数传递逻辑混乱：原代码的选择器与后续处理函数不匹配，导致无法正确定位目标元素。

修正后的代码 #

const reviews = (await page.$$(cssSelectors.REVIEW_CARD_CONTAINER)).slice(3); // 用slice替代splice，避免修改原数组

let reviewsRes = {"reviews": []};
for(const review of reviews.slice(0, 1)){
    // 一次evaluate完成所有提取，减少上下文切换开销
    const reviewData = await page.evaluate(el => {
        // 获取用户名
        const userName = el.querySelector('.c-product-review-user-info__username').textContent.trim();
        // 获取容器内所有层级的<p>标签文本
        const pTexts = Array.from(el.querySelectorAll('p')).map(p => p.textContent.trim());
        return { userName, pTexts };
    }, review);
    
    console.log('用户名:', reviewData.userName);
    console.log('所有p标签内容:', reviewData.pTexts);
    reviewsRes.reviews.push(reviewData);
}

关键优化点 #

• 使用el.querySelectorAll('p')获取容器内所有层级的<p>标签，通过Array.from转为数组后提取文本。
• 合并多次page.evaluate调用为一次，减少Puppeteer与浏览器的上下文切换，提升性能。
• 用slice替代splice操作数组，避免修改原数组导致后续数据丢失。
• 增加trim()去除文本前后多余空格，让结果更整洁。

内容的提问来源于stack exchange，提问作者ChaoticQubit