使用Scrapy无法抓取noon.com商品标题等数据的技术求助

问题原因 #

noon.com的商品列表是通过JavaScript动态加载的，Scrapy默认发送HTTP请求获取到的是未经过前端渲染的初始HTML源码，里面并不包含你在浏览器开发者工具中看到的.eyNLqb这类商品元素，所以无论用什么CSS/XPath选择器都无法匹配到内容。

解决方案 #

方案1：使用Playwright渲染动态页面 #

Playwright是Scrapy官方推荐的处理JavaScript渲染页面的工具，步骤如下：

1. 安装依赖

pip install scrapy-playwright

2. 修改项目settings.py，启用Playwright中间件

DOWNLOAD_HANDLERS = {
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

PLAYWRIGHT_LAUNCH_OPTIONS = {
    "headless": True,  # 设为False可查看浏览器渲染过程
    "timeout": 10000,
}

DOWNLOADER_MIDDLEWARES = {
    "scrapy_playwright.middleware.PlaywrightMiddleware": 543,
}

3. 修改爬虫代码，指定使用Playwright渲染

import scrapy
from ..items import NoonItem


class NoonspiderSpider(scrapy.Spider):
    name = 'noonspider'
    allowed_domains = ['noon.com']
    start_urls = ['https://www.noon.com/uae-en/search/?q=figurine']

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url,
                meta={"playwright": True, "playwright_include_page": True},
                callback=self.parse
            )

    async def parse(self, response):
        items = NoonItem()
        items['title'] = response.css('.eyNLqb span::text').extract()
        yield items

方案2：直接请求商品数据API（更高效） #

打开浏览器开发者工具「网络」面板，筛选XHR请求，找到加载商品数据的API接口（格式类似https://www.noon.com/_svc/search/api/v3/search），直接请求该API获取JSON格式数据，无需处理页面渲染：

示例代码（需根据实际API参数调整）：

import scrapy
from ..items import NoonItem
import json


class NoonspiderSpider(scrapy.Spider):
    name = 'noonspider'
    allowed_domains = ['noon.com']

    def start_requests(self):
        api_url = "https://www.noon.com/_svc/search/api/v3/search?q=figurine&country=ae&language=en"
        yield scrapy.Request(api_url, callback=self.parse_api)

    def parse_api(self, response):
        data = json.loads(response.text)
        items = NoonItem()
        # 根据API返回的JSON结构提取商品标题
        items['title'] = [product['name'] for product in data['products']]
        yield items

注意：API接口的参数、签名规则可能会变化，需自行在浏览器中确认最新的请求地址和参数。

内容的提问来源于stack exchange，提问作者xangetsue