如何用Scrapy调用回调函数抓取多级分类并生成指定JSON？

问题需求 #

实现多级分类递归抓取逻辑：从主分类进入子分类，若子分类下仍有下级分类则持续解析并添加链接，直至遇到包含产品的最终分类时停止；最终输出如下结构的JSON数据：

{
  "url": "Category URL",
  "category_name": "Category name",
  "subcategories": [
    {
      "url": "Subcategory URL",
      "category_name": "Subcategory name",
      "subcategories": [
        
      ]
    },
    ...
  ]
}

现有Scrapy代码无法正确生成嵌套结构，不清楚如何通过回调函数实现子分类的层级解析，当前代码如下：

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
"""
            name_category = item.css('a span::text').get()
            url_category = item.css('a::attr(href)').get()
"""

class CategoryLinkSpider(scrapy.Spider):
    name = "category_link"
    allowed_domains = ["illyushatoys.com.ua"]
    start_urls = ["https://illyushatoys.com.ua/?categoryID=184864"]

    def start_requests(self):
        category_item = {}
        yield scrapy.Request(self.start_urls[0], self.parse, meta={'category_item': category_item})


    def parse(self, response):
        category_item = response.meta.get('category_item')
        links = LinkExtractor(allow=r'.*category/\d+/$', restrict_css='div.inmenu').extract_links(response)
        if not links:
            print('emty_links')
        else:
            for link in links:
                category_item['url'] = link.url
                category_item['title_category'] = link.text
                category_item['subcategories'] = []
                yield response.follow(link, callback=self.parse_subcategory, meta={'category_item': category_item})

    def parse_subcategory(self, response):
        category_item = response.meta['category_item']
        links = LinkExtractor(allow=r'.*category/\d+/$', restrict_css='div.inmenu').extract_links(response)
        if links:
            for link in links:
                category_item['subcategories'].append({
                    'url': link.url,
                    'title_category': link.text
                })
                yield response.follow(link, callback=self.parse_subcategory, meta={'category_item': category_item})
            yield category_item

问题分析 #

现有代码的核心问题：

• 复用同一个category_item字典，递归时会覆盖之前的层级数据，无法生成嵌套结构
• 未区分「当前分类」与「子分类」的层级关系，回调传递的元数据逻辑错误
• 未处理「最终分类（含产品）」的终止条件

修正代码 #

通过递归创建独立的分类对象、维护层级关系，实现正确的嵌套结构生成：

import scrapy
from scrapy.linkextractors import LinkExtractor

class CategoryLinkSpider(scrapy.Spider):
    name = "category_link"
    allowed_domains = ["illyushatoys.com.ua"]
    start_urls = ["https://illyushatoys.com.ua/?categoryID=184864"]

    def parse(self, response):
        # 初始化当前分类对象
        current_category = {
            "url": response.url,
            "category_name": response.css('h1::text').get().strip(),  # 根据页面实际结构调整选择器
            "subcategories": []
        }

        # 提取子分类链接
        sub_links = LinkExtractor(allow=r'.*category/\d+/$', restrict_css='div.inmenu').extract_links(response)
        
        if sub_links:
            # 遍历子分类，递归解析
            for link in sub_links:
                sub_category = {
                    "url": link.url,
                    "category_name": link.text.strip(),
                    "subcategories": []
                }
                current_category["subcategories"].append(sub_category)
                # 传递层级元数据到回调
                yield response.follow(
                    link, 
                    callback=self.parse_subcategory, 
                    meta={"parent_category": current_category, "current_subcat": sub_category}
                )
        else:
            # 无下属分类，解析当前分类的产品
            products = response.css('div.product-item h3::text').getall()
            current_category["products"] = [p.strip() for p in products]
            yield current_category

    def parse_subcategory(self, response):
        parent_category = response.meta["parent_category"]
        current_subcat = response.meta["current_subcat"]

        # 提取当前子分类的下属分类链接
        sub_links = LinkExtractor(allow=r'.*category/\d+/$', restrict_css='div.inmenu').extract_links(response)
        
        if sub_links:
            for link in sub_links:
                child_subcat = {
                    "url": link.url,
                    "category_name": link.text.strip(),
                    "subcategories": []
                }
                current_subcat["subcategories"].append(child_subcat)
                yield response.follow(
                    link, 
                    callback=self.parse_subcategory, 
                    meta={"parent_category": parent_category, "current_subcat": child_subcat}
                )
        else:
            # 解析当前分类下的产品，输出完整分类树
            products = response.css('div.product-item h3::text').getall()
            current_subcat["products"] = [p.strip() for p in products]
            yield parent_category

关键说明 #

1. 层级对象隔离：每个分类创建独立字典，避免递归时数据覆盖
2. 元数据传递：通过meta传递父分类和当前子分类对象，维护嵌套关系
3. 终止逻辑：检测不到子分类链接时，解析产品并输出完整结构
4. 选择器适配：需根据目标网站实际HTML结构，调整category_name和产品提取的CSS选择器

内容的提问来源于stack exchange，提问作者Олександр Митровка