Scrapy JSON导出格式调整求助：将多独立对象转为pages数组

Got it, let's get your Scrapy JSON output formatted exactly how you want! The default Scrapy JSON exporter spits out each page item as a standalone object, but we can adjust it to wrap all your data into a single pages array. Here are two reliable approaches:

Method 1: Custom Item Exporter (Recommended for Permanent Use) #

This method modifies Scrapy's built-in exporter to generate your desired structure directly when you run the spider.

1. Create a custom exporter file
   In your Scrapy project root, make a new file named exporters.py and paste this code:

   from scrapy.exporters import JsonItemExporter
   from scrapy.utils.python import to_bytes
   
   class PagesJsonExporter(JsonItemExporter):
       def __init__(self, file, **kwargs):
           super().__init__(file, **kwargs)
           self.first_item = True
   
       def start_exporting(self):
           # Write the opening of our target structure
           self.file.write(to_bytes('{"pages": [', self.encoding))
   
       def finish_exporting(self):
           # Close the array and outer object cleanly
           self.file.write(to_bytes(']}', self.encoding))
   
       def export_item(self, item):
           # Add a comma before every item except the first to avoid syntax errors
           if not self.first_item:
               self.file.write(to_bytes(',', self.encoding))
           self.first_item = False
           # Let the parent class handle item serialization
           super().export_item(item)
   

2. Configure Scrapy to use your exporter
   Open your project's settings.py and add this line (replace your_project_name with your actual project name):

   FEED_EXPORTERS = {
       'json': 'your_project_name.exporters.PagesJsonExporter',
   }
   

3. Run your spider as normal
   Now when you export to JSON, it'll automatically wrap all items in the pages array:

   scrapy crawl your_spider_name -o output.json
   

Method 2: Post-Processing with jq (Quick One-Time Fix) #

If you don't want to tweak your project code, use the jq command-line tool to reformat the default Scrapy output after exporting.

1. Export items with Scrapy's default JSON lines format

   scrapy crawl your_spider_name -o items.json
   

   This gives you a file with one JSON object per line.

2. Use jq to wrap items into the pages array
   Run this command to transform the output:

   jq -s '{"pages": .}' items.json > final_output.json
   

   The -s flag tells jq to slurp all line-separated objects into a single array, then we wrap that array in the outer {"pages": ...} structure.

Either method will produce the exact structure you're aiming for:

{"pages": [{"title": "x", "text": "x", "tags": "x", "url": "x"}, ...]}

内容的提问来源于stack exchange，提问作者Janne Salmi