Portia/Scrapy：如何替换或添加输出JSON中的值？

Hey there! I’ve got you covered with step-by-step solutions for both Portia and Scrapy to address your two questions. Let’s dive in!

1. Replace Extracted Text in Final JSON #

Portia’s visual editor makes this straightforward using its transform feature for fields:

• Open your Portia project and navigate to the Item you’re scraping.
• Find the field that extracts the "ADD TO CART" text (e.g., a button text field).
• Click on the field to edit it, then look for the Transform section.
• Enter a Python expression to replace the text, like:
  value.replace("ADD TO CART", "IN STOCK")
• Save your changes. Now every time this field is extracted, the text will be replaced before it’s added to the JSON output.

2. Add Custom Data (e.g., "Store name") to Final JSON #

Adding fixed custom data to every scraped item in Portia is easy with a static field:

• In your Item editor, click the Add Field button.
• Name the field something like store_name (matching the JSON key you want).
• For the Value type, select "Static" instead of "Extracted".
• Enter your desired store name (e.g., "My Awesome Store") in the static value field.
• Save the field. Now every item in your final JSON will include this store_name key with your custom value.

Since you already have your Scrapy spider code, here are practical approaches for each question:

1. Replace Extracted Text in Final JSON #

You can do this either directly in your spider’s parse method, or using an Item Pipeline for more reusable logic.

Option 1: Replace in the Spider’s Parse Method #

Modify the extracted text immediately before assigning it to the item:

def parse(self, response):
    item = YourScrapyItem()
    # Extract and clean the button text, then replace it
    add_to_cart_text = response.css('.add-to-cart-button::text').get().strip()
    item['status'] = add_to_cart_text.replace("ADD TO CART", "IN STOCK")
    # Rest of your item extraction code...
    yield item

Option 2: Use an Item Pipeline #

For replacement logic that applies across multiple spiders or fields, create a pipeline:

1. In your project’s pipelines.py, add this class:

class ReplaceTextPipeline:
    def process_item(self, item, spider):
        # Check if the target field exists and needs replacement
        if 'status' in item and item['status'].strip() == "ADD TO CART":
            item['status'] = "IN STOCK"
        return item

2. Enable the pipeline in settings.py:

ITEM_PIPELINES = {
    'your_project_name.pipelines.ReplaceTextPipeline': 300,
}

2. Add Custom Data (e.g., "Store name") to Final JSON #

You have a few flexible options here depending on your needs:

Option 1: Assign Directly in the Spider #

Add the custom field to every item in your parse method:

def parse(self, response):
    item = YourScrapyItem()
    # Your existing extraction code...
    # Add the custom store name
    item['store_name'] = "My Awesome Store"
    yield item

Option 2: Use an Item Pipeline for Global Application #

If you want this custom data added to all items across your project, use a pipeline:

1. In pipelines.py:

class AddCustomDataPipeline:
    def process_item(self, item, spider):
        item['store_name'] = "My Awesome Store"
        return item

2. Enable it in settings.py (adjust the priority number to control execution order):

ITEM_PIPELINES = {
    'your_project_name.pipelines.AddCustomDataPipeline': 200,
    # Other pipelines you may have...
}

Bonus: Set a Default Value in the Item Definition #

You can also define the custom field with a default value in your items.py:

import scrapy

class YourScrapyItem(scrapy.Item):
    # Your existing fields...
    store_name = scrapy.Field(default="My Awesome Store")

This way, the field will be included in every item automatically, even if you don’t explicitly assign it in the spider.

内容的提问来源于stack exchange，提问作者Rafael Souza