Scrapy技术疑问：如何向ItemLoader传递动态变量？

需求背景 #

需要将ItemLoader中url_in处理器里硬编码的https://www.chocolate.co.uk替换为Spider中动态解析到的域名（即parse方法里的url变量），避免硬编码。

可行解决方案 #

方案一：通过ItemLoader初始化参数传递动态域名 #

修改ChocolateProductLoader，让它在初始化时接收base_url参数，并动态设置url_in处理器：

from itemloaders.processors import TakeFirst, MapCompose
from scrapy.loader import ItemLoader

class ChocolateProductLoader(ItemLoader):
    default_output_processor = TakeFirst()
    price_in = MapCompose(lambda x: x.split('£')[-1])

    def __init__(self, *args, base_url=None, **kwargs):
        super().__init__(*args, **kwargs)
        # 利用传入的base_url动态构建url处理器
        self.url_in = MapCompose(lambda x: base_url + x)

在Spider中创建Loader实例时，传入动态获取的base_url：

import scrapy
from urllib.parse import urlparse
from scrapeops_guide.itemsloaders import ChocolateProductLoader
from scrapeops_guide.items import ChocolateProduct

class ChocolateSpider(scrapy.Spider):
    name = 'chocolate_spider'
    allowed_domains = ['chocolate.co.uk']
    start_urls = ['https://www.chocolate.co.uk/collections/all']

    def parse(self, response, **kwargs):
        products = response.css('product-item')
        url = urlparse(response.url)
        base_url = f'{url.scheme}://{url.netloc}'
        for product in products:
            chocolate = ChocolateProductLoader(
                item=ChocolateProduct(),
                selector=product,
                base_url=base_url  # 传入动态域名
            )
            chocolate.add_css('name', 'a.product-item-meta__title::text')
            chocolate.add_css('price', 'span.price',
                              re='<span class="price">\n              '
                                 '<span class="visually-hidden">Sale price'
                                 '</span>(.*)</span>')
            chocolate.add_css('url', 'div.product-item-meta a::attr(href)')
            yield chocolate.load_item()

        next_page = response.css('a[rel="next"]::attr(href)').get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

方案二：利用Scrapy Loader的context属性 #

如果想通过context传递参数，需要定义接收loader_context参数的处理函数（MapCompose会自动将context作为第二个参数传入）：

先修改ItemLoader：

from itemloaders.processors import TakeFirst, MapCompose
from scrapy.loader import ItemLoader

# 定义带loader_context参数的处理函数
def append_base_url(relative_url, loader_context):
    base_url = loader_context.get('base_url')
    return base_url + relative_url

class ChocolateProductLoader(ItemLoader):
    default_output_processor = TakeFirst()
    price_in = MapCompose(lambda x: x.split('£')[-1])
    url_in = MapCompose(append_base_url)  # 使用自定义处理函数

然后在Spider中传入context参数：

import scrapy
from urllib.parse import urlparse
from scrapeops_guide.itemsloaders import ChocolateProductLoader
from scrapeops_guide.items import ChocolateProduct

class ChocolateSpider(scrapy.Spider):
    name = 'chocolate_spider'
    allowed_domains = ['chocolate.co.uk']
    start_urls = ['https://www.chocolate.co.uk/collections/all']

    def parse(self, response, **kwargs):
        products = response.css('product-item')
        url = urlparse(response.url)
        base_url = f'{url.scheme}://{url.netloc}'
        for product in products:
            chocolate = ChocolateProductLoader(
                item=ChocolateProduct(),
                selector=product,
                context={'base_url': base_url}  # 传入context
            )
            chocolate.add_css('name', 'a.product-item-meta__title::text')
            chocolate.add_css('price', 'span.price',
                              re='<span class="price">\n              '
                                 '<span class="visually-hidden">Sale price'
                                 '</span>(.*)</span>')
            chocolate.add_css('url', 'div.product-item-meta a::attr(href)')
            yield chocolate.load_item()

        next_page = response.css('a[rel="next"]::attr(href)').get()
        if next_page:
            yield response.follow(next_page, callback=self.parse)

说明 #

之前尝试context失败，是因为直接用无参数的lambda无法获取loader_context，必须定义接收两个参数的处理函数，MapCompose会自动将Loader的context作为第二个参数传递给处理函数。

内容的提问来源于stack exchange，提问作者Bahia