如何将Scrapy的Response从Spider传递至Pipeline？

问题描述 #

需要在Pipeline中获取包含配置（settings）、URL等完整信息的Scrapy Response，用于创建模型对象。但目前Pipeline仅能获取Item，现有信息无法满足需求，直接调用Response未果，寻求实现方法。

现有代码 #

pipeline.py

class ScraperPipeline(object):
    def process_item(self, item, spider):
        logger = get_task_logger("logs")
        logger.info("Pipeline activated.")
        id = item['id'][0]
        user= item['user'][0]
        text = item['text'][0]
        Mail.objects.create(user=User.objects.get_or_create(
            id=id, user=user),
            text=text, date=today)
        logger.info(f"Pipeline disacvtivated")

spider.py

class Spider(CrawlSpider):
    name = 'spider'
    allowed_domains = ['xxx.com']

    def start_requests(self):
        urls = [
            'xxx.com',
        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse,
                                 headers={'User-Agent':
                                              'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, '
                                              'like Gecko) Chrome/107.0.0.0 Safari/537.36'})

    def parse(self, response):
        item = MailItem()
        for row in response.xpath('xpath thins'):
            ip['id'] = row.xpath('td[1]//text()').extract_first(),
            ip['user'] = row.xpath('td[2]//text()').extract_first(),
            ip['text'] = row.xpath('td[3]//text()').extract_first(),
            yield item

解决方案 #

方法一：将所需Response信息存入Item（推荐） #

直接在Spider的parse方法中，把Response的URL、settings配置等需要的信息添加到Item字段中，Pipeline即可直接读取这些数据。

修改后的spider.py

class Spider(CrawlSpider):
    name = 'spider'
    allowed_domains = ['xxx.com']

    def start_requests(self):
        urls = [
            'xxx.com',
        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse,
                                 headers={'User-Agent':
                                              'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, '
                                              'like Gecko) Chrome/107.0.0.0 Safari/537.36'})

    def parse(self, response):
        item = MailItem()
        # 存入Response的来源URL
        item['source_url'] = response.url
        # 从settings获取自定义配置（替换为你的配置名）
        item['custom_config'] = self.settings.get('YOUR_CUSTOM_SETTING')
        
        for row in response.xpath('xpath thins'):
            # 修正原代码变量名错误：ip改为item
            item['id'] = row.xpath('td[1]//text()').extract_first(),
            item['user'] = row.xpath('td[2]//text()').extract_first(),
            item['text'] = row.xpath('td[3]//text()').extract_first(),
            yield item

修改后的pipeline.py

class ScraperPipeline(object):
    def process_item(self, item, spider):
        logger = get_task_logger("logs")
        logger.info("Pipeline activated.")
        id = item['id'][0]
        user= item['user'][0]
        text = item['text'][0]
        # 获取存入的URL和配置
        source_url = item['source_url']
        custom_config = item['custom_config']
        
        Mail.objects.create(user=User.objects.get_or_create(
            id=id, user=user),
            text=text, date=today, source_url=source_url)
        logger.info(f"Pipeline disactivated")
        return item  # 必须返回item，否则后续Pipeline无法处理

方法二：直接传递整个Response对象（不推荐，影响性能） #

如果确实需要完整的Response对象，可以将其存入Item，但注意Response体积较大，可能降低爬取效率。

步骤1：在Item类中添加对应字段（items.py）

import scrapy

class MailItem(scrapy.Item):
    id = scrapy.Field()
    user = scrapy.Field()
    text = scrapy.Field()
    response = scrapy.Field()  # 用于存储Response对象

步骤2：在Spider的parse方法中存入Response

def parse(self, response):
    item = MailItem()
    # 存入完整Response对象
    item['response'] = response
    
    for row in response.xpath('xpath thins'):
        item['id'] = row.xpath('td[1]//text()').extract_first(),
        item['user'] = row.xpath('td[2]//text()').extract_first(),
        item['text'] = row.xpath('td[3]//text()').extract_first(),
        yield item

步骤3：在Pipeline中使用Response

class ScraperPipeline(object):
    def process_item(self, item, spider):
        logger = get_task_logger("logs")
        logger.info("Pipeline activated.")
        # 获取完整Response对象
        response = item['response']
        source_url = response.url
        # 其他Response相关操作...
        
        id = item['id'][0]
        user= item['user'][0]
        text = item['text'][0]
        Mail.objects.create(user=User.objects.get_or_create(
            id=id, user=user),
            text=text, date=today, source_url=source_url)
        logger.info(f"Pipeline disactivated")
        return item

额外技巧：直接在Pipeline中获取Settings #

如果仅需要settings配置，无需存入Item，可直接通过spider.settings获取：

class ScraperPipeline(object):
    def process_item(self, item, spider):
        # 直接获取settings中的配置
        custom_config = spider.settings.get('YOUR_CUSTOM_SETTING')
        # ...后续逻辑

内容的提问来源于stack exchange，提问作者c0rvvv