Scrapy爬虫如何将yield返回的爬取结果保存到全局变量中

方法1：直接使用全局变量（快速实现） #

直接在解析函数中操作全局变量即可，适合小脚本快速使用，修改后的完整代码如下：

import scrapy
from scrapy.crawler import CrawlerProcess

global_var = [] # 改为列表存储多条爬取结果

class TestSpider(scrapy.Spider):
    name = 'test'
    allowed_domains = ['worldpopulationreview.com']
    start_urls = ['https://worldpopulationreview.com/countries/countries-by-national-debt/']

    def parse(self, response):
        country_data = response.xpath('//tbody/tr')
        for data in country_data:
            name = data.xpath('.//td[1]/a/text()').get()
            debt = data.xpath('.//td[2]/text()').get()
            population = data.xpath('.//td[3]/text()').get()
            link = data.xpath('.//td[1]/a/@href').get()

            item = {'country_name': name, 'country_debt': debt, 'country_population': population, 'country_url': response.urljoin(link)}
            yield item
            # 直接写入全局变量
            global_var.append(item)


process = CrawlerProcess(
    settings={
        "FEEDS": {
            "result.json": {"format": "json"},
            "result.csv": {"format": "csv"},
        },
    }
)

process.crawl(TestSpider)
process.start()

# 爬虫运行结束后即可使用global_var
print(global_var)

方法2：使用Item Pipeline（更规范，适合局部参数传递） #

如果不想污染全局变量，需要把结果作为局部参数传递给其他函数，推荐使用Scrapy原生的Item Pipeline做结果收集，代码如下：

import scrapy
from scrapy.crawler import CrawlerProcess

# 定义结果收集Pipeline
class ResultCollectPipeline:
    def __init__(self):
        self.items = []

    def process_item(self, item, spider):
        self.items.append(item)
        return item

class TestSpider(scrapy.Spider):
    name = 'test'
    allowed_domains = ['worldpopulationreview.com']
    start_urls = ['https://worldpopulationreview.com/countries/countries-by-national-debt/']

    def parse(self, response):
        country_data = response.xpath('//tbody/tr')
        for data in country_data:
            name = data.xpath('.//td[1]/a/text()').get()
            debt = data.xpath('.//td[2]/text()').get()
            population = data.xpath('.//td[3]/text()').get()
            link = data.xpath('.//td[1]/a/@href').get()

            yield {'country_name': name, 'country_debt': debt, 'country_population': population, 'country_url': response.urljoin(link)}


process = CrawlerProcess(
    settings={
        "ITEM_PIPELINES": {
            '__main__.ResultCollectPipeline': 300, # 注册自定义Pipeline
        },
        "FEEDS": {
            "result.json": {"format": "json"},
            "result.csv": {"format": "csv"},
        },
    }
)

# 创建crawler实例方便后续取结果
crawler = process.create_crawler(TestSpider)
process.crawl(crawler)
process.start()

# 从Pipeline中取出爬取结果
result_list = []
for pipeline in crawler.engine.pipelines.middlewares:
    if isinstance(pipeline, ResultCollectPipeline):
        result_list = pipeline.items
        break

# 此时result_list就是所有爬取结果，可直接作为参数传递给其他函数
print(result_list)

注意事项 #

• process.start()是阻塞方法，所有爬虫逻辑运行结束后才会执行后面的代码，读取变量的操作必须放在process.start()之后
• 如果是多进程/分布式爬虫场景，全局变量方案会失效，推荐用Pipeline配合数据库存储收集结果

内容的提问来源于stack exchange，提问作者codecumber