如何在Python Scrapy爬虫中为缺失信息导入空值到CSV？

我正在开发第一个Web Scraping项目，目标爬取Booking.com的纽约酒店数据，重点获取酒店是否包含早餐的信息。需求是每个酒店的早餐信息要么是["Breakfast included"]，要么无相关信息时为[""]，但运行代码后仅能抓取到少数含早餐的酒店数据。

问题核心：当酒店不含早餐时，其property card中不存在类名为e05969d63d的元素（该元素对应早餐包含信息），导致无法生成空值条目。例如，若酒店1和3含早餐、酒店2不含，期望导出["Breakfast included","","Breakfast included"]，但实际仅得到["Breakfast included", "Breakfast included"]。

原代码 #

1. 管道类代码 #

from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"

import scrapy
import logging
from scrapy.crawler import CrawlerProcess
from scrapy.exporters import CsvItemExporter

class CsvPipeline(object):
    def __init__(self):
        self.file = open ('hotel.tmp','wb')
        self.exporter = CsvItemExporter(self.file,str)
        self.exporter.start_exporting()
    def close_spider(self,spider):
        self.exporter.finish_exporting()
        self.file.close()
    def process_items(self,item,spider):
        self.exporter.export_item(item)
        return item

2. 爬虫类代码 #

class hotelsNY(scrapy.Spider):
    name = "hotelsNY"
    start_urls =[]
    for i in range (0, 10):
        start_urls.append('https://www.booking.com/searchresults.pl.html?label=gen173nr-1BCAEoggI46AdIM1gEaLYBiAEBmAEeuAEXyAEM2AEB6AEBiAIBqAIDuALX3uicBsACAdICJGRlODkzYmJmLTIyZjQtNDYwNi04YzYwLWIxOWRlMGU0MmM0MdgCBeACAQ&sid=7ab6fb8585341629f1a790546e37a1c5&aid=304142&ss=Nowy+Jork&ssne=Nowy+Jork&ssne_untouched=Nowy+Jork&lang=pl&sb=1&src_elem=sb&src=index&dest_id=20088325&dest_type=city&checkin=2022-12-30&checkout=2023-01-03&group_adults=2&no_rooms=1&group_children=0&sb_travel_purpose=leisure&offset=' + str(i*25))
    
    custom_settings = {
        'LOG_LEVEL': logging.WARNING,
        'FEED_EXPORTERS': {'csv': 'scrapy.exporters.CsvItemExporter'},
        'FEED_FORMAT': 'csv',
        'FEED_URI': 'hotels_tmp1.csv'
    }

    def parse(self, response):
        nexturl = 'https://www.booking.com/searchresults.pl.html?label=gen173nr-1BCAEoggI46AdIM1gEaLYBiAEBmAEeuAEXyAEM2AEB6AEBiAIBqAIDuALX3uicBsACAdICJGRlODkzYmJmLTIyZjQtNDYwNi04YzYwLWIxOWRlMGU0MmM0MdgCBeACAQ&sid=7ab6fb8585341629f1a790546e37a1c5&aid=304142&ss=Nowy+Jork&ssne=Nowy+Jork&ssne_untouched=Nowy+Jork&lang=pl&sb=1&src_elem=sb&src=index&dest_id=20088325&dest_type=city&checkin=2022-12-30&checkout=2023-01-03&group_adults=2&no_rooms=1&group_children=0&sb_travel_purpose=leisure&offset=0'
        alH = response.xpath('//*[@data-testid="property-card"]').getall()
        for name in alH:
            hotelName = response.xpath('//*[@data-testid="title"]/text()').extract(),
            address = response.xpath('//*[@data-testid="address"]/text()').extract(),
            price = response.xpath('//*[@data-testid="price-and-discounted-price"]/text()').extract(),
            dist = response.xpath('//span[@data-testid="distance"]/text()').extract(),
            breakfast = response.xpath('//span[@class="e05969d63d"]/text()').extract(),
            yield {'hotelName': hotelName, 'address': address, 'price': price, 'dist': dist, 'breakfast': breakfast}

3. 启动代码 #

process = CrawlerProcess(
    {
     'USER_AGENT':'Mozilla/4.0 (comatible;MSIE 7.0;Window NT 5.1)'
     })
process.crawl(hotelsNY)
process.start()

解决方法 #

问题分析 #

原代码的核心错误：

1. 遍历property-card时，直接处理getall()返回的字符串列表，而非单个酒店卡片的选择器对象，导致每次循环都提取页面全量数据，而非单酒店数据。
2. 提取早餐信息时未处理元素不存在的场景，无法生成空值条目。
3. 冗余的CsvPipeline类与Scrapy自带Feed导出功能重复。

修改后的完整代码 #

from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"

import scrapy
import logging
from scrapy.crawler import CrawlerProcess

class hotelsNY(scrapy.Spider):
    name = "hotelsNY"
    start_urls =[]
    for i in range (0, 10):
        start_urls.append('https://www.booking.com/searchresults.pl.html?label=gen173nr-1BCAEoggI46AdIM1gEaLYBiAEBmAEeuAEXyAEM2AEB6AEBiAIBqAIDuALX3uicBsACAdICJGRlODkzYmJmLTIyZjQtNDYwNi04YzYwLWIxOWRlMGU0MmM0MdgCBeACAQ&sid=7ab6fb8585341629f1a790546e37a1c5&aid=304142&ss=Nowy+Jork&ssne=Nowy+Jork&ssne_untouched=Nowy+Jork&lang=pl&sb=1&src_elem=sb&src=index&dest_id=20088325&dest_type=city&checkin=2022-12-30&checkout=2023-01-03&group_adults=2&no_rooms=1&group_children=0&sb_travel_purpose=leisure&offset=' + str(i*25))
    
    custom_settings = {
        'LOG_LEVEL': logging.WARNING,
        'FEED_FORMAT': 'csv',
        'FEED_URI': 'hotels_tmp1.csv'
    }

    def parse(self, response):
        # 遍历单个酒店卡片的选择器对象
        for hotel_card in response.xpath('//*[@data-testid="property-card"]'):
            # 使用相对路径（.//）从当前卡片内提取数据，避免全局提取
            hotel_name = hotel_card.xpath('.//*[@data-testid="title"]/text()').extract_first(default='')
            address = hotel_card.xpath('.//*[@data-testid="address"]/text()').extract_first(default='')
            price = hotel_card.xpath('.//*[@data-testid="price-and-discounted-price"]/text()').extract_first(default='')
            dist = hotel_card.xpath('.//span[@data-testid="distance"]/text()').extract_first(default='')
            
            # 处理早餐信息：存在则取文本，不存在则返回空字符串列表
            breakfast_text = hotel_card.xpath('.//span[@class="e05969d63d"]/text()').extract_first(default='')
            breakfast = [breakfast_text] if breakfast_text else [""]

            yield {
                'hotelName': hotel_name,
                'address': address,
                'price': price,
                'dist': dist,
                'breakfast': breakfast
            }

process = CrawlerProcess(
    {
     'USER_AGENT':'Mozilla/4.0 (compatible;MSIE 7.0;Windows NT 5.1)'
     })
process.crawl(hotelsNY)
process.start()

关键修改说明 #

• 遍历酒店卡片时直接使用response.xpath()返回的选择器对象，而非字符串列表
• 提取单酒店数据时使用相对路径（以.开头），确保只从当前卡片内获取信息
• 用extract_first(default='')替代extract()，避免返回空列表，同时设置默认空值
• 针对早餐信息做特殊处理：存在文本则包装为列表，不存在则返回[""]，完全匹配需求
• 移除冗余的CsvPipeline类，使用Scrapy自带的Feed导出功能
• 修正USER_AGENT中的拼写错误

内容的提问来源于stack exchange，提问作者jl0506