Scrapy爬虫拼接地址报错：TypeError: NoneType与str无法执行+=

问题描述 #

我开发了一个Scrapy爬虫用于抓取黄页中的水管商家信息，地址被拆分为street_adress和locality_两个字段。尝试在Pipeline文件中将这两部分拼接为full_address时，出现错误：

full_address += f'{locailty_} {street_adress}' TypeError: unsupported operand type(s) for +=: 'NoneType' and 'str'

但从爬虫的单条输出结果来看，这两个地址字段均有有效值，请问为何会出现该错误？

爬虫单条输出示例 #

{
 'locality_': 'Manassas, VA 20109',
 'logo': None,
 'name': 'F H Furr Plumbing-Heating & Air Conditioning',
 'number_of_riews': None,
 'payment_mentod': 'check',
 'phone_number': '(571) 234-6893',
 'stars_out_of_five': None,
 'street_adress': '9040 Mike Garcia Dr',
 'website': 'http://www.fhfurr.com'
}

相关代码 #

spider.py #

import scrapy
from yp_scraper.items import PlubmerInfo

class PlumberScraperSpider(scrapy.Spider):
    name = 'plumber_scraper'
    allowed_domains = ['yellowpages.com']
    start_urls = ['https://www.yellowpages.com/']

    def parse(self, response):
        # 以下是地区名称和州缩写，用于拼接起始URL定位水管商家区域
        area = "fairfax"
        state = "va"# 必须是缩写
        # 下面是要抓取的页面数量
        numer_of_pages = 10 
        page_numebr = 1

        #while numer_of_pages > 0:
        url = f'https://www.yellowpages.com/{area}-{state}/plumbers?page={page_numebr}'
            #page_numebr += 1
            #numer_of_pages -= 1
        print('tesssssssssssssssst')
        yield response.follow(url, callback=self.parse_plumbers)

    def parse_plumbers(self, response):
        print('text2')
        plumber_item = PlubmerInfo()
        print('yes_man')
        plumbers = response.css('div.result')

        for plumber in plumbers:
            starter_indidual_url = plumber.css('a.business-name ::attr(href)').get()
            indidual_url = f'https://www.yellowpages.com{starter_indidual_url}'
            yield response.follow(indidual_url, callback=self.parse_indvidual_plumbers)

    def parse_indvidual_plumbers(self, response):
        plumber_item = PlubmerInfo()
        print(response.xpath('//*[@id="default-ctas"]/a[3]/span/text()').get())
        plumber_item['name'] = response.css('h1.business-name ::text').get()
        plumber_item['phone_number'] = response.css('a.phone ::text').get() 
        plumber_item['website'] = response.css('a.website-link ::attr(href)').get()
        plumber_item['genral_info'] = response.css('dd.general-info ::text').get()
        plumber_item['payment_mentod'] = response.css('dd.payment ::text').get()
        plumber_item['stars_out_of_five'] = response.css('div.rating-stars ::attr(class)').get()
        plumber_item['number_of_riews'] = response.css('span.count ::text').get()
        plumber_item['locality_'] = response.xpath('//*[@id="default-ctas"]/a[3]/span/text()').get()
        plumber_item['street_adress'] = response.css('span.address ::text').get()
        #plumber_item['services'] = response.css('div.locality ::text').get()
        plumber_item['email'] = response.css('a.email-business ::attr(href)').get()
        plumber_item['logo'] = response.css('dd.logo ::attr(href)').get()

        yield plumber_item

pipelines.py #

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html


# useful for handling different item types with a single interface
from itemadapter import ItemAdapter


class YpScraperPipeline:
    def process_item(self, item, spider):
        adapter = ItemAdapter(item)

        #combindes locality and street adress
        locailty_ = adapter.get('locality_')
        street_adress = adapter.get('street_adress')
        full_address = adapter.get('full_address')
        if locailty_ is not None:
            for i in street_adress:
                full_address += f'{locailty_} {street_adress}'


        return item

错误原因及解决方案 #

错误原因 #

1. full_address初始值为None：你通过adapter.get('full_address')获取该字段，但爬虫的parse_indvidual_plumbers方法中从未生成过full_address字段，因此初始值是None，无法直接用+=拼接字符串。
2. 多余的循环逻辑：代码中for i in street_adress会遍历street_adress字符串的每个字符，导致多次重复拼接地址，完全没必要。
3. 变量名拼写错误：locailty_少写了一个字母l，正确应为locality_，虽然当前示例中字段有值，但后续空值判断可能出现逻辑错误。
4. 未处理street_adress为空的情况：仅判断了locailty_不为None，但如果street_adress为空，拼接逻辑仍会出现异常。

修正后的Pipeline代码 #

from itemadapter import ItemAdapter


class YpScraperPipeline:
    def process_item(self, item, spider):
        adapter = ItemAdapter(item)

        # 拼接地址字段
        locality_ = adapter.get('locality_')
        street_address = adapter.get('street_adress')
        
        # 初始化full_address为空字符串
        full_address = ""
        # 同时判断两个字段都不为空才拼接
        if locality_ and street_address:
            full_address = f"{street_address} {locality_}"
        elif locality_:
            full_address = locality_
        elif street_address:
            full_address = street_address
        
        # 将拼接后的地址存入item
        adapter['full_address'] = full_address

        return item

内容的提问来源于stack exchange，提问作者Ethan Koch