Scrapy爬取第三页报str与非str参数混合错误问题咨询

运行抛出You can't mix str and non-str arguments是代码里3处核心问题导致的：

• 错误将Scrapy选择器对象当作URL传入拼接方法：urlem = response.urljoin(website)行的website是遍历列表时拿到的Selector对象，不是字符串格式的链接，传入仅接受字符串参数的urljoin()会直接触发类型错误
• 翻页逻辑无边界判断：爬取到最后一页时，下一页选择器PAGETEST会取到None值，非字符串的None传入urljoin()同样会触发该报错
• 附加逻辑bug：parseEmail方法里定义选择器变量为hps，后续却调用未定义的hxs；旧版HtmlXPathSelector在新版Scrapy中已废弃；且提取到website字段后没有发起对应URL的爬取请求，不符合提取该地址下数据的需求

# -*- coding: utf-8 -*-
import scrapy
from scrapy.crawler import CrawlerProcess
from scrapy import Request, Spider


class RynekMainSpider(scrapy.Spider):
    name = "RynekMain"
    start_urls = [
        'https://rynekpierwotny.pl/deweloperzy/?page=1']
    def parse(self, response):
        websites = response.css('div#root')[0]
        PAGETEST = response.xpath('//a[contains(@class,"rp-173nt6g")]/../following-sibling::li').css('a::attr(href)').get()
        for website in websites.css('li.rp-np9kb1'):
            page = website.css('a::attr(href)').get()
            address = website.css('address.rp-o9b83y::text').get()
            name = website.css('h2.rp-69f2r4::text').get()
            params = {
                'address' : address,
                'name' : name,
                'href' : page,
            }
            url  = response.urljoin(page)
            yield Request(url=url, cb_kwargs={'params': params}, callback=self.parseMain)
        # 仅当存在下一页链接时才发起翻页请求
        if PAGETEST:
            yield Request(url=response.urljoin(PAGETEST), callback=self.parse)

    def parseMain(self, response, params=None):
        website_url = response.css('div.rp-l0pkv6 a::attr(href)').get()
        params['website'] = website_url
        yield params
        # 拿到合法的website链接后，再发起对应地址的爬取请求
        if website_url:
            yield Request(url=response.urljoin(website_url), cb_kwargs={'params': params}, callback=self.parseEmail)
    
    def parseEmail(self,response, params=None):
        # 直接使用response内置选择器，替换废弃的HtmlXPathSelector，修复变量名笔误
        email_list = response.xpath('//body').re('([a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+)')
        # 可在此处补充该URL下其他目标字段的提取逻辑
        params['emails'] = email_list
        yield params      

if __name__ == "__main__":
    process =CrawlerProcess(settings={'LOG_LEVEL': 'INFO'})
    process.crawl(RynekMainSpider)
    process.start()

• 删除了错误传入选择器对象的URL拼接逻辑，避免非字符串参数传入链接处理方法
• 给翻页逻辑增加非空判断，最后一页无下一页链接时自动终止翻页循环
• 补全website字段的后续爬取逻辑：拿到合法的站点URL后才发起对应请求，匹配爬取该地址下目标数据的需求
• 修复变量名笔误，替换废弃API，所有传入URL拼接方法的参数都提前做合法性校验，从根源避免类型混合报错

内容的提问来源于stack exchange，提问作者WaterWolf86