Scrapy分页失效求助：仅能爬取目标站点第一页数据

架构老林

2026-8-3

Zapimoveis爬虫分页失效问题排查与解决

核心问题分析

你的代码存在几个关键问题导致分页失效：

分页参数不匹配：API的from参数与page参数未同步递增，初始请求from=24但page=1，后续请求from始终固定为24，导致重复请求或获取错误页码数据。
类属性page线程不安全：Scrapy是异步多线程架构，多个parse请求同时修改self.page会导致页码混乱（如跳页、重复页）。
停止爬虫条件错误：判断len(json.loads(response.body)) ==0不严谨，API不会返回空响应，应检查返回的listings是否为空或基于总数据量计算停止时机。
静态参数可能失效：URL中的__zt是动态生成的验证参数，固定值可能被站点拦截判定为无效请求。

具体修复方案

1. 修正分页参数逻辑

API的from参数规则为(当前页码-1)*每页数据量，每页固定24条数据：

page=1 → from=0
page=2 → from=24
page=3 → from=48

2. 安全传递页码

通过meta参数传递当前页码，避免多线程下类属性被篡改。

3. 正确判断停止时机

从API返回的totalCount计算总页数，当当前页码超过总页数时停止爬虫；或当返回的listings为空时终止。

4. 完善请求头

添加User-Agent、Referer等模拟真实浏览器请求，降低被反爬拦截的概率。

修正后的代码示例

import scrapy
import json
from scrapy.exceptions import CloseSpider
from scrapy.http import urlencode
 
class ImoveisSpider(scrapy.Spider):
    name = 'imoveis'
    size = 24  # 每页数据量
    base_url = 'https://glue-api.zapimoveis.com.br/v2/listings'

    def start_requests(self):
        # 第一页：page=1，from=0
        yield scrapy.Request(
            url=self.build_request_url(page=1, from_offset=0),
            headers={
                'x-domain': 'www.zapimoveis.com.br',
                'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36',
                'Referer': 'https://www.zapimoveis.com.br/'
            },
            callback=self.parse,
            meta={'current_page': 1}
        )

    def build_request_url(self, page, from_offset):
        # 拆分参数便于维护，避免手动拼接长字符串出错
        params = {
            '0': 'U', '1': 'n', '2': 'i', '3': 't', '4': 'T', '5': 'y', '6': 'p', '7': 'e', '8': '_', '9': 'N',
            '10': 'O', '11': 'N', '12': 'E',
            'categoryPage': 'RESULT',
            'business': 'SALE',
            'listingType': 'USED',
            'parentId': 'null',
            'size': str(self.size),
            'from': str(from_offset),
            'page': str(page),
            'includeFields': 'search(result(listings(listing(displayAddressType,amenities,usableAreas,constructionStatus,listingType,description,title,createdAt,floors,unitTypes,nonActivationReason,providerId,propertyType,unitSubTypes,unitsOnTheFloor,legacyId,id,portal,unitFloor,parkingSpaces,updatedAt,address,suites,publicationType,externalId,bathrooms,usageTypes,totalAreas,advertiserId,advertiserContact,whatsappNumber,bedrooms,acceptExchange,pricingInfos,showPrice,resale,buildings,capacityLimit,status),account(id,name,logoUrl,licenseNumber,showAddress,legacyVivarealId,legacyZapId,minisite),medias,accountLink,link)),totalCount),expansion(search(result(listings(listing(displayAddressType,amenities,usableAreas,constructionStatus,listingType,description,title,createdAt,floors,unitTypes,nonActivationReason,providerId,propertyType,unitSubTypes,unitsOnTheFloor,legacyId,id,portal,unitFloor,parkingSpaces,updatedAt,address,suites,publicationType,externalId,bathrooms,usageTypes,totalAreas,advertiserId,advertiserContact,whatsappNumber,bedrooms,acceptExchange,pricingInfos,showPrice,resale,buildings,capacityLimit,status),account(id,name,logoUrl,licenseNumber,showAddress,legacyVivarealId,legacyZapId,minisite),medias,accountLink,link)),totalCount)),nearby(search(result(listings(listing(displayAddressType,amenities,usableAreas,constructionStatus,listingType,description,title,createdAt,floors,unitTypes,nonActivationReason,providerId,propertyType,unitSubTypes,unitsOnTheFloor,legacyId,id,portal,unitFloor,parkingSpaces,updatedAt,address,suites,publicationType,externalId,bathrooms,usageTypes,totalAreas,advertiserId,advertiserContact,whatsappNumber,bedrooms,acceptExchange,pricingInfos,showPrice,resale,buildings,capacityLimit,status),account(id,name,logoUrl,licenseNumber,showAddress,legacyVivarealId,legacyZapId,minisite),medias,accountLink,link)),totalCount)),page,fullUriFragments,developments(search(result(listings(listing(displayAddressType,amenities,usableAreas,constructionStatus,listingType,description,title,createdAt,floors,unitTypes,nonActivationReason,providerId,propertyType,unitSubTypes,unitsOnTheFloor,legacyId,id,portal,unitFloor,parkingSpaces,updatedAt,address,suites,publicationType,externalId,bathrooms,usageTypes,totalAreas,advertiserId,advertiserContact,whatsappNumber,bedrooms,acceptExchange,pricingInfos,showPrice,resale,buildings,capacityLimit,status),account(id,name,logoUrl,licenseNumber,showAddress,legacyVivarealId,legacyZapId,minisite),medias,accountLink,link)),totalCount)),superPremium(search(result(listings(listing(displayAddressType,amenities,usableAreas,constructionStatus,listingType,description,title,createdAt,floors,unitTypes,nonActivationReason,providerId,propertyType,unitSubTypes,unitsOnTheFloor,legacyId,id,portal,unitFloor,parkingSpaces,updatedAt,address,suites,publicationType,externalId,bathrooms,usageTypes,totalAreas,advertiserId,advertiserContact,whatsappNumber,bedrooms,acceptExchange,pricingInfos,showPrice,resale,buildings,capacityLimit,status),account(id,name,logoUrl,licenseNumber,showAddress,legacyVivarealId,legacyZapId,minisite),medias,accountLink,link)),totalCount)),owners(search(result(listings(listing(displayAddressType,amenities,usableAreas,constructionStatus,listingType,description,title,createdAt,floors,unitTypes,nonActivationReason,providerId,propertyType,unitSubTypes,unitsOnTheFloor,legacyId,id,portal,unitFloor,parkingSpaces,updatedAt,address,suites,publicationType,externalId,bathrooms,usageTypes,totalAreas,advertiserId,advertiserContact,whatsappNumber,bedrooms,acceptExchange,pricingInfos,showPrice,resale,buildings,capacityLimit,status),account(id,name,logoUrl,licenseNumber,showAddress,legacyVivarealId,legacyZapId,minisite),medias,accountLink,link)),totalCount))',
            'cityWiseStreet': '1',
            'developmentsSize': '3',
            'superPremiumSize': '3',
            'addressCountry': '', 'addressState': '', 'addressCity': '', 'addressZone': '',
            'addressNeighborhood': '', 'addressStreet': '', 'addressAccounts': '', 'addressType': '',
            'addressLocationId': '', 'addressPointLat': '', 'addressPointLon': '',
            '__zt': 'ama:b'  # 若此参数失效，从浏览器抓包获取最新值
        }
        return f"{self.base_url}?{urlencode(params)}"

    def parse(self, response):
        # 处理JSON解析异常
        try:
            resp_data = json.loads(response.body)
        except json.JSONDecodeError:
            self.logger.error(f"页面{response.meta.get('current_page')}响应解析失败")
            return

        search_result = resp_data.get('search', {}).get('result', {})
        listings = search_result.get('listings', [])
        total_count = search_result.get('totalCount', 0)
        current_page = response.meta.get('current_page', 1)

        # 提取数据
        for item in listings:
            yield {
                'link': item.get('link', {}).get('href'),
                'city': item.get('link', {}).get('data', {}).get('city')
            }

        # 计算总页数并判断是否继续爬取
        total_pages = (total_count + self.size - 1) // self.size
        if current_page >= total_pages:
            self.logger.info(f"已爬取最后一页{current_page}，停止爬虫")
            raise CloseSpider('无更多页面可爬取')

        # 生成下一页请求
        next_page = current_page + 1
        next_from = (next_page - 1) * self.size
        yield scrapy.Request(
            url=self.build_request_url(next_page, next_from),
            headers={
                'x-domain': 'www.zapimoveis.com.br',
                'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36',
                'Referer': 'https://www.zapimoveis.com.br/'
            },
            callback=self.parse,
            meta={'current_page': next_page}
        )

额外排查建议

检查响应状态码：在parse中打印response.status，若出现403/429，说明被反爬拦截，需添加代理、请求延迟或更新请求头。
验证API参数：用Postman测试下一页URL，确认page和from参数的对应关系是否正确。
动态更新__zt参数：打开浏览器开发者工具抓包正常分页请求，复制最新的__zt参数替换到代码中。

内容的提问来源于stack exchange，提问作者Lucas Guidi

Scrapy分页失效求助：仅能爬取目标站点第一页数据

核心问题分析 #

具体修复方案 #

1. 修正分页参数逻辑 #

2. 安全传递页码 #

3. 正确判断停止时机 #

4. 完善请求头 #

修正后的代码示例 #

额外排查建议 #