Scrapy无表单场景下如何构造有效POST请求获取筛选后站点数据

核心错误原因 #

• POST请求地址错误，你调用的是搜索展示页地址，实际提交搜索的接口是https://www.guidestar.org/search/SubmitSearch
• 缺少AJAX请求必需的标识头，站点后端会校验是否为异步请求，无对应标识会直接返回原始页面
• 未提前获取站点会话Cookie，无有效会话的搜索请求会被拦截
• 手动编码表单参数容易出现转义错误

修正后代码 #

import scrapy
from scrapy.http import FormRequest
from scrapy.utils.response import open_in_browser


class NonprofitSpider(scrapy.Spider):
    name = 'nonprofit'
    # 先请求搜索首页获取会话Cookie
    start_urls = ['https://www.guidestar.org/search']

    def parse(self, response):
        # 正确的搜索提交接口地址
        url = 'https://www.guidestar.org/search/SubmitSearch'

        headers = {
            'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
            # 必需的AJAX请求标识
            'X-Requested-With': 'XMLHttpRequest',
            # 必需的来源页标识
            'Referer': 'https://www.guidestar.org/search'
        }

        data = {
            "CurrentPage": "1",
            "SearchType": "org",
            "GroupExemption": "",
            "AffiliateOrgName": "",
            "RelatedOrgName": "",
            "RelatedOrgEin": "",
            "RelationType": "",
            "RelatedOrgs": "",
            "SelectedCityNav[]": "",
            "SelectedCountyNav[]": "",
            "Eins": "",
            "ul": "",
            "PCSSubjectCodes[]": "",
            "PeoplePCSSubjectCodes[]": "",
            "PCSPopulationCodes[]": "",
            "AutoSelectTaxonomyFacet": "",
            "AutoSelectTaxonomyText": "",
            "Keywords": "",
            "State": "Alaska",
            "City": "",
            "PeopleZip": "",
            "PeopleZipRadius": "Zip+Only",
            "PeopleCity": "",
            "PeopleRevenueRangeLow": "$0",
            "PeopleRevenueRangeHigh": "max",
            "PeopleAssetsRangeLow": "$0",
            "PeopleAssetsRangeHigh": "max",
            "Sort": ""
        }
        # 用FormRequest自动处理表单编码，自动携带当前会话的Cookie
        return FormRequest(
            url=url,
            headers=headers,
            formdata=data,
            callback=self.start
        )

    def start(self, response):
        print('response status:', response.status)
        open_in_browser(response)
        # 后续可以加解析逻辑

额外注意事项 #

• 如果后续需要翻页，只需要修改参数里的CurrentPage值即可
• 如果请求被拦截，可以在settings.py里配置USER_AGENT为普通浏览器标识，避免被反爬识别
• 该站点可能有请求频率限制，建议添加下载延迟避免被封IP

内容的提问来源于stack exchange，提问作者Albert