无法从Scrapy API获取数据：JSONDecodeError及500错误排查求助

问题描述 #

尝试通过API请求获取数据时遇到服务器返回500错误，同时抛出JSONDecodeError异常：

raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 2)

目标爬取URL：https://www.machinerytrader.com/listings/for-sale/excavators/1031
现有代码如下：

import scrapy
import json


class ListingSpider(scrapy.Spider):
    name = 'listing'
    allowed_domains = ['www.machinerytrader.com']
    # start_urls = ['https://www.machinerytrader.com/listings/for-sale/excavators/1031']

    def start_requests(self):
        payload = {
            "Category":"1031",
            "sort": "1",
            "page":"2"
            }
        headers= {
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36",
            "authority": "www.machinerytrader.com",
            "method": "GET",
            "path": "/ajax/listings/ajaxsearch?Category=1031&sort=1&page=2",
            "scheme": "https",
            "accept": "application/json, text/plain, */*",
            "accept-encoding": "gzip, deflate, br",
            "cache-control": "no-cache",
            "content-type": "application/json",
            "cookie": "ASP.NET_SessionId=uircx3p1up0gs3we43zfy3xp; Tracking=SessionStarted=1&UserReferrer=&GUID=75541984850425423381; __RequestVerificationToken=CqqhcuoUxcCh_VvGb2QkZPTMG1sygAxcDjWmGutWxYGvIScO7I1rCwBZabShMlyTl9syCA2; BIGipServerwww.machinery_tradesites_http_pool=545368256.20480.0000; ln_or=eyI0MjU0ODkyIjoiZCJ9; AMP_TOKEN=%24NOT_FOUND; _gid=GA1.2.104780578.1678372555; _fbp=fb.1.1678372557791.1218325782; _hjFirstSeen=1; _hjIncludedInSessionSample_1143836=1; _hjSession_1143836=eyJpZCI6IjU1ZGYyOGJmLWQ4YjktNGU2Mi04NjU2LWUwYmJkYzdiNGMxMSIsImNyZWF0ZWQiOjE2NzgzNzI1NTgzNDcsImluU2FtcGxlIjp0cnVlfQ==; _hjIncludedInPageviewSample=1; _hjAbsoluteSessionInProgress=1; __gads=ID=2a38a4e969861bb1:T=1678372561:S=ALNI_MarB5bgIDdhzpQPECKDmX-70INeJg; __gpi=UID=00000bd5f25a7b4b:T=1678372561:RT=1678372561:S=ALNI_MapbcTx6haLt65wewjezZyeMFVtCw; _hjSessionUser_1143836=eyJpZCI6IjI3ODM2ZDdhLTc0YzUtNTIwMi05YjdhLWYxMmM5YTk4ZGNmNiIsImNyZWF0ZWQiOjE2NzgzNzI1NTgzMzksImV4aXN0aW5nIjp0cnVlfQ==; __atuvc=2%7C10; __atuvs=6409eecd9390ad6d001; Top_PopUp=true; reese84=3:MhdsyFtuLMcDbPfjHYfnUQ==:Xnyj2+4WPTbNOTnv4Aj99+6mLrSjYnrQVoSGqCJEwqmN/gdPfQuCPFYN1/1sInEQHaUvLNdN2VbgdxeC96k6tr1MUSbHd2GxI4AKb1CxnkZfLm63/CXWNqJ/vlS66hOTSsEn+gxPb2l3g2TD3RGi0H4PjyhskjDIE10USkPi3mm83aG/xkAYL4khuWtRDaYzyHjzQ76f9yRr0tNnEEbUPbxZTW7BPXcEF606e6mzq6v5/YEy17JScccw/CCkXb4Uv1tzeNYhkMuFj5V5upY0a2tC/MiJeCACNCYnX9obZhGsfPbL6VUYdJDEhmyR8OBJsHuH4BwOdjnbr7pFG+o4AZKqDHliWKhUnDxGAHIhKwzhhq5TFjeJbqRwSLrMXH54WxZZHcuvRtwr734U2F3Pmf8NqW+zavYdB/aYrk+HpA9LfQQQFBGd/1FNRAM0e8fxZpj5U/DxTKPMdvwK5qBnfzQaTzycDwe80G7QRYX9kf4=:gQlue37nFKz2zVkiSWGW9vURldmXHHEIxHz2yiUrtF8=; _uetsid=b29154b0be8711edabac07f5e20bba65; _uetvid=b2917380be8711edbb2587449262974a; _ga=GA1.2.1777003444.1678372555; UserID=ID=n2K08gLg7XRct%2fxxJeWEnGDwbYpUh6vQ%2fiE1eBN%2f25lkMV4lKXFpeoTT54DrUsz9CriJRnchYL4PfEPzqxaCRA%3d%3d&LV=sHEWMHROf%2fDobQFZDWX1nWtv%2bf2Uk9i6YA9N5Sk0lGE%2fWiDudiekp7MPDIUnH0jGKMx9VZbhLzD4VuT7pKbqepCdPLaN274I; UserSettingsCookie=screenSize=1246|947; _ga_27QWK2FVDW=GS1.1.1678372557.1.1.1678373273.60.0.0",
            "pragma": "no-cache",
            "referer": "https://www.machinerytrader.com/listings/for-sale/excavators/1031",
            "sec-ch-ua-mobile": "?0",
            "sec-ch-ua-platform": "Windows",
            "sec-fetch-dest": "empty",
            "sec-fetch-mode": "cors",
            "sec-fetch-site": "same-origin",
            "x-xsrf-token": "lKwor8adm67mnDJjariTC1-_x2sWvmjxDtVZerZ6p03OwqvVc10YVZUQMmD4-pTv7E2cTSN-8rsTW6ISckmZVgBek66eHw3iFUngI3jYt6h_rwqQ3pI_QxPjYH1us7eHyW27lxFL_-wSS3QC0",
            "sec-ch-ua": '"Google Chrome";v="111", "Not(A:Brand";v="8", "Chromium";v="111"',
            }

        yield scrapy.Request(
                url="https://www.machinerytrader.com/ajax/listings/ajaxsearch?Category=1031&sort=1&page=2",
                method="GET",
                headers=headers,
                body=json.dumps(payload),
                callback=self.parse
                )

    def parse(self, response):
        json_resp = json.loads(response.body)
        products = json_resp['Listings']
        
        yield {
            'DealerLocation': products['DealerLocation'],
            }

问题分析与修复方案 #

1. GET请求携带Body违反HTTP规范 #

HTTP标准中GET请求不应包含请求体，你当前代码用method="GET"同时传递body=json.dumps(payload)，会导致服务器无法解析请求，直接返回500错误。

修复方式：

• 保留GET方法，删除body参数，参数已通过URL查询字符串传递，无需重复提交；
• 若API要求POST，则将method改为POST，保留body参数并确保content-type头正确。

2. Headers包含冗余无效字段 #

手动添加的method、path、scheme属于请求元数据，Scrapy会自动处理，硬编码会导致请求头异常。同时GET请求不需要content-type: application/json头，需删除这些冗余字段。

3. Cookie与X-XSRF-Token已过期 #

代码中硬编码的Cookie和XSRF令牌是临时凭证，过期后服务器会拒绝请求，返回500或403错误。

修复方式：

• 先请求目标页面https://www.machinerytrader.com/listings/for-sale/excavators/1031，从响应中动态提取Cookie和XSRF令牌；
• 利用Scrapy的CookiesMiddleware自动管理会话，无需手动硬编码凭证。

4. Parse方法遍历逻辑错误 #

json_resp['Listings']是列表类型，直接通过products['DealerLocation']取值会抛出KeyError，需遍历列表中的每个元素。

修复后的完整代码 #

import scrapy
import json

class ListingSpider(scrapy.Spider):
    name = 'listing'
    allowed_domains = ['www.machinerytrader.com']
    start_urls = ['https://www.machinerytrader.com/listings/for-sale/excavators/1031']

    def parse(self, response):
        # 从页面中提取XSRF令牌
        xsrf_token = response.css('input[name="__RequestVerificationToken"]::attr(value)').get()
        
        # 构造AJAX请求参数
        ajax_url = "https://www.machinerytrader.com/ajax/listings/ajaxsearch"
        params = {
            "Category": "1031",
            "sort": "1",
            "page": "2"
        }
        headers = {
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36",
            "accept": "application/json, text/plain, */*",
            "referer": response.url,
            "x-xsrf-token": xsrf_token,
            "sec-ch-ua": '"Google Chrome";v="111", "Not(A:Brand";v="8", "Chromium";v="111"',
        }

        yield scrapy.Request(
            url=ajax_url,
            method="GET",
            headers=headers,
            params=params,
            callback=self.parse_listings
        )

    def parse_listings(self, response):
        try:
            json_resp = json.loads(response.body)
            # 遍历Listings列表中的每个商品
            for product in json_resp.get('Listings', []):
                yield {
                    'DealerLocation': product.get('DealerLocation'),
                    # 可添加其他需要的字段
                }
        except json.JSONDecodeError as e:
            self.logger.error(f"JSON解析失败: {str(e)}")
            self.logger.error(f"响应内容预览: {response.text[:500]}")

内容的提问来源于stack exchange，提问作者Raisul Islam