使用Scrapy爬取Hepsiburada时遭遇JSONDecodeError求助

三天前还能正常爬取Hepsiburada站点，现在持续出现JSONDecodeError。站点返回Scrapy 200响应，已更换代理和User-Agent，问题依旧。查看源码发现站点JSON格式变更，每个请求会生成唯一UUID作为键（示例格式如下），原解析代码无法适配新结构：

window.MORIA.PRODUCTLIST = Object.assign(window.MORIA.PRODUCTLIST || {}, {
'60cada8e-57dd-466e-f7af-62efca4fa8a8': {

报错信息 #

File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

原解析代码 #

import scrapy
import json
import datetime
import bs4
import re
import time
from requests.models import PreparedRequest
import logging
from hepsibura_spider.items import HepsiburaSpiderItem
from scrapy.crawler import CrawlerProcess

class HepsiburaSpider(scrapy.Spider):
    name = 'hepsibura'
    # allowed_domains = ['www.hepsibura.com']
    handle_httpstatus_list = [301]
    def start_requests(self):
        urls = [
            'https://www.hepsiburada.com/monitor-bilgisayarlar-c-116465?filtreler=satici:Hepsiburada;?_random_number={rn}#tabIndex=0',
            
        ]
        for url in urls:
            params = []
            # added a meta to provide the used url here
            main_url, parameters = url.split('&') if '&' in url else url, None
            parameters = parameters.split(':') if parameters else []
            for parameter in parameters:
                key, value = parameter.split('=')
                params.append((key.strip(), value.strip()))

            # params.append(('main_url', main_url))

            if 'sayfa' not in dict(params):
                params.append(('sayfa', '1'))

            yield scrapy.Request(
                url=url.format(rn=time.time()),
                callback=self.parse_json,
                meta={
                    'main_url': main_url,
                    'params': dict(params),
                },
                headers={
                    'Cache-Control': 'store, no-cache, must-revalidate, post-check=0, pre-check=0',
                    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 11_10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.5134.152 Safari/537.36',
                }
            )
    
    def parse_json(self, response):

        if response.status == 301:
            logging.log(logging.INFO, 'Finished scraping')
            return
        current_url = response.request.url.split('&')[0].strip()
        parameters = response.meta.get('params')

        soup = bs4.BeautifulSoup(response.text,'lxml')
        scripts = soup.select('script')
        data_script = ''
        for script in scripts:
            # print(script.text)
            if 'window.MORIA.PRODUCTLIST = {' in str(script):
                print('Found the data')
                data_script = str(script)
                break
        
        data_script = data_script.replace('<script type="text/javascript">','').replace('window.MORIA = window.MORIA || {};','').replace('window.MORIA.PRODUCTLIST = {','').replace('\'STATE\': ', '').replace('</script>','')[:-4]
        json_data = json.loads(data_script)
        products = json_data['data']['products']
        for product in products:
            item = HepsiburaSpiderItem()

            item['rowid'] = hash(str(datetime.datetime.now()) + str(product['productId']))
            item['date'] = str(datetime.datetime.now())
            item['listing_id'] = product['variantList'][0]["listing"]["listingId"]
            item['product_id'] = product['variantList'][0]["sku"].lower()
            item['product_name'] = product['variantList'][0]['name']
            item['price'] = float(product['variantList'][0]['listing']['priceInfo']['price'])
            item['url'] = 'https://www.hepsiburada.com' + product['variantList'][0]["url"]
            item['merchantName'] = product['variantList'][0]["listing"]["merchantName"].lower()

            yield item
        
        parameters['sayfa'] = int(parameters['sayfa']) + 1
        req = PreparedRequest()
        req.prepare_url(current_url, parameters)

        yield scrapy.Request(
            url=req.url,
            callback=self.parse_json,
            meta={
                'params': parameters,
            },
            headers={
                'Cache-Control': 'store, no-cache, must-revalidate, post-check=0, pre-check=0',
                'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 11_10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.5134.152 Safari/537.36',
            }
        )


if __name__ == '__main__':
    process = CrawlerProcess()
    process.crawl(HepsiburaSpider)
    process.start()

解决方案 #

站点现在用Object.assign赋值window.MORIA.PRODUCTLIST，新增随机UUID作为数据键，原代码的硬编码字符串替换逻辑完全失效。调整解析逻辑如下：

修改后的parse_json方法 #

def parse_json(self, response):
    if response.status == 301:
        logging.log(logging.INFO, 'Finished scraping')
        return
    current_url = response.request.url.split('&')[0].strip()
    parameters = response.meta.get('params')

    soup = bs4.BeautifulSoup(response.text,'lxml')
    scripts = soup.select('script')
    data_script = ''
    for script in scripts:
        script_text = script.get_text(strip=True)
        if 'window.MORIA.PRODUCTLIST = Object.assign' in script_text:
            data_script = script_text
            break
    
    # 正则提取Object.assign中的目标JSON对象
    match = re.search(r'Object.assign\(.*?,\s*({.*?})\);', data_script, re.DOTALL)
    if not match:
        logging.error("Failed to extract product data")
        return
    
    raw_json_str = match.group(1)
    # 修复HTML转义的双引号
    raw_json_str = raw_json_str.replace('"', '"')
    # 解析JSON并获取UUID对应的嵌套数据
    json_data = json.loads(raw_json_str)
    state_data = next(iter(json_data.values()))
    products = state_data['data']['products']
    
    for product in products:
        item = HepsiburaSpiderItem()
        item['rowid'] = hash(str(datetime.datetime.now()) + str(product['productId']))
        item['date'] = str(datetime.datetime.now())
        item['listing_id'] = product['variantList'][0]["listing"]["listingId"]
        item['product_id'] = product['variantList'][0]["sku"].lower()
        item['product_name'] = product['variantList'][0]['name']
        item['price'] = float(product['variantList'][0]['listing']['priceInfo']['price'])
        item['url'] = 'https://www.hepsiburada.com' + product['variantList'][0]["url"]
        item['merchantName'] = product['variantList'][0]["listing"]["merchantName"].lower()
        yield item
    
    parameters['sayfa'] = int(parameters['sayfa']) + 1
    req = PreparedRequest()
    req.prepare_url(current_url, parameters)

    yield scrapy.Request(
        url=req.url,
        callback=self.parse_json,
        meta={
            'params': parameters,
        },
        headers={
            'Cache-Control': 'store, no-cache, must-revalidate, post-check=0, pre-check=0',
            'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 11_10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.5134.152 Safari/537.36',
        }
    )

关键改动说明 #

1. 正则精准提取数据：用re.DOTALL模式匹配Object.assign中的JSON对象，避免原代码硬编码替换的脆弱性。
2. 适配UUID键：通过next(iter(json_data.values()))直接获取随机UUID对应的嵌套数据，无需关心具体UUID值。
3. 修复转义问题：将"转义回正常双引号，确保JSON解析正常。

内容的提问来源于stack exchange，提问作者cevreyolu