Scrapy爬取英国赛马网API失效，求反爬及URL哈希处理指导

数据老陈

2026-6-17

问题描述

我此前通过Scrapy爬取英国赛马网站www.britishhorseracing.com的赛事（fixture）结果数据，目标链接示例为https://www.britishhorseracing.com/racing/results/fixture-results/#!/2024/702，需要提取每场比赛的距离变化数据，后续还需获取场地状况（going）信息。

在他人协助下，我找到可获取结果数据的API：https://api09.horseracing.software/bha/v1/fixtures/2024/702，并编写了如下Scrapy脚本循环爬取所需场次：

import scrapy
import json


class ApicrawlerSpider(scrapy.Spider):
    name = 'apicrawler'
    allowed_domains = ['britishhorseracing.com']
    start_urls = ['http://britishhorseracing.com/']
    allowed_domains = ['www.britishhorseracing.com']
    urls = 'https://api09.horseracing.software/bha/v1/fixtures?page='
    start_urls = []
    page_number1 = 4214
    while page_number1 <= 4220:
        page_number = str(page_number1)
        start_urls.append('https://api09.horseracing.software/bha/v1/fixtures?page=' + page_number)
        page_number1 +=1
    print(start_urls)
    def parse(self, response):
        data = json.loads(response.body)
        print(data)
        data2 = data['data']
        for data3 in data2:
            fixtureID = data3['fixtureId']
            fixtureYear = data3['fixtureYear']
            base_url = 'https://api09.horseracing.software/bha/v1/fixtures'
            races_url = f'{base_url}/{fixtureYear}/{fixtureID}/races'
            #going_url = f'{base_url}/{fixtureYear}/{fixtureID}/going'
            yield{
                'url' : races_url
            }