需前置打开站点的接口数据爬取问题（Scrapy实现）

Hey there! Let's figure out why your Scrapy code is returning empty responses and fix it up.

The Root Cause #

That API endpoint you're targeting isn't just accessible by hitting it directly—it relies on session context (like cookies) or required request headers that your browser automatically gets when you first visit the main site. When you skip visiting the main site and jump straight to the API, the server doesn't recognize your request as a valid, browser-like session, so it returns an empty response.

Fixes to Try #

Here are a few practical approaches to get the data you need:

1. First Visit the Main Site to Grab Session Cookies

Scrapy automatically persists cookies across requests, so start by hitting the main site first, then make your API request. This mimics how a browser works:

import scrapy

class AaidSpider(scrapy.Spider):
    name = 'agm'
    # Start with the main site to establish a valid session
    start_urls = ['https://www.agmgranite.com/']

    def parse(self, response):
        # Now we have the site's cookies—construct the API request
        api_url = 'https://www.agmgranite.com/paginate.php?page=1&lid=3&f=reset&invp='
        yield scrapy.Request(api_url, callback=self.parse_api)

    def parse_api(self, response):
        # Process the API response here
        self.logger.info("API Response: %s", response.body.decode('utf-8'))
        # If it's JSON data, parse it like this:
        # import json
        # data = json.loads(response.body)
        # for item in data:
        #     yield item

2. Add Required Request Headers

Sometimes the server checks for headers like Referer (to confirm the request came from the main site) or a valid User-Agent. Add these to your API request:

import scrapy

class AaidSpider(scrapy.Spider):
    name = 'agm'
    start_urls = ['https://www.agmgranite.com/']

    def parse(self, response):
        api_url = 'https://www.agmgranite.com/paginate.php?page=1&lid=3&f=reset&invp='
        # Mimic browser headers
        headers = {
            'Referer': 'https://www.agmgranite.com/',
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36'
        }
        yield scrapy.Request(api_url, headers=headers, callback=self.parse_api)

    def parse_api(self, response):
        print(response.body.decode('utf-8'))

3. Check for Dynamic Parameters

If the above still doesn't work, open your browser's DevTools again and inspect the API request closely:

• Are there any hidden parameters (like a token or csrf value) that are generated on the main site? If so, you'll need to extract that value from the main site's HTML and include it in your API request.
• Verify the request method (GET/POST)—your code uses GET, but maybe the endpoint expects POST with form data.

Debugging Tips #

• Use scrapy shell to test requests interactively: first run scrapy shell https://www.agmgranite.com/, then run fetch("https://www.agmgranite.com/paginate.php?page=1&lid=3&f=reset&invp=") to see the response.
• Check Scrapy logs (run scrapy crawl agm --log-level=DEBUG) to look for HTTP status codes (like 403 Forbidden) which indicate your request is being blocked.

内容的提问来源于stack exchange，提问作者João Vitor Fontes