使用Requests请求PCComponentes AJAX接口返回403但浏览器可正常访问，如何解决？

问题背景 #

针对带「加载更多」按钮的动态商品页，我找到了目标站点的AJAX加载接口，计划直接调用该接口采集多页商品信息，原有实现代码如下：

import pandas as pd
import requests
from bs4 import BeautifulSoup
import datetime
from time import sleep
import random
from Scrapingtools import joinfiles
from Scrapingtools import uploadfiles

url_list = [
     'https://www.pccomponentes.com/listado/ajax?page=3&order=relevance&gtmTitle=Procesadores%20PC&idFamilies%5B%5D=4',
    'https://www.pccomponentes.com/listado/ajax?page=1&order=relevance&gtmTitle=Placas%20Base&idFamilies%5B%5D=3'
    ]

df_list =[] 
store = 'PCComponentes'
extraction_date = datetime.datetime.today().replace(microsecond=0)

for url in url_list:
    for x in range(0,1):
        headers = ({'user-agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36"})
        r = requests.get(url, headers=headers)
        print()
        soup = BeautifulSoup(r.content,'html.parser')
        items = soup.find_all('div',class_='col-xs-6 col-sm-4 col-md-4 col-lg-4')
        print('Response:'+str(r.status_code)+'.Found '+str(len(items))+' items in '+url + str(x))
        
        for item in items:
            product_name = item.find('h3',class_ = 'c-product-card__title').text.strip()
            try:
                price = item.find('div', class_ = 'c-product-card__prices-actual cy-product-price-normal').text[:-1]
            except AttributeError:
                price = item.find('div', class_ = 'c-product-card__prices-actual c-product-card__prices-actual--discount cy-product-price-discount').text[:-1]
            try:
                old_price = item.find('div',class_ = 'c-product-card__prices-pvp cy-product-price-normal').text[:-1]
            except AttributeError:
                old_price = "No discount"
            try:
                rating = item.find('span',class_ = 'c-star-rating__text cy-product-text').text.strip()
            except AttributeError:
                rating = ""
            try:
                reviews = item.find('span',class_ = 'c-star-rating__text cy-product-rating-result').text.strip()
            except AttributeError:
                reviews = ""
            try:
                brand = item.find('article')['data-brand'] 
            except AttributeError:
                brand = "No brand"
            try:
                category = item.find('article')['data-category']
            except AttributeError:
                category = "No category"
                   
            product_info =  {
                'product_name' : product_name,
                'price' : price,
                'old_price' : old_price,
                'rating' : rating,
                'reviews' : reviews,
                'brand' : brand,
                'category' : category,
                'store' : store,
                'date_extraction' : extraction_date,
            }
            df_list.append(product_info)
            
    sleep(random.uniform(3.5, 7.5))

df = pd.DataFrame(df_list)

遇到的核心问题 #

即使修改了User-Agent请求头，调用接口依然返回403错误，测试代码如下：

import requests
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.76 Safari/537.36'}
response = requests.get('https://www.pccomponentes.com/listado/ajax?page=3&order=relevance&gtmTitle=Procesadores%20PC&idFamilies%5B%5D=4', headers=headers)
print(response.status_code)
print(response.url)

但该接口链接直接在浏览器地址栏粘贴可以正常返回商品数据，需要优先解决403问题，再处理分页迭代逻辑。

403错误解决方案 #

• 补全请求头字段：仅带User-Agent不足以通过站点校验，需要补充Referer、Accept、Accept-Language、Cookie等浏览器请求时携带的完整字段，Referer字段填目标站点的商品列表页地址即可，Cookie可以先通过浏览器访问一次页面后复制到请求头中测试，后续可优化为动态获取。
• 改用会话保持请求：使用requests.Session()创建会话，先访问一次商品列表主页，让会话自动保存站点返回的Cookie，再发起AJAX接口请求，模拟正常用户的访问顺序，不要直接调用AJAX接口。示例实现如下：

import requests

session = requests.Session()
# 先访问主页获取Cookie
home_headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
    "Accept-Language": "es-ES,es;q=0.8,en-US;q=0.5,en;q=0.3"
}
session.get("https://www.pccomponentes.com/procesadores", headers=home_headers)

# 再请求AJAX接口
ajax_headers = home_headers.copy()
ajax_headers["Referer"] = "https://www.pccomponentes.com/procesadores"
ajax_headers["Accept"] = "*/*"
response = session.get("https://www.pccomponentes.com/listado/ajax?page=3&order=relevance&gtmTitle=Procesadores%20PC&idFamilies%5B%5D=4", headers=ajax_headers)
print(response.status_code)

• 调整请求频率：如果还是出现403，可将请求间隔调整到5-10秒，必要时使用代理IP轮换避免IP被封禁。
• 备选方案：如果以上方案都不生效，可以使用Playwright、Selenium等无头浏览器工具，模拟用户点击「加载更多」按钮获取数据，完全模拟浏览器行为，基本可以规避403问题，仅性能略低于直接请求接口。

分页逻辑优化建议 #

无需写死url列表，可以先通过第一页请求获取总页数，然后循环修改URL中的page参数即可实现多页采集。

内容的提问来源于stack exchange，提问作者CMonte2