使用Python的requests/httpx请求时响应时断时续的问题求助

问题描述 #

抓取Blibli网站商品数据时，使用httpx/requests发送请求，响应成功或失败随机出现，重跑失败请求也无法100%成功，寻求问题原因及解决办法。

目标示例链接 #

urllist=['https://www.blibli.com/backend/product-detail/products/ps--RAM-70107-16912/_summary?defaultItemSku=RAM-70107-16912-00001&pickupPointCode=PP-3239816&cnc=false',
'https://www.blibli.com/backend/product-detail/products/ps--RAM-70107-04124/_summary?defaultItemSku=RAM-70107-04124-00001&pickupPointCode=PP-3239816&cnc=false',
'https://www.blibli.com/backend/product-detail/products/ps--RAM-70107-19598/_summary?defaultItemSku=RAM-70107-19598-00001&pickupPointCode=PP-3239816&cnc=false',
'https://www.blibli.com/backend/product-detail/products/ps--RAM-70107-01115/_summary?defaultItemSku=RAM-70107-01115-00001&pickupPointCode=PP-3206709&cnc=false',
'https://www.blibli.com/backend/product-detail/products/ps--RAM-70107-02620/_summary?defaultItemSku=RAM-70107-02620-00001&pickupPointCode=PP-3239816&cnc=false',
'https://www.blibli.com/backend/product-detail/products/ps--RAM-70107-17552/_summary?defaultItemSku=RAM-70107-17552-00001&pickupPointCode=PP-3023611&cnc=false',
'https://www.blibli.com/backend/product-detail/products/ps--RAM-70107-08377/_summary?defaultItemSku=RAM-70107-08377-00001&pickupPointCode=PP-3239816&cnc=false',
'https://www.blibli.com/backend/product-detail/products/ps--RAM-70107-03274/_summary?defaultItemSku=RAM-70107-03274-00001&pickupPointCode=PP-3069769&cnc=false',
'https://www.blibli.com/backend/product-detail/products/ps--RAM-70107-04345/_summary?defaultItemSku=RAM-70107-04345-00001&pickupPointCode=PP-3239816&cnc=false',
'https://www.blibli.com/backend/product-detail/products/ps--RAM-70107-01030/_summary?defaultItemSku=RAM-70107-01030-00001&pickupPointCode=PP-3239816&cnc=false',
'https://www.blibli.com/backend/product-detail/products/ps--RAM-70107-01534/_summary?defaultItemSku=RAM-70107-01534-00001&pickupPointCode=PP-3239816&cnc=false']

当前代码 #

import pandas as pd
from fake_useragent import UserAgent
import random
from time import sleep, randint
import httpx

# 假设random_ip、id、today是已定义变量
rancherrors=pd.DataFrame()
ranchdf=pd.DataFrame()
for url in urllist:
    try:   
        ua = UserAgent()
        USER_AGENT = ua.random
        headers={
            "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
            "accept-encoding": "gzip, deflate, br",
            "accept-language": "id-ID,id;q=0.9,en-US;q=0.8,en;q=0.7",
            "cache-control": "max-age=0",
            "sec-ch-ua": 'Chromium";v="106", "Google Chrome";v="106", "Not;A=Brand";v="99',
            "sec-ch-ua-platform": "Windows",
            "sec-fetch-dest": "document",
            "sec-fetch-mode": "navigate",
            "sec-fetch-site": "none",
            'X-Forwarded-For': f'{random_ip}',
            "user-agent" : f"{USER_AGENT}",
            'referer':'https://www.blibli.com/'
        }
        response = httpx.get(url, headers=headers)

        try:
            price=str(response.json()['data']['price']['listed']).replace(".0","")
            discount=str(response.json()['data']['price']['totalDiscount'])
        except:
            price="0"
            discount="0"
        try:
            unit=str(response.json()['data']['uniqueSellingPoint']).replace("• ","")
        except:
            unit=""

        dat={
            'product_name':response.json()['data']['name'],
            'normal_price':price,
            'discount':discount,
            'competitor_id':response.json()['data']['ean'],
            'url':url,  # 原代码笔误，将input改为url
            'unit':unit,
            'astro_id':id,
            'date_key':today,
            'web':'ranch market'
        }
        dat=pd.DataFrame([dat])
        ranchdf=pd.concat([ranchdf, dat], ignore_index=True)  # 替换已弃用的append方法
        sleep(randint(2,5))
    except Exception as e:
        datum={
            'id':id,
            'url':url
        }
        datum=pd.DataFrame([datum])
        rancherrors=pd.concat([rancherrors, datum], ignore_index=True)
        print(f'{url} error {e}')

问题分析与解决建议 #

1. 请求头的核心问题 #

• sec-fetch-site设置错误：当前请求是Blibli同域接口，sec-fetch-site应设为same-origin，而非none（仅适用于直接打开的页面），异常值会触发反爬拦截。
• sec-ch-ua格式异常：原代码字符串缺少开头双引号，正确格式应为"Chromium";v="106", "Google Chrome";v="106", "Not;A=Brand";v="99"，格式错误会被识别为异常请求。
• X-Forwarded-For无效：随机生成的IP大概率不符合真实IP规则（如保留段、格式错误），直接去掉该字段，或使用真实有效的印尼代理IP。

2. 反爬规避的优化点 #

• UA策略调整：每次循环初始化UserAgent()易生成重复或非目标地区UA，建议预先准备一批印尼用户常用UA列表，随机选取：

  INDONESIA_UAS = [
      "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36",
      "Mozilla/5.0 (Linux; Android 13; SM-G998B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Mobile Safari/537.36",
      "Mozilla/5.0 (iPhone; CPU iPhone OS 16_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5 Mobile/15E148 Safari/604.1"
  ]
  

• 添加智能重试机制：针对临时网络波动或轻度拦截，实现带指数退避的重试（限制3次以内），避免直接丢弃请求：

  max_retries = 3
  for url in urllist:
      success = False
      for attempt in range(max_retries):
          try:
              # 请求逻辑...
              response.raise_for_status()  # 触发4xx/5xx异常
              success = True
              break
          except Exception as e:
              print(f'Attempt {attempt+1} failed: {e}')
              sleep(randint(3, 7))  # 重试间隔延长
      if not success:
          # 记录错误...
  

• 动态调整请求间隔：固定2-5秒间隔易被识别，改为1-6秒随机范围，或根据响应状态码动态调整（如遇429则延长间隔）。

3. 代码缺陷修复 #

• 处理响应状态码：先判断response.status_code == 200，再解析JSON，避免非成功响应触发解析异常。
• 使用会话维持一致性：用httpx.Client替代单次请求，维持Cookie等会话信息，模拟真实用户访问流程：

  with httpx.Client() as client:
      for url in urllist:
          response = client.get(url, headers=headers)
  

4. 其他建议 #

• 使用印尼代理IP池：频繁拦截时，切换高质量印尼代理IP，避免单一IP被封禁。
• 匹配接口Accept类型：接口返回JSON，将accept设为application/json，更符合真实请求逻辑。
• 检查网站规则：查看Blibli的robots.txt，确认目标接口是否允许抓取，避免违反网站协议。

内容的提问来源于stack exchange，提问作者Hal