Python中解决httpx请求Tokopedia时变量模式超时问题

针对你遇到的硬编码参数正常、变量形式超时的问题，以下是具体修复方案：

问题排查 #

1. 未使用随机User-Agent：代码中生成了ua = UserAgent().random但未在请求头中替换硬编码的UA，固定UA易被反爬识别
2. URL分割逻辑不健壮：用字符串分割提取shopdomain和product_key容易出错，建议用URL解析库处理
3. extParam参数缺失：原URL中的extParam=ivf%3Dfalse&src=topads未传入payload，导致请求参数不完整
4. 未设置超时与异常处理：默认超时时间可能不足以应对反爬验证，缺少异常捕获会导致调试困难

修改后的代码 #

!pip install fake_useragent
!pip install httpx
import httpx
from fake_useragent import UserAgent
from urllib.parse import urlparse, parse_qs

tokopedia = ['https://www.tokopedia.com/samudrasembako/regal-marie-roll-230-gram?extParam=ivf%3Dfalse&src=topads']

for url in tokopedia:
    # 生成随机UA并替换请求头
    ua = UserAgent().random
    
    # 用URL解析库提取参数，避免字符串分割出错
    parsed_url = urlparse(url)
    path_parts = parsed_url.path.strip('/').split('/')
    shopdomain = path_parts[0]
    product_key = path_parts[1]
    
    # 提取原URL中的extParam参数
    query_params = parse_qs(parsed_url.query)
    ext_param = query_params.get('extParam', [''])[0]
    
    payload = {
        "operationName": "PDPGetLayoutQuery",
        "variables": {
            "shopDomain": shopdomain,
            "productKey": product_key,
            "layoutID": "",
            "apiVersion": 1,
            "userLocation": {
                "cityID": "176",
                "addressID": "0",
                "districtID": "2274",
                "postalCode": "",
                "latlon": ""
            },
            "extParam": ext_param  # 传入原URL的extParam
        },
        "query": """fragment ProductVariant on pdpDataProductVariant {\n  errorCode\n  parentID\n  defaultChild\n  sizeChart\n  totalStockFmt\n  variants {\n    productVariantID\n    variantID\n    name\n    identifier\n    option {\n      picture {\n        urlOriginal: url\n        urlThumbnail: url100\n        __typename\n      }\n      productVariantOptionID\n      variantUnitValueID\n      value\n      hex\n      stock\n      __typename\n    }\n    __typename\n  }\n  children {\n    productID\n    price\n    priceFmt\n    optionID\n    optionName\n    productName\n    productURL\n    picture {\n      urlOriginal: url\n      urlThumbnail: url100\n      __typename\n    }\n    stock {\n      stock\n      isBuyable\n      stockWordingHTML\n      minimumOrder\n      maximumOrder\n      __typename\n    }\n    isCOD\n    isWishlist\n    campaignInfo {\n      campaignID\n      campaignType\n      campaignTypeName\n      campaignIdentifier\n      background\n      discountPercentage\n      originalPrice\n      discountPrice\n      stock\n      stockSoldPercentage\n      startDate\n      endDate\n      endDateUnix\n      appLinks\n      isAppsOnly\n      isActive\n      hideGimmick\n      isCheckImei\n      minOrder\n      __typename\n    }\n    thematicCampaign {\n      additionalInfo\n      background\n      campaignName\n      icon\n      __typename\n    }\n    __typename\n  }\n  __typename\n}\n\nfragment ProductMedia on pdpDataProductMedia {\n  media {\n    type\n    urlOriginal: URLOriginal\n    urlThumbnail: URLThumbnail\n    urlMaxRes: URLMaxRes\n    videoUrl: videoURLAndroid\n    prefix\n    suffix\n    description\n    variantOptionID\n    __typename\n  }\n  videos {\n    source\n    url\n    __typename\n  }\n  __typename\n}\n\nfragment ProductHighlight on pdpDataProductContent {\n  name\n  price {\n    value\n    currency\n    __typename\n  }\n  campaign {\n    campaignID\n    campaignType\n    campaignTypeName\n    campaignIdentifier\n    background\n    percentageAmount\n    originalPrice\n    discountedPrice\n    originalStock\n    stock\n    stockSoldPercentage\n    threshold\n    startDate\n    endDate\n    endDateUnix\n    appLinks\n    isAppsOnly\n    isActive\n    hideGimmick\n    __typename\n  }\n  thematicCampaign {\n    additionalInfo\n    background\n    campaignName\n    icon\n    __typename\n  }\n  stock {\n    useStock\n    value\n    stockWording\n    __typename\n  }\n  variant {\n    isVariant\n    parentID\n    __typename\n  }\n  wholesale {\n    minQty\n    price {\n      value\n      currency\n      __typename\n    }\n    __typename\n  }\n  isCashback {\n    percentage\n    __typename\n  }\n  isTradeIn\n  isOS\n  isPowerMerchant\n  isWishlist\n  isCOD\n  isFreeOngkir {\n    isActive\n    __typename\n  }\n  preorder {\n    duration\n    timeUnit\n    isActive\n    preorderInDays\n    __typename\n  }\n  __typename\n}\n\nfragment ProductCustomInfo on pdpDataCustomInfo {\n  icon\n  title\n  isApplink\n  applink\n  separator\n  description\n  __typename\n}\n\nfragment ProductInfo on pdpDataProductInfo {\n  row\n  content {\n    title\n    subtitle\n    applink\n    __typename\n  }\n  __typename\n}\n\nfragment ProductDetail on pdpDataProductDetail {\n  content {\n    title\n    subtitle\n    applink\n    showAtFront\n    isAnnotation\n    __typename\n  }\n  __typename\n}\n\nfragment ProductDataInfo on pdpDataInfo {\n  icon\n  title\n  isApplink\n  applink\n  content {\n    icon\n    text\n    __typename\n  }\n  __typename\n}\n\nfragment ProductSocial on pdpDataSocialProof {\n  row\n  content {\n    icon\n    title\n    subtitle\n    applink\n    type\n    rating\n    __typename\n  }\n  __typename\n}\n\nquery PDPGetLayoutQuery($shopDomain: String, $productKey: String, $layoutID: String, $apiVersion: Float, $userLocation: pdpUserLocation, $extParam: String) {\n  pdpGetLayout(shopDomain: $shopDomain, productKey: $productKey, layoutID: $layoutID, apiVersion: $apiVersion, userLocation: $userLocation, extParam: $extParam) {\n    requestID\n    name\n    pdpSession\n    basicInfo {\n      alias\n      createdAt\n      isQA\n      id: productID\n      shopID\n      shopName\n      minOrder\n      maxOrder\n      weight\n      weightUnit\n      condition\n      status\n      url\n      needPrescription\n      catalogID\n      isLeasing\n      isBlacklisted\n      menu {\n        id\n        name\n        url\n        __typename\n      }\n      category {\n        id\n        name\n        title\n        breadcrumbURL\n        isAdult\n        isKyc\n        minAge\n        detail {\n          id\n          name\n          breadcrumbURL\n          isAdult\n          __typename\n        }\n        __typename\n      }\n      txStats {\n        transactionSuccess\n        transactionReject\n        countSold\n        paymentVerified\n        itemSoldFmt\n        __typename\n      }\n      stats {\n        countView\n        countReview\n        countTalk\n        rating\n        __typename\n      }\n      __typename\n    }\n    components {\n      name\n      type\n      position\n      data {\n        ...ProductMedia\n        ...ProductHighlight\n        ...ProductInfo\n        ...ProductDetail\n        ...ProductSocial\n        ...ProductDataInfo\n        ...ProductCustomInfo\n        ...ProductVariant\n        __typename\n      }\n      __typename\n    }\n    __typename\n  }\n}\n"""
    }
    
    headers = {
        'origin': 'https://www.tokopedia.com',
        'referer': url,
        'sec-ch-ua': '"Not?A_Brand";v="8", "Chromium";v="108", "Google Chrome";v="108"',
        'sec-ch-ua-mobile': '?0',
        'sec-ch-ua-platform': "Windows",
        'sec-fetch-dest': 'empty',
        'sec-fetch-mode': 'cors',
        'sec-fetch-site': 'same-site',
        'user-agent': ua,  # 使用随机生成的UA
        'x-device': 'desktop',
        'x-source': 'tokopedia-lite',
        'x-tkpd-akamai': 'pdpGetLayout',
        'x-tkpd-lite-service': 'zeus',
        'x-version': '53ac990'
    }
    
    # 设置超时时间，添加异常捕获
    try:
        with httpx.Client(timeout=15.0) as client:
            resp = client.post(
                "https://gql.tokopedia.com/graphql/PDPGetLayoutQuery",
                json=payload,
                headers=headers
            )
            resp.raise_for_status()  # 抛出HTTP错误
            print(resp.json())
    except httpx.HTTPError as e:
        print(f"请求出错: {e}")
    except Exception as e:
        print(f"其他错误: {e}")

关键修改说明 #

1. 使用URL解析库：用urllib.parse的urlparse和parse_qs替代字符串分割，确保shopdomain、product_key和extParam提取准确
2. 启用随机UA：将生成的随机User-Agent替换到请求头中，降低被反爬识别的概率
3. 补充extParam参数：把原URL中的extParam值传入payload，保证请求参数完整性
4. 添加超时与异常处理：设置15秒超时时间，捕获HTTP错误和其他异常，便于调试
5. 使用上下文管理器：用with httpx.Client()确保客户端正确关闭，避免资源泄漏

内容的提问来源于stack exchange，提问作者Hal