Python爬取Kijiji分页仅第一页有结果后续页面无数据如何解决？

核心原因 #

• Kijiji平台反爬机制要求同一会话翻页必须携带首次访问下发的Cookie，你当前每次使用requests.get()发起无状态请求，没有留存Cookie，第二页起请求会被判定为异常访问，返回无有效列表的页面。
• 同一会话下频繁轮换User-Agent属于异常访问特征，进一步提高了反爬识别概率。
• 未设置请求间隔，短时间高频请求触发频率限制。

修复步骤 #

1. 使用requests.Session()维持会话 #

Session会自动管理同一会话下的Cookie、请求头等信息，符合正常用户的浏览逻辑，同一会话固定使用一个User-Agent即可，无需频繁更换。

2. 增加随机请求间隔 #

模拟正常用户浏览行为，避免触发频率限制。

3. 增加返回内容校验 #

请求完成后先校验返回状态和内容有效性，避免直接解析拦截页导致的结果为空。

修改后的核心代码示例 #

import requests, pandas, random, time
from bs4 import BeautifulSoup
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

user_agent_list = [
  'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Safari/605.1.15',
  'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0',
  'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36',
  'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko/20100101 Firefox/77.0',
  'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36',
]

# 初始化会话，配置重试机制
session = requests.Session()
retry = Retry(
    total=5,
    backoff_factor=1,
    status_forcelist=[429, 500, 502, 503, 504]
)
adapter = HTTPAdapter(max_retries=retry)
session.mount("https://", adapter)
session.mount("http://", adapter)
# 同一会话固定UA
session.headers.update({'User-Agent': random.choice(user_agent_list)})

domain = "https://www.kijiji.ca"
base_url = ("https://www.kijiji.ca/b-cars-trucks/ontario/used/")
listCars = [] 

for x in range(2,4):
    new_url = base_url + f"page-{x}" + "/c174l9004a49"
    try:
        # 用会话发起请求
        r = session.get(new_url, timeout=10)
        r.raise_for_status()
        print(f"\nHttps Request Success: FULL PAGE {x}:  {new_url}\n")
    except Exception as e:
        print(f"\nERROR: Can't Get Page {x}, 错误信息：{str(e)}\n")
        continue
    # 增加随机延迟
    time.sleep(random.uniform(1, 3))
    
    page = r.content 
    soup = BeautifulSoup(page, "html.parser")
    rows = soup.find_all("div", {"class": "regular-ad"})
    print(f"{len(rows)} Results on Page {x}.")
    # 调试用：如果返回0条，打印前1000字符确认是否被拦截
    if len(rows) == 0:
        print("拦截页内容预览：", r.text[:1000])
    
    # 后续详情页请求也改用session.get，保持同一会话
    for row in rows:
        carDictionary = {}
        ad_id = row.get("data-listing-id")
        ad_url = domain + row.get("data-vip-url")
        if "kijijiautos.ca" in ad_url:
            print("Skipped Invalid URL Ad")
            continue
        carDictionary["Ad ID"] = ad_id
        carDictionary["Ad URL"] = ad_url

        # 获取价格
        carPriceRaw = row.find("div", {"class": "price"}).text.replace("\n", "").replace(" ","")
        try:
            if carPriceRaw in ("Please Contact", "PleaseContact"):
                carPrice = None
            else:
                carPrice = int(float(carPriceRaw.replace('$', '').replace(',', '')))
            carDictionary["Price"] = carPrice
        except ValueError:
            continue

        # 请求详情页
        try:
            singlePgContent = session.get(ad_url, timeout=5).content
            print(f"Https Request Success: SINGLE AD:  {ad_url}")
        except Exception as e:
            print(f"ERROR: LINK FAILED - SKIPPED: {ad_url}, 错误信息：{str(e)}")
            continue
        time.sleep(random.uniform(0.5, 2))
        
        single_page = BeautifulSoup(singlePgContent, "html.parser")
        # 原有属性解析逻辑保持不变
        try:
            attributes = single_page.find("div", {"class": "attributeListWrapper-2108313769"})
            print("retrieved attributes")
            columns = attributes.find_all("ul", {"class": "itemAttributeList-1090551278"})
            first_col = columns[0]
            second_col = columns[1]
            first_col_attributes = first_col.find_all("li", {"class": "itemAttributeWrapper-37588635"})
            second_col_attributes = second_col.find_all("li", {"class": "itemAttributeWrapper-37588635"})
        except AttributeError:
            print("Could not retrieve attributes")
            continue

        for attribute_li in first_col_attributes:
            try:
                label = attribute_li.find("dl", {"class": "itemAttribute-3080139557"}).find("dt", {"class": "attributeLabel-240934283"}).text
            except:
                continue
            try:
                value = attribute_li.find("dl", {"class": "itemAttribute-3080139557"}).find("dd", {"class": "attributeValue-2574930263"}).text
                if label == "Year":
                    carDictionary["Year"] = int(value)
                elif label == "Make":
                    carDictionary["Make"] = value
                elif label == "Model":
                    carDictionary["Model"] = value
                elif label == "Trim":
                    carDictionary["Trim"] = value
            except (AttributeError, ValueError):
                carDictionary[label] = None

        for attribute_li in second_col_attributes:
            try:
                label = attribute_li.find("dl", {"class": "itemAttribute-3080139557"}).find("dt", {"class": "attributeLabel-240934283"}).text
            except:
                continue
            try:
                value = attribute_li.find("dl", {"class": "itemAttribute-3080139557"}).find("dd", {"class": "attributeValue-2574930263"}).text
                if label == "Body Type":
                    carDictionary["Body"] = value
                elif label == "Transmission":
                    carDictionary["Transmission"] = value
                elif label == "Kilometers":
                    carDictionary["Kilometers"] = int(value.replace(',', ''))
                elif label == "Drivetrain":
                    carDictionary["Drivetrain"] = value
            except (AttributeError, ValueError):
                carDictionary[label] = None

        print(carDictionary)
        listCars.append(carDictionary)
        
df = pandas.DataFrame(listCars)
df.to_csv('cars.csv')
print(df)

额外说明 #

如果修改后仍出现0条结果，可通过打印返回内容确认是否出现验证码，此时可增加代理IP轮换逻辑规避反爬。

内容的提问来源于stack exchange，提问作者Firas Aboushamalah