为何我的网页爬虫脚本仅抓取到H&M连衣裙页面1000个URL中的500个？

• 需求：获取H&M美国站女装连衣裙页面的全部商品列表URL
• 现状：已编写代码成功抓取前500个URL，但设置page-size为商品总数（约1000）时，仅返回前500条内容，剩余约500个无法获取
• 疑问：推测网站对单请求返回数量有限制，需要拆分请求分批抓取，但不知道如何实现

import requests
from bs4 import BeautifulSoup
import os
import numpy as np
import json
import re
from ast import literal_eval
import requests
import matplotlib.pyplot as plt
from PIL import Image
from io import BytesIO
import random

headers = headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0"
}

base_url = "https://www2.hm.com/en_us/women/products/dresses.html"

response = requests.get(base_url, headers=headers)
soup = BeautifulSoup(response.text, 'lxml')
pagination_element = soup.find('div', class_='filter-pagination')
items_text = pagination_element.get_text(strip=True)
num_listings = int(items_text.split()[0])


params = {
    "sort": "stock",
    "image-size": "small",
    "image": "model",
    "offset": 0,
    "page-size": num_listings
}

response = requests.get(base_url, params=params, headers=headers)
soup = BeautifulSoup(response.content, "lxml")

listing_urls = [a["href"] for a in soup.select(".item-link")]

print(f'Listings collected: {len(listing_urls)}')
print(f"Total number of listings: {num_listings}")
print(f'Listing URLs 1-5: {listing_urls[:5]}')

H&M页面接口限制了单请求最多返回500条商品数据，因此需要通过**分批请求+偏移量（offset）**来获取全部内容。具体实现逻辑：

• 每次请求固定page-size=500
• 通过offset参数控制起始位置，每次递增500
• 循环请求直到获取的商品总数达到或超过页面显示的总数量

修改后的代码（移除了无关导入，添加分批请求逻辑）：

import requests
from bs4 import BeautifulSoup

headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0"
}

base_url = "https://www2.hm.com/en_us/women/products/dresses.html"

# 获取总商品数量
response = requests.get(base_url, headers=headers)
soup = BeautifulSoup(response.text, 'lxml')
pagination_element = soup.find('div', class_='filter-pagination')
items_text = pagination_element.get_text(strip=True)
num_listings = int(items_text.split()[0])

listing_urls = []
batch_size = 500  # 单请求最大返回数量
offset = 0

while offset < num_listings:
    params = {
        "sort": "stock",
        "image-size": "small",
        "image": "model",
        "offset": offset,
        "page-size": batch_size
    }
    response = requests.get(base_url, params=params, headers=headers)
    soup = BeautifulSoup(response.content, "lxml")
    # 收集当前批次的商品URL
    batch_urls = [a["href"] for a in soup.select(".item-link")]
    listing_urls.extend(batch_urls)
    
    # 更新偏移量
    offset += batch_size
    # 可选：添加短暂延迟，避免请求过于频繁被限制
    # import time
    # time.sleep(1)

print(f'Listings collected: {len(listing_urls)}')
print(f"Total number of listings: {num_listings}")
print(f'Listing URLs 1-5: {listing_urls[:5]}')

关键说明 #

1. 分批请求：通过offset和batch_size（固定500）的组合，每次请求获取一段范围的商品数据
2. 循环终止条件：当offset超过总商品数量时停止循环，确保覆盖所有商品
3. 反爬优化：可以添加短暂的请求延迟（如time.sleep(1)），降低被网站反爬机制限制的风险
4. 代码精简：移除了原代码中未使用的库（如numpy、matplotlib等），减少冗余

内容的提问来源于stack exchange，提问作者Danny_webb