Python requests+BeautifulSoup多页爬虫仅抓取单页问题求解

问题原因 #

你的代码有3个核心错误导致只能抓取1页内容：

• 分页URL参数拼接错误：原URL未包含正确的分页参数名，type=simple={page}的写法不符合传参规则，实际站点分页参数应为&page={page}
• 循环缩进完全错误：提取商品链接、抓取商品详情的所有逻辑都写在了分页循环外层，只会处理最后一次分页请求返回的内容
• 存储列表放在分页循环内，每次循环都会清空之前所有已采集的数据

修正后完整代码 #

import requests
from bs4 import BeautifulSoup
import pandas as pd
import time # 增加请求延迟避免被站点反爬拦截

baseurl = 'https://www.tradesupply.eu'
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36'
}

# 存储列表统一在循环外初始化，避免数据被清空
productlinks = []
Title = []
Brand = []
Colour = []
Printer = []
Code = []
Des = []

# 若站点分页从0开始，可将range改为range(0,10)
for page in range(1, 11):
    # 修正分页URL参数格式
    r = requests.get(f'https://www.tradesupply.eu/categories.html?name=drum&type=simple&page={page}', headers=headers)
    if r.status_code != 200:
        print(f'第{page}页请求失败，状态码{r.status_code}')
        continue
    soup = BeautifulSoup(r.content, 'html.parser')
    tra = soup.find_all('li', class_='products__item')
    # 提取当前页商品链接逻辑缩进至分页循环内
    for links in tra:
        for link in links.find_all('a', href=True):
            comp = baseurl + link['href']
            productlinks.append(comp)
    # 分页请求间隔1.5秒
    time.sleep(1.5)

# 遍历所有商品链接抓取详情
for link in productlinks:
    try:
        r = requests.get(link, headers=headers)
        soup = BeautifulSoup(r.content, 'html.parser')
        title = soup.find('h1').text
        Title.append(title)
        brand = soup.find('div', class_="product__brand").text
        Brand.append(brand)
        colour = soup.find('div', class_="product__colour").text 
        Colour.append(colour)
        printer = soup.find('div', class_="product__printer-manufacturer").text
        Printer.append(printer)
        code = soup.find('div', class_='l-product__codeblock').text.strip()
        Code.append(code)
        des = soup.find('div', class_="padding").text.strip()
        Des.append(des)
        time.sleep(1)
    except Exception as e:
        print(f'商品{link}抓取失败：{e}')
        continue

df = pd.DataFrame(
    {"Title": Title, "brand": Brand, "colour": Colour, "printer": Printer, "code": Code, "des": Des}
)
print(df)
# 如需保存数据可取消下一行注释
# df.to_csv('tradesupply_drum.csv', index=False, encoding='utf-8-sig')

额外说明 #

如果运行后发现第一页数据重复或者缺失，可手动访问站点确认分页参数的起始值，调整range的起始数值即可。

内容的提问来源于stack exchange，提问作者Arslan Aziz