Steam社区市场分页爬虫重复数据问题求助

我是首次使用Stack Overflow，表述若有疏漏请见谅。我编写了Python3脚本用于爬取Steam社区市场的图标、名称与价格，数据提取和格式化逻辑正常。但由于网站采用分页机制，我通过for循环将页码插入URL发起多轮GET请求时，发现存储数据的数组中有90%内容完全重复（例如第2页的内容被重复添加7次），不清楚该如何修复以获取正确的请求数据。

源码 #

import requests
from bs4 import BeautifulSoup
import time
import json as json

def main():

    name_arr = []
    img_arr = []
    price_arr = []

    for i in range(1,11): # later change to 169 pages
        
        url = f"https://steamcommunity.com/market/search?q=&category_730_ItemSet%5B%5D=any&category_730_ProPlayer%5B%5D=any&category_730_StickerCapsule%5B%5D=any&category_730_TournamentTeam%5B%5D=any&category_730_Weapon%5B%5D=any&category_730_Exterior%5B%5D=tag_WearCategory2&category_730_Quality%5B%5D=tag_normal&category_730_Quality%5B%5D=tag_unusual&appid=730#p{i}_popular_desc"
        print(url)
        r = requests.get(url)

        print("----------------------------------- on : " + str(i) + "right now")
        print(r.status_code)

        soup = BeautifulSoup(r.content, "html.parser")

        images = soup.find_all("img", class_="market_listing_item_img")
        names = soup.find_all("span", class_="market_listing_item_name")
        prices = soup.find_all("span", class_="sale_price")

        def extract_text(list, list_arr):
            for x in list:
                name_only = x.text.replace("(Field-Tested)", "").strip()
                list_arr.append(name_only)

        def extract_src(list, list_arr):
            for x in list:
                list_arr.append(x["src"])

        extract_text(names, name_arr)
        extract_text(prices,price_arr)
        extract_src(images, img_arr)

        time.sleep(60)

    print(name_arr)
    print(price_arr)
    print(img_arr)

    with open('output.json', 'w') as f:
    # Write the array to file as JSON
        json.dump(name_arr, f)

    # amount = float(dollars.replace("$", "").strip()) 

if __name__ == "__main__":
    main()

终端输出 #

❯ python3 webscrape.py

['P90 | Blind Spot', 'SCAR-20 | Cardiac', 'Five-SeveN | Contractor', 'PP-Bizon | Forest Leaves', 'XM1014 | Urban Perforated', 'Sawed-Off | Irradiated Alert', 'SG 553 | Tornado', 'P250 | Mehndi', 'FAMAS | Commemoration', 'XM1014 | Blaze Orange', 'P90 | Blind Spot', 'SCAR-20 | Cardiac', 'Five-SeveN | Contractor', 'PP-Bizon | Forest Leaves', 'XM1014 | Urban Perforated', 'Sawed-Off | Irradiated Alert', 'SG 553 | Tornado', 'P250 | Mehndi', 'FAMAS | Commemoration', 'XM1014 | Blaze Orange', 'P90 | Blind Spot', 'SCAR-20 | Cardiac', 'Five-SeveN | Contractor', 'PP-Bizon | Forest Leaves', 'XM1014 | Urban Perforated', 'Sawed-Off | Irradiated Alert', 'SG 553 | Tornado', 'P250 | Mehndi', 'FAMAS | Commemoration', 'XM1014 | Blaze Orange', 'Sawed-Off | Highwayman', 'Galil AR | Shattered', 'AUG | Torque', 'SG 553 | Tornado', 'Dual Berettas | Briar', 'SG 553 | Wave Spray', 'Five-SeveN | Kami', 'FAMAS | Contrast Spray', 'MAG-7 | Chainmail', 'Sawed-Off | Serenity', 'P90 | Blind Spot', 'SCAR-20 | Cardiac', 'Five-SeveN | Contractor', 'PP-Bizon | Forest Leaves', 'XM1014 | Urban Perforated', 'Sawed-Off | Irradiated Alert', 'SG 553 | Tornado', 'P250 | Mehndi', 'FAMAS | Commemoration', 'XM1014 | Blaze Orange', 'P90 | Blind Spot', 'SCAR-20 | Cardiac', 'Five-SeveN | Contractor', 'PP-Bizon | Forest Leaves', 'XM1014 | Urban Perforated', 'Sawed-Off | Irradiated Alert', 'SG 553 | Tornado', 'P250 | Mehndi', 'FAMAS | Commemoration', 'XM1014 | Blaze Orange', 'P90 | Blind Spot', 'SCAR-20 | Cardiac', 'Five-SeveN | Contractor', 'PP-Bizon | Forest Leaves', 'XM1014 | Urban Perforated', 'Sawed-Off | Irradiated Alert', 'SG 553 | Tornado', 'P250 | Mehndi', 'FAMAS | Commemoration', 'XM1014 | Blaze Orange', 'Sawed-Off | Highwayman', 'Galil AR | Shattered', 'AUG | Torque', 'SG 553 | Tornado', 'Dual Berettas | Briar', 'SG 553 | Wave Spray', 'Five-SeveN | Kami', 'FAMAS | Contrast Spray', 'MAG-7 | Chainmail', 'Sawed-Off | Serenity', 'Sawed-Off | Highwayman', 'Galil AR | Shattered', 'AUG | Torque', 'SG 553 | Tornado', 'Dual Berettas | Briar', 'SG 553 | Wave Spray', 'Five-SeveN | Kami', 'FAMAS | Contrast Spray', 'MAG-7 | Chainmail', 'Sawed-Off | Serenity', 'P90 | Blind Spot', 'SCAR-20 | Cardiac', 'Five-SeveN | Contractor', 'PP-Bizon | Forest Leaves', 'XM1014 | Urban Perforated', 'Sawed-Off | Irradiated Alert', 'SG 553 | Tornado', 'P250 | Mehndi', 'FAMAS | Commemoration', 'XM1014 | Blaze Orange']

核心问题：URL锚点无效 #

你当前URL中的#p{i}_popular_desc是浏览器端的锚点，服务器不会处理这个部分，所以每次requests.get请求返回的都是第1页的数据，导致数组重复添加相同内容。Steam市场的分页参数是通过URL查询参数page传递的，同时排序规则也要用查询参数指定。

修复步骤 #

1. 修正URL构造
   将原来的锚点替换为查询参数，正确的URL格式如下：

   url = f"https://steamcommunity.com/market/search?q=&category_730_ItemSet%5B%5D=any&category_730_ProPlayer%5B%5D=any&category_730_StickerCapsule%5B%5D=any&category_730_TournamentTeam%5B%5D=any&category_730_Weapon%5B%5D=any&category_730_Exterior%5B%5D=tag_WearCategory2&category_730_Quality%5B%5D=tag_normal&category_730_Quality%5B%5D=tag_unusual&appid=730&page={i}&sort_column=popular&sort_dir=desc"
   

   这里把锚点的p{i}_popular_desc拆分成三个查询参数：

   • page={i}：指定当前页码
   • sort_column=popular：按热度排序
   • sort_dir=desc：降序排列

2. 优化代码规范

   • 把extract_text和extract_src函数移到main函数外部，避免在循环中重复定义函数
   • 添加请求头User-Agent，模拟浏览器请求，避免被Steam拦截

3. 完整修复后代码

import requests
from bs4 import BeautifulSoup
import time
import json

# 提取文本的工具函数，移到外部
def extract_text(list_obj, target_arr):
    for x in list_obj:
        text_only = x.text.replace("(Field-Tested)", "").strip()
        target_arr.append(text_only)

# 提取图片链接的工具函数，移到外部
def extract_src(list_obj, target_arr):
    for x in list_obj:
        target_arr.append(x["src"])

def main():
    name_arr = []
    img_arr = []
    price_arr = []
    # 添加请求头，模拟浏览器
    headers = {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36"
    }

    for i in range(1, 11):  # 后续可修改为169页
        # 使用正确的分页查询参数
        url = f"https://steamcommunity.com/market/search?q=&category_730_ItemSet%5B%5D=any&category_730_ProPlayer%5B%5D=any&category_730_StickerCapsule%5B%5D=any&category_730_TournamentTeam%5B%5D=any&category_730_Weapon%5B%5D=any&category_730_Exterior%5B%5D=tag_WearCategory2&category_730_Quality%5B%5D=tag_normal&category_730_Quality%5B%5D=tag_unusual&appid=730&page={i}&sort_column=popular&sort_dir=desc"
        print(url)
        r = requests.get(url, headers=headers)

        print(f"----------------------------------- 当前页码: {i}")
        print(f"请求状态码: {r.status_code}")

        soup = BeautifulSoup(r.content, "html.parser")

        images = soup.find_all("img", class_="market_listing_item_img")
        names = soup.find_all("span", class_="market_listing_item_name")
        prices = soup.find_all("span", class_="sale_price")

        extract_text(names, name_arr)
        extract_text(prices, price_arr)
        extract_src(images, img_arr)

        time.sleep(60)

    print(name_arr)
    print(price_arr)
    print(img_arr)

    with open('output.json', 'w') as f:
        json.dump(name_arr, f)

if __name__ == "__main__":
    main()

额外建议 #

• 可以改用字典列表存储数据，结构更清晰，比如items = []，每页循环时添加items.append({"name": 名称, "price": 价格, "img": 图片链接})，最后保存JSON时可读性更强
• 可以增加异常处理，比如捕获请求失败的情况，避免脚本中途中断

内容的提问来源于stack exchange，提问作者pablo