aiohttp.ClientSession.get大迭代量后失效问题求助

核心问题判断 #

你的问题是网站反爬机制拦截+无限制异步并发导致的连接阻塞共同作用的结果：

• 一次性发起246个并发请求，远超正常用户的访问频率，极易触发网站流量检测，被判定为恶意爬虫后，服务器会直接拒绝连接或挂起请求；
• 首次运行慢、再次运行快是因为客户端缓存，二次请求时直接读取缓存内容，无需与服务器建立新连接。

代码优化方案 #

针对上述问题，我们需要给请求添加并发限制、浏览器伪装、超时控制，同时优化代码结构：

优化后的完整代码 #

import aiohttp
import asyncio
from bs4 import BeautifulSoup

BASE_URL = 'https://www.thecycledb.com/items'
# 完整物品列表
data = ['Hammer', 'Shattergun', 'Advocate', 'S-576 PDW', 'Kinetic Arbiter', 'Bulldog', 'KOR-47', 'Rusty K-28', 'Rusty B9 Trenchgun', 'KARMA-1', "KM-9 'Scrapper'", 'C-32 Bolt Action', 'Lacerator', 'Voltaic Brute', 'PKR Maelstrom', 'B9 Trenchgun', 'Basilisk', 'K-28', 'Binocular', 'Rusty S-576 PDW', 'Rusty AR-55 Autorifle', 'Scarab', 'Phasic Lancer', 'Manticore', 'ICA Guarantee', 'Gorgon', 'Heavy Mining Tool', 'KOMRAD', 'KBR Longshot', 'Asp Flechette Gun', 'Zeus Beam', 'Flashlight', 'Mineral Scanner', 'AR-55 Autorifle', 'Audio Decoy', 'Combat Stim', 'Weak Stim', 'Grenade', 'Combat Medkit', 'Strong Stim', 'Strong Medkit', 'Smoke Grenade', 'Gas Grenade', 'Weak Medkit', 'Large Backpack', 'Small Backpack', 'Heavy Duty Backpack', 'Worn Emergency Bag', 'Medium Backpack', 'Epic Helmet', 'Common Tactical Helmet', 'Uncommon Tactical Helmet', 'Common Helmet', 'Rare Helmet', 'Rare Tactical Helmet', 'Exotic Helmet', 'Rare Restoration Helmet', 'Legendary NV Helmet', 'Uncommon Helmet', 'Uncommon Restoration Helmet', 
'Marauder Head', 'ICA Scrip', 'NiC Oil Cannister', "Magic-GROW Fertilizer", 'Heavy Strider Flesh', 'Savage Marauder Flesh', 'Resin Gun', 'Smart Mesh', 'Heavy Strider Head', 'Crusher Hide', 'Mature Rattler Head', 'Crusher Flesh', 'Pure Focus Crystal', "Fusion Cartridge Batteries ", 'Salvaged Insulation', 'Rattler Skin', 'Autoloader', 'Spinal Base', 'Mature Rattler Eyes', 'Portable Lab', 'Shard Slicer', 'Copper Wire', 'Hardened Metals', 'Interactive Screen', 'Meteor Core', 'Electronic Cables', 'Shock Absorber', 'Meteor Fragment', 'Titan Ore', 'Clear Veltecite', 'Crusher Head', 'Miniature Reactor', 'Hydraulic Piston', 'Blue Runner Egg', 'Circuit Board', 'Marauder Flesh', 'Strider Head', 'Korolev Scrip', 'Nutritional Bar', 'Dustbloom', 'Letium Clot', 'Radio Equipment', 'Polymetallic Prefabricate', 'Biological Sampler', 'Old Medicine', 'Print Resin', 'Zero Systems CPU', 'Master Unit CPU', 'Ball Bearings', 'Altered Nickel', 'Veltecite Heart', 'Toxic Glands', 'Brittle Titan Ore', 'Nickel', 'Pure Veltecite', 'Aluminum scrap', 'Alpha Crusher Skull', 'Sample Container', 'Medical Supplies', 'Charged Spinal Base', 'Rattler Head', 'Optic Glass', 'Metallic Alloys', 'Old Currency', 'Brightcap Mushroom', 'Jewellery', 'Osiris Scrip', 'Pale Ivy Blossom', 'Waterweed Filament', 'Focus Crystal', 'Textiles', 'Indigenous Fruit', 'Rattler Eyes', 'Alpha Crusher Heart', 'Flawed Veltecite', 'Hardened Bone Plates', 'Strider Flesh', 'Glowy Brightcap Mushroom', 'Co-TEC MultiTool', 'Azure Tree Bark', 'Compound Sheets', 'Gyroscope', 'Savage Marauder Head', 'Magnetic Field Stabilizer', 'Derelict Explosives', 'Cloudy Veltecite', 'Ultralight Stock', 'Tactical Stock', 'Light Converter', '4x Optic', 'MKM Ultralight Stock', 'Small Suppressor', 'Medium Creature Dmg', 'Titan Ore Scanning Module', 'Standard Stock', 'Ergonomic Grip', 'Shotgun Slugs', 'Shotgun Quickdraw', 'Tactical Foregrip', 'Quickdraw Stock', 'Heavy Quickdraw', '8x Optic', 'Medium Muzzle Brake', 'Crude Oil Scanning Module', 'Focus Crystal Scanning Module', 'MKM Tactical Stock', 'Holographic Sight', 'Quickdraw Foregrip', 'Light Extended Quickdraw', 'Shotgun Extended', '2x Optic', 'Veltecite Scanning Module', 'Small Muzzle Brake', 'Medium Quickdraw', 'Angled Foregrip', 'Marksman Stock', 'Tactical Rear Grip', 'Medium Suppressor', 'Shotgun Converter', 'Medium Converter', 'Light Quickdraw', 'Medium Extended Quickdraw', 'Light Creature Dmg', 'Heavy Converter', 'Red Dot Sight', '6x Optic', 'MKM Quickdraw Stock', 'Medium Extended', 'Heavy Extended', 'Light Extended', 'Tactical Light', '2 - 4x Variable Optic', 'Common Shield', 'Uncommon Shield', 'Rare Tactical Shield', 'Rare Shield', 'Common Tactical Shield', 'Epic Shield', 'Uncommon Tactical Shield', 'Uncommon Restoration Shield', 'Rare Restoration Shield', 'Exotic Shield', 'Janitors Key', 'Armory Key', 'Loose House Key', 'Lab Keycard', 'Bar Storage Key', 'Bright Sands Observation Room Key', 'Community Room', 'Overseers Office', 'Tall House Key', 'Mine Access Key', "Boss' Office", 'Garage Office', 'Server Access Key', 'Luggage Saferoom Key', 'Skeleton Key', 'Letium Bio Samples', 'Uncommon Data Drive', 'Notes on Meteor Experiment - 1', 'Letium Coated Helmet', 'Miner Cam #D027', 'Old Bones', 'Data Drive', 'Notes on Meteor Experiment - 2', 'Oil Pump Part', 'Warden Skull', 'Miner Cam #2F53', 'Orbital Cannon Beacon', 'Rare Data Drive', 'Unique Data Drive', 'Miner Cam #A45D', 'Valuable Data Drive', "Dig Site Data Drive", 'Flight Recorder', 'Laser Drill Control Unit', 'Oil Pump Beacon', 'Laser Drill Beacon', 'Alpha Crusher Bait', 'Old Notebook', 'Sign of Life from Stranded Prospector', 'Medium ammo', 'Special ammo', 'Light ammo', 'Shotgun ammo', 'Heavy ammo']

# 并发限制：控制同时发起的请求数，建议5-10，避免触发反爬
SEMAPHORE = asyncio.Semaphore(8)

async def fetch_item(session, item_name):
    # 构造请求URL
    item_slug = item_name.lower().replace("'", "").replace(" ", "-").strip()
    url = f"{BASE_URL[:-1]}/{item_slug}"
    
    async with SEMAPHORE:
        try:
            # 添加请求头伪装成浏览器，设置超时
            headers = {
                'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'
            }
            async with session.get(url, headers=headers, timeout=10) as response:
                # 检查响应状态码，非200直接返回0
                if response.status != 200:
                    return 0
                html = await response.text()
                soup = BeautifulSoup(html, 'lxml')
                price_elem = soup.find('h3', string='Shop Price')
                if price_elem:
                    price_text = price_elem.next_element.next_element.text.replace(',', '')
                    return int(price_text)
                else:
                    return 0
        except Exception as e:
            # 捕获所有异常，避免单个请求失败导致整个程序挂起
            print(f"处理{item_name}时出错: {str(e)}")
            return 0

async def parse():
    async with aiohttp.ClientSession() as session:
        # 创建所有任务
        tasks = [fetch_item(session, item) for item in data]
        # 批量执行任务，返回结果列表
        price_list = await asyncio.gather(*tasks)
        print(price_list)

if __name__ == "__main__":
    asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
    asyncio.run(parse())

关键优化点说明 #

1. 并发限制：用asyncio.Semaphore(8)控制同时发起的请求数，避免短时间内大量请求触发反爬；
2. 请求头伪装：添加User-Agent模拟浏览器请求，降低被识别为爬虫的概率；
3. 超时设置：给每个请求设置10秒超时，避免单个请求挂起导致整个程序卡住；
4. 异常捕获：在单个请求函数内捕获所有异常，确保一个请求失败不会影响其他请求；
5. 状态码检查：非200响应直接返回0，避免无效HTML解析报错；
6. 代码结构优化：将单个请求逻辑封装成独立函数，代码更清晰易维护。

额外建议 #

• 如果仍出现拦截，可添加随机延迟（如await asyncio.sleep(random.uniform(0.5, 1.5))）进一步降低请求频率；
• 若需要更稳定的请求，可尝试使用代理IP，但调整并发和请求头对该网站已足够；
• 若要测试真实请求速度，可在请求头中添加Cache-Control: no-cache禁用缓存。

内容的提问来源于stack exchange，提问作者Michael Nikitin