Python多HTTP请求处理：网页数据爬取场景技术问询

Hey Kiran, nice to see you working on this scraping project! When you're sending multiple requests sequentially (like your current loop over paramList), you're wasting a lot of time waiting for each response to come back—concurrent execution is the way to go here. Let's cover the most practical approaches tailored to your existing code using requests, plus a bonus async method for even better performance.

1. Use concurrent.futures.ThreadPoolExecutor (Great for IO-Bound Tasks) #

Since web requests are IO-bound (most time is spent waiting for the server), a thread pool is perfect. It lets you send multiple requests at once without overcomplicating things.

Here's how to adapt your code:

import requests
from concurrent.futures import ThreadPoolExecutor

# Reuse a Session to keep connections alive (big efficiency win!)
session = requests.Session()

def get_url_data(param):
    post_data = get_post_data(param)
    headers = {
        "Content-Type": "text/xml; charset=UTF-8",
        "Content-Length": str(len(post_data)),
        "Connection": 'Keep-Alive',
        "Cache-Control": 'no-cache'
    }
    try:
        response = session.post(url, data=post_data, headers=headers, timeout=10)
        response.raise_for_status()  # Raise error for bad status codes
        return response.text  # Or process the data as needed
    except requests.exceptions.RequestException as e:
        print(f"Request failed for param {param}: {e}")
        return None

# Process params concurrently
if __name__ == "__main__":
    paramList = [/* your list of params */]
    # Adjust max_workers based on the server's tolerance (start with 5-10)
    with ThreadPoolExecutor(max_workers=8) as executor:
        results = list(executor.map(get_url_data, paramList))
    
    # Now you can iterate over results to handle the scraped data
    for param, result in zip(paramList, results):
        if result:
            print(f"Successfully scraped data for {param}")

Key notes here:

• Using a requests.Session() reuses TCP connections, which cuts down on handshake overhead for multiple requests.
• max_workers should be tuned—don't set it too high (you might get blocked or rate-limited). Start with 5-10 and adjust based on your target server's behavior.
• Added basic error handling to avoid crashing on failed requests.

2. Go Async with aiohttp (Even Higher Throughput) #

For larger numbers of requests (hundreds+), asynchronous requests are more efficient than threads. Here's how to rewrite your logic with aiohttp:

import aiohttp
import asyncio

async def get_url_data_async(session, param):
    post_data = get_post_data(param)
    headers = {
        "Content-Type": "text/xml; charset=UTF-8",
        "Content-Length": str(len(post_data)),
        "Connection": 'Keep-Alive',
        "Cache-Control": 'no-cache'
    }
    try:
        async with session.post(url, data=post_data, headers=headers, timeout=10) as response:
            response.raise_for_status()
            return await response.text()
    except aiohttp.ClientError as e:
        print(f"Async request failed for param {param}: {e}")
        return None

async def main():
    paramList = [/* your list of params */]
    async with aiohttp.ClientSession() as session:
        tasks = [get_url_data_async(session, param) for param in paramList]
        results = await asyncio.gather(*tasks)
    
    for param, result in zip(paramList, results):
        if result:
            print(f"Successfully scraped data for {param}")

if __name__ == "__main__":
    asyncio.run(main())

Additional Optimization Tips #

• Rate Limiting: Add delays between requests (or use libraries like tenacity or ratelimit) to avoid getting blocked by the server.
• Retry Mechanisms: Use tenacity to automatically retry failed requests (e.g., on 5xx errors or timeouts).
• Batch Processing: If your paramList is huge, split it into batches to avoid overwhelming the server or your network.
• Monitor Performance: Use tools like timeit to compare sequential vs concurrent execution and tweak max_workers or async task limits.

内容的提问来源于stack exchange，提问作者Kiran