如何实现并行处理双URL文件的HTTP请求并对比响应？

Got it, let's revamp your serial URL comparison code to run in parallel—this way you won't waste time waiting for one request to finish before starting the next. Here are two solid approaches depending on your needs:

Approach 1: ThreadPoolExecutor (Minimal Code Changes) #

Since requests is synchronous, using Python's built-in concurrent.futures.ThreadPoolExecutor is the easiest way to add parallelism. Threads will free up the CPU while waiting for HTTP responses, letting you run multiple request pairs at once.

First, refactor your logic into a reusable function for each URL pair, then use the executor to run them in parallel:

import requests
from concurrent.futures import ThreadPoolExecutor

def fetch_and_compare(url_pair):
    url1, url2 = url_pair
    url1_clean = url1.rstrip()
    url2_clean = url2.rstrip()
    headers = {"User-Agent": "XY"}
    
    # Fetch both URLs for the pair (you could even parallelize these two if needed)
    resp1 = requests.get(url1_clean, headers=headers)
    resp2 = requests.get(url2_clean, headers=headers)
    
    return f"{url1_clean} equals {url2_clean}" if resp1.text == resp2.text else f"{url1_clean} notequals {url2_clean}"

def compareresponse(f1, f2):
    # Read all URL pairs first to avoid file handling issues in threads
    url_pairs = list(zip(f1, f2))
    
    # Adjust max_workers based on your target site's rate limits (10-20 is a safe start)
    with ThreadPoolExecutor(max_workers=10) as executor:
        # Run all tasks and collect results
        results = executor.map(fetch_and_compare, url_pairs)
        
        # Print outcomes as they complete
        for result in results:
            print(result)

# Safely open files and run the comparison
with open(FileHandler.File1path, 'rt') as f1, open(FileHandler.File2path, 'rt') as f2:
    compareresponse(f1, f2)

Key Notes: #

• Max Workers: Don't set this too high—many websites block excessive concurrent requests. Start with 10 and adjust based on your results.
• File Handling: We read all URL pairs into a list first to avoid threading issues with open file objects.
• Minimal Changes: This keeps your original requests logic intact, so you don't have to learn a new library.

Approach 2: AsyncIO with aiohttp (Maximum Efficiency) #

For even better performance (especially with hundreds/thousands of URLs), use asynchronous IO with aiohttp. This uses a single thread but handles all requests non-blockingly, making it more efficient than threading for large workloads.

First, install aiohttp if you haven't:

pip install aiohttp

Then implement the async version:

import asyncio
import aiohttp

async def fetch(session, url):
    headers = {"User-Agent": "XY"}
    async with session.get(url.rstrip(), headers=headers) as resp:
        return await resp.text()

async def compare_pair(session, url1, url2):
    url1_clean = url1.rstrip()
    url2_clean = url2.rstrip()
    
    # Fetch both URLs in parallel for the pair
    resp1_text, resp2_text = await asyncio.gather(
        fetch(session, url1_clean),
        fetch(session, url2_clean)
    )
    
    if resp1_text == resp2_text:
        print(f"{url1_clean} equals {url2_clean}")
    else:
        print(f"{url1_clean} notequals {url2_clean}")

async def compareresponse_async(f1, f2):
    url_pairs = list(zip(f1, f2))
    async with aiohttp.ClientSession() as session:
        # Create all async tasks
        tasks = [compare_pair(session, url1, url2) for url1, url2 in url_pairs]
        # Wait for all tasks to complete
        await asyncio.gather(*tasks)

# Run the async function
with open(FileHandler.File1path, 'rt') as f1, open(FileHandler.File2path, 'rt') as f2:
    asyncio.run(compareresponse_async(f1, f2))

Key Notes: #

• Non-Blocking: Every request waits for a response without blocking the entire program, so you can handle far more concurrent requests safely.
• Session Reuse: aiohttp.ClientSession reuses connections, which speeds up requests compared to creating a new connection for each request.

Which to Choose? #

• Use ThreadPoolExecutor if you want minimal code changes and have a moderate number of URLs.
• Use aiohttp + AsyncIO if you need maximum speed with a large number of URLs, or want to leverage modern async patterns.

内容的提问来源于stack exchange，提问作者for automation