如何用Python处理80k级CSV URL并检测404错误

Got it, let's break this down into a practical, scalable workflow for your 80k URL CSV. You already nailed the core string manipulation with rsplit—we just need to wrap that into a batch process and add efficient HTTP status checking.

Step 1: Clean the URLs from your CSV #

First, we'll read the CSV, strip off the trailing numeric part, and clean up the extra quotes. I'll include two options: one using pandas (great for large datasets) and a lightweight alternative with Python's built-in csv module.

Option A: Using Pandas (simpler for large datasets) #

import pandas as pd

# Read the CSV (adjust file path as needed; we'll use a generic column name since your CSV has no header)
df = pd.read_csv('your_urls.csv', header=None, names=['full_entry'])

# Clean each entry: split on the last comma, take the first part, then strip surrounding double quotes
df['clean_url'] = df['full_entry'].str.rsplit(',', 1).str[0].str.strip('"')

# Save cleaned URLs to a new CSV (optional but useful for backup)
df['clean_url'].to_csv('cleaned_urls.csv', index=False, header=False)

Option B: Using Python's built-in CSV module (no extra dependencies) #

import csv

with open('your_urls.csv', 'r') as infile, open('cleaned_urls.csv', 'w', newline='') as outfile:
    reader = csv.reader(infile)
    writer = csv.writer(outfile)
    
    for row in reader:
        # Each row contains one entry like "URL","number"
        full_entry = row[0]
        # Split on last comma, keep the first segment, then remove surrounding quotes
        clean_url = full_entry.rsplit(',', 1)[0].strip('"')
        writer.writerow([clean_url])

Step 2: Batch Check HTTP Status Codes (find 404s) #

80k URLs is a lot—don't use synchronous requests (it'll take forever and might get you blocked). We'll use asynchronous requests with aiohttp for speed, and add concurrency limits to avoid overwhelming the target server.

First, install aiohttp if you haven't:

pip install aiohttp

Full Async Checker Code #

import aiohttp
import asyncio
import csv
from aiohttp import ClientTimeout

# Configure settings (tweak these based on the target site's tolerance)
CONCURRENCY_LIMIT = 50  # Start lower (20-30) if the site is strict about request rates
TIMEOUT = ClientTimeout(total=10)  # 10-second timeout per request to avoid hanging
HEADERS = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36'
}

async def check_url(session, url, semaphore):
    async with semaphore:
        try:
            async with session.get(url, headers=HEADERS, timeout=TIMEOUT) as response:
                if response.status == 404:
                    return url
                # Optional: Uncomment to log other status codes for debugging
                # print(f"{url} returned status code {response.status}")
        except Exception as e:
            # Catch common errors (timeout, connection issues, DNS failures)
            print(f"Failed to check {url}: {str(e)}")
            return None

async def main(url_list):
    semaphore = asyncio.Semaphore(CONCURRENCY_LIMIT)
    async with aiohttp.ClientSession() as session:
        # Create async tasks for all URLs
        tasks = [check_url(session, url, semaphore) for url in url_list]
        # Run all tasks and collect results
        results = await asyncio.gather(*tasks)
        # Filter out None values (non-404 URLs or failed requests)
        return [url for url in results if url is not None]

if __name__ == "__main__":
    # Load cleaned URLs from the CSV we created earlier
    with open('cleaned_urls.csv', 'r') as f:
        url_list = [line.strip() for line in f if line.strip()]
    
    # Start the 404 detection process
    print("Starting 404 URL detection...")
    not_found_urls = asyncio.run(main(url_list))
    
    # Save the 404 URLs to a new CSV
    with open('404_urls.csv', 'w', newline='') as f:
        writer = csv.writer(f)
        for url in not_found_urls:
            writer.writerow([url])
    
    print(f"Done! Found {len(not_found_urls)} URLs returning 404. Results saved to 404_urls.csv")

Key Tips for Smooth Execution #

• Backup First: Always work with a copy of your original CSV to avoid accidental data loss.
• Adjust Concurrency: If you get blocked or see lots of connection errors, lower CONCURRENCY_LIMIT to reduce the number of simultaneous requests.
• Request Headers: The User-Agent mimics a browser—many sites block requests without this header.
• Error Logging: The code prints basic error messages, but you can expand the except block to log errors to a file if you need more detailed debugging.
• Performance: For 80k URLs, this async approach should take a few minutes (depending on your internet speed and the target server's response time) instead of hours with synchronous requests.

内容的提问来源于stack exchange，提问作者Kanchan Jha