使用BeautifulSoup遍历URL时遇随机Internal Server Error求解决

Hey Daniel, sorry to hear you're stuck with these random HTTP errors while scraping hundreds of URLs—those can be super frustrating when they don't tie to specific links! Let's break down some fixes that should help resolve this issue, since a basic while loop might not be handling all the edge cases here.

Common Reasons for Random HTTP Errors #

First, let's quickly cover why these might be happening randomly:

• Anti-scraping measures: Servers might flag your requests as bots if you're hitting them too fast or without proper headers.
• Network flakiness: Temporary drops in connectivity or server overload (like 503 Service Unavailable errors) that come and go.
• Incomplete request headers: Your requests might be missing browser-like headers, making servers reject them randomly.

Practical Fixes to Implement #

1. Add Proper Request Headers to Mimic a Browser

Most servers block requests without a valid User-Agent or other common headers. Update your request code to include these:

import requests
from bs4 import BeautifulSoup

headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36",
    "Accept-Language": "en-US,en;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Connection": "keep-alive"
}

def get_soup(url):
    try:
        response = requests.get(url, headers=headers)
        response.raise_for_status()  # Raises HTTPError for bad status codes
        return BeautifulSoup(response.text, 'html.parser')
    except requests.exceptions.HTTPError as e:
        print(f"HTTP Error for {url}: {e}")
        return None

2. Use Exponential Backoff for Retries (Better Than Basic while Loops)

A simple while loop might retry too quickly, triggering more blocks. Instead, use exponential backoff—wait longer between each retry to give the server time to recover. You can use the tenacity library or implement it manually:

Option A: Using tenacity (Recommended)

First install it: pip install tenacity

from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type

@retry(
    stop=stop_after_attempt(5),  # Max 5 retries
    wait=wait_exponential(multiplier=1, min=2, max=10),  # Wait 2s, 4s, 8s, up to 10s
    retry=retry_if_exception_type(requests.exceptions.HTTPError)
)
def get_soup_with_retry(url):
    response = requests.get(url, headers=headers)
    response.raise_for_status()
    return BeautifulSoup(response.text, 'html.parser')

Option B: Manual Exponential Backoff

If you don't want to add a library:

import time

def get_soup_with_backoff(url):
    max_retries = 5
    delay = 2  # Start with 2 seconds
    for attempt in range(max_retries):
        try:
            response = requests.get(url, headers=headers)
            response.raise_for_status()
            return BeautifulSoup(response.text, 'html.parser')
        except requests.exceptions.HTTPError as e:
            print(f"Attempt {attempt+1} failed for {url}: {e}")
            if attempt < max_retries - 1:
                time.sleep(delay)
                delay *= 2  # Double the delay each time
            else:
                print(f"All retries failed for {url}")
                return None

3. Add Random Delays Between Requests

To avoid triggering anti-scraping systems, add a random delay before each request:

import random

# Add this before making a request
time.sleep(random.uniform(1, 3))  # Wait between 1-3 seconds

4. Log Errors for Further Debugging

Even if errors are random, logging them can help you spot patterns (e.g., do they happen more often at certain times? With specific domains?). Add logging to your code:

import logging

logging.basicConfig(filename='scraping_errors.log', level=logging.ERROR)

# Inside your error handler
logging.error(f"HTTP Error for {url}: {str(e)}")

5. Check for IP Blocking

If errors persist, your IP might be temporarily blocked by some servers. Try using a proxy service or switching your network (e.g., from home Wi-Fi to mobile data) to test this.

Final Notes #

Random HTTP errors often boil down to either anti-scraping defenses or temporary server issues. Combining proper headers, exponential backoff, and random delays should cover most cases. If you still run into issues, check your error logs for any patterns—sometimes even "random" errors have hidden triggers!

内容的提问来源于stack exchange，提问作者Daniel Slätt