使用BeautifulSoup爬取Danbooru图片遇SSLEOFError问题求助

问题背景 #

原用Selenium爬取Danbooru图片，大数据集下耗时过长，改用BeautifulSoup+requests后出现以下错误：

Error processing get_images_srcs: HTTPSConnectionPool(host='danbooru.donmai.us', port=443): Max retries exceeded with url: /posts/'my url' (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1002)')))

该URL可在浏览器正常访问，排除网络问题。相关代码如下：

获取图片源函数 #

def get_image_src(self, post_id):
    image_src = []  
    search_url = f"{self.base_url}/posts/{post_id}?q={quote(self.tag)}"
    try:
        response = self.session.get(search_url) # self.session is initialized already
        if response.status_code != 200:
            print(f"{response.status_code} is the status code : not 200")
            return image_src 
        soup = BeautifulSoup(response.text, 'html.parser')
        image = soup.find("img", class_="fit-width")
        if image:
            image_src.append(image.get("src"))
    except Exception as e:
        print(f"Error processing get_images_srcs {post_id}: {str(e)}")
    return image_src

会话初始化函数 #

def _make_session(self):
    session = requests.Session()
    adapter = HTTPAdapter(
        pool_connections=25,
        pool_maxsize=25,
        max_retries=Retry(
            total=4,
            backoff_factor=1, 
            status_forcelist=[443,503,504] 
        )
    )
    session.headers.update({
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
        'Referer': 'https://danbooru.donmai.us'
    })
    session.mount('http://', adapter)
    session.mount('https://', adapter)
    return session

解决方案 #

1. 修复重试策略与SSL配置 #

• 错误点修正：status_forcelist中包含443是错误的，443是HTTPS端口而非HTTP状态码，应替换为Danbooru常见限流/错误状态码（如403、429、503、504）。
• SSL适配：指定TLS版本解决协议不兼容问题；必要时可临时关闭SSL验证（仅测试用，生产环境不推荐）。
• 调整重试参数：增大backoff_factor避免短时间重复请求触发反爬。

修改后的会话初始化函数：

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
import ssl
from requests.packages.urllib3.poolmanager import PoolManager

class TLSAdapter(HTTPAdapter):
    def init_poolmanager(self, connections, maxsize, block=False):
        ctx = ssl.create_default_context()
        ctx.set_ciphers('DEFAULT@SECLEVEL=1')
        ctx.options |= 0x4  # 禁用TLSv1.3，适配部分服务器协议
        self.poolmanager = PoolManager(
            num_pools=connections,
            maxsize=maxsize,
            block=block,
            ssl_version=ssl.PROTOCOL_TLSv1_2,
            ssl_context=ctx
        )

def _make_session(self):
    session = requests.Session()
    # 使用自定义TLS适配器
    adapter = TLSAdapter(
        pool_connections=10,  # 降低连接池大小，减少触发限流概率
        pool_maxsize=10,
        max_retries=Retry(
            total=3,
            backoff_factor=2,  # 指数退避，间隔1s、2s、4s
            status_forcelist=[403, 429, 503, 504],
            allowed_methods=["GET"]
        )
    )
    session.headers.update({
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',  # 更新UA到新版本
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
        'Accept-Language': 'en-US,en;q=0.9',
        'Referer': 'https://danbooru.donmai.us',
        'DNT': '1'  # 增加Do Not Track头，模拟浏览器行为
    })
    session.mount('https://', adapter)
    session.mount('http://', adapter)
    return session

2. 优化请求逻辑与频率控制 #

• 修正URL构造：访问单帖页面无需添加q={tag}参数，该参数用于搜索列表页，单帖URL应为f"{self.base_url}/posts/{post_id}"，多余参数可能触发反爬或无效请求。
• 添加请求延迟：Danbooru对未登录用户有严格请求频率限制（约1请求/秒），在请求间添加随机延迟避免触发429限流。

修改后的获取图片源函数：

import time
import random

def get_image_src(self, post_id):
    image_src = []  
    # 修正URL，移除多余的q参数
    post_url = f"{self.base_url}/posts/{post_id}"
    try:
        # 添加随机延迟0.8-1.2秒
        time.sleep(random.uniform(0.8, 1.2))
        response = self.session.get(post_url, timeout=10)  # 添加超时时间
        if response.status_code == 429:
            print(f"Rate limited, waiting 10s before retry...")
            time.sleep(10)
            response = self.session.get(post_url, timeout=10)
        if response.status_code != 200:
            print(f"Status code {response.status_code} for post {post_id}")
            return image_src 
        soup = BeautifulSoup(response.text, 'html.parser')
        # 适配Danbooru多类图片选择器
        image = soup.find("img", class_="fit-width") or soup.find("img", class_="original-file-thumbnail")
        if image:
            src = image.get("src")
            # 替换缩略图为原图（可选）
            if "/thumbnails/" in src:
                src = src.replace("/thumbnails/", "/images/").replace("_thumbnail", "")
            image_src.append(src)
    except Exception as e:
        print(f"Error processing post {post_id}: {str(e)}")
        # 遇到SSL错误时重试一次
        time.sleep(2)
        try:
            response = self.session.get(post_url, timeout=10)
            soup = BeautifulSoup(response.text, 'html.parser')
            image = soup.find("img", class_="fit-width") or soup.find("img", class_="original-file-thumbnail")
            if image:
                src = image.get("src")
                if "/thumbnails/" in src:
                    src = src.replace("/thumbnails/", "/images/").replace("_thumbnail", "")
                image_src.append(src)
        except Exception as retry_e:
            print(f"Retry failed for post {post_id}: {str(retry_e)}")
    return image_src

3. 进阶优化：使用官方API替代页面解析 #

Danbooru提供官方API（/posts.json），直接请求API比解析HTML更高效稳定，返回结构化数据，避免页面结构变化导致的解析失败。示例：

def get_image_src_via_api(self, post_id):
    image_src = []
    api_url = f"{self.base_url}/posts/{post_id}.json"
    try:
        time.sleep(random.uniform(0.8, 1.2))
        response = self.session.get(api_url, timeout=10)
        if response.status_code == 200:
            post_data = response.json()
            # 获取原图URL
            if post_data.get("file_url"):
                image_src.append(post_data["file_url"])
        else:
            print(f"API request failed for post {post_id}, status {response.status_code}")
    except Exception as e:
        print(f"API error for post {post_id}: {str(e)}")
    return image_src

注意：API请求同样受频率限制，未登录用户上限为2请求/秒，登录后可提升至20请求/秒（需在请求头中添加login和api_key参数，可在Danbooru账户设置中获取）。

内容的提问来源于stack exchange，提问作者Acno_Sama