为何浏览器可加载RSS feed，aiohttp请求却返回503错误？

1. 优化请求头，贴近真实浏览器行为 #

当前请求头存在两处可能触发反爬的问题：一是使用Firefox的User-Agent但实际Chrome能正常访问，二是Accept字段未优先匹配RSS格式，且包含不必要的Upgrade-Insecure-Requests字段。修改后的请求头可参考Chrome实际请求参数（可通过Chrome开发者工具Network面板查看）：

request_headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    "Accept": "application/rss+xml,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.5",
    "Accept-Encoding": "gzip, deflate, br",
    "Connection": "keep-alive",
    "DNT": "1",
    "Cache-Control": "max-age=0",
}

同时移除ssl=False参数，禁用SSL验证容易被服务器识别为异常请求。

2. 复用ClientSession，维持会话一致性 #

每次请求新建ClientSession会导致连接无状态，容易被反爬拦截。改为在main函数中创建一次会话，传给fetch_url复用：

async def fetch_url(session, url, save_to_file=False):
    try:
        print('loading...', url)
        async with session.get(url, headers=request_headers, raise_for_status=True) as response:
            text = await response.text('utf-8')
            print('success...', url)
            if save_to_file:
                current_timestamp = datetime.now().strftime('%m_%d_%Y_%H_%M_%S')
                name_part = urlparse(url).netloc
                file_name = os.path.join(os.getcwd(), 'extras', 'feeds', current_timestamp, name_part)
                os.makedirs(os.path.dirname(file_name), exist_ok=True)
                with open(file_name, 'w+') as f:
                    f.write(text)
            return (url, response.status, len(text))
    except Exception as e:
        print('failure...', url, e)
        return (url, -1)

async def main(urls, save_to_file):
    timeout = aiohttp.ClientTimeout(total=30)
    async with aiohttp.ClientSession(timeout=timeout) as session:
        tasks=[fetch_url(session, url, save_to_file=save_to_file) for url in urls]
        results=await asyncio.gather(*tasks, return_exceptions=True)
    return results

3. 调整请求频率，增加重试机制 #

每2分钟一次的请求频率对于个人网站RSS源过高，建议改为15-30分钟一次。同时添加失败重试逻辑，重试时加入随机延迟避免被识别：

from random import randint
async def fetch_url(session, url, save_to_file=False, retries=3):
    for attempt in range(retries):
        try:
            print('loading...', url, f'(attempt {attempt+1})')
            async with session.get(url, headers=request_headers, raise_for_status=True) as response:
                text = await response.text('utf-8')
                print('success...', url)
                if save_to_file:
                    current_timestamp = datetime.now().strftime('%m_%d_%Y_%H_%M_%S')
                    name_part = urlparse(url).netloc
                    file_name = os.path.join(os.getcwd(), 'extras', 'feeds', current_timestamp, name_part)
                    os.makedirs(os.path.dirname(file_name), exist_ok=True)
                    with open(file_name, 'w+') as f:
                        f.write(text)
                return (url, response.status, len(text))
        except Exception as e:
            print('failure...', url, e)
            if attempt < retries - 1:
                await asyncio.sleep(randint(2,5)) # 随机延迟2-5秒重试
    return (url, -1)

4. 检查cron执行环境 #

cron的执行IP可能被网站拉黑，或环境变量与本地测试不同。在cron任务中添加日志输出，方便排查问题：

# cron示例（调整为30分钟一次）
*/30 * * * * /usr/bin/python3 /path/to/your/script.py >> /var/log/rss_fetch.log 2>&1

若日志显示IP被限制，可考虑使用代理IP请求。

内容的提问来源于stack exchange，提问作者PirateApp