FastAPI课程异步网页爬虫报httpx 301重定向错误如何解决？

数据老陈

2026-9-27

问题修复方案

错误原因说明

语法错误：代码中async def get_title_range()函数定义末尾缺少冒号，运行时会首先触发语法报错。
301重定向报错：httpx的异步客户端默认关闭自动重定向跟随，目标网站对仅带节目编号的短URL做了301永久跳转处理，因此直接请求会抛出httpx.HTTPStatusError。

修复步骤

补全get_title_range函数定义末尾的冒号
调用client.get()时添加follow_redirects=True参数，开启自动重定向跟随

修复后完整代码

import asyncio
import datetime

import httpx
import bs4
from colorama import Fore

global loop
    
async def get_html(episode_number: int) -> str:
    print(Fore.YELLOW + f"Getting HTML for episode {episode_number}", flush=True)

    url = f"https://talkpython.fm/episodes/show/{episode_number}"

    async with httpx.AsyncClient() as client:
        # 添加follow_redirects参数开启重定向跟随
        resp = await client.get(url, follow_redirects=True)
        resp.raise_for_status()

        return resp.text


def get_title(html: str, episode_number: int) -> str:
    print(Fore.CYAN + f"Getting TITLE for episode {episode_number}", flush=True)
    soup = bs4.BeautifulSoup(html, 'html.parser')
    header = soup.select_one('h1')
    if not header:
        return "MISSING"

    return header.text.strip()


def main():
    t0 = datetime.datetime.now()

    global loop
    loop = asyncio.get_event_loop()
    loop.run_until_complete(get_title_range())

    dt = datetime.datetime.now() - t0
    print(f"Done in {dt.total_seconds():.2f} sec.")

# 补全函数定义末尾的冒号
async def get_title_range():
    tasks = []
    for n in range(270, 280):
        tasks.append((n, loop.create_task(get_html(n))))

    for n, t in tasks:
        html = await t
        title = get_title(html, n)
        print(Fore.WHITE + f"Title found: {title}", flush=True)


if __name__ == '__main__':
    main()