Python requests.get()请求URL跳转引发UnicodeDecodeError问题求助

环境与问题描述 #

• 开发环境：Visual Studio Code，Python 3.12.2，BeautifulSoup 4.12.3，Windows 11，文件编码UTF-8
• 问题：构造的URL https://www.scale-rc-car.com/forum/showthread.php?t=13717&pp=1&page=1 打印输出正常，但调用requests.get()时，页面跳转至含特殊字符编码（如é转%E9、è转%E8）的地址，触发UnicodeDecodeError，完整错误栈如下：

PS C:\xampp\htdocs\python> python dumpy.py
https://www.scale-rc-car.com/forum/showthread.php?t=13717&pp=1&page=1
Traceback (most recent call last):
  File "C:\xampp\htdocs\python\dumpy.py", line 10, in <module>
    html_content = requests.get(url)
                   ^^^^^^^^^^^^^^^^^
  File "C:\Users\bartz\AppData\Roaming\Python\Python312\site-packages\requests\api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bartz\AppData\Roaming\Python\Python312\site-packages\requests\api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bartz\AppData\Roaming\Python\Python312\site-packages\requests\sessions.py", line 589, in request     
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bartz\AppData\Roaming\Python\Python312\site-packages\requests\sessions.py", line 725, in send        
    history = [resp for resp in gen]
              ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bartz\AppData\Roaming\Python\Python312\site-packages\requests\sessions.py", line 175, in resolve_redirects
    url = self.get_redirect_target(resp)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bartz\AppData\Roaming\Python\Python312\site-packages\requests\sessions.py", line 124, in get_redirect_target
    return to_native_string(location, "utf8")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\bartz\AppData\Roaming\Python\Python312\site-packages\requests\_internal_utils.py", line 33, in to_native_string
    out = string.decode(encoding)
          ^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 64: invalid continuation byte
PS C:\xampp\htdocs\python>

解决方法 #

方案1：自定义Session重定向逻辑（推荐） #

自定义Session的重定向处理，针对跳转URL的编码问题添加容错逻辑，仅影响当前Session，不会全局修改Requests行为：

import requests
from requests.sessions import Session
from requests.models import Response

def custom_redirect_handler(session: Session, resp: Response, **kwargs):
    location = resp.headers.get('Location')
    if not location:
        return []
    
    # 尝试用UTF-8解码，失败则用Latin-1兜底
    try:
        decoded_loc = location.encode('latin-1').decode('utf-8')
    except UnicodeDecodeError:
        decoded_loc = location.encode('latin-1').decode('latin-1')
    
    # 调用原重定向处理逻辑
    return original_resolve(session, resp, decoded_loc, **kwargs)

# 保存原重定向方法
original_resolve = requests.Session.resolve_redirects
# 创建自定义Session并替换重定向处理
session = requests.Session()
session.resolve_redirects = lambda resp, **kwargs: custom_redirect_handler(session, resp, **kwargs)

# 发送请求
target_url = "https://www.scale-rc-car.com/forum/showthread.php?t=13717&pp=1&page=1"
try:
    response = session.get(target_url)
    print(f"请求成功，状态码：{response.status_code}")
except Exception as e:
    print(f"请求失败：{str(e)}")

方案2：禁用自动重定向，手动处理跳转 #

关闭Requests的自动重定向功能，手动提取跳转地址并处理编码：

import requests
from urllib.parse import unquote

target_url = "https://www.scale-rc-car.com/forum/showthread.php?t=13717&pp=1&page=1"
# 禁用自动重定向
initial_resp = requests.get(target_url, allow_redirects=False)

# 处理3xx跳转
if 300 <= initial_resp.status_code < 400:
    location = initial_resp.headers.get('Location')
    if location:
        # 手动解码URL，用replace模式处理无法解码的字符
        decoded_location = unquote(location, encoding='utf-8', errors='replace')
        # 发送请求到解码后的地址
        final_resp = requests.get(decoded_location)
        print(f"跳转后请求成功，状态码：{final_resp.status_code}")
else:
    print(f"无跳转，状态码：{initial_resp.status_code}")

方案3：临时修改Requests内部解码函数（不推荐） #

通过替换Requests内部的to_native_string函数，增加解码容错，但会全局影响Requests的行为，仅适合临时应急：

import requests
from requests._internal_utils import to_native_string as original_to_native

def patched_to_native_string(string, encoding='utf-8'):
    try:
        return original_to_native(string, encoding)
    except UnicodeDecodeError:
        # 解码失败时用Latin-1兜底
        return string.decode('latin-1')

# 替换原函数
requests._internal_utils.to_native_string = patched_to_native_string

# 发送请求
target_url = "https://www.scale-rc-car.com/forum/showthread.php?t=13717&pp=1&page=1"
try:
    response = requests.get(target_url)
    print(f"请求成功，状态码：{response.status_code}")
except Exception as e:
    print(f"请求失败：{str(e)}")

方案对比 #

• 方案1：隔离性好，仅作用于自定义Session，不会影响其他请求，是最优选择。
• 方案2：灵活性高，可完全控制跳转逻辑，但需自行处理多轮重定向的情况。
• 方案3：全局修改风险高，可能引发其他未知问题，仅临时使用。

内容的提问来源于stack exchange，提问作者Bart Zakkenwasser