Python写入保存HTML文件遇UnicodeEncodeError，如何修复？

问题 #

我是Python新手，编写了一段使用requests获取Yandex音乐网页内容并保存为HTML文件的代码，但运行时出现UnicodeEncodeError错误，提示'charmap'编码无法转义字符'\u20bd'，请问需要如何修改代码？

原代码 #

from bs4 import BeautifulSoup
import requests

def get_data(url):
   headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.124 YaBrowser/22.9.3.886 Yowser/2.5 Safari/537.36"
   }

   r = requests.get(url, headers=headers)
   print(r.text)

   with open("projects.html", "w") as file:
    file.write(r.text)


get_data("https://music.yandex.ru/non-music/editorial/album/keep_calm")

报错信息 #

Traceback (most recent call last):
  File "d:\python\BestParser\parser.py", line 17, in <module>
    get_data("https://music.yandex.ru/non-music/editorial/album/keep_calm")
  File "d:\python\BestParser\parser.py", line 14, in get_data
    file.write(r.text)
  File "D:\python\lib\encodings\cp1251.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u20bd' in position 20952: character maps to <undefined>  

解决方案 #

错误根源是Windows默认文件编码（如cp1251）无法识别网页中的特殊Unicode字符（\u20bd为俄罗斯卢布符号），以下两种方法可解决：

• 方法一：指定UTF-8编码写入文本文件
  在open函数中添加encoding="utf-8"参数，确保支持全Unicode字符：

  from bs4 import BeautifulSoup
  import requests
  
  def get_data(url):
     headers = {
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.124 YaBrowser/22.9.3.886 Yowser/2.5 Safari/537.36"
     }
  
     r = requests.get(url, headers=headers)
     print(r.text)
  
     # 指定utf-8编码打开文件
     with open("projects.html", "w", encoding="utf-8") as file:
      file.write(r.text)
  
  
  get_data("https://music.yandex.ru/non-music/editorial/album/keep_calm")
  

• 方法二：直接写入二进制字节流
  使用r.content获取网页的二进制内容，以二进制模式(wb)写入文件，无需手动处理编码：

  from bs4 import BeautifulSoup
  import requests
  
  def get_data(url):
     headers = {
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.124 YaBrowser/22.9.3.886 Yowser/2.5 Safari/537.36"
     }
  
     r = requests.get(url, headers=headers)
     print(r.text)
  
     # 二进制模式写入原始内容
     with open("projects.html", "wb") as file:
      file.write(r.content)
  
  
  get_data("https://music.yandex.ru/non-music/editorial/album/keep_calm")
  

内容的提问来源于stack exchange，提问作者arseniybr