Python爬虫导出CSV触发charmap编码UnicodeEncodeError错误求解

• 基于Python开发爬虫脚本抓取Indeed网站多页Web开发类招聘信息，未加入招聘要求（requirements）字段导出逻辑时，程序运行无异常
• 将抓取到的requirements字段加入CSV导出列表后，程序运行抛出编码错误，移除该字段导出逻辑后程序恢复正常
• 运行时抛出的错误信息：

UnicodeEncodeError: 'charmap' codec can't encode character '\ufb02' in position 582: character maps to

原始实现代码如下：

import requests
from bs4 import BeautifulSoup
import csv
from itertools import zip_longest

job_title = []
company_name = []
location_name = []
job_skill = []
links = []
salary = []
requirements = []
date = []
page_num = 0
while page_num != 20:

    result = requests.get(f"https://www.indeed.com/jobs?q=web%20development&start={page_num}")
    source = result.content
    soup = BeautifulSoup(source, "lxml")

    job_titles = soup.find_all("a", {"class", "jcs-JobTitle"})
    company_names = soup.find_all("span", {"class": "companyName"})
    location_names = soup.find_all("div", {"class": "companyLocation"})
    job_skills = soup.find_all("div", {"class": "job-snippet"})
    dates = soup.find_all("span", {"class": "date"})

    for i in range(len(job_titles)):
        job_title.append(job_titles[i].text.strip())
        links.append("https://www.indeed.com" + job_titles[i].attrs["href"])
        company_name.append(company_names[i].text.strip())
        location_name.append(location_names[i].text.strip())
        job_skill.append(job_skills[i].text.strip())
        date.append(dates[i].text.strip())

    page_num += 10
    print("Page switched...")

for link in links:
    result = requests.get(link)
    source = result.content
    soup = BeautifulSoup(source, "lxml")
    salaries = soup.find("span", {"class": "icl-u-xs-mr--xs attribute_snippet"})
    salary.append(salaries.text if salaries else "None".strip())
    requirement = soup.find("div", {"id": "jobDescriptionText", "class": "jobsearch-jobDescriptionText"}).ul
    requirements_text = ""
    if requirement:
        for li in requirement.find_all("li"):
            requirements_text += li.text + "| "
    else:
        requirements_text += "None"
    requirements_text = requirements_text[:-2]
    requirements.append(requirements_text)


my_file = [job_title, company_name, location_name, job_skill, salary, links, date, requirements]
exported = zip_longest(*my_file)
with open("/Users/Rich/Desktop/testing/indeed.csv", "w") as myfile:
    writer = csv.writer(myfile)
    writer.writerow(["Job titles", "Company names", "Location names", "Job skills", "Salaries", "Links", "Dates", "Requirements"])
    writer.writerows(exported)

1. 之前抓取的职位名称、公司名、地点等字段都是列表页的短文本，基本都是普通ASCII英文字符，没有特殊Unicode符号，写入文件时不会触发编码问题。
2. requirements字段是职位详情页抓取的大段职位描述文本，包含特殊Unicode字符\ufb02（拉丁字母fl连字）。打开CSV文件时未指定编码，Python会默认调用系统的charmap编码（Windows系统默认对应cp1252/GBK编码），这类编码的字符集没有收录该特殊字符，无法完成编码映射，因此抛出编码错误。

修改CSV文件打开的代码行，显式指定支持全量Unicode字符的UTF-8编码，同时补充csv模块官方要求的newline=''参数避免跨系统写入出现多余空行，使用utf-8-sig编码可以保证Windows/macOS下用Excel直接打开CSV不会出现乱码。

将原代码中打开文件的行：

with open("/Users/Rich/Desktop/testing/indeed.csv", "w") as myfile:

替换为：

with open("/Users/Rich/Desktop/testing/indeed.csv", "w", encoding="utf-8-sig", newline="") as myfile:

修改后即可正常写入包含特殊字符的requirements字段，不会再抛出编码错误。

内容的提问来源于stack exchange，提问作者Wisam