Webscraping导出CSV编码异常：特殊字符乱码求助

问题分析 #

你遇到的撇号显示为’的乱码，本质是编码不匹配：

1. 硬设置response.encoding = 'utf-8'可能覆盖了requests自动检测的正确编码，导致响应内容解码错误。
2. Excel默认不识别无BOM的UTF-8文件，会将其当作ANSI编码解析，引发乱码。

解决方案 #

1. 修正响应编码处理 #

不要硬编码设置response.encoding，改用response.apparent_encoding让requests自动检测内容的正确编码；同时用response.text（已解码的字符串）传给BeautifulSoup，避免字节解码二次出错。

2. 导出CSV时添加UTF-8 BOM #

将CSV文件的编码改为utf-8-sig，Excel会识别这个BOM头，正确解析UTF-8内容。

修改后的代码 #

import requests
from bs4 import BeautifulSoup
import csv

def scrape_report_content(url):
    response = requests.get(url)
    # 改用自动检测的编码，替换硬设的utf-8
    response.encoding = response.apparent_encoding
    # 使用response.text（已解码的字符串）初始化soup
    soup = BeautifulSoup(response.text, 'html.parser')
    content_div = soup.find('div', class_='usa-prose pep-prose margin-top-6')
    if content_div:
        paragraphs = content_div.find_all('p')
        content = '\n'.join([p.get_text(strip=True) for p in paragraphs])
        return content.strip()
    else:
        return None

base_url = "https://oig.hhs.gov/reports-and-publications/all-reports-and-publications/"
page_limit = 50

report_data = []

for page in range(1, page_limit + 1):
    page_url = f"{base_url}?page={page}"
    print(f"Scraping page {page}...")

    response = requests.get(page_url)
    # 同样改用自动检测编码
    response.encoding = response.apparent_encoding
    soup = BeautifulSoup(response.text, "html.parser")

    reports = soup.find_all("div", class_="usa-card__container")

    for report in reports:
        title_tag = report.find("a")
        title = title_tag.get_text(strip=True)
        link = "https://oig.hhs.gov" + title_tag['href']
        audit, agency, date = [e.get_text(strip=True) for e in report.find_all("dd")]

        content = scrape_report_content(link)

        report_data.append({
            "Title": title,
            "URL": link,
            "Report Number(s)": audit,
            "Agency": agency,
            "Date": date,
            "Content": content
        })

# Export to CSV
csv_file = "OIG_Reports.csv"
# 改用utf-8-sig编码，添加BOM头
with open(csv_file, mode='w', newline='', encoding='utf-8-sig') as file:
    fieldnames = ["Title", "URL", "Report Number(s)", "Agency", "Date", "Content"]
    writer = csv.DictWriter(file, fieldnames=fieldnames)
    writer.writeheader()
    for data in report_data:
        writer.writerow(data)

print(f"Data exported to {csv_file}")

关键修改点说明 #

• response.encoding = response.apparent_encoding：让requests分析响应内容的字节流，自动确定最准确的编码，避免硬编码导致的解码错误。
• soup = BeautifulSoup(response.text, ...)：直接使用已正确解码的字符串初始化BeautifulSoup，跳过字节到字符串的二次解码步骤，减少编码出错概率。
• encoding='utf-8-sig'：在文件开头添加UTF-8 BOM标记，Excel会以此识别文件为UTF-8编码，彻底解决中文/特殊字符乱码问题。

内容的提问来源于stack exchange，提问作者user23471091