如何修改Python爬虫代码以正确爬取并保留带重音符号的文本

核心原因 #

你代码中使用的unidecode库会自动将带重音的Unicode字符转换为无重音的ASCII字符，这是导致重音丢失的直接原因。

修改步骤 #

• 删除from unidecode import unidecode导入语句
• 写入CSV时，去掉title、pretext、maintext三个字段外层的unidecode()包裹，直接使用原始文本
• 可选优化：将文件打开的编码参数改为utf-8-sig，添加UTF-8 BOM标识，适配Windows环境下文本分析工具的读取规则
• 可选优化：每次发起请求获取响应后，添加r.encoding = r.apparent_encoding指定响应编码，避免BeautifulSoup解析阶段出现编码异常

调整后完整代码示例 #

import requests
from bs4 import BeautifulSoup
import json
import csv


def datetonumeric(stringdate):
    spliteddate=stringdate.split()
    data=(spliteddate[0]).split(".")
    day=int(data[0])
    themonth=int(data[1])
    year=int(data[2])
    if(themonth >10 or themonth < 7):
        return ""
    if(themonth==10 and day>7):
        return ""
    return f'{day}/{themonth}/{year}'


count=0
# 编码修改为utf-8-sig
with open('parlamentnilistyoutput.csv', 'w', newline='', encoding="utf-8-sig") as csvfile:
    spamwriter = csv.writer(csvfile, delimiter=',')
    spamwriter.writerow(['id_clanku','zdroj','datum','title','perex','text','url'])
    for i in range(20,70):
        r=requests.get(f'https://www.parlamentnilisty.cz/special/Volby%202021?p={i}')
        # 新增编码指定避免解析乱码
        r.encoding = r.apparent_encoding
        soup = BeautifulSoup(r.text, 'html.parser')
        articles=soup.select(".articles-list ul.list-unstyled li")
        for article in articles:
            try:
                id_clanku=f'PA000{count+1}'
                urlselector=(article.select("a"))
                url=f"https://www.parlamentnilisty.cz{(urlselector[0])['href']}"
                r=requests.get(url)
                # 新增编码指定避免解析乱码
                r.encoding = r.apparent_encoding
                soup=BeautifulSoup(r.text, 'html.parser')
                dateselector=soup.select('div.time')
                date=(dateselector[0]).get_text()
                date=datetonumeric(date)
                print(date)
                if(date!=""):
                    titleselector=soup.select('.article-header h1')
                    title=titleselector[0].get_text()
                    title=title.replace(","," ")
                    pretextselector=soup.select("p.brief")
                    pretext=pretextselector[0].get_text()
                    pretext=pretext.replace(","," ")
                    alltext=soup.select('.article-content > p')
                    maintext=""
                    for text in alltext:
                        maintext=maintext + (text.get_text()).replace("\n"," ")
                        maintext=maintext+"\n"
                    maintext=maintext.replace(","," ")
                    # 移除unidecode调用保留原始字符
                    spamwriter.writerow([id_clanku,'parlamentnilisty',date,title,pretext,maintext,url])
                    count=count+1
            except:
                print("wrong request")
            break

内容的提问来源于stack exchange，提问作者Veronika K