如何解决Python读取CSV文件时出现的UnicodeDecodeError编码错误？

问题 #

尝试用以下Python代码遍历指定目录下的CSV文件：

import csv
import os, sys

directory = "/Users/aliharam/Desktop/Lamis File"
files = []
for filename in os.listdir(directory):
    f = os.path.join(directory, filename)
    # checking if it is a file
    if os.path.isfile(f):
        files.append(f)
files.pop()

for i in files:
    with open(i, 'r') as csvfile:
        datareader = csv.reader(csvfile)
        for row in datareader:
            print(row)

运行后触发Unicode解码错误：

Traceback (most recent call last):
  File "/Users/aliharam/PycharmProjects/LamisTasks/Normalization.py", line 16, in <module>
    for row in datareader:
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbf in position 355: invalid start byte
['\tAli Haram                                             \tAli Haram                                             ']

Process finished with exit code 1

已尝试以下两种方法但未解决问题：

dataset = pd.read_csv(i, header= 0,
                          encoding= 'unicode_escape')

with io.open(filename, 'r', encoding='utf-8') as fn:
  lines = fn.readlines()

解决方案 #

1. 指定匹配的文件编码 #

错误中的0xbf字节是Windows-1252（或ISO-8859-1）编码的典型特征，直接用该编码打开文件即可：

CSV模块修改版

for i in files:
    with open(i, 'r', encoding='windows-1252') as csvfile:
        datareader = csv.reader(csvfile)
        for row in datareader:
            print(row)

Pandas修改版

dataset = pd.read_csv(i, header=0, encoding='windows-1252')

2. 添加错误处理机制 #

如果暂时无法确定编码，可以通过错误处理跳过或替换无效字节，避免程序崩溃：

CSV模块版本

for i in files:
    with open(i, 'r', encoding='utf-8', errors='replace') as csvfile:
        datareader = csv.reader(csvfile)
        for row in datareader:
            print(row)

（errors='replace'会把无效字节替换为�，也可使用errors='ignore'直接忽略）

Pandas版本

dataset = pd.read_csv(i, header=0, encoding='utf-8', on_bad_lines='skip')

3. 自动检测文件编码 #

使用chardet库自动识别文件编码，步骤如下：

1. 安装依赖库：

pip install chardet

2. 检测并读取文件：

import chardet

for i in files:
    # 读取二进制内容检测编码
    with open(i, 'rb') as f:
        detect_result = chardet.detect(f.read())
    # 用检测到的编码打开文件
    with open(i, 'r', encoding=detect_result['encoding']) as csvfile:
        datareader = csv.reader(csvfile)
        for row in datareader:
            print(row)

内容的提问来源于stack exchange，提问作者AbuGalamba