使用AzureOpenAI与Langchain读取CSV时遇UnicodeDecodeError求助

尝试基于CSV文件，使用AzureOpenAI（LLM）和Langchain框架开发机器人时，执行出现以下错误：

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x92 in position 12062: invalid start byte

代码片段 #

from langchain_experimental.agents.agent_toolkits.csv.base import create_csv_agent
from langchain_openai import AzureOpenAI
from dotenv import load_dotenv
import os
import streamlit as st

def main():
    load_dotenv()
    OPENAI_API_KEY = ""
    OPENAI_API_VERSION = "0301"
    
    os.environ["OPENAI_API_TYPE"] = "azure"
    os.environ["OPENAI_API_VERSION"] = "2023-05-15"
    os.environ["AZURE_OPENAI_ENDPOINT"] = "xxxx"
    os.environ["OPENAI_API_KEY"] = "xxxx"
    
    st.set_page_config(page_title="XXXXX")
    st.header("XXXXX ")
    
    llm = AzureOpenAI(
        deployment_name="name",
        model_name="gpt-3.5-turbo",
    )
  
    agent = create_csv_agent(llm, 'Data.csv')
    user_question = st.text_input("Ask your question ")
    
    if user_question is not None and user_question != "":
        with st.spinner(text="In progress..."):
            st.write(agent.run(user_question))
    
if __name__ == "__main__":
    main()

错误回溯信息 #

File "C:\Program Files\Python39\lib\site-packages\streamlit\runtime\scriptrunner\script_runner.py", line 535, in _run_script
    exec(code, module.__dict__)
File "C:\UsersXXXmain.py", line 38, in <module>
    main()
File "C:\Users\XXXX\XXXX\main.py", line 27, in main
    agent = create_csv_agent(llm, 'Data.csv')
File "C:\Program Files\Python39\lib\site-packages\langchain_experimental\agents\agent_toolkits\csv\base.py", line 28, in create_csv_agent
    df = pd.read_csv(path, **_kwargs)
File "C:\Program Files\Python39\lib\site-packages\pandas\io\parsers\readers.py", line 1024, in read_csv
    return _read(filepath_or_buffer, kwds)
File "C:\Program Files\Python39\lib\site-packages\pandas\io\parsers\readers.py", line 618, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
File "C:\Program Files\Python39\lib\site-packages\pandas\io\parsers\readers.py", line 1618, in __init__
    self._engine = self._make_engine(f, self.engine)
File "C:\Program Files\Python39\lib\site-packages\pandas\io\parsers\readers.py", line 1896, in _make_engine
    return mapping[engine](f, **self.options)
File "C:\Program Files\Python39\lib\site-packages\pandas\io\parsers\c_parser_wrapper.py", line 93, in __init__
    self._reader = parsers.TextReader(src, **kwds)
File "parsers.pyx", line 574, in pandas._libs.parsers.TextReader.__cinit__
File "parsers.pyx", line 663, in pandas._libs.parsers.TextReader._get_header
File "parsers.pyx", line 874, in pandas._libs.parsers.TextReader._tokenize_rows
File "parsers.pyx", line 891, in pandas._libs.parsers.TextReader._check_tokenize_status
File "parsers.pyx", line 2053, in pandas._libs.parsers.raise_parser_error

解决方案 #

1. 核心原因 #

错误中的0x92字节是Windows-1252编码的智能单引号（’），说明CSV文件并非UTF-8编码，而是使用了Windows系统常见的编码格式。

2. 直接修复代码 #

create_csv_agent函数支持通过_kwargs参数传递pandasread_csv的配置项，只需指定正确的编码即可：

# 修改agent创建的代码行，添加encoding参数
agent = create_csv_agent(llm, 'Data.csv', _kwargs={'encoding': 'windows-1252'})

如果windows-1252无效，可尝试iso-8859-1编码。

3. 备选：自动检测编码 #

如果不确定文件编码，可使用chardet库检测：

• 先安装库：pip install chardet
• 检测编码：

import chardet

with open('Data.csv', 'rb') as f:
    result = chardet.detect(f.read())
print(result['encoding'])  # 输出文件实际编码，比如windows-1252

将检测到的编码填入_kwargs即可。

内容的提问来源于stack exchange，提问作者Ramdas Metla