使用pandas读取CSV遇Cannot convert numpy.ndarray错误求助

仅使用pd.read_csv("xxx.csv")读取CSV文件时触发报错，该文件在Python 3.9环境中可正常读取，但通过conda create -n test_env python=3.11 -y创建的虚拟环境中运行时出错，报错信息如下：

Python 3.11.11 (main, Dec 11 2024, 10:25:04) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> df = pd.read_csv("xxxxxx.csv")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/hhung/miniconda3/envs/rag_poc/lib/python3.11/site-packages/pandas/io/parsers/readers.py", line 1026, in read_csv
    return _read(filepath_or_buffer, kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hhung/miniconda3/envs/rag_poc/lib/python3.11/site-packages/pandas/io/parsers/readers.py", line 626, in _read
    return parser.read(nrows)
           ^^^^^^^^^^^^^^^^^^
  File "/Users/hhung/miniconda3/envs/rag_poc/lib/python3.11/site-packages/pandas/io/parsers/readers.py", line 1968, in read
    df = DataFrame(
         ^^^^^^^^^^
  File "/Users/hhung/miniconda3/envs/rag_poc/lib/python3.11/site-packages/pandas/core/frame.py", line 778, in __init__
    mgr = dict_to_mgr(data, index, columns, dtype=dtype, copy=copy, typ=manager)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hhung/miniconda3/envs/rag_poc/lib/python3.11/site-packages/pandas/core/internals/construction.py", line 443, in dict_to_mgr
    arrays = Series(data, index=columns, dtype=object)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hhung/miniconda3/envs/rag_poc/lib/python3.11/site-packages/pandas/core/series.py", line 490, in __init__
    index = ensure_index(index)
            ^^^^^^^^^^^^^^^^^^^
  File "/Users/hhung/miniconda3/envs/rag_poc/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 7647, in ensure_index
    return Index(index_like, copy=copy, tupleize_cols=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hhung/miniconda3/envs/rag_poc/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 565, in __new__
    arr = sanitize_array(data, None, dtype=dtype, copy=copy)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hhung/miniconda3/envs/rag_poc/lib/python3.11/site-packages/pandas/core/construction.py", line 654, in sanitize_array
    subarr = maybe_convert_platform(data)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/hhung/miniconda3/envs/rag_poc/lib/python3.11/site-packages/pandas/core/dtypes/cast.py", line 138, in maybe_convert_platform
    arr = lib.maybe_convert_objects(arr)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "lib.pyx", line 2538, in pandas._libs.lib.maybe_convert_objects
TypeError: Cannot convert numpy.ndarray to numpy.ndarray

解决思路 #

• 检查pandas与numpy版本兼容性：Python3.11环境下的pandas版本可能和numpy版本不匹配，导致内部类型转换逻辑冲突。可尝试降级/升级两者到适配Python3.11的稳定组合，比如执行：

  conda install pandas=2.1.0 numpy=1.26.0 -y
  

• 重新构建虚拟环境：删除现有环境后重新创建，同时指定pandas和numpy版本，避免conda自动安装的依赖版本冲突：

  conda remove -n test_env --all -y
  conda create -n test_env python=3.11 pandas=2.1 numpy=1.26 -y
  

• 指定读取参数规避转换：在pd.read_csv中添加dtype=object强制所有列以对象类型读取，跳过自动类型转换步骤，尝试绕过报错：

  df = pd.read_csv("xxxxxx.csv", dtype=object)
  

• 检查CSV文件的特殊格式：虽然在Python3.9能读取，但Python3.11+pandas新版本对某些特殊格式（比如含嵌套数组的单元格、非常规编码）的处理逻辑可能变化，可以先用csv模块读取几行测试内容：

  import csv
  with open("xxxxxx.csv", "r") as f:
      reader = csv.reader(f)
      for i, row in enumerate(reader):
          print(row)
          if i == 5:
              break
  

内容的提问来源于stack exchange，提问作者TripleH