Julia中使用CSV.read读取大DataFrame失败求助

问题描述 #

使用CSV.read(file, DataFrame)读取大型CSV文件时，抛出如下错误：

ERROR: TaskFailedException

    nested task error: BoundsError: attempt to access 308-element Vector{BigFloat} at index [0]
    Stacktrace:
      [1] getindex
        @ .\array.jl:861 [inlined]
      [2] _scale(#unused#::Type{Float64}, v::BigInt, exp::Int64, neg::Bool)
        @ Parsers C:\Users\mazoi\.julia\packages\Parsers\KmPKe\src\floats.jl:524
      [3] scale(#unused#::Type{Float64}, v::BigInt, exp::Int64, neg::Bool)
        @ Parsers C:\Users\mazoi\.julia\packages\Parsers\KmPKe\src\floats.jl:408
      [4] parseexp
        @ C:\Users\mazoi\.julia\packages\Parsers\KmPKe\src\floats.jl:356 [inlined]
      [5] parsefrac
        @ C:\Users\mazoi\.julia\packages\Parsers\KmPKe\src\floats.jl:320 [inlined]
      [6] parsedigits
        @ C:\Users\mazoi\.julia\packages\Parsers\KmPKe\src\floats.jl:251 [inlined]
      [7] _parsedigits(#unused#::Type{Float64}, source::Vector{UInt8}, pos::Int64, len::Int64, b::UInt8, code::Int16, options::Parsers.Options, digits::BigInt, neg::Bool, startpos::Int64)
        @ Parsers C:\Users\mazoi\.julia\packages\Parsers\KmPKe\src\floats.jl:186
      [8] parsedigits
        @ C:\Users\mazoi\.julia\packages\Parsers\KmPKe\src\floats.jl:210 [inlined]
      [9] _parsedigits(#unused#::Type{Float64}, source::Vector{UInt8}, pos::Int64, len::Int64, b::UInt8, code::Int16, options::Parsers.Options, digits::UInt128, neg::Bool, startpos::Int64)
        @ Parsers C:\Users\mazoi\.julia\packages\Parsers\KmPKe\src\floats.jl:186
     [10] parsedigits
        @ C:\Users\mazoi\.julia\packages\Parsers\KmPKe\src\floats.jl:210 [inlined]
     [11] typeparser
        @ C:\Users\mazoi\.julia\packages\Parsers\KmPKe\src\floats.jl:179 [inlined]
     [12] xparse(::Type{Float64}, source::Vector{UInt8}, pos::Int64, len::Int64, options::Parsers.Options, ::Type{Float64})
        @ Parsers C:\Users\mazoi\.julia\packages\Parsers\KmPKe\src\Parsers.jl:316
     [13] xparse
        @ C:\Users\mazoi\.julia\packages\Parsers\KmPKe\src\Parsers.jl:266 [inlined]
     [14] detect
        @ C:\Users\mazoi\.julia\packages\CSV\jFiCn\src\utils.jl:470 [inlined]
     [15] detect
        @ C:\Users\mazoi\.julia\packages\CSV\jFiCn\src\utils.jl:459 [inlined]
     [16] findchunkrowstart(ranges::Vector{Int64}, i::Int64, buf::Vector{UInt8}, opts::Parsers.Options, typemap::Dict{Type, Type}, downcast::Bool, ncols::Int64, rows_to_check::Int64, columns::Vector{CSV.Column}, origcoltypes::Vector{Type}, columnlock::ReentrantLock, stringtype::Any, totalbytes::Base.Threads.Atomic{Int64}, totalrows::Base.Threads.Atomic{Int64}, succeeded::Base.Threads.Atomic{Bool})
        @ CSV C:\Users\mazoi\.julia\packages\CSV\jFiCn\src\detection.jl:383
     [17] macro expansion
        @ C:\Users\mazoi\.julia\packages\CSV\jFiCn\src\detection.jl:470 [inlined]
     [18] (::CSV.var"#16#17"{Vector{UInt8}, Parsers.Options, Vector{Int64}, Int64, Vector{CSV.Column}, DataType, Dict{Type, Type}, Bool, Int64, Vector{Type}, ReentrantLock, Base.Threads.Atomic{Bool}, Base.Threads.Atomic{Int64}, Base.Threads.Atomic{Int64}, Int64})()
        @ CSV .\threadingconstructs.jl:178
Stacktrace:
 [1] sync_end(c::Channel{Any})
   @ Base .\task.jl:381
 [2] macro expansion
   @ .\task.jl:400 [inlined]
 [3] findrowstarts!(buf::Vector{UInt8}, opts::Parsers.Options, ranges::Vector{Int64}, ncols::Int64, columns::Vector{CSV.Column}, stringtype::Any, typemap::Dict{Type, Type}, downcast::Bool, rows_to_check::Int64)
   @ CSV C:\Users\mazoi\.julia\packages\CSV\jFiCn\src\detection.jl:468
 [4] CSV.Context(source::CSV.Arg, header::CSV.Arg, normalizenames::CSV.Arg, datarow::CSV.Arg, skipto::CSV.Arg, footerskip::CSV.Arg, transpose::CSV.Arg, comment::CSV.Arg, ignoreemptyrows::CSV.Arg, ignoreemptylines::CSV.Arg, select::CSV.Arg, drop::CSV.Arg, limit::CSV.Arg, buffer_in_memory::CSV.Arg, threaded::CSV.Arg, ntasks::CSV.Arg, tasks::CSV.Arg, rows_to_check::CSV.Arg, lines_to_check::CSV.Arg, missingstrings::CSV.Arg, missingstring::CSV.Arg, delim::CSV.Arg, ignorerepeated::CSV.Arg, quoted::CSV.Arg, quotechar::CSV.Arg, openquotechar::CSV.Arg, closequotechar::CSV.Arg, escapechar::CSV.Arg, dateformat::CSV.Arg, dateformats::CSV.Arg, decimal::CSV.Arg, truestrings::CSV.Arg, falsestrings::CSV.Arg, stripwhitespace::CSV.Arg, type::CSV.Arg, types::CSV.Arg, typemap::CSV.Arg, pool::CSV.Arg, downcast::CSV.Arg, lazystrings::CSV.Arg, stringtype::CSV.Arg, strict::CSV.Arg, silencewarnings::CSV.Arg, maxwarnings::CSV.Arg, debug::CSV.Arg, parsingdebug::CSV.Arg, validate::CSV.Arg, streaming::CSV.Arg)
   @ CSV C:\Users\mazoi\.julia\packages\CSV\jFiCn\src\context.jl:608
 [5] #File#25
   @ C:\Users\mazoi\.julia\packages\CSV\jFiCn\src\file.jl:221 [inlined]
 [6] CSV.File(source::String)
   @ CSV C:\Users\mazoi\.julia\packages\CSV\jFiCn\src\file.jl:221
 [7] top-level scope
   @ REPL[16]:1

推测错误源于部分行存在类型不匹配，Python读取时，问题列显示\N。尝试设置escapechar='\'未解决问题，需要找到绕过问题行的方法，或替代CSV.jl的更健壮工具。

解决方法 #

1. 正确识别缺失值标记\N #

CSV.jl默认缺失值标记不含\N，需手动指定missingstrings参数：

using CSV, DataFrames
df = CSV.read("your_file.csv", DataFrame, missingstrings = ["\\N"])

若存在其他缺失值格式，可一并添加，例如：

df = CSV.read("your_file.csv", DataFrame, missingstrings = ["\\N", "", "NA", "null"])

2. 禁用多线程解析 #

错误栈显示问题出在多线程任务中，临时禁用多线程可绕过解析bug：

df = CSV.read("your_file.csv", DataFrame, threaded=false, missingstrings = ["\\N"])

3. 强制指定列类型 #

若已知问题列的类型，通过types参数强制解析，允许缺失值：

# 示例：指定第3列类型为Union{Float64, Missing}
df = CSV.read("your_file.csv", DataFrame, 
              types = Dict(3 => Union{Float64, Missing}), 
              missingstrings = ["\\N"])

# 或给所有列统一指定允许缺失的类型
df = CSV.read("your_file.csv", DataFrame, 
              type=Union{Float64, Int64, String, Missing}, 
              missingstrings = ["\\N"])

4. 使用替代CSV解析包 #

若CSV.jl仍无法处理，可尝试以下工具：

• DelimitedFiles.jl（Julia标准库，适合结构简单的CSV）：

using DelimitedFiles, DataFrames
data, header = readdlm("your_file.csv", ',', header=true)
df = DataFrame(data, vec(header))

• CSVFiles.jl（需先安装：using Pkg; Pkg.add("CSVFiles")）：

using CSVFiles, DataFrames
df = load("your_file.csv") |> DataFrame

5. 跳过验证或问题行 #

• 关闭解析验证（谨慎使用，可能引入脏数据）：

df = CSV.read("your_file.csv", DataFrame, validate=false, missingstrings = ["\\N"])

• 若已知问题行位置，用skipto和limit分段读取：

# 示例：跳过前100行，读取接下来的10000行
df = CSV.read("your_file.csv", DataFrame, skipto=101, limit=10000, missingstrings = ["\\N"])

内容的提问来源于stack exchange，提问作者Marius Zoican