Rust中CSV转Apache Arrow时解析报错问题求助

需要将CSV文件转换为Apache Arrow格式，CSV结构如下：

Date,Value,High,Low,Entry
1209920400,1413.50,1413.50,1412.75,1413.00
1209920580,1413.25,1414.00,1413.25,1413.75
1209921240,1413.75,1414.00,1413.25,1413.50
1209921300,1413.25,1413.25,1413.00,1413.00
1209921600,1413.25,1413.25,1412.75,1412.75
1209921780,1413.00,1413.00,1413.00,1413.00
1209921900,1413.00,1413.00,1412.75,1412.75
1209921960,1412.50,1412.50,1412.50,1412.50
1209922800,1412.75,1412.75,1412.75,1412.75
1209923100,1412.75,1413.50,1412.75,1413.25
1209923400,1412.75,1412.75,1412.50,1412.50
1209926940,1413.75,1414.00,1413.50,1413.50
1209930420,1413.75,1414.25,1413.75,1414.00

编写的Rust转换代码如下：

use arrow::{
    error::ArrowError,
    csv::ReaderBuilder,
    ipc::writer::FileWriter
};
use std::sync::Arc;
use std::{fs::File};

fn main() -> Result<(), ArrowError> {

    let input = "my_data.csv";
    let output = "my_data.arrow";
    let delimiter: u8 = b',';
    let max_read_records: Option<usize> = Some(100);
    let has_header = true;

    let schema = arrow_csv::reader::infer_schema_from_files(&[input.to_string()], delimiter, max_read_records, has_header).unwrap();

    println!("{:?}", schema);

    let file = File::open(input).unwrap();
    let csv_reader = ReaderBuilder::new(Arc::new(schema)).build(file).unwrap();

    let mut writer = FileWriter::try_new(File::create(output)?, csv_reader.schema().as_ref())?;

    for batch in csv_reader {
        match batch {
            Ok(batch) => writer.write(&batch)?,
            Err(error) => return Err(error),
        }
    }

    let _ = writer.finish();

    Ok(())
}

运行后控制台打印的Schema：

Schema {
  fields:[
    Field { name: "Date", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} },
    Field { name: "Value", data_type: Float64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} },
    Field { name: "High", data_type: Float64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} },
    Field { name: "Low", data_type: Float64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} },
    Field { name: "Entry", data_type: Float64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }
  ],
  metadata: {} 
}

同时抛出解析错误：

Error: ParseError("Error while parsing value Date for column 0 at line 0")

推断的Schema符合预期，但无法解析对应值，简化CSV结构后问题依然存在，求解决思路。

• 问题本质：infer_schema_from_files 方法会自动跳过CSV表头行来推断字段类型，但后续创建ReaderBuilder时，没有明确声明文件包含表头，导致读取器把第一行（表头）当作数据行解析，而字符串"Date"无法转换为Int64类型，因此报错。

• 修复方法：
  在创建ReaderBuilder时，添加.has_header(true)配置，明确告知读取器跳过表头行：

  let csv_reader = ReaderBuilder::new(Arc::new(schema))
      .has_header(true)
      .build(file)
      .unwrap();
  

• 额外优化建议：

  • 替换代码中的unwrap()为?，统一错误传播逻辑，提升代码健壮性；
  • 可通过.batch_size(1024)指定批处理大小，优化大文件转换的内存占用和处理速度。

内容的提问来源于stack exchange，提问作者Jona Rodrigues