如何用Rust版Polars像DataFusion或Spark那样读取JSON文件？

待读取的JSON文件为**换行分隔JSON（NDJSON）**格式，每行一个独立JSON对象：

{<json object>},
{<json object>}

以下是对应你提供的DataFusion实现的Polars（Rust）代码，实现读取文件并选择指定列的功能：

use polars::prelude::*;
use std::time::Instant;
use env_logger;

fn main() -> Result<()> {
    env_logger::init();
    let start = Instant::now();
    let file_path = "datalayers/landing/Toys_and_Games_5.json";
    // let file_path = "datalayers/landing/test_file.json";
    let df = read_data(file_path)?;
    let duration = start.elapsed();
    log::info!("Pipeline executed successfully!");
    log::info!("Pipeline Execution time: {:?}", duration);
    Ok(())
}

fn read_data(path: &str) -> Result<DataFrame> {
    let selected_columns = vec![
        "asin",
        "vote",
        "verified",
        "unixReviewTime",
        "reviewTime",
        "reviewText",
    ];
    // 读取换行分隔格式的JSON文件
    let df = JsonReader::from_path(path)
        .with_json_format(JsonFormat::LineDelimited)
        .finish()?;
    
    // 筛选目标列
    let df = df.select(selected_columns)?;
    log::info!("Data loading plan created successfully!");
    Ok(df)
}

关键说明 #

• Polars通过JsonReader::with_json_format(JsonFormat::LineDelimited)明确指定读取NDJSON格式，和DataFusion的NdJsonReadOptions作用完全一致。
• 若文件是.gz压缩格式，Polars支持直接读取，无需手动解压，直接传入压缩文件路径即可。

示例文件下载命令 #

wget -P datalayers/landing http://deepyeti.ucsd.edu/jianmo/amazon/categoryFilesSmall/Toys_and_Games_5.json.gz

内容的提问来源于stack exchange，提问作者DataPsycho