Apache Arrow Java实现Parquet与CSV互转示例及参数配置咨询

一、依赖准备 #

在项目中引入 Arrow 核心依赖（以 Maven 为例）：

<dependencies>
    <!-- Arrow CSV 处理模块 -->
    <dependency>
        <groupId>org.apache.arrow</groupId>
        <artifactId>arrow-csv</artifactId>
        <version>15.0.0</version>
    </dependency>
    <!-- Arrow Parquet 处理模块 -->
    <dependency>
        <groupId>org.apache.arrow</groupId>
        <artifactId>arrow-parquet</artifactId>
        <version>15.0.0</version>
    </dependency>
    <!-- Arrow 内存管理依赖 -->
    <dependency>
        <groupId>org.apache.arrow</groupId>
        <artifactId>arrow-memory-netty</artifactId>
        <version>15.0.0</version>
    </dependency>
</dependencies>

二、CSV 转 Parquet（含分隔符、日期时间格式配置） #

核心配置要点 #

• 分隔符：通过 CsvReadOptions.Builder.setDelimiter() 配置，逗号传入 ","，制表符传入 "\t"
• 日期/时间戳格式：通过 setDateFormat() 和 setTimestampFormat() 指定自定义格式，遵循 Java SimpleDateFormat 语法

完整代码示例 #

import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.memory.RootAllocator;
import org.apache.arrow.vector.VectorSchemaRoot;
import org.apache.arrow.adapter.csv.CsvReader;
import org.apache.arrow.adapter.csv.CsvReadOptions;
import org.apache.arrow.parquet.ParquetWriter;
import java.io.File;
import java.io.IOException;
import java.nio.charset.StandardCharsets;

public class CsvToParquet {
    public static void main(String[] args) throws IOException {
        // 初始化内存分配器，务必用try-with-resources自动释放
        try (BufferAllocator allocator = new RootAllocator(Long.MAX_VALUE)) {
            // 配置CSV读取参数
            CsvReadOptions readOptions = CsvReadOptions.builder()
                    // 设置制表符分隔，替换为","则是逗号分隔
                    .setDelimiter("\t".getBytes(StandardCharsets.UTF_8))
                    // 指定日期格式
                    .setDateFormat("yyyy-MM-dd")
                    // 指定时间戳格式
                    .setTimestampFormat("yyyy-MM-dd HH:mm:ss")
                    // 跳过CSV表头行
                    .setSkipHeader(true)
                    .build();

            // 读取CSV文件
            File csvInput = new File("input.csv");
            try (CsvReader csvReader = CsvReader.from(csvInput, allocator, readOptions);
                 VectorSchemaRoot root = csvReader.read()) {
                // 写入Parquet文件
                File parquetOutput = new File("output.parquet");
                try (ParquetWriter<VectorSchemaRoot> writer = ParquetWriter.builder(root)
                        .withPath(parquetOutput)
                        .withAllocator(allocator)
                        .build()) {
                    writer.write(root);
                }
            }
        }
    }
}

三、Parquet 转 CSV（含分隔符配置） #

核心配置要点 #

• 输出分隔符：通过 CsvWriteOptions.Builder.setDelimiter() 配置，支持逗号、制表符等自定义分隔符

完整代码示例 #

import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.memory.RootAllocator;
import org.apache.arrow.vector.VectorSchemaRoot;
import org.apache.arrow.adapter.csv.CsvWriter;
import org.apache.arrow.adapter.csv.CsvWriteOptions;
import org.apache.arrow.parquet.ParquetReader;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.nio.charset.StandardCharsets;

public class ParquetToCsv {
    public static void main(String[] args) throws IOException {
        try (BufferAllocator allocator = new RootAllocator(Long.MAX_VALUE)) {
            // 读取Parquet文件
            File parquetInput = new File("input.parquet");
            try (ParquetReader<VectorSchemaRoot> reader = ParquetReader.builder(allocator, parquetInput).build();
                 VectorSchemaRoot root = reader.read()) {
                // 配置CSV写入参数
                CsvWriteOptions writeOptions = CsvWriteOptions.builder()
                        // 设置逗号分隔，替换为"\t"则是制表符分隔
                        .setDelimiter(",")
                        // 输出CSV表头
                        .setIncludeHeader(true)
                        .build();

                // 写入CSV文件
                File csvOutput = new File("output.csv");
                try (FileWriter fileWriter = new FileWriter(csvOutput, StandardCharsets.UTF_8);
                     CsvWriter csvWriter = CsvWriter.create(fileWriter, root, writeOptions)) {
                    csvWriter.write(root);
                }
            }
        }
    }
}

四、关键配置补充说明 #

• 分隔符细节：读取时需传入字节数组格式的分隔符，写入时直接传入字符串即可
• 日期时间格式兼容：若CSV中存在多种日期时间格式，可先将字段读取为字符串类型，再自行转换为对应Arrow类型
• 内存管理：所有Arrow资源（分配器、Reader、Writer等）必须通过try-with-resources管理，避免内存泄漏

内容的提问来源于stack exchange，提问作者YaOg