Java/Scala环境下CSV转Avro文件的实现及可用技术库咨询

Hey there! I get it—hunting for the right libraries to convert CSV to Avro in Java or Scala can feel like looking for a needle in a haystack, especially if your initial Google searches came up empty. Let me break down the solid, usable options you can leverage:

Java Implementation Options #

1. Apache Avro + Apache Commons CSV (Manual, Lightweight) #

This is the foundational approach if you want full control over the conversion process. You’ll parse CSV rows manually and map them to Avro-generated classes.

Steps:

• Define your Avro schema (save as a .avsc file, e.g., user.avsc)
• Use the Avro tools to generate Java classes from the schema
• Use Apache Commons CSV to read the input CSV
• Map each CSV record to the generated Avro class and write to an Avro file

Sample Code:

import org.apache.avro.file.DataFileWriter;
import org.apache.avro.io.DatumWriter;
import org.apache.avro.specific.SpecificDatumWriter;
import org.apache.commons.csv.CSVFormat;
import org.apache.commons.csv.CSVParser;
import org.apache.commons.csv.CSVRecord;

import java.io.File;
import java.nio.charset.StandardCharsets;

// Assume this class is generated from your Avro schema
import com.yourpackage.User;

public class CsvToAvro {
    public static void main(String[] args) throws Exception {
        // Read CSV with headers
        CSVParser parser = CSVParser.parse(
                new File("input.csv"),
                StandardCharsets.UTF_8,
                CSVFormat.DEFAULT.withHeader()
        );

        // Set up Avro writer
        DatumWriter<User> datumWriter = new SpecificDatumWriter<>(User.class);
        try (DataFileWriter<User> dataFileWriter = new DataFileWriter<>(datumWriter)) {
            dataFileWriter.create(User.getClassSchema(), new File("output.avro"));

            // Map CSV records to Avro objects
            for (CSVRecord record : parser) {
                User user = new User();
                user.setId(Integer.parseInt(record.get("id")));
                user.setName(record.get("name"));
                user.setEmail(record.get("email"));
                // Add other field mappings as needed
                dataFileWriter.append(user);
            }
        }
    }
}

2. Apache Spark (Big Data-Friendly, Zero-Boilerplate) #

If you’re working with large datasets, Spark is the way to go—it handles CSV to Avro conversion in just a few lines of code, with built-in support via the spark-avro module.

Sample Code:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;

public class SparkCsvToAvro {
    public static void main(String[] args) {
        SparkSession spark = SparkSession.builder()
                .appName("CSV-to-Avro")
                .master("local[*]") // Remove this for cluster deployment
                .getOrCreate();

        // Read CSV file
        Dataset<Row> csvData = spark.read()
                .option("header", "true")
                .option("inferSchema", "true")
                .csv("input.csv");

        // Write as Avro
        csvData.write()
                .format("avro")
                .save("output.avro");

        spark.stop();
    }
}

Required Maven Dependency:

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-avro_2.12</artifactId>
    <version>3.5.0</version>
    <scope>provided</scope>
</dependency>

Scala Implementation Options #

Scala is fully compatible with Java libraries, so the above Avro/Commons CSV and Spark approaches work here too. Plus, there are Scala-specific tools for a more idiomatic experience:

1. Apache Spark Scala API (Concise, Scalable) #

The Spark Scala API is even more streamlined than the Java version:

import org.apache.spark.sql.SparkSession

object CsvToAvroScala {
  def main(args: Array[String]): Unit = {
    val spark = SparkSession.builder()
      .appName("CSV-to-Avro")
      .master("local[*]")
      .getOrCreate()

    val csvData = spark.read
      .option("header", "true")
      .option("inferSchema", "true")
      .csv("input.csv")

    csvData.write.format("avro").save("output.avro")

    spark.stop()
  }
}

2. Scavro + Scala-CSV (Idiomatic Scala) #

Scavro simplifies Avro integration with Scala (supports case classes directly), and scala-csv is a lightweight CSV parser for Scala. Together, they make conversion clean and type-safe.

Sample Code:

import com.github.tototoshi.csv._
import com.elbrains.scavro.{Scavro, SchemaFor}
import java.io.File

case class User(id: Int, name: String, email: String)
// Generate Avro schema from the case class
implicit val schemaForUser: SchemaFor[User] = SchemaFor.gen[User]

object CsvToAvroScala {
  def main(args: Array[String]): Unit = {
    // Read CSV with headers
    val reader = CSVReader.open(new File("input.csv"))
    val csvRecords = reader.allWithHeaders()

    // Map CSV rows to User case classes
    val users = csvRecords.map(row => User(
      row("id").toInt,
      row("name"),
      row("email")
    ))

    // Write to Avro file
    Scavro.writeToFile(users, new File("output.avro"))
    reader.close()
  }
}

Required SBT Dependencies:

libraryDependencies += "com.elbrains" %% "scavro" % "1.0.0"
libraryDependencies += "com.github.tototoshi" %% "scala-csv" % "1.3.10"

Why Might Your Initial Search Miss These? #

It’s easy to miss these options if you use overly generic keywords. Try narrowing your search to things like:

• "Java CSV to Avro with Apache Avro"
• "Scala CSV to Avro Spark example"
• "Scala Avro case class conversion"

The official Apache Avro and Spark docs also have detailed guides that might have flown under your radar!

内容的提问来源于stack exchange，提问作者Nitish Kumar