在Apache Spark中读取Avro文件时出现java.lang.NoSuchMethodError错误的技术求助

Hey there, let’s tackle this java.lang.NoSuchMethodError you’re facing when reading Avro files with Apache Spark. This error usually boils down to version mismatches between Spark core and the Avro-related dependencies, so let’s walk through how to diagnose and fix it.

Why This Error Occurs #

The method org.apache.spark.sql.internal.SQLConf.avroFilterPushDown() was introduced in Spark 3.0 and later. If you’re seeing this error, it means your runtime is trying to use a version of Spark (or its Avro module) that either doesn’t have this method, or there’s a conflicting dependency loading an older version of the class that lacks it.

Troubleshooting Steps #

• Verify Spark and spark-avro versions match

  • First, confirm your core Spark version: run spark-submit --version in your terminal, or add println(SparkVersion.getVersion()) to your code if you’re working locally.
  • Check your project’s dependency config (pom.xml for Maven, build.sbt for SBT) to ensure the spark-avro module version exactly matches your core Spark version. Mismatched versions here are the #1 cause of this error.

• Check for dependency conflicts

  • Use dependency tree commands to spot conflicting libraries:
    • For Maven: mvn dependency:tree
    • For SBT: sbt dependencyTree
  • Look for multiple entries of spark-sql, spark-avro, or avro with different versions—these can cause the classloader to pick an older, incompatible version of the class.

Fixes to Try #

1. Align Spark and spark-avro Versions #

Make sure every Spark-related dependency in your project uses the exact same version. Here’s how to set this up:

Maven Example

<properties>
    <spark.version>3.3.0</spark.version> <!-- Match your actual Spark version -->
</properties>
<dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.12</artifactId>
        <version>${spark.version}</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.12</artifactId>
        <version>${spark.version}</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-avro_2.12</artifactId>
        <version>${spark.version}</version>
        <scope>provided</scope>
    </dependency>
</dependencies>

SBT Example

val sparkVersion = "3.3.0" // Match your actual Spark version
libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion % Provided,
  "org.apache.spark" %% "spark-sql" % sparkVersion % Provided,
  "org.apache.spark" %% "spark-avro" % sparkVersion % Provided
)

2. Exclude Conflicting Dependencies #

If your dependency tree shows other libraries pulling in older versions of Spark or Avro modules, exclude those conflicting dependencies from your project’s dependencies. For example:

<!-- Maven example excluding conflicting Spark/Avro from another library -->
<dependency>
    <groupId>com.example</groupId>
    <artifactId>some-third-party-lib</artifactId>
    <version>1.0.0</version>
    <exclusions>
        <exclusion>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.12</artifactId>
        </exclusion>
        <exclusion>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro</artifactId>
        </exclusion>
    </exclusions>
</dependency>

3. Validate Runtime Environment #

• Cluster runs: Ensure your Spark cluster’s version matches the version you used to package your application. A cluster running Spark 3.2 can’t properly run an app built with Spark 3.3’s spark-avro module.
• Local/IDE runs: Double-check your IDE’s project dependencies—sometimes old JARs can linger in your build path. Clean your project (Maven: mvn clean install, SBT: sbt clean compile) and rebuild to eliminate stale files.

4. Correctly Load spark-avro in Interactive Sessions #

If you’re using spark-shell or pyspark to read Avro, make sure you load the spark-avro module with a version matching your Spark version:

spark-shell --packages org.apache.spark:spark-avro_2.12:3.3.0

内容的提问来源于stack exchange，提问作者Paulo Moreira