Elasticsearch Hadoop 7.10.0与Spark 3.0.1兼容吗？及报错求助

Hey there, let's tackle your problems one by one:

1. Fixing the java.lang.NoClassDefFoundError: scala/Product$class Error #

That error is almost always tied to Scala version mismatches between your dependencies. Here's the breakdown:

• Your Spark 3.0.1 uses Scala 2.12 by default
• The elasticsearch-hadoop 7.1.0 you're using is compiled against Scala 2.11, which creates a class loading conflict when trying to access Scala's Product trait (a core part of Scala's collection and case class system)

Solution: Use the Correct Dependency Artifact #

You need to swap out your current elasticsearch-hadoop dependency for one that:

• Matches your Scala 2.12 version
• Supports Spark 3.x
• Aligns with your Elasticsearch 7.10.0 version (always keep ES and ES-Hadoop versions in sync to avoid API conflicts)

For Maven:

<dependency>
    <groupId>org.elasticsearch</groupId>
    <artifactId>elasticsearch-spark-30_2.12</artifactId>
    <version>7.10.0</version>
</dependency>

For SBT:

libraryDependencies += "org.elasticsearch" % "elasticsearch-spark-30_2.12" % "7.10.0"

Key notes about the artifact name:

• spark-30 = targets Spark 3.x versions
• _2.12 = compiled for Scala 2.12

2. Compatibility Between Elasticsearch Hadoop 7.10.0 and Spark 3.0.1 #

Yes, they are fully compatible! According to Elastic's official compatibility matrix, Elasticsearch Hadoop 7.10.0 supports Spark 3.0.x (and 3.1.x) as long as you use the correct elasticsearch-spark-30_<scala-version> artifact we mentioned above.

Final Checks #

• Double-check your build file to ensure no leftover Scala 2.11 dependencies are present
• Make sure your Elasticsearch cluster is running version 7.10.0 (matching the ES-Hadoop version)
• If you're submitting a Spark job via spark-submit, include the correct ES-Hadoop jar using --packages org.elasticsearch:elasticsearch-spark-30_2.12:7.10.0 to avoid runtime classpath issues

内容的提问来源于stack exchange，提问作者Charles