Spark 2.0使用spark-infotheoretic-feature-selection报NoSuchMethodError求助

This error almost always boils down to dependency version conflict between the Breeze library bundled with Spark 2.0 and the version required by the spark-infotheoretic-feature-selection package. Here's how to resolve it step by step:

Why This Happens #

Spark 2.0 ships with an older version of Breeze (typically around 0.11.x), while the spark-infotheoretic-feature-selection library likely depends on a newer Breeze release (0.12+). When running on the cluster, Spark's classloader prioritizes its own bundled libraries over the ones you added to the classpath—so the third-party package can't find the newer method in the outdated Breeze jar. Your local environment works because it uses the correct Breeze version that matches the library's requirements.

Solutions #

1. Resolve Conflicts During Job Submission #

Use spark-submit with package exclusion to force loading the correct Breeze version alongside the feature selection library:

spark-submit \
  --exclude-packages org.scalanlp:breeze,org.scalanlp:breeze-macros \
  --packages com.github.sramirez:spark-infotheoretic-feature-selection:[YOUR_LIB_VERSION],org.scalanlp:breeze:[COMPATIBLE_BREEZE_VERSION],org.scalanlp:breeze-macros:[COMPATIBLE_BREEZE_VERSION] \
  your-spark-job.jar

• Replace [YOUR_LIB_VERSION] with the spark-infotheoretic-feature-selection version you're using.
• Replace [COMPATIBLE_BREEZE_VERSION] with the Breeze version the library requires (check its pom.xml or build.sbt file for details).

2. Build an Uber Jar (Fat Jar) #

Create a self-contained jar that includes the feature selection library and compatible Breeze version, excluding Spark's bundled dependencies:

• For Maven: Add <scope>provided</scope> to Spark dependencies, exclude Breeze from Spark's transitive dependencies, then use the maven-shade-plugin to package all required libraries.
• For sbt: Mark spark-core, spark-sql, etc., as Provided, exclude Breeze via excludeDependencies, then run sbt assembly to build the fat jar.

Submit this fat jar to the cluster—this ensures the correct dependencies are loaded before Spark's bundled ones.

3. Update Cluster Spark Classpath (Caution) #

If you have cluster admin access, you can replace the older Breeze jars in Spark's $SPARK_HOME/jars directory with the compatible version. Note: This may break other jobs that depend on Spark's original Breeze version, so test thoroughly first.

Verification #

Run your job with the --verbose flag in spark-submit to check loaded dependencies. Look for lines confirming that the correct Breeze jar is being loaded instead of Spark's bundled one.

内容的提问来源于stack exchange，提问作者Ahsan