Spark Streaming Twitter报java.lang.AbstractMethodError错误求助

Hey there, that AbstractMethodError you're hitting is almost certainly a dependency version mismatch—a super common gotcha when working with Spark Streaming and third-party connectors like the Twitter one from Bahir. Let's break down how to fix this:

Core Cause #

This error pops up when the version of spark-streaming-twitter you're using doesn't align with your installed Spark version. The Bahir connectors are tightly coupled to Spark's release versions—their version numbers mirror Spark's (e.g., 2.1.0 for Spark 2.1.x, 2.3.0 for Spark 2.3.x). When there's a mismatch, the connector tries to call methods that no longer exist (or have changed) in your Spark installation.

Step-by-Step Fixes #

1. Verify Your Spark Version #

First, check exactly which Spark version you're running:

spark-submit --version

Note the major/minor version (e.g., 2.3.0).

2. Align Bahir Connector Version #

Update your spark-submit command to use the Bahir version that matches your Spark version. For example:

• If Spark is 2.3.0 → use org.apache.bahir:spark-streaming-twitter_2.11:2.3.0
• If Spark is 2.2.0 → use org.apache.bahir:spark-streaming-twitter_2.11:2.2.0

Your corrected spark-submit command might look like this (adjust the version to match your Spark):

spark-submit --packages org.apache.bahir:spark-streaming-twitter_2.11:2.3.0 --class com.andrewrgoss.spark.PopularHashtags ".\target\scala-2.11\twitter_streaming_2.11-0.1.jar"

(Note: Wrapped the jar path in quotes to avoid issues with spaces in Windows paths)

3. Sync Dependencies in build.sbt #

Make sure your project's build.sbt uses the same Spark and Bahir versions as your local installation. Here's an example for Spark 2.3.0:

name := "twitter_streaming"
version := "0.1"
scalaVersion := "2.11.8" // Match this to your Spark's Scala version (Spark 2.x uses 2.11)

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.3.0" % Provided
libraryDependencies += "org.apache.spark" %% "spark-streaming" % "2.3.0" % Provided
libraryDependencies += "org.apache.bahir" %% "spark-streaming-twitter" % "2.3.0"

4. Rebuild and Retest #

Clean your previous build artifacts and repackage:

sbt clean package

Then run the corrected spark-submit command again.

Bonus: Check Twitter4J Conflicts #

If you still run into issues, there might be a conflict with the Twitter4J library (used by the Bahir connector). Add explicit Twitter4J dependencies to your build.sbt to lock in a compatible version:

libraryDependencies += "org.twitter4j" % "twitter4j-core" % "4.0.7"
libraryDependencies += "org.twitter4j" % "twitter4j-stream" % "4.0.7"

Don't forget to double-check that your Twitter API credentials (consumer key/secret, access token/secret) are correctly configured in your code—while that won't cause this specific error, it's a common pitfall for Twitter streaming projects.

内容的提问来源于stack exchange，提问作者smilinghulkress