Hey there, I’ve tackled this exact Spark import issue in Cloudera’s Top-Gear Hadoop environment before—let’s get your Eclipse setup sorted out step by step.

Required JARs & Their Exact Paths in Cloudera #

First, never mix community Spark JARs with Cloudera’s customized ones—stick strictly to the JARs installed on your cluster to avoid compatibility headaches. Here’s the full list of core dependencies you need:

Core Spark JARs (Mandatory)

All these live in /usr/lib/spark/jars/ (for CDH 6.x+; CDH 5.x uses /usr/lib/spark/lib/):

• spark-core_2.11-<spark-version>.jar (e.g., spark-core_2.11-2.4.0-cdh6.3.2.jar — match your cluster’s Spark version)
• spark-sql_2.11-<spark-version>.jar (if you’re working with DataFrames/Datasets)
• spark-catalyst_2.11-<spark-version>.jar (underpins Spark SQL functionality)
• spark-network-common_2.11-<spark-version>.jar (handles inter-node network communication)
• spark-unsafe_2.11-<spark-version>.jar (optimized low-level memory operations)

Hadoop Compatibility JARs

These ensure Spark integrates smoothly with Cloudera’s Hadoop stack, found in /usr/lib/hadoop/ and /usr/lib/hadoop/client/:

• hadoop-common-<hadoop-version>.jar
• hadoop-client-<hadoop-version>.jar

Optional JARs (If Needed)

• spark-mllib_2.11-<spark-version>.jar (for machine learning APIs)
• spark-streaming_2.11-<spark-version>.jar (for real-time streaming workloads)

Critical Setup Steps to Avoid Pitfalls #

1. Verify Your Cluster’s Spark/Scala Version
   Run spark-submit --version on any cluster node to get the exact Spark version and its bundled Scala version (Cloudera Spark 2.4.x uses Scala 2.11, for example). Make sure your Eclipse project’s Scala compiler version matches this exactly—mismatched versions are a top cause of import failures.

2. Add JARs to Eclipse Build Path
   You can either:

   • Add them as external JARs pointing directly to the cluster paths (if you have access via a shared filesystem), or
   • Copy the required JARs to your project’s lib folder and add them to the build path via Project > Properties > Java Build Path > Libraries.

3. Clean & Rebuild Your Project
   Eclipse often holds onto stale cache files that cause false "missing import" errors. Go to Project > Clean and rebuild your project after adding the JARs—this usually resolves lingering issues.

Quick Troubleshoot #

If imports still fail:

• Double-check that you didn’t miss any core Spark JARs from /usr/lib/spark/jars/—Cloudera’s Spark distro includes several supporting JARs that are easy to overlook.
• Ensure no conflicting JARs (like older Spark versions or community builds) are present in your project’s build path.

内容的提问来源于stack exchange，提问作者Ashutosh Ranghar