Spark 2.4.0与2.4.0.cloudera2差异及依赖适配问题咨询

Let's tackle your two questions about Spark and dependency management in a Cloudera environment—here's what you need to know:

The Cloudera-distributed version of Spark isn't just a rebranded upstream release; it's tailored for their ecosystem. Here are the core distinctions:

• Backported fixes & stability patches: Cloudera adds bug fixes and performance improvements to the base 2.4.0 code that aren't present in the upstream Apache Spark release. These are often focused on making Spark work reliably with other Cloudera tools like Hadoop, Hive, and Cloudera Manager.
• Aligned transitive dependencies: As you saw with RoaringBitmap, Cloudera adjusts the versions of nested dependencies to avoid conflicts across their entire stack. For example, they might upgrade a library to a version that resolves compatibility issues with HDFS or YARN.
• Ecosystem integration: The Cloudera version comes pre-configured to play nice with Cloudera's management tools and CDH (Cloudera Distribution including Hadoop). Upstream Spark 2.4.0 requires manual configuration to integrate with these services.
• Long-term support: Cloudera provides ongoing support for their Spark distributions, while upstream Apache Spark 2.4.0 is end-of-life and no longer receives updates.

If your current POM uses upstream Central repo dependencies but needs to run on Cloudera's Spark, here are three actionable solutions:

Option 1: Switch to Cloudera's Repository & Dependency Versions #

This is the most straightforward approach to ensure full compatibility:

1. Add Cloudera's Maven repository to your pom.xml (prioritize it over Central to pull Cloudera-specific versions first):

   <repositories>
       <repository>
           <id>cloudera-releases</id>
           <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
           <releases>
               <enabled>true</enabled>
           </releases>
           <snapshots>
               <enabled>false</enabled>
           </snapshots>
       </repository>
       <repository>
           <id>central</id>
           <url>https://repo1.maven.org/maven2/</url>
       </repository>
   </repositories>
   

2. Update your Spark dependencies to use the Cloudera version. For example:

   <dependency>
       <groupId>org.apache.spark</groupId>
       <artifactId>spark-core_2.11</artifactId>
       <version>2.4.0.cloudera2</version>
       <scope>provided</scope> <!-- Spark is typically provided by the Cloudera cluster -->
   </dependency>
   

   This will automatically pull in compatible transitive dependencies like RoaringBitmap 0.7.45.

Option 2: Exclude Conflicting Dependencies & Override with Cloudera's Versions #

If you need to keep some upstream dependencies but fix conflicts:

• Identify conflicting libraries (like RoaringBitmap) and exclude the upstream version from your Spark dependency, then explicitly add the Cloudera-compatible version:

  <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.11</artifactId>
      <version>2.4.0</version>
      <exclusions>
          <exclusion>
              <groupId>org.roaringbitmap</groupId>
              <artifactId>RoaringBitmap</artifactId>
          </exclusion>
      </exclusions>
  </dependency>
  <dependency>
      <groupId>org.roaringbitmap</groupId>
      <artifactId>RoaringBitmap</artifactId>
      <version>0.7.45</version>
  </dependency>
  

  Note: This can be risky because you might miss other hidden dependency conflicts. Use this only if you have a specific reason not to switch to the full Cloudera dependency set.

Option 3: Use Cloudera's BOM for Version Alignment #

For the most reliable dependency management, use Cloudera's Bill of Materials (BOM) to automatically sync all versions with their stack:

1. Add the BOM to your dependencyManagement section:

   <dependencyManagement>
       <dependencies>
           <dependency>
               <groupId>com.cloudera.spark</groupId>
               <artifactId>spark-2.4.0.cloudera2-bom</artifactId>
               <version>1.0.0</version>
               <type>pom</type>
               <scope>import</scope>
           </dependency>
       </dependencies>
   </dependencyManagement>
   

2. Declare Spark dependencies without specifying versions—they'll inherit the correct Cloudera versions from the BOM:

   <dependency>
       <groupId>org.apache.spark</groupId>
       <artifactId>spark-core_2.11</artifactId>
       <scope>provided</scope>
   </dependency>
   

   This ensures every dependency matches what Cloudera's Spark expects, eliminating version mismatches entirely.

内容的提问来源于stack exchange，提问作者BlazerHe