Ubuntu Eclipse运行Spark SQL程序出现Scala类找不到的运行时错误求助

数据老陈

2026-7-20

Spark SQL程序运行时出现scala.$less$colon$less类找不到错误

环境信息

Ubuntu Linux 22.04 LTS
Eclipse IDE
Oracle Sun JDK 1.8.0_361
Apache Spark 3.4.0

问题现象

普通Spark程序可正常运行，但Spark SQL程序执行时抛出运行时错误，项目构建过程无异常。

出错的Spark SQL代码

package org.example;
import org.apache.log4j.Level;
import org.apache.log4j.Logger;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;

import static org.apache.spark.sql.functions.avg;
import static org.apache.spark.sql.functions.col;
import static org.apache.spark.sql.functions.max;

public class HousePriceSolution {

    private static final String PRICE = "Price";
    private static final String PRICE_SQ_FT = "Price SQ Ft";

    public static void main(String[] args) throws Exception 
    {
        Logger.getLogger("org").setLevel(Level.ERROR);
        SparkSession session = SparkSession.builder().appName("HousePriceSolution").master("local[1]").getOrCreate();

        Dataset<Row> realEstate = session.read().option("header", "true").csv("src/main/resources/RealEstate.csv");

        Dataset<Row> castedRealEstate = realEstate.withColumn(PRICE, col(PRICE).cast("long"))
                                                  .withColumn(PRICE_SQ_FT, col(PRICE_SQ_FT).cast("long"));

        castedRealEstate.groupBy("Location")
                        .agg(avg(PRICE_SQ_FT), max(PRICE))
                        .orderBy(col("avg(" + PRICE_SQ_FT + ")").desc())
                        .show();
    }
}

项目pom.xml配置

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>sparkwordcount</groupId>
  <artifactId>sparkwordcount</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <packaging>jar</packaging>
  <name>"Spark Word Count"</name>
  
  <repositories>
    <repository>
      <id>scala-tools.org</id>
      <name>Scala-tools Maven2 Repository</name>
      <url>http://scala-tools.org/repo-releases</url>
    </repository>
  </repositories>

  <pluginRepositories>
    <pluginRepository>
      <id>scala-tools.org</id>
      <name>Scala-tools Maven2 Repository</name>
      <url>http://scala-tools.org/repo-releases</url>
    </pluginRepository>
  </pluginRepositories>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
  </properties>

  <build>
    <plugins>
<!-- this plugin is for scala code. It uses the version of the Scala library dependency to pick the Scala version -->
<!--
      <plugin>
    <groupId>net.alchim31.maven</groupId>
    <artifactId>scala-maven-plugin</artifactId>
    <version>4.4.0</version>
        <executions>
          <execution>
            <goals>
              <goal>compile</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      -->
    
<!-- this plugin is for java code. the source and target versions are Java versions -->
      <plugin>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.8.1</version>
        <configuration>
          <source>1.8</source>
          <target>1.8</target>
        </configuration>
      </plugin>
    </plugins>  
  </build>

  <dependencies>
     
    <dependency>
      <groupId>org.scala-lang</groupId>
      <artifactId>scala-library</artifactId>
      <version>2.12.17</version>
    </dependency>
   
    <dependency>
      <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.13</artifactId>
        <version>3.4.0</version>
      <scope>provided</scope>
    </dependency>
    <!-- the following aren't needed for the word count demo, but
     will be for more complex things.
    -->
    <dependency>
      <groupId>org.apache.commons</groupId>
      <artifactId>commons-text</artifactId>
      <version>1.6</version>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
    <artifactId>spark-sql_2.13</artifactId>
    <version>3.4.0</version>
      <scope>provided</scope>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-streaming_2.13</artifactId>
    <version>3.4.0</version>
      <scope>provided</scope>
    </dependency>
    
    <dependency>
    <groupId>org.apache.commons</groupId>
    <artifactId>commons-lang3</artifactId>
    <version>3.12.0</version>
</dependency>
<!-- the following artifacts are also avaiable, depending upon
     what you're doing. They would use  the same groupId and version as the ones above:
 spark-mllib
 spark-hive
 spark-catalyst
 spark-streaming-kafka
 spark-repl
 spark-graphx
-->
  </dependencies>
</project>

运行时错误信息

Exception in thread "main" java.lang.NoClassDefFoundError: scala/$less$colon$less
    at org.example.HousePriceSolution.main(HousePriceSolution.java:22)
Caused by: java.lang.ClassNotFoundException: scala.$less$colon$less
    at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
    ... 1 more

解决方案

问题根源

Scala版本不匹配：Spark 3.4.0官方适配的Scala版本是2.13.x，但pom.xml中scala-library用的是2.12.17，版本不一致导致类加载失败。
依赖Scope设置问题：Spark核心和SQL依赖的scope设为provided，本地运行时Eclipse不会将这些依赖加入类路径，而Spark SQL的运行依赖Scala库，从而触发类找不到错误。

修复步骤

统一Scala版本：将scala-library的版本修改为Spark 3.4.0对应的Scala版本（推荐2.13.10）：

<dependency>
  <groupId>org.scala-lang</groupId>
  <artifactId>scala-library</artifactId>
  <version>2.13.10</version>
</dependency>

调整Spark依赖Scope：本地调试时，将spark-core_2.13和spark-sql_2.13的scope从provided改为compile（如果后续要提交到Spark集群，再改回provided）：

<dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-core_2.13</artifactId>
  <version>3.4.0</version>
  <scope>compile</scope>
</dependency>
<dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-sql_2.13</artifactId>
  <version>3.4.0</version>
  <scope>compile</scope>
</dependency>

刷新Maven依赖：在Eclipse中右键项目 → Maven → Update Project，勾选"Force Update of Snapshots/Releases"，点击确定。
清理并重新构建：执行mvn clean install命令，或在Eclipse中右键项目 → Run As → Maven Clean，再Run As → Maven Install。

完成以上步骤后，重新运行Spark SQL程序即可解决该错误。

内容的提问来源于stack exchange，提问作者Ashu

Ubuntu Eclipse运行Spark SQL程序出现Scala类找不到的运行时错误求助

环境信息 #

问题现象 #

出错的Spark SQL代码 #

项目pom.xml配置 #

运行时错误信息 #

解决方案 #

问题根源 #

修复步骤 #