Apache Spark 3.4.0在Java 17下报sun.nio.ch.DirectBuffer访问错误求解决

问题 #

在Eclipse中创建Maven项目，使用JavaSE-17/JDK17与Apache Spark 3.4.0，代码通过JdbcRDD读取MySQL数据并过滤。运行时抛出IllegalAccessError，提示org.apache.spark.storage.StorageUtils$无法访问sun.nio.ch.DirectBuffer，已添加JVM参数--add-exports java.base/sun.nio.ch=ALL-UNNAMED但无效。

项目代码 #

public static void main(String[] args) {
    String url = "jdbc:mysql://localhost:3306/test";
    Properties props = new Properties();
    props.setProperty("user", "root");
    props.setProperty("password", "root");
    props.setProperty("driver", "com.mysql.cj.jdbc.Driver");

    SparkConf conf = new SparkConf().setAppName("NameFilteringApp").setMaster("local");
    JavaSparkContext sc = new JavaSparkContext(conf);

    JavaRDD<String> namesRDD = JdbcRDD.create(sc,
            ()->DriverManager.getConnection(url, props),
            "select name from test", 1, 100, 2,
            (ResultSet rs)->rs.getString("name"));  
    
    JavaRDD<String> filteredNamesRDD = namesRDD.filter(name ->name.startsWith("D"));
    
    List<String> filteredNames = filteredNamesRDD.collect();
    System.out.println(filteredNames);
}

pom.xml依赖 #

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-core_2.13</artifactId>
    <version>3.4.0</version>
</dependency>

控制台异常信息 #

Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties 23/05/09 13:09:36 INFO SparkContext: Running Spark version 3.4.0 23/05/09 13:09:36 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 23/05/09 13:09:36 INFO ResourceUtils: ============================================================== 23/05/09 13:09:36 INFO ResourceUtils: No custom resources configured for spark.driver. 23/05/09 13:09:36 INFO ResourceUtils: ============================================================== 23/05/09 13:09:36 INFO SparkContext: Submitted application: NameFilteringApp 23/05/09 13:09:36 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0) 23/05/09 13:09:36 INFO ResourceProfile: Limiting resource is cpu 23/05/09 13:09:36 INFO ResourceProfileManager: Added ResourceProfile id: 0 23/05/09 13:09:36 INFO SecurityManager: Changing view acls to: sankarsh.b 23/05/09 13:09:36 INFO SecurityManager: Changing modify acls to: sankarsh.b 23/05/09 13:09:36 INFO SecurityManager: Changing view acls groups to:  23/05/09 13:09:36 INFO SecurityManager: Changing modify acls groups to:  23/05/09 13:09:36 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: sankarsh.b; groups with view permissions: EMPTY; users with modify permissions: sankarsh.b; groups with modify permissions: EMPTY 23/05/09 13:09:36 INFO Utils: Successfully started service 'sparkDriver' on port 49985. 23/05/09 13:09:36 INFO SparkEnv: Registering MapOutputTracker 23/05/09 13:09:37 INFO SparkEnv: Registering BlockManagerMaster 23/05/09 13:09:37 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 23/05/09 13:09:37 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up Exception in thread "main" java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x5afa3c9) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x5afa3c9     at org.apache.spark.storage.StorageUtils$.<clinit>(StorageUtils.scala:213)  at org.apache.spark.storage.BlockManagerMasterEndpoint.<init>(BlockManagerMasterEndpoint.scala:114)     at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:358)     at org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:295)    at org.apache.spark.SparkEnv$.create(SparkEnv.scala:344)    at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:196)   at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:279)     at org.apache.spark.SparkContext.<init>(SparkContext.scala:464)     at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)     at one.MainClass.main(MainClass.java:24)

解决方案 #

1. 正确配置Eclipse的VM参数 #

参数无效大概率是位置不对，必须添加到VM arguments区域：

• 打开Run Configurations → 找到你的Java应用 → 切换到Arguments标签
• 在VM arguments框中添加完整参数：

  --add-exports java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-exports java.base/java.nio=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED
  

• 注意：不要把这些参数写在Program arguments里，VM arguments是专门给JVM设置的区域

2. 改用SparkSession替代JavaSparkContext #

Spark新版本推荐使用SparkSession，API更简洁且能避免部分模块访问问题，修改后的代码示例：

import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.Encoders;
import java.util.Properties;
import java.util.List;

public static void main(String[] args) {
    String url = "jdbc:mysql://localhost:3306/test";
    Properties props = new Properties();
    props.setProperty("user", "root");
    props.setProperty("password", "root");
    props.setProperty("driver", "com.mysql.cj.jdbc.Driver");

    SparkSession spark = SparkSession.builder()
            .appName("NameFilteringApp")
            .master("local")
            .getOrCreate();

    Dataset<Row> df = spark.read().jdbc(url, "test", props);
    Dataset<String> namesDS = df.select("name").as(Encoders.STRING());
    Dataset<String> filteredNamesDS = namesDS.filter(name -> name.startsWith("D"));

    List<String> filteredNames = filteredNamesDS.collectAsList();
    System.out.println(filteredNames);

    spark.stop();
}

3. 补充缺失的MySQL驱动依赖 #

你的pom.xml里缺少MySQL JDBC驱动，添加后才能正常连接数据库：

<dependency>
    <groupId>mysql</groupId>
    <artifactId>mysql-connector-java</artifactId>
    <version>8.0.33</version>
</dependency>

4. 验证JDK版本与Spark的兼容性 #

Spark 3.4.0官方支持JDK 17，但如果还是有模块访问问题，可以尝试降级到JDK 11（Spark对JDK11的兼容性更成熟），或者升级到Spark 3.5.0及以上版本，新版本对JDK17的适配更好。

内容的提问来源于stack exchange，提问作者Raging Horse