本地Scala Spark应用初始化SparkContext遇UnresolvedAddressException错误

本地MacBook Pro（Intel Core i7 x64）上运行Scala Spark应用时出现疑似网络相关错误：本周起所有Spark应用均失败，此前运行正常且未变更配置，代码在GitHub CI及其他机器上可正常运行。该错误已阻碍代码编译（编译需执行单元测试），最初仅影响终端sbt使用，现已波及IntelliJ Idea内置运行/测试工具。

错误回溯（已隐去部分标识信息） #

[info] Loading project definition from /Users/carlosplanelles/path/to/repo/project
[info] Set current project to ProjectName (in build file:/Users/carlosplanelles/path/to/repo/)
[info] Compiling 7 Scala sources to /Users/carlosplanelles/path/to/repo/target/scala-2.12/classes...
[info] Compiling 9 Scala sources to /Users/carlosplanelles/path/to/repo/target/scala-2.12/test-classes...
[warn] 7 deprecations (since 1.3.2); re-run with -deprecation for details
[warn] one warning found
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/Users/carlosplanelles/.ivy2/cache/ch.qos.logback/logback-classic/jars/logback-classic-1.2.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/Users/carlosplanelles/.ivy2/cache/org.slf4j/slf4j-log4j12/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
[2024-02-09 16:43:27,956] [pool-4-thread-1] WARN  o.a.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
[2024-02-09 16:43:29,158] [pool-4-thread-1] ERROR org.apache.spark.SparkContext - Error initializing SparkContext. 
java.nio.channels.UnresolvedAddressException: null
        at sun.nio.ch.Net.checkAddress(Net.java:100)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:220)
        at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:134)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:550)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1334)
        at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:506)
        at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:491)
        at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:973)
        at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:248)
        at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:356)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:750)
[info] TestClassName:
[info] com.unit.testing.package *** ABORTED ***
[info]   java.nio.channels.UnresolvedAddressException:
[info]   at sun.nio.ch.Net.checkAddress(Net.java:100)
[info]   at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:220)
[info]   at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:134)
[info]   at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:550)
[info]   at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1334)
[info]   at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:506)
[info]   at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:491)
[info]   at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:973)
[info]   at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:248)
[info]   at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:356)
[info]   ...
[info] Run completed in 3 seconds, 451 milliseconds.
[info] Total number of tests run: 0
[info] Suites: completed 0, aborted 1
[info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
[info] *** 1 SUITE ABORTED ***
[error] Error during tests:
[error]         com.unit.testing.package
[error] (test:testOnly) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 36 s, completed 9-Feb-2024 4:43:30 PM

SparkSession初始化代码 #

已尝试使用local和local[1]作为master url：

import org.apache.spark.sql.SparkSession

trait SharedSparkContext {
  implicit val _spark = SparkSession
    .builder()
    .master("local[*]")
    .getOrCreate()
}

已尝试的解决方案 #

• 重新安装Java（JDK和JRE）
• 重装sbt
• 执行sbt clean
• 清理IntelliJ Idea缓存并重启软件和电脑

内容的提问来源于stack exchange，提问作者Carlos Planelles