Spark3.2（conda安装）运行报'DGEMV'第6参数非法值错误求助

通过conda方式部署的Spark 3.2版本完成升级后，运行任务触发异常，抛出如下错误：
java.lang.IllegalArgumentException: ** On entry to 'DGEMV' parameter number 6 had an illegal value
根据栈追踪定位，错误触发于BucketedRandomProjectionLSHModel的哈希计算环节，调用BLAS的dgemv方法时参数校验失败。
完整驱动栈追踪如下：

Driver stacktrace:
        at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2403)
        at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2352)
        at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2351)
        at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
        at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
        at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2351)        at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1109)
        at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1109)
        at scala.Option.foreach(Option.scala:407)
        at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1109)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2591)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2533)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2522)
        at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
Caused by: org.apache.spark.SparkException: Failed to execute user defined function (LSHModel$$Lambda$3787/1853968316: (struct<type:tinyint,size:int,indices:array<int>,values:array<double>>) => array<struct<type:tinyint,size:int,indices:array<int>,values:array<double>>>)
        at org.apache.spark.sql.errors.QueryExecutionErrors$.failedExecuteUserDefinedFunctionError(QueryExecutionErrors.scala:136)
        at org.apache.spark.sql.errors.QueryExecutionErrors.failedExecuteUserDefinedFunctionError(QueryExecutionErrors.scala)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.processNext(Unknown Source)        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
        at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:759)
        at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
        at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:140)
        at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
        at org.apache.spark.scheduler.Task.run(Task.scala:131)
        at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.IllegalArgumentException: ** On entry to 'DGEMV' parameter number 6 had an illegal value
        at dev.ludovic.netlib.blas.AbstractBLAS.checkArgument(AbstractBLAS.java:46)
        at dev.ludovic.netlib.blas.AbstractBLAS.dgemv(AbstractBLAS.java:345)
        at dev.ludovic.netlib.blas.Java8BLAS.dgemv(Java8BLAS.java:30)
        at dev.ludovic.netlib.blas.AbstractBLAS.dgemv(AbstractBLAS.java:336)
        at dev.ludovic.netlib.blas.Java8BLAS.dgemv(Java8BLAS.java:30)
        at org.apache.spark.ml.linalg.BLAS$.gemvImpl(BLAS.scala:631)
        at org.apache.spark.ml.linalg.BLAS$.gemv(BLAS.scala:614)
        at org.apache.spark.ml.linalg.BLAS$.gemv(BLAS.scala:594)
        at org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel.hashFunction(BucketedRandomProjectionLSH.scala:87)
        at org.apache.spark.ml.feature.LSHModel.$anonfun$transform$1(LSH.scala:99)
        ... 15 more

• 命令行查询Java版本为OpenJDK 1.8.0_332，符合Spark官方版本兼容要求，排除Java版本不匹配问题
• 初始状态未主动启用原生BLAS，尝试手动配置export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.332.b09-1.el7_9.x86_64/jre/后问题无改善
• 运行日志存在NativeBLAS加载失败告警，提示无法加载dev.ludovic.netlib.blas.JNIBLAS与dev.ludovic.netlib.blas.ForeignLinkerBLAS实现
• 参考公开教程配置Spark使用原生BLAS库后，问题仍未解决

Java版本信息 #

java -version
openjdk version "1.8.0_332"
OpenJDK Runtime Environment (build 1.8.0_332-b09)
OpenJDK 64-Bit Server VM (build 25.332-b09, mixed mode)

日志告警信息 #

WARN InstanceBuilder$NativeBLAS: Failed to load implementation from:dev.ludovic.netlib.blas.JNIBLAS
22/07/03 14:00:07 WARN InstanceBuilder$NativeBLAS: Failed to load implementation from:dev.ludovic.netlib.blas.ForeignLinkerBLAS

conda分发的Spark 3.2版本默认依赖的dev.ludovic.netlib纯Java BLAS实现（Java8BLAS）存在参数校验逻辑bug，在LSH计算处理稀疏向量场景下，会误判dgemv方法的第6个传入参数非法，触发异常。由于conda环境的库路径默认不在系统加载路径中，即使安装了OpenBLAS也无法被正常加载，最终回退到有bug的纯Java实现触发报错。

按落地成本从低到高排序：

1. 强制使用F2J纯Java BLAS实现绕过bug
   在Spark配置spark-defaults.conf中添加如下参数，或在spark-submit提交任务时传入，跳过有bug的Java8BLAS实现：

   spark.driver.extraJavaOptions=-Ddev.ludovic.netlib.blas.impl=dev.ludovic.netlib.blas.F2jBLAS
   spark.executor.extraJavaOptions=-Ddev.ludovic.netlib.blas.impl=dev.ludovic.netlib.blas.F2jBLAS
   

2. 正确配置conda环境内的原生OpenBLAS
   先找到conda环境下的libopenblas.so文件路径，在Spark配置中添加LD_PRELOAD参数强制加载该库，避免回退到Java实现：

   spark.driverEnv.LD_PRELOAD=/your/conda/env/path/lib/libopenblas.so
   spark.executorEnv.LD_PRELOAD=/your/conda/env/path/lib/libopenblas.so
   

   配置完成后重启Spark任务，观察日志中NativeBLAS加载失败的告警消失即配置生效，原生BLAS性能远高于Java实现，适合大数据量场景。
3. 调整Spark版本绕开bug
   若上述配置均不生效，可将Spark降级到3.1.x稳定版，或升级到3.3及以上版本，上述版本线不存在该netlib依赖的校验bug。

内容的提问来源于stack exchange，提问作者Matt Andruff