无法启用SparkR Arrow：配置设置后验证失败报错

问题重现 #

尝试启用SparkR的Arrow功能，使用代码如下：

library(SparkR)
Sys.setenv(SPARK_HOME = "/home/spark/spark-3.3.2-bin-hadoop3")
install.packages("arrow", repos="https://cloud.r-project.org/")
library(arrow)
library(sparklyr)
sparkR.session(master = "local[*]",
               sparkConfig = list(spark.sql.execution.arrow.sparkr.enabled = "true"))

启动会话时已将Arrow配置设为true，但执行以下代码读取配置时：

sparkR.conf("spark.sql.execution.arrow.sparkR.enabled")

触发大量报错，核心提示为Config 'spark.sql.execution.arrow.sparkR.enabled' is not set，完整报错信息如下：

> 23/03/21 09:41:32 ERROR RBackendHandler: get on 3 failed
java.lang.reflect.InvocationTargetException
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.apache.spark.api.r.RBackendHandler.handleMethodCall(RBackendHandler.scala:164)
    at org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:105)
    at org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:39)
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.util.NoSuchElementException: spark.sql.execution.arrow.sparkR.enabled
    at org.apache.spark.sql.errors.QueryExecutionErrors$.noSuchElementExceptionError(QueryExecutionErrors.scala:1678)
    at org.apache.spark.sql.internal.SQLConf.$anonfun$getConfString$3(SQLConf.scala:4577)
    at scala.Option.getOrElse(Option.scala:189)
    at org.apache.spark.sql.internal.SQLConf.getConfString(SQLConf.scala:4577)
    at org.apache.spark.sql.RuntimeConfig.get(RuntimeConfig.scala:72)
    ... 37 more
Error in value[[3L]](cond) : 
  Config 'spark.sql.execution.arrow.sparkR.enabled' is not set
23/03/21 11:21:32 WARN RBackendHandler: Ignoring read timeout in RBackendHandler

解决办法 #

1. 修正参数大小写：Spark配置参数大小写敏感，你配置时用的是spark.sql.execution.arrow.sparkr.enabled（小写r），读取时用的是spark.sql.execution.arrow.sparkR.enabled（大写R），两者不匹配导致找不到配置。统一参数的大小写即可，比如把启动会话的配置改成spark.sql.execution.arrow.sparkR.enabled = "true"。
2. 移除冲突库：同时加载SparkR和sparklyr可能引发兼容性问题，建议只保留需要的库（如果使用SparkR就不要加载sparklyr）。
3. 验证配置：启动会话后，执行sparkR.conf()不带参数可以列出所有生效的配置，确认目标参数是否存在。

修改后的代码示例：

library(SparkR)
Sys.setenv(SPARK_HOME = "/home/spark/spark-3.3.2-bin-hadoop3")
install.packages("arrow", repos="https://cloud.r-project.org/")
library(arrow)
# 移除sparklyr加载，避免冲突
sparkR.session(master = "local[*]",
               sparkConfig = list(spark.sql.execution.arrow.sparkR.enabled = "true"))

# 验证配置是否生效
sparkR.conf("spark.sql.execution.arrow.sparkR.enabled")

内容的提问来源于stack exchange，提问作者Vortenzie