PyCharm通过databricks-connect运行LightGBMRegressor遇JsonWriter缺失错误

在PyCharm中使用databricks-connect连接Databricks，运行Spark的LightGBMRegressor时，调用fit方法拟合数据出现NoClassDefFoundError: spray/json/JsonWriter错误。

运行代码 #

if "DATABRICKS_RUNTIME_VERSION" not in os.environ:
    from pyspark.sql import SparkSession
    from pyspark.dbutils import DBUtils
    spark = SparkSession.builder \
            .config("spark.jars.packages", "com.microsoft.azure:synapseml_2.12:0.9.5") \
            .config("spark.jars.repositories", "https://mmlspark.azureedge.net/maven") \
            .getOrCreate()
from pyspark.ml.evaluation import RegressionEvaluator
train_data = featurizer.transform(x_trn)[experiment.config.target_col, 'features']
test_data = featurizer.transform(x_tst)[experiment.config.target_col, 'features']
train_data.groupBy(experiment.config.target_col)
model = splightgbm.LightGBMRegressor(
    numIterations=500,
    learningRate=0.05,
    featuresCol="features", labelCol=experiment.config.target_col
)
model.fit(
    train_data
)

报错堆栈信息 #

py4j.protocol.Py4JJavaError: An error occurred while calling o1676.fit.
: java.lang.NoClassDefFoundError: spray/json/JsonWriter
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at org.apache.spark.util.Utils$.classForName(Utils.scala:242)
    at org.apache.spark.sql.util.SparkServiceObjectInputStream.readResolveClassDescriptor(SparkServiceObjectInputStream.scala:60)
    at org.apache.spark.sql.util.SparkServiceObjectInputStream.readClassDescriptor(SparkServiceObjectInputStream.scala:55)

解决方法 #

这个错误是因为缺少spray-json依赖包，SynapseML的LightGBM组件依赖该包，需在SparkSession配置中补充对应依赖：

1. 修改spark.jars.packages配置，追加io.spray:spray-json_2.12:1.3.6（Scala版本需与SynapseML保持一致，此处为2.12）
   修改后的SparkSession构建代码：

if "DATABRICKS_RUNTIME_VERSION" not in os.environ:
    from pyspark.sql import SparkSession
    from pyspark.dbutils import DBUtils
    spark = SparkSession.builder \
            .config("spark.jars.packages", "com.microsoft.azure:synapseml_2.12:0.9.5,io.spray:spray-json_2.12:1.3.6") \
            .config("spark.jars.repositories", "https://mmlspark.azureedge.net/maven") \
            .getOrCreate()

2. 额外注意事项：

• 确认Databricks Runtime版本与SynapseML版本兼容，0.9.5版本的SynapseML适配Databricks Runtime 10.4+（Scala 2.12）
• databricks-connect版本必须与Databricks集群版本完全一致，否则极易出现依赖不兼容问题

内容的提问来源于stack exchange，提问作者eliavs