PySpark写入GCS/BigQuery时“无法获取公共无参构造函数”错误解决

问题场景 #

已配置PySpark会话，代码如下：

self.spark_session = SparkSession.builder.appName(
            "Example Session"
        ).config("spark.jars", "../../.jars/spark-bigquery-with-dependencies_2.13-0.28.0.jar")\
            .config("spark.hadoop.fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem")\
         .config("spark.driver.extraClassPath", "../../.jars/gcs-connector-hadoop3-latest.jar")\
         .config("spark.executor.extraClassPath", "../../.jars/gcs-connector-hadoop3-latest.jar").getOrCreate()

可以正常读取并转换数据集，但执行写入BigQuery（临时存储于GCS）的代码时报错：

dataframe.write.format("bigquery").option("temporaryGcsBucket", bucket_path).save(table_name)

报错信息 #

E                   py4j.protocol.Py4JJavaError: An error occurred while calling o65.json.
E                   : java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: com.google.cloud.spark.bigquery.BigQueryRelationProvider Unable to get public no-arg constructor
E                       at java.base/java.util.ServiceLoader.fail(ServiceLoader.java:582)
E                       at java.base/java.util.ServiceLoader.getConstructor(ServiceLoader.java:673)
E                       at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(ServiceLoader.java:1233)
E                       at java.base/java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(ServiceLoader.java:1265)
E                       at java.base/java.util.ServiceLoader$2.hasNext(ServiceLoader.java:1300)
E                       at java.base/java.util.ServiceLoader$3.hasNext(ServiceLoader.java:1385)
E                       at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:45)
E                       at scala.collection.Iterator.foreach(Iterator.scala:943)
E                       at scala.collection.Iterator.foreach$(Iterator.scala:943)
E                       at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
E                       at scala.collection.IterableLike.foreach(IterableLike.scala:74)
E                       at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
E                       at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
E                       at scala.collection.TraversableLike.filterImpl(TraversableLike.scala:303)
E                       at scala.collection.TraversableLike.filterImpl$(TraversableLike.scala:297)
E                       at scala.collection.AbstractTraversable.filterImpl(Traversable.scala:108)
E                       at scala.collection.TraversableLike.filter(TraversableLike.scala:395)
E                       at scala.collection.TraversableLike.filter$(TraversableLike.scala:395)
E                       at scala.collection.AbstractTraversable.filter(Traversable.scala:108)
E                       at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:657)
E                       at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSourceV2(DataSource.scala:725)
E                       at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:207)
E                       at org.apache.spark.sql.DataFrameReader.json(DataFrameReader.scala:361)
E                       at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
E                       at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
E                       at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
E                       at java.base/java.lang.reflect.Method.invoke(Method.java:566)
E                       at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
E                       at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
E                       at py4j.Gateway.invoke(Gateway.java:282)
E                       at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
E                       at py4j.commands.CallCommand.execute(CallCommand.java:79)
E                       at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
E                       at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
E                       at java.base/java.lang.Thread.run(Thread.java:829)
E                   Caused by: java.lang.NoClassDefFoundError: scala/$less$colon$less
E                       at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
E                       at java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:3137)
E                       at java.base/java.lang.Class.getConstructor0(Class.java:3342)
E                       at java.base/java.lang.Class.getConstructor(Class.java:2151)
E                       at java.base/java.util.ServiceLoader$1.run(ServiceLoader.java:660)
E                       at java.base/java.util.ServiceLoader$1.run(ServiceLoader.java:657)
E                       at java.base/java.security.AccessController.doPrivileged(Native Method)
E                       at java.base/java.util.ServiceLoader.getConstructor(ServiceLoader.java:668)
E                       ... 33 more
E                   Caused by: java.lang.ClassNotFoundException: scala/$less$colon$less
E                       at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
E                       at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
E                       at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
E                       ... 41 more

报错分析 #

核心错误是scala/$less$colon$less类找不到，说明Scala版本不匹配或者缺少对应版本的Scala标准库依赖。你使用的spark-bigquery-with-dependencies_2.13-0.28.0.jar基于Scala 2.13编译，但Spark环境可能使用了不同版本的Scala，或者类路径中未包含对应版本的Scala库。

另外，配置中同时使用spark.jars和spark.driver/executor.extraClassPath可能导致依赖加载冲突，BigQuery连接器的带依赖包已经包含了GCS相关组件，无需单独指定GCS连接器的extraClassPath。

解决方案 #

• 统一依赖管理，避免冲突
  删除spark.driver.extraClassPath和spark.executor.extraClassPath配置，直接通过spark.jars加载BigQuery连接器的带依赖包：

  self.spark_session = SparkSession.builder.appName(
      "Example Session"
  ).config("spark.jars", "../../.jars/spark-bigquery-with-dependencies_2.13-0.28.0.jar")\
   .config("spark.hadoop.fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem")\
   .getOrCreate()
  

• 确保Scala版本匹配
  检查Spark版本对应的Scala版本：

  • Spark 3.0+默认使用Scala 2.13，若使用Spark 2.x，需下载对应Scala 2.12版本的BigQuery连接器（如spark-bigquery-with-dependencies_2.12-0.28.0.jar）
  • 确认本地Scala库版本与连接器版本一致，若缺失Scala标准库，添加对应版本的scala-library.jar到spark.jars路径中。

• 验证依赖完整性
  重新从官方渠道下载对应版本的spark-bigquery-with-dependencies包，确保包内包含所有依赖项。

内容的提问来源于stack exchange，提问作者OpenDataAlex