Spark写入BigQuery遇InvocationTargetException及VerifyError求助

问题背景 #

已尝试以下所有配置组合，均出现相同错误：

• Python与Scala
• JDK 8与JDK 11
• Spark 3.1.2与Spark 3.3.1

以Scala + JDK 11 + Spark 3.3.1为例，操作流程及错误如下：

1. 启动spark-shell配置 #

export JAVA_HOME=$(/usr/libexec/java_home -v 11)
export SPARK_HOME=~/opt/spark/spark-3.3.1-bin-hadoop3-scala2.13
$SPARK_HOME/bin/spark-shell \
  -c spark.hadoop.fs.AbstractFileSystem.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS \
  -c spark.hadoop.fs.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem \
  --packages "com.google.cloud.spark:spark-bigquery-with-dependencies_2.13:0.28.0,com.google.cloud.bigdataoss:gcs-connector:hadoop3-2.2.10"

2. 创建测试DataFrame #

import org.apache.spark.sql._
import org.apache.spark.sql.types._

val df = spark.createDataFrame(
  java.util.List.of(
    Row(1, "foo"),
    Row(2, "bar")
  ), StructType(
    StructField("a", IntegerType) ::
    StructField("b", StringType) ::
    Nil))

df.show()

输出正常：

+---+---+
|  a|  b|
+---+---+
|  1|foo|
|  2|bar|
+---+---+

3. 写入BigQuery时触发错误 #

df.write.
  format("bigquery").
  mode("overwrite").
  option("project", "<redacted>").
  option("parentProject", "<redacted>").
  option("dataset", "<redacted>").
  option("credentials", bigquery_credentials_b64).
  option("temporaryGcsBucket", "<redacted>").
  save("test_table")

错误信息：

java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
  at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3467)
  at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
  at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
  at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
  at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
  at com.google.cloud.spark.bigquery.SparkBigQueryUtil.getUniqueGcsPath(SparkBigQueryUtil.java:127)
  at com.google.cloud.spark.bigquery.SparkBigQueryUtil.createGcsPath(SparkBigQueryUtil.java:108)
  ... 75 elided
Caused by: java.lang.reflect.InvocationTargetException: java.lang.VerifyError: Bad type on operand stack
Exception Details:
  Location:
    com/google/api/ClientProto.registerAllExtensions(Lcom/google/protobuf/ExtensionRegistryLite;)V @4: invokevirtual
  Reason:
    Type 'com/google/protobuf/GeneratedMessage$GeneratedExtension' (current frame, stack[1]) is not assignable to 'com/google/protobuf/ExtensionLite'
  Current Frame:
    bci: @4
    flags: { }
    locals: { 'com/google/protobuf/ExtensionRegistryLite' }
    stack: { 'com/google/protobuf/ExtensionRegistryLite', 'com/google/protobuf/GeneratedMessage$GeneratedExtension' }
  Bytecode:
    0000000: 2ab2 0002 b600 032a b200 04b6 0003 2ab2
    0000010: 0005 b600 03b1

  at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
  at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
  at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:135)
  ... 83 more
Caused by: java.lang.VerifyError: Bad type on operand stack
Exception Details:
  Location:
    com/google/api/ClientProto.registerAllExtensions(Lcom/google/protobuf/ExtensionRegistryLite;)V @4: invokevirtual
  Reason:
    Type 'com/google/protobuf/GeneratedMessage$GeneratedExtension' (current frame, stack[1]) is not assignable to 'com/google/protobuf/ExtensionLite'
  Current Frame:
    bci: @4
    flags: { }
    locals: { 'com/google/protobuf/ExtensionRegistryLite' }
    stack: { 'com/google/protobuf/ExtensionRegistryLite', 'com/google/protobuf/GeneratedMessage$GeneratedExtension' }
  Bytecode:
    0000000: 2ab2 0002 b600 032a b200 04b6 0003 2ab2
    0000010: 0005 b600 03b1

  ... 5 elided and 88 more

问题原因 #

这个错误是Protobuf版本冲突导致的：Spark发行版自带的Protobuf版本与BigQuery/GCS连接器依赖的Protobuf版本不兼容，类加载时出现类型不匹配，触发VerifyError。

解决方案 #

方案1：排除冲突的Protobuf依赖 #

在启动spark-shell时，显式排除Spark自带的旧版Protobuf，强制使用连接器兼容的版本。修改启动命令如下：

export JAVA_HOME=$(/usr/libexec/java_home -v 11)
export SPARK_HOME=~/opt/spark/spark-3.3.1-bin-hadoop3-scala2.13
$SPARK_HOME/bin/spark-shell \
  -c spark.hadoop.fs.AbstractFileSystem.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS \
  -c spark.hadoop.fs.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem \
  --packages "com.google.cloud.spark:spark-bigquery-with-dependencies_2.13:0.28.0,com.google.cloud.bigdataoss:gcs-connector:hadoop3-2.2.10" \
  --exclude-packages "com.google.protobuf:protobuf-java"

方案2：使用匹配的连接器版本组合 #

根据Spark版本选择官方推荐的BigQuery连接器版本，避免版本不兼容：

• Spark 3.3.x 搭配 spark-bigquery-with-dependencies_2.13:0.30.0+
• Hadoop 3环境下，GCS连接器选择 hadoop3-2.2.10+

修改后的启动命令示例：

$SPARK_HOME/bin/spark-shell \
  -c spark.hadoop.fs.AbstractFileSystem.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS \
  -c spark.hadoop.fs.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem \
  --packages "com.google.cloud.spark:spark-bigquery-with-dependencies_2.13:0.30.0,com.google.cloud.bigdataoss:gcs-connector:hadoop3-2.2.11" \
  --exclude-packages "com.google.protobuf:protobuf-java"

方案3：手动替换Spark中的Protobuf jar #

如果上述方案无效，可手动将Spark安装目录下的旧版Protobuf jar（如protobuf-java-*.jar）替换为连接器依赖的兼容版本（可通过mvn dependency:tree查看连接器所需的Protobuf版本）。

验证 #

修改配置后重新启动spark-shell，执行写入BigQuery的代码，确认错误消失，数据成功写入。

内容的提问来源于stack exchange，提问作者clay