Spark环境下Kubernetes Java Client(OpenAPI)JSON反序列化失败

环境与依赖 #

Spark依赖 #

"org.apache.spark"   %% "spark-hadoop-cloud"        % "3.3.0"
"org.apache.spark"   %% "spark-sql"                 % "3.3.0"
"org.apache.spark"   %% "spark-sql-kafka-0-10"      % "3.3.0"

Kubernetes客户端API依赖 #

"io.kubernetes" % "client-java" % "19.0.1"
"io.kubernetes" % "client-java-api" % "19.0.1"

报错信息 #

执行应用时，调用io.kubernetes.client.openapi.ApiClient.handleResponse(Response response, Type returnType)方法，内部触发反序列化错误：

Exception in thread "main" com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 1 column 11921
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:176)
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:40)
at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:187)
at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:145)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172)
at com.google.gson.internal.bind.TypeAdapterRuntimeTypeWrapper.read(TypeAdapterRuntimeTypeWrapper.java:40)
at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:81)
at com.google.gson.internal.bind.CollectionTypeAdapterFactory$Adapter.read(CollectionTypeAdapterFactory.java:60)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:93)
at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:172)
at com.google.gson.Gson.fromJson(Gson.java:803)
at com.google.gson.Gson.fromJson(Gson.java:768)
at com.google.gson.Gson.fromJson(Gson.java:717)
at io.kubernetes.client.openapi.JSON.deserialize(JSON.java:168)
at io.kubernetes.client.openapi.ApiClient.deserialize(ApiClient.java:775)
at io.kubernetes.client.openapi.ApiClient.handleResponse(ApiClient.java:978)

已排查情况 #

• 相同的Kubernetes客户端逻辑在独立（非Spark）Java应用中运行正常
• 确认Kubernetes库和Spark依赖的GSON版本均为2.10.1，排除Spark的GSON依赖后问题仍存在
• 将Kubernetes客户端逻辑单独隔离到Spark项目中测试（直接发送请求并反序列化响应），执行成功

解决思路 #

1. 排查类加载器冲突 #

Spark Driver的类加载机制与普通Java应用不同，可能存在类加载优先级问题：

• 在Spark启动参数中添加--driver-class-path，明确指定Kubernetes客户端和GSON的jar包路径，强制优先加载这些依赖
• 开启spark.driver.userClassPathFirst=true配置，让用户自定义类优先于Spark内置类加载

2. 验证K8s API响应内容差异 #

Spark环境中可能获取到异常的API响应：

• 在Spark Driver中捕获K8s API的完整响应body，输出到日志或本地文件，与独立应用中的响应对比，确认JSON格式是否一致
• 检查请求Header，确保Accept设置为application/json，避免返回非JSON格式的错误内容

3. 隔离GSON实例 #

Spark内部可能存在自定义GSON实例干扰反序列化：

• 在调用Kubernetes客户端前，显式创建独立的GSON实例替换默认处理器：

  ApiClient client = Configuration.getDefaultApiClient();
  JSON customJsonHandler = new JSON(new GsonBuilder().create());
  client.setJSON(customJsonHandler);
  

4. 检查Spark Operator的环境配置 #

Spark Operator初始化的上下文可能存在特殊配置：

• 检查Driver Pod的环境变量，确认HTTP_PROXY/HTTPS_PROXY配置正确，避免代理篡改响应内容
• 验证Driver Pod的ServiceAccount权限，确保能正常访问K8s API，避免返回权限错误的非JSON响应

内容的提问来源于stack exchange，提问作者Jelly