使用AWS EKS令牌提交Spark任务时遭遇IllegalArgumentException错误求助

问题场景 #

从EC2执行spark-submit提交任务到EKS集群时，执行以下命令生成令牌：

aws eks get-token --cluster-name spark-on-eks --region eu-west-2 > token

生成的token文件是完整JSON结构：

{
    "kind": "ExecCredential",
    "apiVersion": "client.authentication.k8s.io/v1beta1",
    "spec": {},
    "status": {
        "expirationTimestamp": "2023-03-30T10:02:17Z",
        "token": "k8s-aws-v1.aHR0cHM6Ly9zdHMuZXUtd2VzdC0yLmFtYXpvbmF3cy5jb20vP0FjdGlvbj1HZXRDYWxsZXJJZGVudGl0eSZWZXJzaW9uPTIwMTEtMDYtMTUmWC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBUldRUzZVQ0JKTjdWQjZFQiUyRjIwMjMwMzMwJTJGZXUtd2VzdC0yJTJGc3RzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyMzAzMzBUMDk0ODE3WiZYLUFtei1FeHBpcmVzPTYwJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCUzQngtazhzLWF3cy1pZCZYLUFtei1TaWduYXR1cmU9NGUxZGE5MzkxYjZmMGFhM2M3ZDQyM2EyMGQyNTRhMWU4NWQ4M2IwNmEzNTRiNmExOWRkMWIxNmMzODZiYTA0YQ"
    }
}

在spark-submit中传递令牌时使用：

export K8S_TOKEN=/var/run/secrets/kubernetes.io/serviceaccount/token
--conf spark.kubernetes.authenticate.submission.oauthTokenFile=$K8S_TOKEN 

执行后抛出错误：

Exception in thread "main" java.lang.IllegalArgumentException: Unexpected char 0x0a at 8 in Authorization value: Bearer {
    "kind": "ExecCredential",
    "apiVersion": "client.authentication.k8s.io/v1beta1",
    "spec": {},
    "status": {
        "expirationTimestamp": "2023-03-30T10:02:17Z",
        "token": "k8s-aws-v1.aHR0cHM6Ly9zdHMuZXUtd2VzdC0yLmFtYXpvbmF3cy5jb20vP0FjdGlvbj1HZXRDYWxsZXJJZGVudGl0eSZWZXJzaW9uPTIwMTEtMDYtMTUmWC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBUldRUzZVQ0JKTjdWQjZFQiUyRjIwMjMwMzMwJTJGZXUtd2VzdC0yJTJGc3RzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyMzAzMzBUMDk0ODE3WiZYLUFtei1FeHBpcmVzPTYwJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCUzQngtazhzLWF3cy1pZCZYLUFtei1TaWduYXR1cmU9NGUxZGE5MzkxYjZmMGFhM2M3ZDQyM2EyMGQyNTRhMWU4NWQ4M2IwNmEzNTRiNmExOWRkMWIxNmMzODZiYTA0YQ"
    }
}

        at okhttp3.Headers.checkValue(Headers.java:272)
        at okhttp3.Headers$Builder.add(Headers.java:312)
        at okhttp3.Request$Builder.addHeader(Request.java:196)
        at io.fabric8.kubernetes.client.utils.HttpClientUtils.lambda$createHttpClient$3(HttpClientUtils.java:150)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257)
        at okhttp3.RealCall.execute(RealCall.java:93)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:490)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:451)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:252)
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:879)
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:341)
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:84)
        at org.apache.spark.deploy.k8s.submit.Client.run(KubernetesClientApplication.scala:139)
        at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$3(KubernetesClientApplication.scala:213)
        at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.$anonfun$run$3$adapted(KubernetesClientApplication.scala:207)
        at org.apache.spark.util.Utils$.tryWithResource(Utils.scala:2611)
        at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.run(KubernetesClientApplication.scala:207)
        at org.apache.spark.deploy.k8s.submit.KubernetesClientApplication.start(KubernetesClientApplication.scala:179)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1030)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1039)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
23/03

问题原因 #

错误提示里的Unexpected char 0x0a是换行符，说明Spark的oauthTokenFile配置要求文件中只能包含纯令牌字符串，但你传入的是完整JSON结构（带换行、JSON字段）。Spark把整个JSON内容当作Authorization头的值，导致HTTP请求头格式非法。

这不是令牌无效，也不是令牌生成语法有问题，是你没正确提取JSON里的令牌值。

解决步骤 #

1. 提取纯令牌值到文件：用jq工具从aws eks get-token的输出中提取status.token字段的值，保存到文件：

aws eks get-token --cluster-name spark-on-eks --region eu-west-2 | jq -r '.status.token' > token

此时token文件里只有一行纯令牌字符串，没有换行和JSON结构。

2. 修改spark-submit配置：将K8S_TOKEN指向这个正确的令牌文件，示例：

export K8S_TOKEN=/path/to/your/token
spark-submit \
  --conf spark.kubernetes.authenticate.submission.oauthTokenFile=$K8S_TOKEN \
  # 其他spark-submit参数

或者直接通过环境变量传递令牌值，不需要文件：

export K8S_TOKEN=$(aws eks get-token --cluster-name spark-on-eks --region eu-west-2 | jq -r '.status.token')
spark-submit \
  --conf spark.kubernetes.authenticate.submission.oauthToken=$K8S_TOKEN \
  # 其他spark-submit参数

内容的提问来源于stack exchange，提问作者Mich Talebzadeh