AKS集群容器中通过代码启动Job遇Forbidden权限异常

问题场景 #

在AKS集群内的容器中，使用.NET 8环境的F#代码启动Kubernetes Job时抛出Forbidden异常，但相同的Job YAML文件通过kubectl可以正常创建和运行。

F#代码片段 #

let startJob (yaml: string) = 
    let job = KubernetesYaml.Deserialize<V1Job>(yaml)
    let client = new Kubernetes(config)
    client.BatchV1.ReplaceNamespacedJob(job, job.Metadata.Name, job.Metadata.Namespace()) |> ignore

异常信息 #

[01:06:24 ERR] Connection id "0HMRLS62EJ1DS", Request id "0HMRLS62EJ1DS:00000001": An unhandled exception was thrown by the application. # {"EventId": {"Id": 13, "Name": "ApplicationError"}, "SourceContext": "Microsoft.AspNetCore.Server.Kestrel", "RequestId": "0HMRLS62EJ1DS:00000001", "RequestPath": "/train-study"}
k8s.Autorest.HttpOperationException: Operation returned an invalid status code 'Forbidden'
at k8s.Kubernetes.SendRequestRaw(String requestContent, HttpRequestMessage httpRequest, CancellationToken cancellationToken)
at k8s.AbstractKubernetes.k8s.IBatchV1Operations.ReplaceNamespacedJobWithHttpMessagesAsync(V1Job body, String name, String namespaceParameter, String dryRun, String fieldManager, String fieldValidation, Nullable1 pretty, IReadOnlyDictionary2 customHeaders, CancellationToken cancellationToken)
at k8s.BatchV1OperationsExtensions.ReplaceNamespacedJobAsync(IBatchV1Operations operations, V1Job body, String name, String namespaceParameter, String dryRun, String fieldManager, String fieldValidation, Nullable1 pretty, CancellationToken cancellationToken) at k8s.BatchV1OperationsExtensions.ReplaceNamespacedJob(IBatchV1Operations operations, V1Job body, String name, String namespaceParameter, String dryRun, String fieldManager, String fieldValidation, Nullable1 pretty)
at Yatp.Adapter.Kubernetes.K8s.startJob(String yaml) in /src/Adapter/Yatp.Adapter.Kubernetes/Kubernetes.fs:line 63
at Yatp.Adapter.Kubernetes.Jobs.TrainStudy.train@17.MoveNext() in /src/Yatp.Domain.Extensions/K8s/TrainStudy.fs:line 28
at lambda_method48(Closure, Object)

已尝试的RBAC配置 #

已测试以下Role/ClusterRole与RoleBinding的所有组合（|表示不同的测试选项），但问题未解决：

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole|Role
metadata:
  name: yatp-role
  namespace: yatp|default
rules:
- apiGroups: [""]
  resources: ["jobs"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: yatp-role-binding
  namespace: yatp|default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole|Role
  name: yatp-role
subjects:
- kind: ServiceAccount
  name: default
  namespace: yatp|default

解决方向 #

1. 修正RBAC规则的API组 #

Kubernetes Job属于batch API组，而非核心API组（空字符串""）。你的RBAC规则中apiGroups配置错误，导致权限不匹配。修正后的规则：

rules:
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

2. 验证ServiceAccount的实际权限 #

• 确认容器使用的ServiceAccount：如果Deployment未指定serviceAccountName，默认使用所在Namespace的default ServiceAccount，需确保绑定的RBAC规则对应正确的Namespace。
• 用以下命令验证权限（替换NAMESPACE和SA_NAME为实际值）：

  kubectl auth can-i create jobs -n NAMESPACE --as=system:serviceaccount:NAMESPACE:SA_NAME
  kubectl auth can-i update jobs -n NAMESPACE --as=system:serviceaccount:NAMESPACE:SA_NAME
  

  若返回no，说明RBAC绑定仍存在问题，需检查RoleBinding的subjects和roleRef配置是否正确。

3. 调整代码的Job操作逻辑 #

代码中使用ReplaceNamespacedJob，该操作要求目标Job已存在（对应update权限）。如果Job尚未创建，应先调用CreateNamespacedJob。修改后的代码逻辑：

let startJob (yaml: string) = 
    let job = KubernetesYaml.Deserialize<V1Job>(yaml)
    let client = new Kubernetes(config)
    try
        // 检查Job是否存在
        let existingJob = client.BatchV1.GetNamespacedJob(job.Metadata.Name, job.Metadata.Namespace())
        // 存在则更新
        client.BatchV1.ReplaceNamespacedJob(job, job.Metadata.Name, job.Metadata.Namespace()) |> ignore
    with
    | :? HttpOperationException as ex when ex.Response.StatusCode = System.Net.HttpStatusCode.NotFound ->
        // 不存在则创建
        client.BatchV1.CreateNamespacedJob(job, job.Metadata.Namespace()) |> ignore

4. 排查AKS特定权限限制 #

AKS可能存在额外的权限控制：

• 如果使用Managed Identity，需确保Identity已被授予对应的Kubernetes RBAC权限。
• 检查集群是否启用了Pod Security Policy或其他访问控制机制，限制了ServiceAccount的操作。

内容的提问来源于stack exchange，提问作者Franco Tiveron