K3s集群中通过Traefik Ingress暴露Thanos Sidecar gRPC服务失败求助

环境信息 #

Pod详情（Thanos Sidecar作为Prometheus的Sidecar运行） #

$ kubectl describe pod prometheus-monitoring-prometheus-0 -n monitoring
Name:             prometheus-monitoring-prometheus-0
Namespace:        monitoring
Priority:         0
Service Account:  monitoring-prometheus
Node:             k3s-node-1/12.345.678.910
Start Time:       Wed, 26 Jul 2023 18:35:38 +0000
Labels:           app.kubernetes.io/instance=monitoring-prometheus
                  app.kubernetes.io/managed-by=prometheus-operator
                  app.kubernetes.io/name=prometheus
                  ...
                  prometheus=monitoring-prometheus
                  statefulset.kubernetes.io/pod-name=prometheus-monitoring-prometheus-0
Annotations:      kubectl.kubernetes.io/default-container: prometheus
Status:           Running
IP:               10.42.0.200
IPs:
  IP:           10.42.0.200
Controlled By:  StatefulSet/prometheus-monitoring-prometheus
...
Containers:
  ...
  thanos-sidecar:
    Container ID:  containerd://bdc1bbfe53bf1ea260c47a44ab26110432388fe5592e037c83da5c6b6c5f696f
    Image:         quay.io/thanos/thanos:v0.31.0 
    Image ID:      quay.io/thanos/thanos@sha256:e7d337d6ac24233f0f9314ec9830291789e16e2b480b9d353be02d05ce7f2a7e
    Ports:         10902/TCP, 10901/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      sidecar
      --prometheus.url=http://127.0.0.1:9090/
      --prometheus.http-client={"tls_config": {"insecure_skip_verify":true}}
      --grpc-address=:10901
      --http-address=:10902
      --objstore.config=$(OBJSTORE_CONFIG)
      --tsdb.path=/prometheus
      --log.level=info
      --log.format=logfmt
    State:          Running
      Started:      Wed, 26 Jul 2023 18:35:41 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      OBJSTORE_CONFIG:  <set to the key 'objstore.yml' in secret 'my-s3-bucket'>  Optional: false
    Mounts:
      /prometheus from prometheus-monitoring-prometheus-db (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-slz8t (ro)
...

Service配置 #

$ kubectl describe svc monitoring-thanos-discovery -n monitoring
Name:              monitoring-thanos-discovery
Namespace:         monitoring
Labels:            app=monitoring-thanos-discovery
                   app.kubernetes.io/instance=monitoring
                   app.kubernetes.io/managed-by=Helm
                   app.kubernetes.io/part-of=monitoring
                   app.kubernetes.io/version=47.2.0
                   chart=kube-prometheus-stack-47.2.0
                   heritage=Helm
                   release=monitoring
Annotations:       meta.helm.sh/release-name: monitoring
                   meta.helm.sh/release-namespace: monitoring
                   traefik.ingress.kubernetes.io/service.serversscheme: h2c
Selector:          app.kubernetes.io/name=prometheus,prometheus=monitoring-prometheus
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                None
IPs:               None
Port:              grpc  10901/TCP
TargetPort:        grpc/TCP
Endpoints:         10.42.0.200:10901
Port:              http  10902/TCP
TargetPort:        http/TCP
Endpoints:         10.42.0.200:10902
Session Affinity:  None
Events:            <none>

Ingress相关配置 #

用于获取TLS证书的Dummy Ingress (thanos-ingress-dummy.yaml)

# 用于为指定域名获取证书（配合IngressRoute使用）
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: thanos-discovery-ingress-dummy
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  rules:
    - host: "thanos-gateway.monitoring.domain.com"
      http:
        paths:
          - path: /cert-placeholder
            pathType: Prefix
            backend:
              service:
                name: monitoring-thanos-discovery
                port:
                  name: grpc
  tls:
    - hosts:
        - "thanos-gateway.monitoring.domain.com"
      secretName: thanos-sidecar-grpc-tls

Traefik IngressRoute配置 (thanos-ingressroute.yaml)

# 用于暴露gRPC服务（支持HTTP2上的gRPC）
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: thanos-discovery-ingress
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`thanos-gateway.monitoring.domain.com`)
      kind: Rule
      services:
        - name: monitoring-thanos-discovery
          port: grpc
  tls:
    secretName: thanos-sidecar-grpc-tls

问题描述 #

• 集群内部可正常访问：通过Service域名调用gRPC健康检查接口成功

$ kubectl exec -it debian-debug -- bash
root@debian-debug:/# grpcurl -plaintext monitoring-thanos-discovery.monitoring.svc.cluster.local:10901 grpc.health.v1.Health.Check
{
  "status": "SERVING"
}

• 外部访问失败：
  • gRPC调用报错：

  $ grpcurl --plaintext thanos-gateway.monitoring.domain.com:443 list
  Failed to list services: server does not support the reflection API
  

  • Curl请求返回错误：

  $ curl https://thanos-gateway.monitoring.domain.com
  Internal Server Error
  
  $ curl http://thanos-gateway.monitoring.domain.com
  404 page not found
  

排查与修复建议 #

1. 修正IngressRoute的gRPC服务转发配置 #

Thanos Sidecar的gRPC端口使用明文HTTP/2（h2c），当前IngressRoute未明确指定服务通信协议，导致Traefik可能用HTTP/1.1转发请求引发500错误。需在IngressRoute的service配置中添加scheme: h2c：

修改后的thanos-ingressroute.yaml：

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: thanos-discovery-ingress
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`thanos-gateway.monitoring.domain.com`)
      kind: Rule
      services:
        - name: monitoring-thanos-discovery
          port: grpc
          scheme: h2c  # 新增：指定使用h2c协议转发到后端服务
  tls:
    secretName: thanos-sidecar-grpc-tls

2. 修正外部gRPC调用命令 #

外部访问使用443端口的HTTPS服务，不能用--plaintext参数，正确调用命令：

# 调用健康检查接口
grpcurl thanos-gateway.monitoring.domain.com:443 grpc.health.v1.Health.Check

# 若需查看服务列表，需先给Thanos Sidecar添加启动参数--grpc-reflection启用反射API，再执行：
# grpcurl thanos-gateway.monitoring.domain.com:443 list

3. 验证Traefik的HTTP/2支持 #

确保Traefik的websecure入口点已启用HTTP/2，检查静态配置（ConfigMap或启动参数）：

entryPoints:
  websecure:
    address: ":443"
    http:
      tls: {}
      http2: {}  # 确认HTTP/2已启用

4. 检查AWS NLB配置 #

• 确认NLB目标组为TCP模式（而非HTTP），保证Traefik处理TLS终止后，gRPC转发的TCP连接完整性。
• 确保NLB安全组允许443端口入站流量。
• 验证NLB目标健康检查正常，Traefik Pod的443端口处于健康状态。

5. 查看Traefik详细日志 #

启用Traefik debug日志获取请求处理细节：

# 修改Traefik Deployment添加debug日志参数
kubectl edit deployment traefik -n kube-system

# 实时查看Traefik日志
kubectl logs -n kube-system -l app=traefik -f

内容的提问来源于stack exchange，提问作者Beefcake