Grafana数据源配置重载失败：HTTP请求发送至HTTPS服务器

• 部署用于添加数据源的ConfigMap后无任何变化，该ConfigMap已存在于集群的正确命名空间中。
• 修改ConfigMap后，查看grafana-sc-datasources容器日志发现错误：POST请求发送至http://localhost:3000/api/admin/provisioning/datasources/reload，响应为400 Bad Request Client sent an HTTP request to an HTTPS server，推测请求失败导致数据源未更新。
• 将grafana.ini中server.protocol改为http时，修改ConfigMap后容器日志显示请求成功（响应200 OK），但无法打开Grafana网页。
• 需求：让Grafana的数据源重载POST请求使用HTTPS发送，同时保证网页正常访问。

相关配置 #

待配置的ConfigMap #

apiVersion: v1
kind: ConfigMap
metadata:
  name: jaeger-${NACKLE_ENV}-grafana-datasource
  labels:
    grafana_datasource: '1'
data:
  jaeger-datasource.yaml: |-
    apiVersion: 1
    datasources:
    - name: Jaeger-${NACKLE_ENV}
      type: jaeger
      access: browser
      url: http://jaeger-${NACKLE_ENV}-query.${NACKLE_ENV}.svc.cluster.local:16690
      version: 1
      basicAuth: false

当前Grafana Values文件 #

# use 1 replica when using a StatefulSet
# If we need more than 1 replica, then we'll have to:
# - remove the `persistence` section below
# - use an external database for all replicas to connect to (refer to Grafana Helm chart docs)
replicas: 1

image:
  pullSecrets:
    - docker-hub

affinity:
  nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 1
      preference:
        matchExpressions:
        - key: eks.amazonaws.com/capacityType
          operator: In
          values:
          - ON_DEMAND

persistence:
  enabled: true
  type: statefulset
  storageClassName: biw-durable-gp2

podDisruptionBudget:
  maxUnavailable: 1

admin:
  existingSecret: grafana

sidecar:
  datasources:
    enabled: true
    label: grafana_datasource
  dashboards:
    enabled: true
    label: grafana_dashboard
    labelValue: 1
dashboardProviders:
  dashboardproviders.yaml:
    apiVersion: 1
    providers:
    - name: 'default'
      orgId: 1
      folder: ''
      type: file
      disableDeletion: false
      editable: true
      options:
        path: /var/lib/grafana/dashboards/default

dashboards:
  default:
    node-exporter:
      gnetId: 1860
      revision: 23
      datasource: Prometheus
    core-dns:
      gnetId: 12539
      revision: 5
      datasource: Prometheus
    fluentd:
      gnetId: 7752
      revision: 6
      datasource: Prometheus

ingress:
  apiVersion: networking.k8s.io/v1
  enabled: true
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/healthcheck-port: traffic-port
    alb.ingress.kubernetes.io/healthcheck-path: '/api/health'
    alb.ingress.kubernetes.io/healthcheck-protocol: HTTPS
    alb.ingress.kubernetes.io/backend-protocol: HTTPS
    # Redirect to HTTPS at the ALB
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
spec:
  rules:
    - http:
        paths:
          - path: /*
            pathType: ImplementationSpecific
            backend:
              service:
                name: ssl-redirect
                port:
                  name: use-annotation
  defaultBackend:
    service:
      name: grafana
      port:
        number: 80


livenessProbe: { "httpGet": { "path": "/api/health", "port": 3000, "scheme": "HTTPS" }, "initialDelaySeconds": 60, "timeoutSeconds": 30, "failureThreshold": 10 }
readinessProbe: { "httpGet": { "path": "/api/health", "port": 3000, "scheme": "HTTPS" } }

service:
  type: NodePort
  name: grafana
  rolePrefix: app-role
  env: eks-test

serviceAccount:
  name: grafana
  annotations:
    eks.amazonaws.com/role-arn: ""

pod:
  spec:
    serviceAccountName: grafana 

grafana.ini:
  server:
    # don't use enforce_domain - it causes an infinite redirect in our setup
    # enforce_domain: true
    enable_gzip: true
    # NOTE - if I set the protocol to http I do see it make changes to datasources but I can not see the website 
    protocol: https
    cert_file: /biw-cert/domain.crt
    cert_key: /biw-cert/domain.key
  users:
    auto_assign_org_role: Editor
  # https://grafana.com/docs/grafana/v6.5/auth/gitlab/
  auth.gitlab:
    enabled: true
    allow_sign_up: true
    org_role: Editor
    scopes: read_api
    auth_url: https://gitlab.biw-services.com/oauth/authorize
    token_url: https://gitlab.biw-services.com/oauth/token
    api_url: https://gitlab.biw-services.com/api/v4
    allowed_groups: nackle-teams/devops

securityContext:
  fsGroup: 472
  runAsUser: 472
  runAsGroup: 472

extraConfigmapMounts:
  - name: "cert-configmap"
    mountPath: "/biw-cert"
    subPath: ""
    configMap: biw-grafana-cert
    readOnly: true

在Grafana的Helm Values文件中，修改sidecar.datasources配置，指定HTTPS的重载URL并跳过本地SSL证书验证：

sidecar:
  datasources:
    enabled: true
    label: grafana_datasource
    # 替换为HTTPS的重载请求地址
    url: https://localhost:3000/api/admin/provisioning/datasources/reload
    # 跳过自签证书的SSL验证（如果使用集群内自签证书）
    skipTLSVerify: true

说明 #

• url参数覆盖sidecar默认的HTTP请求地址，与Grafana运行的HTTPS协议匹配，避免协议不兼容导致的400错误
• skipTLSVerify用于跳过对本地自签证书的信任校验，因为sidecar容器默认不会信任集群内生成的自签证书，开启后可避免SSL握手失败

修改后重新部署Grafana Helm Chart，即可实现ConfigMap更新后，sidecar通过HTTPS发送重载请求，同时Grafana网页可正常通过HTTPS访问。

内容的提问来源于stack exchange，提问作者ErnieAndBert