如何基于Grafana仪表板变量动态设置Cortex数据源HTTP头

问题说明 #

使用Cortex多租户架构时，所有读写请求必须携带X-Scope-OrgID HTTP请求头。写入流程正常，但无法将仪表板下拉框选择的租户名称动态作为该请求头的值传递给Cortex数据源。当前Cortex数据源配置中该头值为硬编码的cust-3b74e7dd-808d-default，可正常加载对应租户指标，需求是切换下拉租户（如cust-test-1234）时，自动切换X-Scope-OrgID的值，租户名称来源于Prometheus数据源。

当前配置 #

Grafana数据源配置 #

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: "http://prom-kube-prometheus-stack-prometheus.observability:9090"
  - name: Cortex
    type: prometheus
    url: "http://cortex-query-frontend-headless.cortex:8080/prometheus"
    isDefault: true
    jsonData:
      httpHeaderName1: 'X-Scope-OrgID'
    secureJsonData:
      httpHeaderValue1: 'cust-3b74e7dd-808d-default'

仪表板JSON配置 #

{
  "annotations": {
    "list": [
      {
        "builtIn": 1,
        "datasource": {
          "type": "grafana",
          "uid": "-- Grafana --"
        },
        "enable": true,
        "hide": true,
        "iconColor": "rgba(0, 211, 255, 1)",
        "name": "Annotations & Alerts",
        "target": {
          "limit": 100,
          "matchAny": false,
          "tags": [],
          "type": "dashboard"
        },
        "type": "dashboard"
      }
    ]
  },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 0,
  "id": 2,
  "links": [],
  "liveNow": false,
  "panels": [
    {
      "datasource": {
        "type": "prometheus",
        "uid": "P6693426190CB2316"
      },
      "fieldConfig": {
        "defaults": {
          "color": {
            "mode": "palette-classic"
          },
          "custom": {
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
            "axisPlacement": "auto",
            "barAlignment": 0,
            "drawStyle": "line",
            "fillOpacity": 0,
            "gradientMode": "none",
            "hideFrom": {
              "legend": false,
              "tooltip": false,
              "viz": false
            },
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
            "scaleDistribution": {
              "type": "linear"
            },
            "showPoints": "auto",
            "spanNulls": false,
            "stacking": {
              "group": "A",
              "mode": "none"
            },
            "thresholdsStyle": {
              "mode": "off"
            }
          },
          "mappings": [],
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 80
              }
            ]
          }
        },
        "overrides": []
      },
      "gridPos": {
        "h": 9,
        "w": 12,
        "x": 7,
        "y": 0
      },
      "id": 2,
      "options": {
        "legend": {
          "calcs": [],
          "displayMode": "list",
          "placement": "bottom",
          "showLegend": true
        },
        "tooltip": {
          "mode": "single",
          "sort": "none"
        }
      },
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "P6693426190CB2316"
          },
          "editorMode": "code",
          "expr": "sum by (id) (dummy_foo)",
          "legendFormat": "__auto",
          "range": true,
          "refId": "A"
        }
      ],
      "title": "Foo",
      "type": "timeseries"
    }
  ],
  "schemaVersion": 37,
  "style": "dark",
  "tags": [],
  "templating": {
    "list": [
      {
        "current": {
          "selected": true,
          "text": "cust-15276cac-808d-default",
          "value": "cust-15276cac-808d-default"
        },
        "datasource": {
          "type": "prometheus",
          "uid": "PBFA97CFB590B2093"
        },
        "definition": "label_values(namespace)",
        "hide": 0,
        "includeAll": false,
        "multi": false,
        "name": "tenant",
        "options": [],
        "query": {
          "query": "label_values(namespace)",
          "refId": "StandardVariableQuery"
        },
        "refresh": 1,
        "regex": "cust-.*",
        "skipUrlSync": false,
        "sort": 0,
        "type": "query"
      }
    ]
  },
  "time": {
    "from": "now-5m",
    "to": "now"
  },
  "timepicker": {},
  "timezone": "",
  "title": "My dashboard",
  "uid": "nqdvuvK4z",
  "version": 1,
  "weekStart": ""
}

解决方案 #

1. 修改Cortex数据源配置，启用动态请求头 #

将硬编码的X-Scope-OrgID值替换为仪表板的tenant变量，同时开启允许在HTTP头中使用变量的设置：

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: "http://prom-kube-prometheus-stack-prometheus.observability:9090"
  - name: Cortex
    type: prometheus
    url: "http://cortex-query-frontend-headless.cortex:8080/prometheus"
    isDefault: true
    jsonData:
      httpHeaderName1: 'X-Scope-OrgID'
      httpHeaderValue1: '$tenant' # 绑定仪表板的tenant变量
      allowVariablesInHeaders: true # 必须开启此选项让变量在请求头中生效
    # 移除硬编码的secureJsonData配置

2. 确认仪表板租户变量配置 #

当前仪表板已通过Prometheus的label_values(namespace)获取租户列表并过滤cust-.*，变量名称为tenant，与数据源中的变量引用一致，无需额外修改。

3. 调整面板数据源指向Cortex #

将面板的数据源从Prometheus切换为Cortex，确保请求发送到Cortex并携带动态头：
修改面板的targets及顶层datasource配置：

"datasource": {
  "type": "prometheus",
  "uid": "<你的Cortex数据源UID>" // 替换为实际的Cortex数据源UID
},
"targets": [
  {
    "datasource": {
      "type": "prometheus",
      "uid": "<你的Cortex数据源UID>"
    },
    "editorMode": "code",
    "expr": "sum by (id) (dummy_foo)",
    "legendFormat": "__auto",
    "range": true,
    "refId": "A"
  }
]

4. 测试验证 #

• 切换仪表板的租户下拉框，选择不同租户名称
• 打开浏览器开发者工具，查看Cortex请求的HTTP头，确认X-Scope-OrgID值与选中租户一致
• 验证对应租户的指标是否正常加载

内容的提问来源于stack exchange，提问作者Zulhilmi Zainudin