安装Helm Chart时ConfigMap变量未生效问题排查

背景信息 #

手动创建的Helm Chart目录结构如下：

├── Chart.yaml
├── Dockerfile
├── locustfile.py
├── requirements.in
├── requirements.txt
├── templates
│   ├── ConfigMap.yaml
│   └── deployment.yaml
└── values.yaml

执行安装命令：

helm upgrade --install loadgenerator .

Pod启动失败，事件报错如下：

Events:
  Type     Reason         Age                    From               Message
  ----     ------         ----                   ----               -------
  Normal   Scheduled      47m                    default-scheduler  Successfully assigned default/loadgenerator-66c46bd489-5rm8v to minikube
  Warning  Failed         45m (x12 over 47m)     kubelet            Error: InvalidImageName
  Warning  InspectFailed  2m17s (x214 over 47m)  kubelet            Failed to apply default image tag "$(image_init_container)": couldn't parse image reference "$(image_init_container)": invalid reference format

现有ConfigMap配置：

apiVersion: v1
kind: ConfigMap
metadata:
  name: app
data:
  image_init_container: "busybox:latest"
  image_app: "loadgenerator"
  namespace: default
  env: prod

现有Deployment配置：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: loadgenerator  
spec:
  selector:
    matchLabels:
      app: loadgenerator
  replicas: 1
  template:
    metadata:
      labels:
        app: loadgenerator        
      annotations:
        sidecar.istio.io/rewriteAppHTTPProbers: "true"
    spec:
      serviceAccountName: default
      terminationGracePeriodSeconds: 5
      restartPolicy: Always
      securityContext:
        fsGroup: 1000
        runAsGroup: 1000
        runAsNonRoot: true
        runAsUser: 1000
      initContainers:
      - command:
        - /bin/sh
        - -exc
        - |
          echo "Init container pinging frontend: ${FRONTEND_ADDR}..."
          STATUSCODE=$(wget --server-response http://${FRONTEND_ADDR} 2>&1 | awk '/^  HTTP/{print $2}')
          if test $STATUSCODE -ne 200; then
              echo "Error: Could not reach frontend - Status code: ${STATUSCODE}"
              exit 1
          fi
        name: frontend-check
        image: $(image_init_container)
        envFrom:
          - configMapRef:
              name: app
        env:
        - name: FRONTEND_ADDR
          value: "frontend:80"
      containers:
      - name: main
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
              - all
          privileged: false
          readOnlyRootFilesystem: true
        image: $(image_app)
        envFrom:
          - configMapRef:
              name: app
        env:
        - name: FRONTEND_ADDR
          value: "frontend:80"
        - name: USERS
          value: "10"
        resources:
          requests:
            cpu: 300m
            memory: 256Mi
          limits:
            cpu: 500m
            memory: 512Mi

错误原因 #

1. Kubernetes不支持在Pod的image字段中直接引用环境变量：你在Deployment里写的$(image_init_container)和$(image_app)不会被解析成ConfigMap中的值，kubelet会直接把这个字符串当作镜像名称，导致镜像格式错误。
2. envFrom的作用范围是容器内部进程：ConfigMap通过envFrom注入的环境变量，只有容器启动后的进程能读取，而镜像名称是kubelet在启动容器前就需要确定的参数，不属于容器内部环境的范畴，无法通过这种方式引用。

正确配置方案 #

利用Helm的模板渲染能力，将镜像配置统一放在values.yaml中，再在ConfigMap和Deployment中引用这些模板变量：

1. 更新values.yaml #

添加镜像相关配置：

# values.yaml
images:
  initContainer: busybox:latest
  app: loadgenerator

2. 修改ConfigMap.yaml #

用Helm模板变量引用values中的值：

apiVersion: v1
kind: ConfigMap
metadata:
  name: app
data:
  image_init_container: {{ .Values.images.initContainer }}
  image_app: {{ .Values.images.app }}
  namespace: default
  env: prod

3. 修改Deployment.yaml #

将原来的$(xxx)替换为Helm模板变量，确保渲染后的Deployment使用正确的镜像名称：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: loadgenerator  
spec:
  selector:
    matchLabels:
      app: loadgenerator
  replicas: 1
  template:
    metadata:
      labels:
        app: loadgenerator        
      annotations:
        sidecar.istio.io/rewriteAppHTTPProbers: "true"
    spec:
      serviceAccountName: default
      terminationGracePeriodSeconds: 5
      restartPolicy: Always
      securityContext:
        fsGroup: 1000
        runAsGroup: 1000
        runAsNonRoot: true
        runAsUser: 1000
      initContainers:
      - command:
        - /bin/sh
        - -exc
        - |
          echo "Init container pinging frontend: ${FRONTEND_ADDR}..."
          STATUSCODE=$(wget --server-response http://${FRONTEND_ADDR} 2>&1 | awk '/^  HTTP/{print $2}')
          if test $STATUSCODE -ne 200; then
              echo "Error: Could not reach frontend - Status code: ${STATUSCODE}"
              exit 1
          fi
        name: frontend-check
        image: {{ .Values.images.initContainer }}
        envFrom:
          - configMapRef:
              name: app
        env:
        - name: FRONTEND_ADDR
          value: "frontend:80"
      containers:
      - name: main
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
              - all
          privileged: false
          readOnlyRootFilesystem: true
        image: {{ .Values.images.app }}
        envFrom:
          - configMapRef:
              name: app
        env:
        - name: FRONTEND_ADDR
          value: "frontend:80"
        - name: USERS
          value: "10"
        resources:
          requests:
            cpu: 300m
            memory: 256Mi
          limits:
            cpu: 500m
            memory: 512Mi

4. 验证 #

重新执行安装命令：

helm upgrade --install loadgenerator .

此时Helm会将模板变量渲染为实际的镜像名称，kubelet能正常解析镜像并启动Pod，同时ConfigMap中也会保存镜像相关的环境变量供容器内部使用。

内容的提问来源于stack exchange，提问作者Iceforest