ArgoCD ApplicationSet跨子生成器参数无法使用问题求助

问题 #

需要在所有集群以ArgoCD ApplicationSet形式部署多个Operator，同时支持灵活跳过特定集群的部分Operator部署。配置了带matrix生成器的ApplicationSet后，部署时报错：

Status:
  Conditions:
    Last Transition Time:  2024-07-31T05:22:57Z
    Message:               failed to get params for second generator in the matrix generator: child generator returned an error on parameter generation: failed to replace parameters in generator: failed to deeply replace JSON field contents: failed to execute go template {{.enableMetricsServer}}: template: :1:2: executing "" at <.enableMetricsServer>: map has no entry for key "enableMetricsServer"
    Reason:                ApplicationGenerationFromParamsError
    Status:                True
    Type:                  ErrorOccurred
    Last Transition Time:  2024-07-31T05:22:57Z
    Message:               failed to get params for second generator in the matrix generator: child generator returned an error on parameter generation: failed to replace parameters in generator: failed to deeply replace JSON field contents: failed to execute go template {{.enableMetricsServer}}: template: :1:2: executing "" at <.enableMetricsServer>: map has no entry for key "enableMetricsServer"
    Reason:                ErrorOccurred
    Status:                False
    Type:                  ParametersGenerated
    Last Transition Time:  2024-07-31T05:22:57Z
    Message:               failed to get params for second generator in the matrix generator: child generator returned an error on parameter generation: failed to replace parameters in generator: failed to deeply replace JSON field contents: failed to execute go template {{.enableMetricsServer}}: template: :1:2: executing "" at <.enableMetricsServer>: map has no entry for key "enableMetricsServer"
    Reason:                ApplicationGenerationFromParamsError
    Status:                False
    Type:                  ResourcesUpToDate

原配置如下：

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: core-operators
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - matrix:
        generators:
          - merge:
              mergeKeys: 
                - server
              generators:
                - clusters: {}
                - list:
                    elements:
                      - server: '{{.server}}'
                        clusterName: '{{.name}}'
                        enableMetricsServer: true
                        enablePrometheus: true
                      - server: https://kubernetes.default.svc
                        clusterName: in-cluster
                        enableMetricsServer: true
                        enablePrometheus: false
          - list:
              elements:
                - operatorName: metrics-server
                  namespace: monitoring
                  enable: '{{.enableMetricsServer}}'
                - operatorName: prometheus
                  namespace: monitoring
                  enable: '{{.enablePrometheus}}'
  template:
    metadata:
      name: '{{.name}}-{{.operatorName}}-{{.enable}}'
      annotations:
        argocd.argoproj.io/manifest-generate-paths: ".;.."
    spec:
      project: default
      source:
        repoURL: git@github.com/operators.git
        targetRevision: HEAD
        path: null-deployment
      destination:
        server: '{{.server}}'
        namespace: '{{.namespace}}'
      syncPolicy:
        syncOptions:
          - CreateNamespace=true
          - ServerSideApply=true
          - Replace=true
          - Force=true
        retry:
          limit: 2
  templatePatch: |
    {{- if eq .enable "true" }}
    spec:
      source:
        repoURL: git@github.com/operators.git
        targetRevision: HEAD
        path: {{.operatorName}}
        helm:
          releaseName: "{{.operatorName}}-parent"
          values: |
            clusterName: {{.name}}
      syncPolicy:
        automated:
          prune: true
    {{- end }}

使用ArgoCD版本v2.11.7，曾看到相关PR标注修复但问题依旧。

解决方案 #

错误根源在于matrix生成器的子生成器是独立执行的，第二个list生成器无法直接引用第一个merge生成器的参数，导致模板渲染时找不到对应键。以下是两种可行的修正方案：

方案一：动态键引用修正配置 #

将Operator的启用状态改为通过键名动态获取，避免跨生成器直接引用参数：

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: core-operators
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - matrix:
        generators:
          # 合并集群列表与自定义启用规则
          - merge:
              mergeKeys: [server]
              generators:
                - clusters: {}
                - list:
                    elements:
                      - server: '{{.server}}'
                        enableMetricsServer: true
                        enablePrometheus: true
                      - server: https://kubernetes.default.svc
                        enableMetricsServer: true
                        enablePrometheus: false
          # 定义Operator列表，传递对应的启用键名
          - list:
              elements:
                - operatorName: metrics-server
                  namespace: monitoring
                  enableKey: "enableMetricsServer"
                - operatorName: prometheus
                  namespace: monitoring
                  enableKey: "enablePrometheus"
  template:
    metadata:
      name: '{{.name}}-{{.operatorName}}'
      annotations:
        argocd.argoproj.io/manifest-generate-paths: ".;.."
    spec:
      project: default
      source:
        repoURL: git@github.com/operators.git
        targetRevision: HEAD
        path: null-deployment
      destination:
        server: '{{.server}}'
        namespace: '{{.namespace}}'
      syncPolicy:
        syncOptions:
          - CreateNamespace=true
          - ServerSideApply=true
          - Replace=true
          - Force=true
        retry:
          limit: 2
  templatePatch: |
    {{- $enable := index . .enableKey }}
    {{- if $enable }}
    spec:
      source:
        repoURL: git@github.com/operators.git
        targetRevision: HEAD
        path: {{.operatorName}}
        helm:
          releaseName: "{{.operatorName}}-parent"
          values: |
            clusterName: {{.name}}
      syncPolicy:
        automated:
          prune: true
    {{- end }}

关键修改点

• Operator生成器传递enableKey而非直接引用具体参数，在templatePatch中用index . .enableKey动态提取当前集群的启用状态
• 简化Application名称，去掉冗余的{{.enable}}后缀
• 避免直接引用未定义的参数键，解决模板渲染报错问题

方案二：通过ConfigMap注入集群配置 #

如果需要更灵活的集群规则管理，可以将启用配置存储在ConfigMap中，通过clusters生成器的valuesFrom注入：

# 先创建集群配置ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-operator-configs
  namespace: argocd
data:
  in-cluster: |
    enableMetricsServer: true
    enablePrometheus: false
  default: |
    enableMetricsServer: true
    enablePrometheus: true

对应的ApplicationSet配置：

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: core-operators
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - matrix:
        generators:
          - clusters:
              valuesFrom:
                - configMapRef:
                    name: cluster-operator-configs
                    key: '{{.name}}'
                    defaultKey: "default"
          - list:
              elements:
                - operatorName: metrics-server
                  namespace: monitoring
                  enableKey: "enableMetricsServer"
                - operatorName: prometheus
                  namespace: monitoring
                  enableKey: "enablePrometheus"
  template:
    metadata:
      name: '{{.name}}-{{.operatorName}}'
      annotations:
        argocd.argoproj.io/manifest-generate-paths: ".;.."
    spec:
      project: default
      source:
        repoURL: git@github.com/operators.git
        targetRevision: HEAD
        path: null-deployment
      destination:
        server: '{{.server}}'
        namespace: '{{.namespace}}'
      syncPolicy:
        syncOptions:
          - CreateNamespace=true
          - ServerSideApply=true
          - Replace=true
          - Force=true
        retry:
          limit: 2
  templatePatch: |
    {{- $enable := index . .enableKey }}
    {{- if $enable }}
    spec:
      source:
        repoURL: git@github.com/operators.git
        targetRevision: HEAD
        path: {{.operatorName}}
        helm:
          releaseName: "{{.operatorName}}-parent"
          values: |
            clusterName: {{.name}}
      syncPolicy:
        automated:
          prune: true
    {{- end }}

内容的提问来源于stack exchange，提问作者Devam Parikh