使用CDK时，如何等待Helm安装的Operator就绪后再应用清单？

问题场景 #

使用CDK将External Secrets Operator（ESO）与Apache Pinot部署到EKS集群时，CDK会在ESO完全启动运行前尝试创建SecretStore资源，导致部署失败。手动分两次部署（先部署ESO和Pinot，再部署SecretStore）可正常运行。

相关代码 #

// install Pinot
const pinot = cluster.addHelmChart('Pinot', {
    chartAsset: new Asset(this, 'PinotChartAsset', { path: path.join(__dirname, '../pinot') }),
    release: 'pinot',
    namespace: 'pinot',
    createNamespace: true
});


// install the External Secrets Operator
const externalSecretsOperator = cluster.addHelmChart('ExternalSecretsOperator', {
    chart: 'external-secrets',
    release: 'external-secrets',
    repository: 'https://charts.external-secrets.io',
    namespace: 'external-secrets',
    createNamespace: true,
    values: {
        installCRDs: true,
        webhook: {
            port: 9443
        }
    }
});


// create a Fargate Profile
const fargateProfile = cluster.addFargateProfile('FargateProfile', {
    fargateProfileName: 'externalsecrets',
    selectors: [{ 'namespace': 'external-secrets' }]
});


// create the Service Account used by the Secret Store
const serviceAccount = cluster.addServiceAccount('ServiceAccount', {
    name: 'eso-service-account',
    namespace: 'external-secrets'
});
serviceAccount.addToPrincipalPolicy(new iam.PolicyStatement({
    effect: iam.Effect.ALLOW,
    actions: [
        'secretsmanager:GetSecretValue',
        'secretsmanager:DescribeSecret'
    ],
    resources: [
        'arn:aws:secretsmanager:us-east-1:<MY-ACCOUNT-ID>:secret:*'
    ]
}))
serviceAccount.node.addDependency(externalSecretsOperator);


// create the Secret Store, an ESO Resource
const secretStoreManifest = getSecretStoreManifest(serviceAccount);
const secretStore = cluster.addManifest('SecretStore', secretStoreManifest);
secretStore.node.addDependency(serviceAccount);
secretStore.node.addDependency(fargateProfile);


// create the External Secret, another ESO resource
const externalSecretManifest = getExternalSecretManifest(secretStoreManifest)
const externalSecret = cluster.addManifest('ExternalSecret', externalSecretManifest)
externalSecret.node.addDependency(secretStore);
externalSecret.node.addDependency(pinot);

部署错误信息 #

Received response status [FAILED] from custom resource. Message returned: Error: b'Error from server (InternalError): error when creating "/tmp/manifest.yaml": Internal error occurred:
failed calling webhook "validate.clustersecretstore.external-secrets.io": Post "https://external-secrets-webhook.external-secrets.svc:443/validate-external-secrets-io-v1beta1-clustersecretstore?timeout=5s": no endpoints available for service "external-secrets-webhook"
'

解决方案 #

1. 用Kubernetes Job等待ESO Webhook就绪 #

创建一个Job，持续检查ESO的webhook服务是否有可用端点，直到就绪后再继续部署SecretStore。

在CDK中添加这个Job的manifest，并让SecretStore依赖该Job：

// 创建等待ESO就绪的Job
const waitForEsoJob = cluster.addManifest('WaitForEsoWebhook', {
  apiVersion: 'batch/v1',
  kind: 'Job',
  metadata: {
    name: 'wait-for-eso-webhook',
    namespace: 'external-secrets'
  },
  spec: {
    template: {
      spec: {
        containers: [
          {
            name: 'wait-for-webhook',
            image: 'bitnami/kubectl:latest',
            command: [
              'sh', '-c',
              'until kubectl get endpoints external-secrets-webhook -n external-secrets | grep -E \'READY.*[1-9]\'; do sleep 5; done'
            ]
          }
        ],
        restartPolicy: 'OnFailure'
      }
    },
    backoffLimit: 10
  }
});

// 让SecretStore依赖这个Job
waitForEsoJob.node.addDependency(externalSecretsOperator);
secretStore.node.addDependency(waitForEsoJob);

2. 拆分CDK栈 #

将ESO、Pinot和Fargate Profile部署到基础栈，将ServiceAccount、SecretStore和ExternalSecret部署到依赖栈。通过CDK的栈依赖机制，确保基础栈完全部署完成后再部署依赖栈。

示例：

• 基础栈（EsoPinotStack）：包含ESO Helm Chart、Pinot Helm Chart、Fargate Profile
• 依赖栈（EsoSecretsStack）：包含ServiceAccount、SecretStore、ExternalSecret，且依赖EsoPinotStack

3. 自定义CDK等待资源 #

编写一个Lambda函数，用于检查ESO的Pod和webhook服务状态，直到就绪。将这个Lambda包装成CDK自定义资源，让SecretStore依赖该资源。

核心逻辑示例（Lambda代码）：

import boto3
import subprocess
import time

eks_client = boto3.client('eks')

def lambda_handler(event, context):
    cluster_name = event['ClusterName']
    # 获取kubeconfig
    eks_client.update_kubeconfig(name=cluster_name)
    
    # 等待ESO Pod就绪
    while True:
        result = subprocess.run(['kubectl', 'get', 'pods', '-n', 'external-secrets', '-l', 'app.kubernetes.io/name=external-secrets'], capture_output=True, text=True)
        if 'Running' in result.stdout and '1/1' in result.stdout:
            break
        time.sleep(5)
    
    # 等待webhook端点就绪
    while True:
        result = subprocess.run(['kubectl', 'get', 'endpoints', 'external-secrets-webhook', '-n', 'external-secrets'], capture_output=True, text=True)
        if 'READY' in result.stdout and any(c.isdigit() and int(c) > 0 for c in result.stdout.split() if c.isdigit()):
            break
        time.sleep(5)
    
    return {'status': 'ready'}

在CDK中创建自定义资源，并让SecretStore依赖它。

内容的提问来源于stack exchange，提问作者James Kelleher