使用Launch Template创建AWS EKS节点组失败求助
Could not join nodes to cluster 我在使用Launch Template创建EKS节点组时遇到错误Could not join nodes to cluster。
如果不指定Launch Template,直接在创建节点组控制台配置中选择AMI,则可以成功创建节点组。请注意其他所有配置均保持一致(VPC、私有子网、安全组、IAM角色、实例类型、最小/最大实例数)。此时我选择的AMI ID为ami-06be503a3852b6423,使用g4dn.2xlarge按需实例。
启动模板详情
{ "LaunchTemplateVersions": [ { "LaunchTemplateId": "TEMPLATEID", "LaunchTemplateName": "eks-gpu-optimized-template-engine-primary", "VersionNumber": 2, "VersionDescription": "adding key", "CreateTime": "2024-01-17T21:13:31+00:00", "CreatedBy": "arn:aws:iam::USER", "DefaultVersion": false, "LaunchTemplateData": { "BlockDeviceMappings": [ { "DeviceName": "/dev/sdb", "VirtualName": "ephemeral0" } ], "ImageId": "ami-099c85b23bfc2fd16", "InstanceType": "g4dn.2xlarge", "KeyName": "XXXXXXXXX", "SecurityGroupIds": [ "sg-XXXXXXXXXXXXX" ] } } ] }
IAM角色与安全组详情
EKS集群IAM角色
{ "Role": { "Path": "/", "RoleName": "eksClusterRole", "RoleId": "ROLEID", "Arn": "arn:aws:iam::XXXXX:role/eksClusterRole", "CreateDate": "2023-11-27T17:57:08+00:00", "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": [ "ec2.amazonaws.com", "eks.amazonaws.com" ] }, "Action": "sts:AssumeRole" } ] }, "Description": "Allows access to other AWS service resources that are required to operate clusters managed by EKS.", "MaxSessionDuration": 3600, "RoleLastUsed": { "LastUsedDate": "2024-01-17T22:40:41+00:00", "Region": "us-east-2" } } }
该角色附加了以下策略:
- AmazonEC2ContainerRegistryReadOnly(AWS托管策略)
- AmazonEKS_CNI_Policy(AWS托管策略)
- AmazonEKSClusterPolicy(AWS托管策略)
- AmazonEKSServicePolicy(AWS托管策略)
- AmazonEKSVPCResourceController(AWS托管策略)
- AmazonEKSWorkerNodePolicy(AWS托管策略)
注意:我同时将此角色用于EKS集群和节点组
我知道这不是最佳实践,存在冗余权限,但该权限范围对集群和节点组均适用,且不使用Launch Template创建节点组时此配置可正常工作。
安全组
我在该环境中所有资源均使用同一个安全组。我知道这不是最佳实践,但默认配置允许所有入站和出站流量,仅用于排查错误来源。
VPC与子网
我在EKS集群所在的同一个VPC中创建节点组。节点组被分配到2个私有子网——EKS集群配置为使用VPC中的所有子网。
注意:我已在EKS集群的endpoint_access管理中启用私有访问。
注意:我的私有子网已关联NAT网关,可访问公共子网和互联网网关
再次说明,使用相同子网和角色配置但不使用启动模板时可正常工作
EKS集群详情
{ "cluster": { "name": "us-2-prod-primary-cluster", "arn": "arn:aws:eks:us-east-2:XXXXXX:cluster/us-2-prod-primary-cluster", "createdAt": "2024-01-11T14:18:07.322000+00:00", "version": "1.28", "endpoint": "https://XXXXXXXXXXXXX.gr7.us-east-2.eks.amazonaws.com", "roleArn": "arn:aws:iam::792342206980:role/eksClusterRole", "resourcesVpcConfig": { "subnetIds": [ "subnet-xdd5", "subnet-x9bb", "subnet-xef7", "subnet-x5e7", "subnet-x84e", "subnet-xcf4", "subnet-xbc4", "subnet-x785", "subnet-x04c" ], "securityGroupIds": [ "sg-XXXXX" ], "clusterSecurityGroupId": "sg-XXX", "vpcId": "vpc-0f7294aef4d6f6e6f", "endpointPublicAccess": false, "endpointPrivateAccess": true, "publicAccessCidrs": [] }, "kubernetesNetworkConfig": { "serviceIpv4Cidr": "10.100.0.0/16", "ipFamily": "ipv4" }, "logging": { "clusterLogging": [ { "types": [ "api", "audit",
其他信息
我的启动模板中使用的是自定义AMI。
- 我首先从不使用Launch Template创建节点组时可正常工作的同一个AMI(
AMI ID: ami-06be503a3852b6423)创建了一个实例 - 我连接到这个新创建的实例,创建了容器化应用所需的目录,并安装了DeepGram部署所需的所有模型
- 然后我基于该实例制作了一个AMI(以便节点组在启动模板中使用此AMI,使所有新节点都预装模型和目录)
我已尝试的操作
- 使用仅配置Amazon Linux 2 EKS优化GPU AMI(
ami-06be503a3852b6423)的新启动模板 - 移除启动模板中除AMI外的所有配置,在节点组中定义这些配置
- 将节点组部署到公共子网
- 从不使用启动模板创建的节点组生成的EC2实例制作AMI,再基于该AMI创建启动模板
我无法理解为何使用Launch Template时始终无法创建节点组。
任何建议或新的排查方向都将不胜感激。
补充说明
明确我的EKS集群/控制平面Kubernetes版本为1.28,与我使用的EKS优化GPU节点AMI(ami-06be503a3852b6423 - amazon-eks-gpu-node-1.28-v20240110)版本匹配。
此外,我在kube-apiserver日志中持续看到以下错误:
2024-01-17T22:53:08.000-06:00 Copy W0118 04:53:08.826989 9 logging.go:59] [core] [Channel #26636 SubChannel #26637] grpc: addrConn.createTransport failed to connect to {Addr: "10.0.x.x:xxxx", ServerName: "10.x.xx.xx", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.0.32.16:2379: operation was canceled" W0118 04:53:08.826989 9 logging.go:59] [core] [Channel #26636 SubChannel #26637] grpc: addrConn.createTransport failed to connect to {Addr: "10.0.xx.xx:xxxx", ServerName: "10.0.xx.xx", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.0.xx.xx:xxxx: operation was canceled"
内容的提问来源于stack exchange,提问作者Gene

