GKE Autopilot是否会主动终止Pod?关键服务如何避免非预期重启?
GKE Autopilot集群Redis Pod异常重启引发服务宕机故障排查记录
故障现象
- 部署在Kubernetes上的Redis服务长时间不可用,导致Django及相关关联服务全部无法连接Redis,大量作业丢失,整体故障持续时长10分钟
- 经数小时排查暂未定位根因,仅确认Redis Deployment无对应事件记录
Pod重启前后日志
重启前首条日志

重启后首条日志

当前排查进展
- 所用集群为GKE Autopilot,初步推测Pod重启为集群主动触发,但该Pod CPU、内存使用率仅约1%,远低于申请的资源配额,触发原因暂不明确
- 暂未找到可让Autopilot不操作特定Deployment的注解
关联Redis配置文件
--- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: redis-disk spec: accessModes: - ReadWriteOnce storageClassName: gce-ssd resources: requests: storage: "2Gi" --- apiVersion: v1 kind: Service metadata: name: redis labels: app: redis spec: ports: - port: 6379 name: redis clusterIP: None selector: app: redis --- apiVersion: apps/v1 kind: Deployment metadata: name: redis labels: app: redis spec: replicas: 1 selector: matchLabels: app: redis template: metadata: labels: app: redis spec: volumes: - name: redis-volume persistentVolumeClaim: claimName: redis-disk readOnly: false terminationGracePeriodSeconds: 5 containers: - name: redis image: redis:6-alpine command: ["sh"] args: ["-c", 'exec redis-server --requirepass "$REDIS_PASSWORD"'] resources: requests: memory: "512Mi" cpu: "500m" ephemeral-storage: "1Gi" envFrom: - secretRef: name: env-secrets volumeMounts: - name: redis-volume mountPath: /data subPath: data
内容的提问来源于stack exchange,提问作者yspreen
相关产品推荐
相关产品推荐

