GKE集群部署ELK栈遇Pod异常问题求助

用Flask开发了一个将GET请求键值打印到控制台的应用，目标是在Google Cloud创建的1主2从Kubernetes集群（4核8GB内存）中部署该应用与ELK栈用于日志收集。已将Flask应用打包为Docker镜像并上传至Docker Hub，最初尝试通过Helm默认Chart部署ELK栈，但Pod一直处于Pending状态。随后改用YAML文件部署Elasticsearch，却出现Pod CrashLoopBackOff的情况，kubectl describe显示容器因OOMKilled终止。

Elasticsearch Deployment配置文件 #

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: elasticsearch
  namespace: elk-stack
spec:
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
        - name: elasticsearch
          image: elasticsearch:7.6.2
          resources:
            requests:
              memory: 1Gi
              cpu: 1
            limits:
              memory: 2Gi
              cpu: 2

kubectl get pods 输出 #

NAME                            READY   STATUS             RESTARTS      AGE   IP           NODE                                            NOMINATED NODE   READINESS GATES
elasticsearch-c49749bbc-db9pt   0/1     CrashLoopBackOff   3 (35s ago)   95s   10.84.0.11   gke-devops-cluster-default-pool-8155544c-978l   <none>           <none>

kubectl top node 输出 #

NAME                                            CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
gke-devops-cluster-default-pool-8155544c-3mkr   73m          7%     1318Mi          46%       
gke-devops-cluster-default-pool-8155544c-978l   94m          10%    1194Mi          42%

kubectl describe pods 输出 #

Name:             elasticsearch-c49749bbc-db9pt
Namespace:        elk-stack
Priority:         0
Service Account:  default
Node:             gke-devops-cluster-default-pool-8155544c-978l/10.156.0.24
Start Time:       Mon, 31 Jul 2023 21:15:00 +0300
Labels:           app=elasticsearch
                  pod-template-hash=c49749bbc
Annotations:      <none>
Status:           Running
IP:               10.84.0.11
IPs:
  IP:           10.84.0.11
Controlled By:  ReplicaSet/elasticsearch-c49749bbc
Containers:
  elasticsearch:
    Container ID:   containerd://c2532904c4a2551f1ff2c8df924fb7704186a9924cfb6b491e7d6996407a3aa5
    Image:          elasticsearch:7.6.2
    Image ID:       docker.io/library/elasticsearch@sha256:1b09dbd93085a1e7bca34830e77d2981521a7210e11f11eda997add1c12711fa
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Mon, 31 Jul 2023 21:18:27 +0300
      Finished:     Mon, 31 Jul 2023 21:18:31 +0300
    Ready:          False
    Restart Count:  5
    Limits:
      cpu:     500m
      memory:  700Mi
    Requests:
      cpu:        300m
      memory:     500Mi
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7kjrr (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-7kjrr:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                     From               Message
  ----     ------     ----                    ----               -------
  Normal   Scheduled  4m41s                   default-scheduler  Successfully assigned elk-stack/elasticsearch-c49749bbc-db9pt to gke-devops-cluster-default-pool-8155544c-978l
  Normal   Pulled     2m48s (x5 over 4m41s)   kubelet            Container image "elasticsearch:7.6.2" already present on machine
  Normal   Created    2m48s (x5 over 4m41s)   kubelet            Created container elasticsearch
  Normal   Started    2m48s (x5 over 4m41s)   kubelet            Started container elasticsearch
  Warning  BackOff    2m16s (x10 over 4m30s)  kubelet            Back-off restarting failed container elasticsearch in pod elasticsearch-c49749bbc-db9pt_elk-stack(90993e32-bcd8-4e96-b27b-d5b6d532a470)

1. 调整Elasticsearch JVM堆内存 #

Elasticsearch默认分配1GB JVM堆内存，但你的容器内存限制仅700Mi，直接触发OOM。需显式设置JVM堆大小，建议为容器内存限制的50%左右（最大不超过32GB）。修改Deployment配置，添加环境变量：

containers:
  - name: elasticsearch
    image: elasticsearch:7.6.2
    env:
      - name: ES_JAVA_OPTS
        value: "-Xms300m -Xmx300m" # 对应700Mi容器内存，设置300Mi堆内存
    resources:
      requests:
        memory: 500Mi
        cpu: 300m
      limits:
        memory: 700Mi
        cpu: 500m

2. 修正资源配置不一致问题 #

你提供的Deployment配置中resources是1Gi内存/1核CPU，但实际Pod的limits是700Mi内存/500mCPU，说明配置未生效（可能部署了旧版本或被其他控制器覆盖）。确保部署的YAML配置与实际期望一致，重新应用配置：

kubectl apply -f elasticsearch-deployment.yaml -n elk-stack

3. 修复Helm部署Pending问题 #

最初Helm默认Chart部署Pending，核心原因是默认资源请求过高，超出集群节点可用资源。修改Helm values.yaml，降低各组件资源请求：

elasticsearch:
  resources:
    requests:
      cpu: "300m"
      memory: "500Mi"
    limits:
      cpu: "500m"
      memory: "700Mi"
kibana:
  resources:
    requests:
      cpu: "100m"
      memory: "200Mi"
logstash:
  resources:
    requests:
      cpu: "200m"
      memory: "400Mi"

重新部署Helm Chart：

helm repo add elastic https://helm.elastic.co
helm upgrade --install elk elastic/elasticsearch -f values.yaml -n elk-stack --create-namespace

4. 集群资源优化建议 #

• 开启Elasticsearch内存锁定：在Pod配置中添加securityContext: privileged: true和环境变量bootstrap.memory_lock: true，避免JVM内存交换导致性能下降
• 日志采集轻量化：用Filebeat替代Logstash做日志采集，降低资源消耗，之后再将数据转发给Elasticsearch
• 节点资源匹配：如果集群长期运行ELK，建议升级节点配置（比如每个节点4核8GB），避免资源瓶颈

内容的提问来源于stack exchange，提问作者stratovic