Kubernetes节点资源使用疑问、优化及集群最佳实践咨询

Hey there! Let's tackle your Kubernetes questions one by one—nice work getting your 2-node cluster up and running with services deployed already, that's a great start!

1. Does MEMORY(%) in kubectl top nodes refer to the node's current memory usage percentage? #

Yes, exactly. The MEMORY% column from kubectl top nodes shows the percentage of the node's total available memory that's currently in use. This includes memory consumed by:

• All running Pods across all namespaces (the command shows node-wide usage, not just a single namespace)
• Kubernetes system components on the node, like kubelet, kube-proxy, and containerd
• The node's underlying OS processes and kernel-level memory (such as page cache, buffers, etc.)

2. Why is the node memory usage high, but the total Pod memory only adds up to ~2500Mi? #

This is a common point of confusion for new Kubernetes users! The key takeaway is that nodes use memory for far more than just your application Pods. Here's what's likely filling the gap:

• System & K8s core components: Nodes need to run essential services like kubelet (manages Pods locally) and containerd (the container runtime). These don't show up in your services-namespace Pod stats—you'd need to run kubectl top pods -n kube-system to see their resource usage.
• Page cache/buffers: Linux automatically uses free memory for disk caching (page cache) to boost performance. This counts towards the node's total memory usage but isn't "locked" by Pods and can be freed up if applications need it. You can check this by running free -h directly on the node.
• Pods in other namespaces: Your kubectl top pods command is limited to services-namespace, but there might be Pods in kube-system, default, or other namespaces consuming memory on the node.
• Metric measurement differences: kubectl top pods reports the working set/RSS memory of Pods, while node memory usage includes all memory allocated by the OS—these are slightly different metrics.

For your second node (gke-kubernetes-cluster-n-default-pool-bbbbbbbb-6g58) showing 110% memory usage, that means it's overcommitted: total memory used (all the above combined) exceeds the node's physical memory capacity.

3. How to address memory and CPU issues? #

Let's break this into memory and CPU-specific steps:

Memory Troubleshooting & Fixes #

• Check non-Pod memory usage: SSH into the problematic node (for GKE, use gcloud compute ssh <node-name>) and run top, htop, or free -h to identify memory hogs. Use ps aux --sort=-%mem to find top memory-consuming processes. Don't forget to check kube-system Pods with kubectl top pods -n kube-system.
• Spot memory leaks: If a Pod's memory usage keeps growing over time, it might have a leak. Use monitoring tools (like GKE's built-in Cloud Monitoring) to track memory trends for each Pod.
• Set resource requests and limits: Define resources.requests.memory and resources.limits.memory for all your Pods. This tells Kubernetes how much memory to reserve for each Pod (requests) and the maximum it can use (limits). Example:

  resources:
    requests:
      memory: "256Mi"
    limits:
      memory: "300Mi"
  

• Fix overcommitted nodes: For the node at 110% memory, you can:
  • Let Kubernetes automatically evict low-priority Pods via the kubelet's eviction policy
  • Reschedule some Pods to the first node (it has more available memory)
  • Add a new node to the cluster to spread the load
• Temporary cache cleanup: If page cache is taking up excess memory, run sync && echo 1 > /proc/sys/vm/drop_caches on the node. Note this is a quick fix, not a long-term solution.

CPU Troubleshooting & Fixes #

• Check node-level CPU usage: Use top or mpstat on the node to see if system processes or K8s components are using high CPU.
• Analyze Pod CPU trends: Keep an eye on Pods with consistent high CPU usage (your current stats look low, but tracking over time helps catch spikes).
• Set CPU requests and limits: Similar to memory, define resources.requests.cpu and resources.limits.cpu for Pods. This helps Kubernetes schedule Pods on nodes with available CPU and prevents single Pods from hogging resources.
• Balance load: If one node is at 28% CPU and the other at 16%, adjust Pod scheduling (using node affinity or taints/tolerations) to spread CPU-heavy workloads across nodes.

4. Kubernetes Cluster Best Practices #

Here are key practices to keep your cluster stable and manageable:

• Enforce resource requests/limits: Always define CPU and memory requests/limits for every Pod—this prevents resource starvation and helps the scheduler make optimal decisions.
• Implement monitoring & alerting: Use tools like Prometheus + Grafana (or GKE's Cloud Monitoring) to track cluster health. Set alerts for high CPU/memory usage, Pod restarts, and node failures.
• Add health checks: Configure livenessProbe (restarts unhealthy Pods) and readinessProbe (stops sending traffic to unready Pods) for all application Pods.
• Use namespaces effectively: Split your cluster into namespaces for different environments (dev/staging/prod) or teams. This simplifies permission management, resource isolation, and visibility.
• Optimize Pod scheduling: Use node affinity/anti-affinity to place Pods on appropriate nodes (e.g., memory-heavy Pods on nodes with more RAM). Use taints and tolerations to reserve nodes for specific workloads.
• Centralize logging: Deploy a logging solution (like Fluentd or GKE's Cloud Logging) to collect and aggregate Pod logs—this makes troubleshooting far easier.
• Clean up unused resources: Regularly delete unused Deployments, Pods, Services, and ConfigMaps to avoid clutter and wasted resources.
• Version control your manifests: Store all Kubernetes resource YAML files in Git. Use CI/CD pipelines to deploy changes—this ensures traceability and simplifies rollbacks.

内容的提问来源于stack exchange，提问作者Rams