Alright, let's break down the best practices to set up a dedicated monitoring node in your Kubernetes cluster that's immune to auto-scaling changes. I've helped several teams implement this, so here's what works reliably:

1. Label and Taint the Node to Reserve It #

First, you need to mark the node as a monitoring-only resource to keep other workloads off it:

• Add a descriptive label so monitoring tools can target this node specifically:

  kubectl label nodes <your-monitoring-node-name> node-role.kubernetes.io/monitoring=active
  

• Apply a taint to prevent non-monitoring pods from being scheduled here by default:

  kubectl taint nodes <your-monitoring-node-name> node-role.kubernetes.io/monitoring=NoSchedule
  

  This taint acts as a "block" — only pods with a matching toleration can run on this node.

2. Configure Monitoring Workloads to Target the Node #

Next, update your monitoring deployments (Prometheus, Grafana, Alertmanager, etc.) to include both a node selector and toleration, so they can be scheduled on the dedicated node:
Here's an example snippet for a Prometheus Deployment:

spec:
  template:
    spec:
      # Target the labeled monitoring node
      nodeSelector:
        node-role.kubernetes.io/monitoring: "active"
      # Tolerate the taint we added earlier
      tolerations:
      - key: "node-role.kubernetes.io/monitoring"
        operator: "Equal"
        value: "active"
        effect: "NoSchedule"
      # Optional: Add a priority class to ensure monitoring pods aren't evicted
      priorityClassName: monitoring-high-priority

3. Exclude the Node from Auto-Scaling #

This is critical to prevent the cluster autoscaler from scaling down or removing the node. The approach varies based on your cluster setup:

• Managed clusters (EKS, GKE, AKS):
  • If the node is part of a dedicated node pool, set the pool's minimum and maximum replica count to 1 (so it can't scale up/down).
  • Alternatively, remove the node from the auto-scaling group (ASG) entirely if it's a standalone instance.
• Self-managed clusters with Cluster Autoscaler:
  • Add a label to the node to disable scaling down:

    kubectl label nodes <your-monitoring-node-name> cluster-autoscaler.kubernetes.io/scale-down-disabled=true
    

  • You can also configure the Cluster Autoscaler to skip nodes with local storage (if your monitoring tools use persistent local disks) by adding the --skip-nodes-with-local-storage flag to its deployment.

4. Optional: Use Node Affinity for Flexibility #

If you might add multiple monitoring nodes later, node affinity is more flexible than a simple node selector. Here's how to configure it:

spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node-role.kubernetes.io/monitoring
                operator: In
                values:
                - "active"

This ensures pods only schedule on nodes with the monitoring=active label, even if you add more nodes later.

5. Protect Monitoring Pods from Eviction #

To make sure your monitoring stack stays up during resource crunches, create a high-priority class and assign it to your monitoring pods:

1. Create the PriorityClass:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: monitoring-high-priority
value: 1000000 # High value ensures it takes precedence over most workloads
globalDefault: false
description: "Priority class for critical monitoring workloads"

2. Reference it in your monitoring pod templates as shown in step 2.

By combining these steps, you'll have a fully isolated monitoring node that's protected from auto-scaling and non-monitoring workloads.

内容的提问来源于stack exchange，提问作者hatem