Great question! When using kops to set up your AWS EC2 cluster, there are actually a couple of straightforward ways to configure Spot instances—both during cluster creation and for modifying existing clusters. Let me break this down for you:

1. Specify Spot Instances When Creating a New Cluster #

You can either set all nodes to use Spot instances, or target specific instance groups (like worker nodes while keeping masters on On-Demand):

• All nodes as Spot instances: Add the --spot-price flag to your cluster creation command. Leave the value empty to use the current On-Demand price as your maximum bid, or set a custom limit:

  kops create cluster \
    --name your-cluster.example.com \
    --state s3://your-kops-state-store \
    --zones us-east-1a,us-east-1b \
    --node-count 3 \
    --node-size t3.medium \
    --spot-price "0.01"
  

• Specific instance groups as Spot: First create a base cluster (with masters on On-Demand), then edit the worker instance group to enable Spot:

  # Create the initial cluster
  kops create cluster \
    --name your-cluster.example.com \
    --state s3://your-kops-state-store \
    --zones us-east-1a,us-east-1b \
    --node-count 3 \
    --node-size t3.medium
  
  # Edit the worker instance group (replace with your IG name)
  kops edit ig nodes-us-east-1a
  

  In the editor, add the spotPrice field under the spec section:

  spec:
    image: kope.io/k8s-1.26-debian-bullseye-amd64-hvm-ebs-2023-05-17
    machineType: t3.medium
    maxSize: 3
    minSize: 1
    spotPrice: "0.01" # Leave empty to use On-Demand price as bid limit
    subnets:
    - us-east-1a
  

  Save your changes, then apply and roll out the update:

  kops update cluster --yes
  kops rolling-update cluster --yes
  

2. Convert Existing Cluster Instance Groups to Spot #

If your cluster is already running, you can modify existing instance groups to use Spot instances:

# List all instance groups to find your target IG
kops get ig

# Edit the target instance group
kops edit ig nodes-us-east-1a

Add the spotPrice field to the spec section as shown above, then run the update and rolling-update commands to apply changes.

3. Kubernetes-Level Optimizations for Spot Instances #

Once your Spot nodes are up, consider these tweaks to improve reliability:

• Pod Disruption Budgets (PDBs): Define PDBs for critical applications to ensure enough replicas remain available if a Spot node is terminated.
• Taints/Tolerations: Mark Spot nodes with taints to control which pods can schedule on them, then add matching tolerations to your pods:

  # Add taint to a Spot node
  kubectl taint nodes <spot-node-name> node.kubernetes.io/spot=true:NoSchedule
  

  Add this toleration to your pod spec:

  tolerations:
  - key: "node.kubernetes.io/spot"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"
  

• Cluster Autoscaler: Enable the Cluster Autoscaler to automatically replace terminated Spot nodes with new ones, maintaining your desired cluster size.

内容的提问来源于stack exchange，提问作者user1650281