请求协助：基于Kubernetes部署Elasticsearch并实现自动扩缩容

Got it, let's walk through this step by step to get your existing Elasticsearch cluster migrated to Kubernetes with the auto-scaling behavior you need. I'll break this down into actionable parts so you can follow along easily:

First, you need to get your full ES data into a storage system that Kubernetes can mount. Two reliable options here:

• Take an Elasticsearch Snapshot (best for multi-node clusters):

  1. Mount a shared storage volume (like NFS) to your existing ES node's /mnt/es_backup directory.
  2. Register the snapshot repository with your existing ES:

     curl -X PUT "http://your-existing-es-ip:9200/_snapshot/es_backup" -H 'Content-Type: application/json' -d'
     {
       "type": "fs",
       "settings": {
         "location": "/mnt/es_backup",
         "compress": true
       }
     }'
     

  3. Run the full snapshot and wait for it to complete:

     curl -X PUT "http://your-existing-es-ip:9200/_snapshot/es_backup/full_cluster_backup?wait_for_completion=true"
     

• Copy the Data Directory (for single-node instances):
  If you're running a single ES node, just tar up the default data directory (/var/lib/elasticsearch) and transfer it to a storage volume that Kubernetes can access (like a pre-provisioned PersistentVolume).

Elasticsearch is a stateful application, so StatefulSet is strongly recommended over a ReplicationController (it manages persistent volumes and stable network identities for each node automatically). Here's how to set it up:

Step 2.1: Create a PersistentVolumeClaim (PVC) #

First, define a PVC to mount your backed-up data. Adjust storage size and storage class to match your cluster's setup:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: es-data-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi # Match your data size
  storageClassName: nfs-storage # Use your cluster's storage class name

Apply it with: kubectl apply -f es-pvc.yaml

Step 2.2: Deploy the StatefulSet #

Use this YAML to deploy ES with your data volume mounted. Make sure to use the same ES version as your existing cluster:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
spec:
  serviceName: elasticsearch
  replicas: 1 # Start with 1 instance as required
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:7.17.0 # Match your existing ES version
        resources:
          requests:
            memory: "8Gi"
            cpu: "2"
          limits:
            memory: "16Gi"
            cpu: "4"
        ports:
        - containerPort: 9200
          name: http
        - containerPort: 9300
          name: transport
        volumeMounts:
        - name: es-data
          mountPath: /usr/share/elasticsearch/data
        env:
        - name: discovery.type
          value: single-node # Initial single-node setup
        - name: ES_JAVA_OPTS
          value: "-Xms8g -Xmx8g" # Set to ~50% of container memory limit
  volumeClaimTemplates:
  - metadata:
      name: es-data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 100Gi
      storageClassName: nfs-storage

Apply it with: kubectl apply -f es-statefulset.yaml

Step 2.3: Restore Your Data #

If you used a snapshot:

1. Exec into the running ES pod: kubectl exec -it elasticsearch-0 -- bash
2. Register the same snapshot repository (since we mounted the shared storage):

   curl -X PUT "http://localhost:9200/_snapshot/es_backup" -H 'Content-Type: application/json' -d'
   {
     "type": "fs",
     "settings": {
       "location": "/mnt/es_backup",
       "compress": true
     }
   }'
   

3. Restore the snapshot:

   curl -X POST "http://localhost:9200/_snapshot/es_backup/full_cluster_backup/_restore"
   

If you copied the data directory, just ensure the tarred files are extracted into the PVC's underlying storage path before starting the pod.

To handle CPU/memory-based scaling, we'll use Kubernetes' Horizontal Pod Autoscaler (HPA). First, make sure your cluster has Metrics Server deployed (it's required for HPA to fetch resource usage metrics).

Step 3.1: Create the HPA Configuration #

This YAML will scale your ES cluster to 3 instances when CPU/memory hits 90% utilization, and scale back down to 1 when usage drops:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: es-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: StatefulSet
    name: elasticsearch
  minReplicas: 1
  maxReplicas: 3
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 90
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 90

Apply it with: kubectl apply -f es-hpa.yaml

Step 3.2: Update ES for Multi-Node Discovery #

When HPA scales up to multiple nodes, you need to update ES's discovery settings to form a cluster:

1. Edit the StatefulSet: kubectl edit statefulset elasticsearch
2. Remove the discovery.type: single-node environment variable.
3. Add these env vars for K8s service-based discovery:

   - name: discovery.seed_hosts
     value: "elasticsearch-0.elasticsearch,elasticsearch-1.elasticsearch,elasticsearch-2.elasticsearch"
   - name: cluster.initial_master_nodes
     value: "elasticsearch-0,elasticsearch-1,elasticsearch-2"
   

4. Save the changes—Kubernetes will roll out the updated config to all pods.

• Persistent Storage for Each Node: StatefulSet automatically creates a unique PVC for each ES node, which prevents data conflicts. Avoid using a single shared PVC with ReplicationController unless you're sure you need it.
• JVM Heap Size: Never set ES's JVM heap above 32GB—this breaks Java's pointer compression and hurts performance. Stick to ~50% of the container's memory limit.
• Safe Scaling Down: When HPA scales back down, ES needs to migrate data off the terminating node. You can enable automatic data migration by ensuring cluster.routing.allocation.enable is set to all (default).
• Monitoring: Add a tool like Metricbeat to monitor ES cluster health and resource usage—this helps you fine-tune your HPA thresholds if needed.

内容的提问来源于stack exchange，提问作者soundararajan.c