如何解决Kubernetes中Cassandra Pod处于CrashLoopBackOff状态的问题

Hey there, let's dig into why your Cassandra pods are stuck in CrashLoopBackOff and get them up and running properly. First, let's start with the most critical step to pinpoint the root cause:

Step 1: Check Pod Logs for Crash Details #

The first thing you need to do is see exactly why the pod is crashing. Run these commands to pull the logs:

# List all Cassandra pods to get their names
kubectl get pods -l app=cassandra

# View logs for a specific pod (replace <pod-name> with your actual pod name)
kubectl logs <pod-name>

# If the pod crashed multiple times, check the logs from its previous run
kubectl logs <pod-name> --previous

This will show you specific errors—common culprits here are out-of-memory kills, permission issues, or Cassandra cluster initialization failures.

Common Causes & Fixes #

1. You're Using a Deployment for a Stateful Application #

Cassandra is a stateful distributed database, and Deployments are designed for stateless workloads. Deployments don't provide the stable network identities or persistent storage guarantees that Cassandra needs to form and maintain a cluster. You should use a StatefulSet instead—it's purpose-built for stateful apps like databases.

2. Insufficient Resource Allocation #

Cassandra requires a decent amount of memory to run. By default, Kubernetes doesn't set resource limits, so the pod might get killed by the OS due to OOM (Out of Memory). Add resource requests and limits to your container spec to prevent this:

resources:
  requests:
    memory: "2Gi"
    cpu: "1"
  limits:
    memory: "4Gi"
    cpu: "2"

Adjust these values based on the available resources on your VMs.

3. Outdated or Problematic Image #

The image gcr.io/google_containers/cassandra:v5 is quite old and may have compatibility issues with modern Kubernetes versions. Switch to an official, up-to-date Cassandra image from Docker Hub, like cassandra:4.1 (check for the latest stable release).

4. Missing Persistent Storage #

If Cassandra can't write to a persistent volume, it might crash on startup. StatefulSets make it easy to attach persistent volume claims (PVCs) to each pod, ensuring data persists across restarts.

Example Working StatefulSet Configuration #

Here's a simplified StatefulSet setup for Cassandra, paired with a Headless Service (required for StatefulSet network identity):

Headless Service (cassandra-headless.yaml) #

apiVersion: v1
kind: Service
metadata:
  name: cassandra-headless
  labels:
    app: cassandra
spec:
  clusterIP: None
  ports:
  - port: 9042
    name: cql
  selector:
    app: cassandra

StatefulSet (cassandra-statefulset.yaml) #

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: cassandra
  labels:
    app: cassandra
spec:
  serviceName: cassandra-headless
  replicas: 3
  selector:
    matchLabels:
      app: cassandra
  template:
    metadata:
      labels:
        app: cassandra
    spec:
      containers:
      - name: cassandra
        image: cassandra:4.1
        ports:
        - containerPort: 9042
          name: cql
        resources:
          requests:
            memory: "2Gi"
            cpu: "1"
          limits:
            memory: "4Gi"
            cpu: "2"
        volumeMounts:
        - name: cassandra-data
          mountPath: /var/lib/cassandra
  volumeClaimTemplates:
  - metadata:
      name: cassandra-data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 10Gi

Deployment Steps #

1. Delete your existing Deployment first:

kubectl delete deployment cassandra

2. Create the Headless Service:

kubectl create -f cassandra-headless.yaml

3. Create the StatefulSet:

kubectl create -f cassandra-statefulset.yaml

4. Watch the pod status to confirm they start successfully:

kubectl get pods -w -l app=cassandra

The pods will start one by one and form a Cassandra cluster automatically.

Final Notes #

• If you still run into issues, check the pod logs again for specific errors (like gossip failure or configuration mismatches).
• You can keep your original NodePort Service if you need external access to Cassandra—just make sure its selector matches the StatefulSet pod labels.

内容的提问来源于stack exchange，提问作者Paloma Gómez