Kubernetes上Neo4j集群性能过慢问题排查求助

Hey there, let's break down what might be causing this massive performance gap between your test and production Neo4j clusters. Here are the key areas to investigate step by step:

• Check T2 Instance Burst Credits: AWS t2.xlarge is a burstable performance instance. If your cluster nodes have exhausted CPU credits, they'll get throttled—this is catastrophic for a causal cluster that needs constant inter-node sync. Verify this with:

  # Check CPU credit balance via CloudWatch (replace region and instance ID)
  aws cloudwatch get-metric-statistics --namespace AWS/EC2 --metric-name CPUCreditBalance --dimensions Name=InstanceId,Value=i-xxxxxx --start-time $(date -d "-1 hour" +%Y-%m-%dT%H:%M:%SZ) --end-time $(date +%Y-%m-%dT%H:%M:%SZ) --period 300 --statistics Average
  

  Or run htop directly on the EC2 nodes to spot CPU throttling.
• Validate Pod Affinity & AZ Placement: Even with pod affinity configured, double-check if core nodes and their paired replicas are in the same AWS Availability Zone. Cross-AZ network latency can kill cluster sync and query routing times. Use this command to confirm pod node topology:

  kubectl describe pod <neo4j-pod-name> | grep -A5 "Node:"
  

• Review Client Connection Strategy: Ensure your Node.js app is routing read requests (like login queries) to read replicas instead of core nodes. If all traffic hits core nodes, you're overloading the cluster's decision-makers. In the Neo4j Node.js driver, set:

  const driver = neo4j.driver(uri, auth, {
    defaultAccessMode: 'READ',
    consistency: 'ANY' // Allows reading from the closest available replica
  });
  

• Fix Heap vs. Pagecache Allocation: t2.xlarge has 8GB total memory. Your current heap settings (4GB for cores, 2GB for replicas) are eating into the file system cache—Neo4j relies heavily on this cache to speed up queries. Adjust these values in your Helm Chart's neo4j.conf:
  • Core nodes: dbms.memory.heap.max_size=3G, dbms.memory.pagecache.size=4G
  • Replicas: dbms.memory.heap.max_size=2G, dbms.memory.pagecache.size=5G
• Check Consistency Level: For login (a read-heavy, non-critical consistency operation), avoid using strong consistency levels like READ_COMMITTED. Switching to ANY lets queries return immediately from the nearest healthy replica instead of waiting for core node confirmation.
• Monitor Garbage Collection: Frequent full GC pauses will cripple performance. Check Neo4j logs for GC warnings:

  kubectl logs <neo4j-core-pod> neo4j | grep -i "gc"
  

  If you see frequent Full GC events, tweak heap settings or switch to G1GC with appropriate tuning flags.

• Profile the Login Query: Run the exact login query in both test and production with PROFILE to compare execution plans. For example:

  PROFILE MATCH (u:User {username: $username}) WHERE u.password = $hashedPassword RETURN u.id, u.email
  

  If production isn't using indexes (check with SHOW INDEXES), a full graph scan on a larger dataset will explain the 7x slowdown. Create missing indexes immediately:

  CREATE INDEX user_username FOR (u:User) ON (u.username);
  

• Trim Unnecessary Data: Ensure your login query only returns the fields you need (e.g., user ID, email) instead of the entire user node and its relationships. Extra data transfer adds unnecessary latency.

• Set CPU Requests/Limits: Don't rely on default resource settings. For t2.xlarge nodes:
  • Core pods: resources: requests: {cpu: "2", memory: "7G"}, limits: {cpu: "4", memory: "8G"}
  • Replica pods: resources: requests: {cpu: "1", memory: "6G"}, limits: {cpu: "2", memory: "7G"}
    This prevents Kubernetes from throttling CPU or memory for your Neo4j pods.
• Upgrade Storage IOPS: If you're using default gp2 EBS volumes, their low baseline IOPS (3IOPS/GB) will bottleneck cluster sync and disk reads. Switch to gp3 volumes with a minimum of 3000 IOPS for better performance.

• Check Cluster Status: Confirm all nodes are online and syncing properly:

  kubectl exec <neo4j-core-pod> -- cypher-shell "CALL dbms.cluster.overview()"
  

  Look for any nodes marked OFFLINE or with high lag values—these will cause query routing delays.
• Watch for Leader Flaps: Frequent leader elections (check logs for "Leader changed" messages) mean your cluster is unstable, leading to intermittent request timeouts and slowdowns. Ensure core nodes have stable network connectivity and sufficient resources.

内容的提问来源于stack exchange，提问作者Swapnil B.