生产环境4节点Kafka集群消息流入速率失衡问题排查求助

Hey there, let's walk through troubleshooting this Kafka message inflow imbalance issue on your kafka01 node. I've dealt with similar scenarios before, so here's a structured approach to get to the root cause and fix it.

1. First, Verify Leader Partition Distribution Across Nodes #

Your cluster uses 64 partitions, and with 4 nodes, you'd expect 16 leader partitions per node (since producers send traffic directly to leader partitions, not replicas). If kafka01 has way more or fewer leader partitions than the others, that's an immediate cause of imbalance.

To check this:

• Run the Kafka topic describe command:

  kafka-topics.sh --describe --topic YOUR_TOPIC_NAME --bootstrap-server YOUR_BROKER_URL
  

• Look at the Leader column for all 64 partitions, then count how many are assigned to kafka01 vs the other nodes.
• If counts are uneven, you might have had a failed partition rebalance, or leader elections that didn't reset to a balanced state.

2. Audit Your Custom Partitioner Logic #

You're using an ID mod 64 to assign partitions—this only works if your ID values are evenly distributed across all 64 mod results. If recent traffic has a skewed ID pattern, that'll funnel messages to a subset of partitions (and thus kafka01 if those partitions are hosted there).

Things to check:

• Sample recent message IDs: Pull a sample of messages from the last week, calculate their ID mod 64 values, and see if the results cluster in the range of partitions assigned to kafka01.
• Check for partitioner bugs: Did anyone modify the partitioner code recently? For example, accidentally converting the ID to a string before taking mod (which can lead to unexpected skews) or using the wrong modulus value.
• ID generation changes: Has your application's ID generation logic shifted? For example, switching from random IDs to sequential IDs could lead to mod results clustering in a small range temporarily.

3. Check kafka01's Health & Resource Constraints #

Even if partitions are balanced, resource bottlenecks on kafka01 can cause apparent imbalance (either producers slow down sending to it, or it's struggling to process traffic and metrics look off). Use your Datadog dashboards to check:

• CPU/memory/disk metrics: Is kafka01 hitting 100% CPU, swapping memory, or experiencing high disk IO latency? These can throttle producer traffic or make metrics report incorrectly.
• Kafka-specific metrics: Look at producer.request.delay.ms (high delays mean producers are waiting for kafka01 to respond), network.io.ratio (indicates network saturation), and under_replicated_partitions (signals replica sync issues that might affect leader stability).
• Broker logs: Check kafka01's server logs for errors like leader election failures, disk errors, or connection timeouts—these can disrupt normal traffic flow.

4. Validate Producer Behavior & Traffic Patterns #

Sometimes the issue isn't with Kafka itself, but with how producers are sending data:

• New producers or traffic spikes: Did you onboard a new service or run a batch job in the last week that's sending a large volume of messages with IDs that mod to kafka01's partitions?
• Producer configuration changes: Have any producers adjusted their acks setting, batch size, or retries? For example, producers using acks=0 might send traffic faster to kafka01 if it's less constrained, leading to higher reported rates.

Fixes Based on Root Cause #

Once you've identified the issue, here's how to resolve it:

• Uneven partition distribution: Use the kafka-reassign-partitions.sh tool to rebalance leader partitions across all 4 nodes, ensuring each has exactly 16 leaders.
• ID/partitioner skew:
  • If business logic allows, switch to a consistent hashing strategy instead of simple mod to spread traffic more evenly, even if IDs are skewed.
  • Adjust your ID generation logic to ensure mod 64 results are distributed uniformly.
• Resource bottlenecks:
  • Scale up kafka01's resources (CPU, memory, faster disks) to match traffic demands.
  • If possible, migrate a few leader partitions from kafka01 to other nodes temporarily to relieve pressure.
• Producer-related issues:
  • Work with the application team to adjust producer logic (e.g., add a secondary field to the partition key to spread traffic) or throttle batch jobs that are causing skews.

内容的提问来源于stack exchange，提问作者palash kulshreshtha