Kafka 2.0（HDP）副本无法加入ISR问题排查及解决咨询

问题现象 #

我们在基于HDP的Kafka 2.0.0.3.1.0.0-78版本中遇到ISR异常：部分分区的ISR数量未达到设置的副本因子。具体示例如下：

Topic:yb_test3  PartitionCount:3    ReplicationFactor:3
    Topic: yb_test3 Partition: 0    Leader: 1002    Replicas: 1002,1005,1008    Isr: 1002,1008,1005
    Topic: yb_test3 Partition: 1    Leader: 1005    Replicas: 1005  Isr: 1005
    Topic: yb_test3 Partition: 2    Leader: 1010    Replicas: 1010,1005,1008    Isr: 1010,1005,1008

我们已尝试分区重分配、首选副本选举操作，但均未解决问题。

排查过程 #

在尝试将副本1002和1008加入分区yb_test3-1的ISR列表时，发现异常行为：

1. 初始状态：

Topic:yb_test3  PartitionCount:3    ReplicationFactor:3
    Topic: yb_test3 Partition: 0    Leader: 1002    Replicas: 1002,1005,1008    Isr: 1002,1008,1005
    Topic: yb_test3 Partition: 1    Leader: 1005    Replicas: 1005  Isr: 1005
    Topic: yb_test3 Partition: 2    Leader: 1010    Replicas: 1010,1005,1008    Isr: 1010,1005,1008

2. Zookeeper中该分区的状态：

[zk: prod-nn-0006.pv.mts.ru:2181(CONNECTED) 0] get /brokers/topics/yb_test3/partitions/1/state
{"controller_epoch":163,"leader":1005,"version":1,"leader_epoch":90,"isr":[1005]}
cZxid = 0x1a054a0c57
ctime = Thu Feb 16 11:06:41 MSK 2023
mZxid = 0x1a0d0b6c0e
mtime = Thu Mar 28 13:24:09 MSK 2024
pZxid = 0x1a054a0c57
cversion = 0
dataVersion = 262
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 81
numChildren = 0

此时该分区的leader_epoch为90。

3. 执行分区重分配：通过kafka-reassign-partitions.sh工具将分区1的副本设置为(1005,1002,1008)，但操作后1002和1008仍未加入ISR：

Topic:yb_test3  PartitionCount:3    ReplicationFactor:3
    Topic: yb_test3 Partition: 0    Leader: 1002    Replicas: 1002,1005,1008    Isr: 1002,1008,1005
    Topic: yb_test3 Partition: 1    Leader: 1005    Replicas: 1005,1002,1008    Isr: 1005
    Topic: yb_test3 Partition: 2    Leader: 1010    Replicas: 1010,1005,1008    Isr: 1010,1005,1008

4. 日志异常：查看Broker 1002上的/var/log/kafka/state-change.log日志，发现控制器向Broker 1002发送了两次LeaderAndIsr请求，第二次请求使用了旧的leader_epoch值90，导致Broker忽略该请求：

[2024-03-28 13:28:10,792] WARN [Broker id=1002] Ignoring LeaderAndIsr request from controller 1002 with correlation id 2186 epoch 163 for partition yb_test3-1 since its associated leader epoch 90 is not higher than the current leader epoch 91 (state.change.logger)

核心疑问与求助 #

• 为何控制器会发送两次LeaderAndIsr请求，且第二次使用旧的leader_epoch？
• 如何修复该ISR异常？

我们已尝试切换控制器，但无效果。目前想到的临时方案是开启unclean.leader.election.enable并重启Broker 1005，但希望先明确问题根因。

内容的提问来源于stack exchange，提问作者yabel