基于ZooKeeper的Quartz分布式锁：解决K8s多Pod任务重复执行问题

问题诊断 #

你当前基于ZooKeeper的分布式锁实现存在几个关键问题，导致锁无法阻止多Pod重复执行任务：

• 锁路径未与任务绑定：所有任务共享同一个/admintasks锁路径，易引发不同任务间的锁冲突，同时也可能导致同一任务的锁逻辑失效
• Curator客户端配置不完善：缺少会话超时、连接超时等关键参数，客户端连接不稳定会直接导致锁机制失效
• 锁操作日志不明确：未对锁获取失败的场景做明确日志输出，难以排查问题原因

修复方案 #

1. 完善Curator客户端配置 #

确保ZooKeeper客户端稳定连接，增加连接等待逻辑：

@Configuration
public class ZookeeperConfig {

    @Value("${zookeeper.connect-string:10.200.XX.XX:2181}")
    private String connectString;

    @Value("${zookeeper.session-timeout:30000}")
    private int sessionTimeout;

    @Value("${zookeeper.connection-timeout:10000}")
    private int connectionTimeout;

    @Bean
    public CuratorFramework client() {
        CuratorFramework client = CuratorFrameworkFactory.builder()
                .connectString(connectString)
                .sessionTimeoutMs(sessionTimeout)
                .connectionTimeoutMs(connectionTimeout)
                .retryPolicy(new ExponentialBackoffRetry(1000, 3))
                .build();
        client.start();
        
        // 等待客户端连接就绪，避免锁操作时客户端未初始化完成
        try {
            if (!client.blockUntilConnected(10, TimeUnit.SECONDS)) {
                throw new RuntimeException("Failed to connect to ZooKeeper");
            }
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            throw new RuntimeException("ZooKeeper connection interrupted", e);
        }
        return client;
    }
}

2. 为每个任务生成唯一锁路径 #

让同一任务的多Pod实例争抢同一把锁，同时避免不同任务间的锁干扰：

@Autowired
CuratorFramework client;

@Override
public final void execute() throws TaskException {
    executionThread = Thread.currentThread();
    String taskName = getTaskName();
    // 为每个任务生成唯一锁路径
    String lockPath = "/admintasks/" + taskName;

    // 先检查ZooKeeper客户端连接状态
    if (client.getState() != CuratorFrameworkState.STARTED) {
        LOG.error("ZooKeeper client not connected, skip task {}", taskName);
        return;
    }

    try {
        InterProcessMutex lock = new InterProcessMutex(client, lockPath);
        // 缩短锁获取超时时间，避免多个Pod长时间等待
        if (lock.acquire(5000, TimeUnit.MILLISECONDS)) {
            try {
                LOG.info("Pod {} acquired lock for task {}, start execution", 
                         System.getenv("HOSTNAME"), taskName);
                execute(taskName);
            } finally {
                try {
                    lock.release();
                    LOG.info("Pod {} released lock for task {}", 
                             System.getenv("HOSTNAME"), taskName);
                } catch (Exception e) {
                    LOG.error("Failed to release lock for task {}", taskName, e);
                }
            }
        } else {
            // 明确打印锁获取失败日志，便于排查
            LOG.info("Pod {} failed to acquire lock for task {}, skip execution", 
                     System.getenv("HOSTNAME"), taskName);
        }
    } catch (Exception e) {
        LOG.error("Exception when executing task {}", taskName, e);
    }
}

3. 验证与排查 #

• 登录ZooKeeper服务器，使用ls /admintasks命令查看任务锁节点是否正确生成
• 查看Pod日志，确认只有一个Pod输出acquired lock日志，其他Pod输出failed to acquire lock日志
• 模拟Pod重启，验证任务是否会自动切换到其他可用Pod执行

内容的提问来源于stack exchange，提问作者Rohit