如何捕获HikariCP连接泄漏事件并构建告警邮件？

问题背景 #

希望在HikariCP发生连接泄漏时发送告警邮件，目前能看到异常日志，但无法有效捕获这些异常以构建精准告警内容。不想使用通用的@ExceptionHandler，因为HikariCP抛出的是通用异常，需要获取底层DataSource等对象来构建有意义的告警信息。

泄漏日志示例 #

[2023-09-06 05:11:01.137] WARN  [HikariPool-24--[DBConfig.class]--ReadOnly housekeeper] com.zaxxer.hikari.pool.ProxyLeakTask - Connection leak detection triggered for org.postgresql.jdbc.PgConnection@443be07d on thread taskExecutor-2, stack trace follows
java.lang.Exception: Apparent connection leak detected
    at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
    at net.ttddyy.dsproxy.support.ProxyDataSource.getConnection(ProxyDataSource.java:72)
    at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
    at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38)
    at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:108)
    at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:138)
    at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getConnectionForTransactionManagement(LogicalConnectionManagedImpl.java:276)
    at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:284)
    at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:246)
    at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:83)
    at com.company.solr.SolrImportationTask.getDelta(SolrImportationTask.java:233)
    at com.company.solr.SolrImportationTask.run(SolrImportationTask.java:141)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)

可行解决方案 #

1. 自定义HikariCP的LeakDetectionListener #

HikariCP 4.0+版本支持LeakDetectionListener接口，可直接实现该接口捕获连接泄漏事件，无需解析日志。

• 实现监听器：

import com.zaxxer.hikari.HikariPool;
import com.zaxxer.hikari.leak.LeakDetectionListener;
import java.sql.Connection;
import java.util.concurrent.TimeUnit;

public class CustomLeakListener implements LeakDetectionListener {
    @Override
    public void onLeakDetected(HikariPool pool, Connection connection, long leakDuration, TimeUnit timeUnit) {
        // 获取连接池核心信息
        String poolName = pool.getPoolName();
        String jdbcUrl = pool.getConfig().getJdbcUrl();
        long leakMinutes = timeUnit.toMinutes(leakDuration);

        // 构建精准告警内容
        String alertContent = String.format("连接池[%s]发生连接泄漏\nJDBC地址：%s\n泄漏时长：%d分钟\n连接对象：%s",
                poolName, jdbcUrl, leakMinutes, connection);

        // 调用邮件发送逻辑
        sendAlertEmail(alertContent);
    }

    private void sendAlertEmail(String content) {
        // 实现你的邮件发送代码
    }
}

• 注册到HikariCP配置：

HikariConfig config = new HikariConfig();
config.setLeakDetectionThreshold(60000); // 设置泄漏检测阈值（毫秒）
config.setLeakDetectionListener(new CustomLeakListener());
HikariDataSource dataSource = new HikariDataSource(config);

2. 自定义日志Appender拦截告警日志 #

如果使用Logback/Log4j2，可自定义Appender拦截HikariCP的泄漏告警日志，解析后发送邮件。

以Logback为例：

• 自定义Appender：

import ch.qos.logback.classic.spi.ILoggingEvent;
import ch.qos.logback.core.AppenderBase;

public class HikariLeakAlertAppender extends AppenderBase<ILoggingEvent> {
    @Override
    protected void append(ILoggingEvent event) {
        // 过滤HikariCP泄漏告警日志
        if ("com.zaxxer.hikari.pool.ProxyLeakTask".equals(event.getLoggerName()) 
            && event.getLevel().equals(ch.qos.logback.classic.Level.WARN)
            && event.getMessage().contains("Connection leak detection triggered")) {
            
            // 提取日志和堆栈信息
            String logMsg = event.getFormattedMessage();
            String stackTrace = event.getThrowableProxy() != null ? 
                event.getThrowableProxy().getStackTraceElementProxyArray().toString() : "";
            
            // 构建告警内容
            String alertContent = "HikariCP连接泄漏告警：\n" + logMsg + "\n\n堆栈信息：\n" + stackTrace;
            
            // 发送邮件
            sendAlertEmail(alertContent);
        }
    }

    private void sendAlertEmail(String content) {
        // 邮件发送逻辑
    }
}

• 在logback.xml中配置：

<configuration>
    <appender name="HIKARI_LEAK_ALERT" class="com.company.logging.HikariLeakAlertAppender"/>
    <logger name="com.zaxxer.hikari.pool.ProxyLeakTask" level="WARN">
        <appender-ref ref="HIKARI_LEAK_ALERT"/>
    </logger>
</configuration>

3. 基于DataSource代理实现泄漏监控 #

利用你已使用的datasource-proxy，自定义监听逻辑监控连接的获取与关闭：

• 自定义ConnectionListener：

import net.ttddyy.dsproxy.listener.ConnectionListener;
import java.sql.Connection;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;

public class LeakDetectionConnectionListener implements ConnectionListener {
    private final Map<Connection, Long> connAcquireTimeMap = new ConcurrentHashMap<>();
    private final ScheduledExecutorService scheduler = Executors.newSingleThreadScheduledExecutor();
    private final long leakThresholdMs = 60000; // 1分钟阈值

    public LeakDetectionConnectionListener() {
        // 定时检查泄漏
        scheduler.scheduleAtFixedRate(this::checkLeaks, leakThresholdMs, leakThresholdMs, TimeUnit.MILLISECONDS);
    }

    @Override
    public void afterGetConnection(Connection connection) {
        connAcquireTimeMap.put(connection, System.currentTimeMillis());
    }

    @Override
    public void afterCloseConnection(Connection connection) {
        connAcquireTimeMap.remove(connection);
    }

    private void checkLeaks() {
        long now = System.currentTimeMillis();
        for (Map.Entry<Connection, Long> entry : connAcquireTimeMap.entrySet()) {
            if (now - entry.getValue() > leakThresholdMs) {
                String alertContent = String.format("检测到连接泄漏\n连接对象：%s\n已持有时长：%d毫秒",
                        entry.getKey(), now - entry.getValue());
                sendAlertEmail(alertContent);
                connAcquireTimeMap.remove(entry.getKey()); // 避免重复告警
            }
        }
    }

    private void sendAlertEmail(String content) {
        // 邮件发送逻辑
    }
}

• 绑定到ProxyDataSource：

ProxyDataSource proxyDataSource = new ProxyDataSource();
proxyDataSource.setDataSource(hikariDataSource);
proxyDataSource.addListener(new LeakDetectionConnectionListener());

内容的提问来源于stack exchange，提问作者ahrooran