如何区分Spring Boot应用异常终止与正常结束并实现Splunk告警

核心思路 #

当前你的应用仅在正常完成时输出successfully completed日志用于Splunk告警，但异常终止时无对应日志。要实现异常终止的告警，需要捕获所有异常终止场景并记录专属错误日志，再通过Splunk监听这类日志触发告警。

具体实现步骤 #

1. 捕获启动与运行时未处理异常 #

在main方法中用try-catch包裹SpringApplication.run()，捕获所有未处理的Throwable，记录带堆栈信息的异常终止日志：

@SpringBootApplication
public class SpringBootHelloWorldExampleApplication {
    private static final Logger log = LoggerFactory.getLogger(SpringBootHelloWorldExampleApplication.class);

    public static void main(String[] args) {
        try {
            SpringApplication.run(SpringBootHelloWorldExampleApplication.class, args);
            log.debug("successfully completed");
        } catch (Throwable e) {
            log.error("Application terminated abnormally due to unexpected error", e);
            throw e; // 保留原有异常抛出，不改变JVM退出状态
        }
    }
}

2. 处理JVM强制退出/崩溃场景 #

对于kill -9、OOM这类直接导致JVM崩溃的场景，注册JVM关闭钩子，并通过自定义标志位区分正常/异常关闭：

@SpringBootApplication
public class SpringBootHelloWorldExampleApplication {
    private static final Logger log = LoggerFactory.getLogger(SpringBootHelloWorldExampleApplication.class);

    public static void main(String[] args) {
        // 注册JVM关闭钩子
        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            if (!ShutdownStatus.isNormal()) {
                log.error("Application terminated abnormally (JVM shutdown hook triggered)");
            }
        }));

        try {
            SpringApplication.run(SpringBootHelloWorldExampleApplication.class, args);
            ShutdownStatus.markAsNormal();
            log.debug("successfully completed");
        } catch (Throwable e) {
            ShutdownStatus.markAsAbnormal();
            log.error("Application terminated abnormally due to unexpected error", e);
            throw e;
        }
    }

    // 静态内部类记录关闭状态
    private static class ShutdownStatus {
        private static volatile boolean isNormalShutdown = false;

        public static boolean isNormal() {
            return isNormalShutdown;
        }

        public static void markAsNormal() {
            isNormalShutdown = true;
        }

        public static void markAsAbnormal() {
            isNormalShutdown = false;
        }
    }
}

3. 监听Spring容器异常关闭 #

通过Spring事件监听容器关闭事件，进一步确认异常关闭场景：

@Component
public class AppShutdownListener implements ApplicationListener<ContextClosedEvent> {
    private static final Logger log = LoggerFactory.getLogger(AppShutdownListener.class);

    @Override
    public void onApplicationEvent(ContextClosedEvent event) {
        if (!SpringBootHelloWorldExampleApplication.ShutdownStatus.isNormal()) {
            log.error("Application context closed abnormally");
        }
    }
}

4. Splunk告警配置 #

在Splunk中新增两条告警规则：

• 正常终止告警：保持现有配置，监听debug级别且包含successfully completed的日志
• 异常终止告警：监听error级别且包含Application terminated abnormally或Application context closed abnormally的日志，触发告警通知

特殊场景补充 #

• OOM等JVM直接崩溃场景：关闭钩子可能无法执行，需添加JVM参数-XX:+HeapDumpOnOutOfMemoryError生成堆转储，同时Splunk可通过监控应用进程状态（结合日志长时间缺失）触发告警。
• 正常停止场景：执行kill -TERM或通过Actuator关闭应用时，markAsNormal()会被调用，关闭钩子和事件监听不会记录异常日志，确保正常告警逻辑不受影响。

内容的提问来源于stack exchange，提问作者tarun singh