Java调用Linux命令/脚本时定时任务每6小时执行异常排查

Java调用Shell命令的代码 #

public String executeShellCommand(String linuxCommand) {

        String commandOutput;
        ProcessBuilder processBuilder;
        String [] command = {
                "/bin/bash",
                "-c",
                linuxCommand
        };

        try {
            log.debug("Making system call : [{}]",String.join(" ", command));
            // Execute the shell command
            processBuilder = new ProcessBuilder(command);
            Process p = processBuilder.start();
            BufferedReader br = new BufferedReader(
                    new InputStreamReader(p.getInputStream()));
            if ((commandOutput = br.readLine()) != null) {
                p.waitFor();
                log.trace("exit: " + p.exitValue());
                p.destroy();
                return commandOutput;
            }

        } catch (Exception e) {
            log.error("Exception in running shell command {}", linuxCommand, e);
        
            return null;
        }
       
        log.trace("There was no output for the given command {} that was executed", linuxCommand);
        return "";
    }

异常现象 #

• JVM首次启动后执行该方法成功
• 定时任务设为每6小时执行时，首次调度执行就失败，后续每次调度都失败
• 定时任务设为每15或30分钟执行时，24小时内所有执行都正常成功

错误日志 #

java.io.IOException: Cannot run program "/bin/bash": error=0, Failed to exec spawn helper: pid: 2009, exit value: 127
        at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1143) ~[na:na]
        at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073) ~[na:na]
        at com.oracle.gbucs.database.exporter.provisioning.service.ShellCommandService.executeShellCommand(ShellCommandService.java:41) ~[classes!/:2024.2.6]
        at com.oracle.gbucs.database.exporter.provisioning.service.ProvisioningService.runExporterProvisioningScript(ProvisioningService.java:395) ~[classes!/:2024.2.6]
        at com.oracle.gbucs.database.exporter.provisioning.service.ProvisioningService.provisionExporterForOneDatabase(ProvisioningService.java:167) ~[classes!/:2024.2.6]
        at com.oracle.gbucs.database.exporter.provisioning.controller.ProvisioningController.provisionExporters(ProvisioningController.java:106) ~[classes!/:2024.2.6]
        at com.oracle.gbucs.database.exporter.provisioning.controller.ProvisioningController.scheduled(ProvisioningController.java:142) ~[classes!/:2024.2.6]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na]
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
        at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na]
        at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) ~[spring-context-5.3.21.jar!/:5.3.21]
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.3.21.jar!/:5.3.21]
        at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:95) ~[spring-context-5.3.21.jar!/:5.3.21]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[na:na]
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
        at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]
Caused by: java.io.IOException: error=0, Failed to exec spawn helper: pid: 2009, exit value: 127
        at java.base/java.lang.ProcessImpl.forkAndExec(Native Method) ~[na:na]
        at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:314) ~[na:na]
        at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:244) ~[na:na]
        at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1110) ~[na:na]
        ... 19 common frames omitted

示例命令 #

统计服务器当前运行的oracledb_exporter脚本数量：

ps -ef | grep oracledb_exporter | grep -v grep | wc -l

该命令直接在命令行执行正常，但通过上述Java方法调用时失败。

环境信息 #

• Java版本：17.0.5
• 运行环境：远程服务器（SSH会话），已确认服务器存在/bin/bash

这个错误Failed to exec spawn helper是Java 17+版本的已知问题，结合调度间隔的差异，核心原因和解决方案如下：

核心原因 #

Java 9+引入了新的进程生成机制，当JVM长时间闲置（比如6小时未执行进程调用）后，系统可能清理临时文件，或者JVM内部的spawn helper相关资源被回收/失效，导致再次调用ProcessBuilder.start()时找不到spawn helper程序，抛出127错误（命令未找到）。而短间隔调度时，JVM频繁调用进程，相关资源不会被清理，因此能正常执行。

另外，SSH会话中运行的Java进程，可能因会话长时间闲置导致环境变量变化，进而让JVM无法定位spawn helper的路径。

解决方案 #

1. 添加JVM参数禁用新进程生成机制 #

启动Java进程时添加以下参数，强制使用旧的fork/exec机制，绕过spawn helper的问题：

-Djdk.lang.Process.launchMechanism=vfork

或：

-Djdk.lang.Process.launchMechanism=fork

2. 修复Java安装目录权限 #

Java 17的spawn helper程序通常位于$JAVA_HOME/lib/jspawnhelper（64位系统可能在$JAVA_HOME/lib/amd64/jspawnhelper），确保该文件有执行权限：

chmod +x $JAVA_HOME/lib/jspawnhelper

3. 优化Java调用Shell的代码 #

原代码存在输出读取不完整、未处理错误流的问题，可能加剧异常情况，修改后的代码示例：

public String executeShellCommand(String linuxCommand) {
    StringBuilder output = new StringBuilder();
    StringBuilder errorOutput = new StringBuilder();
    ProcessBuilder processBuilder = new ProcessBuilder("/bin/bash", "-c", linuxCommand);
    
    try {
        log.debug("Making system call : [{}]", String.join(" ", processBuilder.command()));
        Process p = processBuilder.start();
        
        // 读取标准输出
        try (BufferedReader br = new BufferedReader(new InputStreamReader(p.getInputStream()))) {
            String line;
            while ((line = br.readLine()) != null) {
                output.append(line).append("\n");
            }
        }
        
        // 读取错误输出，避免进程阻塞
        try (BufferedReader br = new BufferedReader(new InputStreamReader(p.getErrorStream()))) {
            String line;
            while ((line = br.readLine()) != null) {
                errorOutput.append(line).append("\n");
            }
        }
        
        int exitCode = p.waitFor();
        log.trace("exit: {}", exitCode);
        if (exitCode != 0) {
            log.error("Command failed with exit code {}: {}", exitCode, errorOutput);
            return null;
        }
        p.destroy();
        return output.length() > 0 ? output.toString().trim() : "";
        
    } catch (Exception e) {
        log.error("Exception in running shell command {}", linuxCommand, e);
        return null;
    }
}

4. 改用后台服务运行Java进程 #

不要在SSH会话中直接运行Java进程，改用systemd等后台服务启动，保证进程运行在稳定环境中，避免因SSH会话超时导致环境变量丢失或资源被回收。

内容的提问来源于stack exchange，提问作者helloworld