如何在Java中执行spark-submit运行Scala编写的Spark应用Jar包？

Absolutely, your approach is completely feasible—and it’s actually a smart way to avoid the messy classpath conflicts you ran into earlier when trying to bundle Spark dependencies into your Java JAR. By calling spark-submit directly, you leverage Spark’s native submission mechanism, which handles all the dependency management and cluster orchestration for you.

How to Execute spark-submit in Java #

You don’t have to use Runtime.exec() (though it works), but ProcessBuilder is the recommended approach—it’s more flexible, safer, and easier to maintain. Here’s a practical example tailored to your use case:

import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class SparkJobLauncher {

    // Submit Spark job asynchronously to keep your API responsive
    public void submitSparkJobAsync(String sparkHome, String scalaJarPath, String mainClass, String appArgs) {
        ExecutorService executor = Executors.newSingleThreadExecutor();
        executor.submit(() -> {
            try {
                runSparkJob(sparkHome, scalaJarPath, mainClass, appArgs);
            } catch (IOException | InterruptedException e) {
                // Log errors or propagate failure status to your GUI/API
                e.printStackTrace();
            } finally {
                executor.shutdown();
            }
        });
    }

    private void runSparkJob(String sparkHome, String scalaJarPath, String mainClass, String appArgs) throws IOException, InterruptedException {
        // Build spark-submit command with separate arguments (avoids command injection risks)
        ProcessBuilder pb = new ProcessBuilder(
                sparkHome + "/bin/spark-submit",
                "--class", mainClass,
                "--master", "local[*]", // Adjust to your cluster setup (e.g., yarn, spark://master:7077)
                scalaJarPath,
                appArgs
        );

        // Optional: Set working directory to where your Scala JAR is located
        pb.directory(new File(scalaJarPath.substring(0, scalaJarPath.lastIndexOf("/"))));

        // Start the process
        Process process = pb.start();

        // Critical: Read stdout/stderr to prevent process hanging from buffer overflow
        try (BufferedReader stdoutReader = new BufferedReader(new InputStreamReader(process.getInputStream()));
             BufferedReader stderrReader = new BufferedReader(new InputStreamReader(process.getErrorStream()))) {

            // Capture or log Spark job output (parse this for results if needed)
            String line;
            while ((line = stdoutReader.readLine()) != null) {
                System.out.println("Spark Job Output: " + line);
                // If your Spark job outputs JSON to stdout, capture this for your GUI
            }

            // Handle error logs
            while ((line = stderrReader.readLine()) != null) {
                System.err.println("Spark Job Error: " + line);
                // Log errors or send failure alerts to your API
            }

            // Wait for job completion and check exit code
            int exitCode = process.waitFor();
            if (exitCode != 0) {
                throw new RuntimeException("Spark job failed with exit code: " + exitCode);
            }
        }
    }
}

Key Considerations for Your Setup #

• Spark Path Configuration: Ensure your Java process can access spark-submit—either use the full path (e.g., /opt/spark/bin/spark-submit) or add Spark’s bin directory to the system PATH environment variable.
• Cluster Compatibility: If running on a cluster (not local), update the --master argument to match your cluster manager (e.g., yarn for Hadoop YARN, spark://your-master:7077 for standalone clusters). Also, ensure your Scala JAR is accessible to all cluster nodes (e.g., store it in HDFS or a shared network drive).
• Permission Checks: The user running your Java API needs execute permissions for spark-submit, plus read access to the Scala JAR and any resources the Spark job uses.
• Result Handling: To pass Spark job results back to your GUI, try these options:
  1. Have your Spark job write results to shared storage (e.g., Redis, PostgreSQL, temporary files) and have your Java API read from it once the job finishes.
  2. Configure your Spark job to output JSON directly to stdout, then capture and parse that output in your Java code to return to the GUI.
• Async Execution: As shown in the example, running the Spark job in a separate thread prevents your Java API from blocking during long-running jobs—critical for keeping your GUI responsive.

Do You Have to Use Runtime.exec()? #

No, you don’t. ProcessBuilder is preferred because:

• It lets you set environment variables, working directories, and redirect streams more cleanly.
• It avoids command injection risks by accepting arguments as a list (instead of a single concatenated string).
• It provides better control over the process lifecycle.

That said, Runtime.exec() can work if you prefer, but it’s less flexible and more error-prone for complex commands like spark-submit.

内容的提问来源于stack exchange，提问作者Benny Suryajaya