Spark新手求助：spark-submit本地模式运行后如何查看输出结果？

Hey there! As someone new to Spark and Scala, it's totally normal to wonder where your job's output ends up—let's break down all the places you can check, plus fix why you're not seeing it in the Spark History UI.

1. 直接查看提交命令的终端输出 #

When you run spark-submit in local mode (--master local[*]), the default behavior is to print all your program's output (like println() statements) and Spark's logs directly to the terminal window where you executed the command.

If you didn't see anything here, double-check:

• Did you run the command in the background (with & at the end)? If so, you might have redirected output to a file or lost it—try running the job in the foreground first to confirm.
• Does your sample program actually produce output? Maybe add a simple println("Hello Spark!") to test if it shows up here.

2. 修复Spark历史Web UI的显示问题 #

You've already set the right configs for event logging (spark.eventLog.enabled=true and spark.eventLog.dir=/tmp/spark-events), but there's one key step you might have missed: starting the Spark History Server. Here's what to do:

• First, make sure the /tmp/spark-events directory exists and has read/write permissions for the user running Spark. You can create it with mkdir -p /tmp/spark-events if needed.
• Start the history server with this command:

  start-history-server.sh
  

• Once it's running, open http://localhost:18080 in your browser. You should see your completed job listed here—click into it to view logs, task details, and any output captured by Spark.

A quick note: The History UI only shows logs for completed jobs. If your job is still running, use the Spark driver's real-time UI at http://localhost:4040 (this port is only available while the job is active).

3. 检查Spark的本地日志文件 #

Spark also writes detailed logs to local files, usually in the $SPARK_HOME/logs directory (replace $SPARK_HOME with your actual Spark installation path). Look for files named like:

spark-<your-username>-org.apache.spark.deploy.SparkSubmit-<process-id>.out

These files contain the same output as the terminal, plus additional debug logs that might be helpful if you run into issues later.

4. 为什么Consul看不到输出？ #

Consul is primarily for service discovery—it doesn't collect Spark job output by default. Unless you've explicitly configured an integration (like pushing Spark logs to a Consul-connected logging system), it's expected that you won't see your job's output there. You can ignore this for now while you're testing local Spark jobs.

内容的提问来源于stack exchange，提问作者mahzad.mah