Apache Spark 2.0.2应用级内存跟踪求助：获取PageRank内存数据并导出

Absolutely! You can totally export Spark application-specific storage and execution memory metrics to text files—no need to rely on external tools like Ganglia or Graphite that only give system-level data. Here are a few straightforward methods tailored to your Standalone cluster setup:

1. Use Spark's Built-in REST API #

Spark Standalone Master exposes a REST API that serves up detailed app-level metrics. You can use simple command-line tools to fetch this data and save it directly to a text file.

• First, note your Master node's REST endpoint (default is http://<master-ip>:8080/api/v1/applications/<your-app-id>)
• To grab storage memory details (like RDD cache usage), append /storage/rdd to the endpoint:

  curl http://<master-ip>:8080/api/v1/applications/<your-app-id>/storage/rdd > app-storage-metrics.txt
  

• For execution memory stats (executor task memory usage, free/used memory per executor), use the /executors endpoint:

  curl http://<master-ip>:8080/api/v1/applications/<your-app-id>/executors > app-execution-metrics.txt
  

• If you want continuous tracking over time, use a simple loop to periodically fetch and append data to a log file:

  while true; do
    echo "=== $(date) ===" >> app-metrics.log
    curl http://<master-ip>:8080/api/v1/applications/<your-app-id>/executors >> app-metrics.log
    sleep 10  # Adjust interval as needed
  done
  

2. Download Metrics Directly from the Spark UI #

The Spark UI gives you a visual way to access metrics, and you can download them as CSV or text files with just a few clicks:

• Cluster-level UI: Access http://<master-ip>:8080, find your PageRank application, and click into its details.
  • Go to the Storage tab: Click the "Download" button at the top to get a CSV file with all RDD storage memory statistics.
  • Go to the Executors tab: Use the "Download" button here to grab executor-level execution memory data (total, used, free memory per executor).
• App-specific UI: While the application is running, each driver exposes a UI at http://<driver-ip>:4040—this has the same download options, and is great for real-time metrics during the PageRank run.
• Note: The 4040 UI disappears once the app finishes, so make sure to download data before it completes (or enable the Spark History Server to retain past app data).

3. Custom Logging with Spark Listeners (For Granular Control) #

If you need super detailed, real-time memory events during the PageRank execution, you can build a custom Spark Listener to capture and write metrics to a file.

• Create a Scala or Java class that extends SparkListener, then override methods like onTaskEnd (to track task execution memory) or onBlockUpdated (to track storage memory changes).
• Package this listener into a JAR, then add it to your Spark submit command:

  spark-submit --class org.apache.spark.examples.PageRank \
    --conf spark.extraListeners=com.yourcompany.CustomMemoryListener \
    --jars your-listener.jar \
    spark-examples_2.12-3.5.0.jar input output
  

• Your custom listener can write metrics directly to a local file on the Master node, or to a shared HDFS path (since you have Hadoop integrated).

Quick Tips #

• Ensure you have network access to the Master's 8080 port and the driver/executors' 4040 ports.
• For completed apps, the Spark History Server (if enabled) lets you access past metrics via its UI or REST API.
• The REST API returns JSON data, which you can parse later with tools like jq if you need to filter or analyze specific metrics.

内容的提问来源于stack exchange，提问作者Assia6