多节点MapR集群Spark-shell启动报错及数据分发测试咨询

Part 1: Fixing Spark Shell Launch Failures #

Hey there, let’s work through your spark-shell issue first—based on your setup details, these are the most likely fixes to try:

1. Address Java Version Compatibility #

Spark 2.2.0 only officially supports Java 7 and 8—Java 9’s module system introduces compatibility conflicts with Spark’s older codebase, which is probably a key issue here.

• Recommended fix: Downgrade to Java 8, then update your spark-env.sh to point to the correct Java home:

  export JAVA_HOME=/usr/lib/jvm/java-8-oracle/
  

• If you must keep Java 9, add this flag when launching spark-shell to bypass module errors:

  ./spark-shell --add-modules java.xml.bind
  

2. Validate Spark Environment Configs #

Double-check your spark-env.sh entries for typos or invalid paths:

• Run echo $JAVA_HOME and scala -version in the terminal to confirm JAVA_HOME and SCALA_HOME point to working directories
• Verify SPARK_MASTER_IP is the correct IP of your MapR Master node (ping it from all slaves to confirm connectivity)
• Ensure SPARK_WORKER_DIR exists and has write permissions for your user:

  mkdir -p /home/administrator/spark-2.2.0-bin-hadoop2.7 && chown -R administrator:administrator $_
  

3. Confirm Passwordless SSH Between Nodes #

Spark requires passwordless SSH access from the Master to all Slave nodes. Test this from the Master:

ssh slave-node-ip

If prompted for a password, set up SSH keys:

# On Master node
ssh-keygen -t rsa -P ""
ssh-copy-id administrator@slave-node-1-ip
ssh-copy-id administrator@slave-node-2-ip

4. Check Logs for Specific Errors #

If the above steps don’t resolve the issue, dig into the logs for details:

• Check Spark’s log directory (default: $SPARK_HOME/logs) for spark-master-*.out and spark-worker-*.out files
• Look for console errors like Connection refused (Master unreachable) or NoClassDefFoundError (version mismatch)

After making changes, restart the Spark cluster:

$SPARK_HOME/sbin/stop-all.sh
$SPARK_HOME/sbin/start-all.sh

Part 2: Testing MapR Cluster Data Distribution #

MapR’s data distribution relies on its distributed file system (MapR-FS). Here’s how to verify it’s working as expected:

1. Upload a Test File & Check Block Distribution #

• Upload a large test file (50MB+) to MapR-FS (replace <cluster-name> with your actual cluster name):

  hadoop fs -put /path/to/local/test-file /mapr/<cluster-name>/test-data/
  

• Use MapR’s CLI to see where the file’s blocks are stored:

  maprcli file info -path /mapr/<cluster-name>/test-data/test-file
  

  Look for the blocklocations section—you should see blocks spread across your Slave nodes (and possibly the Master, depending on your volume config).

2. Verify Distributed Processing with Spark #

Once spark-shell is working, run a simple job to confirm data is processed across nodes:

// Read the test file from MapR-FS
val testData = sc.textFile("/mapr/<cluster-name>/test-data/test-file")
// Trigger a distributed count
testData.count()

• Open the Spark UI at http://<master-ip>:4040, go to the Stages tab. You’ll see tasks split across your Slave nodes, each processing a portion of the file’s blocks.

3. Test Fault Tolerance & Replication #

MapR-FS replicates blocks (default: 3 copies) across nodes. Test this resilience:

• Stop one Slave node (either shut it down or stop MapR services: maprcli node services -nodes <slave-ip> -name fs -action stop)
• Try reading the test file again via Spark or hadoop fs -cat /mapr/<cluster-name>/test-data/test-file—it should load without errors, as MapR will fetch blocks from remaining replicas.
• Check your volume’s replication settings:

  maprcli volume info -name <your-volume-name>
  

  Look for the replication field to confirm the number of copies.

4. Inspect Local Storage on Nodes #

For hands-on verification, check the local MapR storage directory on each node (common paths: /opt/mapr/dfs/volumes/ or /mapr/<cluster-name>/). You’ll see the file blocks stored on nodes that hold replicas.

内容的提问来源于stack exchange，提问作者Vaasir Nisaar