请求CentOS7下1主2从Spark集群+YARN+Livy搭建详细步骤

Hey there! I've walked through this exact setup dozens of times for new teams, so let's break it down step by step—no confusing jargon, just clear actions you can follow on your CentOS 7 nodes. We'll start with the basics, build out Hadoop/YARN, set up Spark, then add Livy for REST access. Let's go!

These are the foundational steps every node needs to complete before we start installing cluster software.

• Set Hostnames & Hosts Mapping

  1. Set hostname for Master:

     hostnamectl set-hostname spark-master
     

     For Slave1:

     hostnamectl set-hostname spark-slave1
     

     Slave2:

     hostnamectl set-hostname spark-slave2
     

  2. Edit /etc/hosts on all nodes to map IPs to hostnames (replace with your actual IPs):

     192.168.1.100 spark-master
     192.168.1.101 spark-slave1
     192.168.1.102 spark-slave2
     

  3. Restart network to apply changes:

     systemctl restart network
     

• Configure SSH Passwordless Login (Master → All Nodes)
  This lets the Master control Slave nodes without typing passwords every time.

  1. Generate SSH key pair on Master (press Enter for all prompts, no passphrase):

     ssh-keygen -t rsa
     

  2. Copy public key to all nodes (including Master itself):

     ssh-copy-id spark-master
     ssh-copy-id spark-slave1
     ssh-copy-id spark-slave2
     

  3. Test it: ssh spark-slave1 should log you in immediately without a password.

• Install JDK 8 (Required for Spark & Hadoop)
  Spark and Hadoop work best with JDK 8—skip newer versions to avoid compatibility issues.

  1. Install via yum:

     yum install java-1.8.0-openjdk-devel -y
     

  2. Verify installation:

     java -version
     

     You should see output for openjdk version "1.8.x".
  3. Set environment variables by editing /etc/profile:

     export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
     export PATH=$PATH:$JAVA_HOME/bin
     

  4. Apply changes:

     source /etc/profile
     

Spark runs on YARN, so we need a working Hadoop cluster first (HDFS + YARN).

2.1 Download & Extract Hadoop (Master Node Only) #

1. Grab a compatible Hadoop version (I recommend 3.3.6 for Spark 3.x) from the official Apache archive, then extract it:

   tar -zxvf hadoop-3.3.6.tar.gz -C /usr/local/
   

2. Rename for easier access:

   mv /usr/local/hadoop-3.3.6 /usr/local/hadoop
   

2.2 Configure Hadoop Environment Variables #

Edit /etc/profile on Master, add:

export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

Apply changes: source /etc/profile

2.3 Modify Hadoop Configuration Files (Master Node) #

All configs live in $HADOOP_HOME/etc/hadoop:

1. hadoop-env.sh #

Add these lines to set core paths:

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

2. core-site.xml #

Replace the existing <configuration> block with:

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://spark-master:9000</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/local/hadoop/tmp</value>
    </property>
</configuration>

3. hdfs-site.xml #

Set replication count (match your number of Slaves) and storage paths:

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>2</value> <!-- 2 Slaves = 2 replicas -->
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/usr/local/hadoop/hdfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>/usr/local/hadoop/hdfs/data</value>
    </property>
</configuration>

4. yarn-site.xml #

Configure YARN Resource Manager settings:

<configuration>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>spark-master:8032</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>spark-master:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>spark-master:8031</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>spark-master:8033</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>spark-master:8088</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
</configuration>

5. mapred-site.xml #

First copy the template:

cp mapred-site.xml.template mapred-site.xml

Then add:

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

6. workers (Hadoop 3.x uses workers; 2.x uses slaves) #

Edit the file to list your Slave nodes:

spark-slave1
spark-slave2

2.4 Sync Hadoop to All Slaves (Master Node) #

Copy the Hadoop directory and updated profile to Slaves:

scp -r /usr/local/hadoop spark-slave1:/usr/local/
scp -r /usr/local/hadoop spark-slave2:/usr/local/

scp /etc/profile spark-slave1:/etc/
scp /etc/profile spark-slave2:/etc/

On each Slave node, run source /etc/profile to apply environment variables.

2.5 Initialize & Start Hadoop Cluster (Master Node) #

1. Format HDFS (⚠️ Only run this once! Re-formatting will delete all HDFS data):

   hdfs namenode -format
   

2. Start HDFS:

   start-dfs.sh
   

3. Start YARN:

   start-yarn.sh
   

4. Verify:
   • Visit http://spark-master:50070 to check HDFS status
   • Visit http://spark-master:8088 to check YARN Resource Manager
   • Run jps on nodes:
     • Master: Should show NameNode, ResourceManager, SecondaryNameNode
     • Slaves: Should show DataNode, NodeManager

Now we'll configure Spark to run on top of our YARN cluster.

3.1 Download & Extract Spark (Master Node Only) #

Grab a Spark version pre-built for your Hadoop version (e.g., 3.5.0 for Hadoop 3.x):

tar -zxvf spark-3.5.0-bin-hadoop3.tgz -C /usr/local/
mv /usr/local/spark-3.5.0-bin-hadoop3 /usr/local/spark

3.2 Configure Spark Environment Variables #

Edit /etc/profile on Master, add:

export SPARK_HOME=/usr/local/spark
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin

Apply changes: source /etc/profile, then sync the profile to Slaves and apply there too.

3.3 Modify Spark Configuration Files (Master Node) #

Configs live in $SPARK_HOME/conf:

1. spark-env.sh #

Copy the template:

cp spark-env.sh.template spark-env.sh

Add these lines (adjust memory values based on your server specs):

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export SPARK_MASTER_HOST=spark-master
export SPARK_WORKER_MEMORY=2g # Allocate 2GB per Slave worker
export SPARK_DRIVER_MEMORY=1g

2. workers #

List your Slave nodes:

spark-slave1
spark-slave2

3. spark-defaults.conf #

Copy the template:

cp spark-defaults.conf.template spark-defaults.conf

Add YARN-specific configs:

spark.master                     yarn
spark.driver.memory              1g
spark.executor.memory            1g
spark.executor.cores             1
spark.yarn.jars                  hdfs://spark-master:9000/spark-jars/*

Then upload Spark jars to HDFS so YARN can access them:

hdfs dfs -mkdir -p /spark-jars
hdfs dfs -put $SPARK_HOME/jars/* /spark-jars/

3.4 Sync Spark to All Slaves (Master Node) #

scp -r /usr/local/spark spark-slave1:/usr/local/
scp -r /usr/local/spark spark-slave2:/usr/local/

3.5 Start Spark Cluster (Master Node) #

start-all.sh

Verify:

• Visit http://spark-master:8080 to see Spark cluster status (you should see 2 Slave workers)
• Run jps on nodes: Master shows Master, Slaves show Worker

Livy lets you submit Spark jobs via REST API—we only need to install it on the Master node.

4.1 Download & Extract Livy (Master Node Only) #

Grab Livy 0.8.0 (compatible with Spark 3.x):

unzip livy-0.8.0-incubating-bin.zip -C /usr/local/
mv /usr/local/livy-0.8.0-incubating-bin /usr/local/livy

4.2 Configure Livy Environment Variables #

Edit /etc/profile on Master, add:

export LIVY_HOME=/usr/local/livy
export PATH=$PATH:$LIVY_HOME/bin

Apply changes: source /etc/profile

4.3 Modify Livy Configuration Files (Master Node) #

Configs live in $LIVY_HOME/conf:

1. livy-env.sh #

Copy the template:

cp livy-env.sh.template livy-env.sh

Add:

export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export SPARK_HOME=/usr/local/spark
export SPARK_CONF_DIR=$SPARK_HOME/conf
export LIVY_LOG_DIR=/usr/local/livy/logs
export LIVY_PID_DIR=/usr/local/livy/pids

Create required directories:

mkdir -p $LIVY_LOG_DIR $LIVY_PID_DIR

2. livy.conf #

Copy the template:

cp livy.conf.template livy.conf

Add key settings:

livy.server.host = spark-master
livy.server.port = 8998
livy.spark.master = yarn
livy.spark.deployMode = cluster
livy.yarn.jar = hdfs://spark-master:9000/livy/livy-server.jar
livy.repl.enableHiveContext = true

Upload Livy's server jar to HDFS:

hdfs dfs -mkdir -p /livy
hdfs dfs -put $LIVY_HOME/jars/livy-server.jar /livy/

4.4 Start Livy Service (Master Node) #

livy-server start

Verify:

• Visit http://spark-master:8998 to see Livy's web UI
• Test the REST API by submitting a sample job:

  curl -X POST --data '{"file": "/usr/local/spark/examples/src/main/python/pi.py", "args": ["10"]}' -H "Content-Type: application/json" http://spark-master:8998/batches
  

  Check job status with:

  curl http://spark-master:8998/batches
  

  You should see the job marked as success once it finishes.

• Stop Services in Order: Livy → Spark → YARN → HDFS. Commands:

  livy-server stop
  stop-all.sh
  stop-yarn.sh
  stop-dfs.sh
  

• Adjust Memory Settings: If you get OutOfMemory errors, tweak memory values in spark-env.sh and spark-defaults.conf to match your server's available RAM.
• Firewall Rules: Open these ports on all nodes if firewalld is enabled:

  # HDFS
  firewall-cmd --add-port=9000/tcp --permanent
  firewall-cmd --add-port=50070/tcp --permanent
  # YARN
  firewall-cmd --add-port=8088/tcp --permanent
  # Spark
  firewall-cmd --add-port=8080/tcp --permanent
  # Livy
  firewall-cmd --add-port=8998/tcp --permanent
  firewall-cmd --reload
  

• Log Debugging: If services fail to start, check logs:
  • Spark: $SPARK_HOME/logs
  • Livy: $LIVY_LOG_DIR
  • Hadoop: $HADOOP_HOME/logs

内容的提问来源于stack exchange，提问作者Rajat