Spark集群搭建

一、前提

1、安装hadoop集群

2、安装scala

3、假设三个节点：master、slave1、slave2

二、Spark集群搭建

1、建立文件夹存放spark压缩包

2、在本文件夹下解压

3、将解压得到的文件夹名重命名

语言 方法

7755 ebbg8K93h4

qaY7t最挣钱没人干的行业

9427 2012.04.15 20-35-29

4、进入spark-2.20\conf文件夹，修改spark-env.sh.template为spark-env.sh

5、在spark-env.sh中添加以下内容：

export JAVA_HOME=/opt/softWare/java/jdk1.8.0_141export SCALA_HOME=/opt/software/scala/scala-2.12.4export HADOOP_HOME=/opt/software/hadoop/hadoop-2.7.3export HADOOP_CONF_DIR=/opt/softWare/hadoop/hadoop-2.7.3/etc/hadoopexport SPARK_MASTER_IP=192.168.XXX.XX#export SPARK_WORKER_INSTANCES=1 //每个Slave中启动几个Worker实例export SPARK_WORKER_MEMORY=1g #export SPARK_DIST_CLASSPATH=$(/home/hadoop/hadoop-2.7.2/bin/hadoop classpath)

注意：前面有#的可以不用添加

6、进入spark-2.20\conf文件夹，修改slaves.template为slave，添加如下内容：

masterslave1slave2 7、赋权限

chmod 777 /opt/software/spark/spark-2.20/bin/*chmod 777 /opt/software/spark/spark-2.20/sbin/* 8、scp到其他两台机器。

scp -r /opt/software/spark slave1:/opt/software/sparkscp -r /opt/software/spark slave2:/opt/software/spark 9、在每台主机配置/etc/profile中配置环境变量

#sparkexport SPARK_HOME=/opt/software/spark/spark-2.20PATH=$SPARK_HOME/bin:$PATH

然后：source /etc/profile

10、启动spark

cd /opt/software/spark/spark-2.20/sbin/./start-all.sh 11、查看jps

12、查看spark Web

ip：8080