1.1下载spark
地址：https://mirrors.tuna.tsinghua.edu.cn/apache/spark/spark-2.3.0/
1.2上传解压；
xiaoye@ubuntu:~/Downloads$ lsapache-activemq-5.15.3-bin.tar.gz hive-0.13.1-cdh5.2.0.tar.gzflume-ng-1.6.0-cdh5.10.1.tar.gz spark-2.3.0-bin-hadoop2.7.tgzhadoop-2.5.0-cdh5.2.0.tar.gz zookeeper-3.4.5-cdh5.10.2.tar.gz
xiaoye@ubuntu:~/Downloads$ tar -zxvf spark-2.3.0-bin-hadoop2.7.tgz -C ../
1.3为安装包建立一个软连接：
xiaoye@ubuntu:~$ ln -s spark-2.3.0-bin-hadoop2.7/ spark
1.4进入spark/conf修改配置文件
xiaoye@ubuntu:~$ cd sparkxiaoye@ubuntu:~/spark$ cd confxiaoye@ubuntu:~/spark/conf$ lsdocker.properties.template metrics.properties.template spark-env.sh.templatefairscheduler.xml.template slaves.templatelog4j.properties.template spark-defaults.conf.template
xiaoye@ubuntu:~/spark/conf$
复制spark-env.sh.template并重命名为spark-env.sh，并在文件最后添加配置内容
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64 export HADOOP_HOME=/home/xiaoye/hadoop export HADOOP_CONF_DIR=/home/xiaoye/hadoop/etc/hadoop export SPARK_WORKER_MEMORY=500m export SPARK_WORKER_CORES=1 export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=ubuntu:2181,ubuntu2:2181,ubuntu3:2181 -Dspark.deploy.zookeeper.dir=/spark"解释：
集群搭建时配置的spark参数可能和现在的不一样，主要是考虑个人电脑配置问题，如果memory配置太大，机器运行很慢。 说明： -Dspark.deploy.recoveryMode=ZOOKEEPER #说明整个集群状态是通过zookeeper来维护的，整个集群状态的恢复也是通过zookeeper来维护的。就是说用zookeeper做了spark的HA配置，Master(Active)挂掉的话，Master(standby)要想变成Master（Active）的话，Master(Standby)就要像zookeeper读取整个集群状态信息，然后进行恢复所有Worker和Driver的状态信息，和所有的Application状态信息； -Dspark.deploy.zookeeper.url=
buntu:2181,ubuntu2:2181,ubuntu3:2181 #将所有配置了zookeeper，并且在这台机器上有可能做master(Active)的机器都配置进来；（我用了3台，就配置了3台）-Dspark.deploy.zookeeper.dir=/spark 这里的dir和zookeeper配置文件zoo.cfg中的dataDir的区别？？？ -Dspark.deploy.zookeeper.dir是保存spark的元数据，保存了spark的作业运行状态； zookeeper会保存spark集群的所有的状态信息，包括所有的Workers信息，所有的Applactions信息，所有的Driver信息,如果集群
复制slaves.template成slaves:
xiaoye@ubuntu:~/spark/conf$ cp slaves.template slaves
修改slaves:
xiaoye@ubuntu:~/spark/conf$ vi slaves
ubuntu ubuntu2 ubuntu31.5 一台机子配置完后分发给其他节点
xiaoye@ubuntu:~$ scp -r spark-2.3.0-bin-hadoop2.7/ xiaoye@192.168.26.141:/home/xiaoye
xiaoye@ubuntu:~$ scp -r spark-2.3.0-bin-hadoop2.7/ xiaoye@192.168.26.142:/home/xiaoye
1.6为每台机子创建软连接
xiaoye@ubuntu2:~$ ln -s spark-2.3.0-bin-hadoop2.7/ spark
xiaoye@ubuntu3:~$ ln -s spark-2.3.0-bin-hadoop2.7/ spark
1.7为每台机子配置环境变量
xiaoye@ubuntu:~$ vim .bashrc
#Spark export SPARK_HOME=/home/xiaoye/spark export PATH=$PATH:$SPARK_HOME/bin配置好后立即生效：
xiaoye@ubuntu:~$ source .bashrc
2试着启动spark集群
2.1启动zookeeper集群：
每台机子执行如下操作：
xiaoye@ubuntu:~$ ./zookeeper/sbin/zkServer.sh startJMX enabled by defaultUsing config: /home/xiaoye/zookeeper/sbin/../conf/zoo.cfgStarting zookeeper ... STARTEDxiaoye@ubuntu:~$ ./zookeeper/sbin/zkServer.sh statusJMX enabled by defaultUsing config: /home/xiaoye/zookeeper/sbin/../conf/zoo.cfg
Mode: leader
2.2启动hdfs:
在一个节点执行以下命令即可：
xiaoye@ubuntu:~$ ./hadoop/sbin/start-dfs.sh 18/06/09 21:36:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Starting namenodes on [ubuntu ubuntu2] ubuntu: Warning: Permanently added 'ubuntu,192.168.26.140' (ECDSA) to the list of known hosts. ubuntu2: Warning: Permanently added 'ubuntu2,192.168.26.141' (ECDSA) to the list of known hosts. ubuntu2: starting namenode, logging to /home/xiaoye/hadoop/logs/hadoop-xiaoye-namenode-ubuntu2.out ubuntu: starting namenode, logging to /home/xiaoye/hadoop/logs/hadoop-xiaoye-namenode-ubuntu.out ubuntu2: Warning: Permanently added 'ubuntu2,192.168.26.141' (ECDSA) to the list of known hosts. ubuntu: Warning: Permanently added 'ubuntu,192.168.26.140' (ECDSA) to the list of known hosts. ubuntu3: Warning: Permanently added 'ubuntu3,192.168.26.142' (ECDSA) to the list of known hosts. ubuntu: starting datanode, logging to /home/xiaoye/hadoop/logs/hadoop-xiaoye-datanode-ubuntu.out ubuntu2: starting datanode, logging to /home/xiaoye/hadoop/logs/hadoop-xiaoye-datanode-ubuntu2.out ubuntu3: starting datanode, logging to /home/xiaoye/hadoop/logs/hadoop-xiaoye-datanode-ubuntu3.out Starting journal nodes [ubuntu ubuntu2 ubuntu3] ubuntu3: Warning: Permanently added 'ubuntu3,192.168.26.142' (ECDSA) to the list of known hosts. ubuntu: Warning: Permanently added 'ubuntu,192.168.26.140' (ECDSA) to the list of known hosts. ubuntu2: Warning: Permanently added 'ubuntu2,192.168.26.141' (ECDSA) to the list of known hosts. ubuntu: starting journalnode, logging to /home/xiaoye/hadoop/logs/hadoop-xiaoye-journalnode-ubuntu.out ubuntu3: starting journalnode, logging to /home/xiaoye/hadoop/logs/hadoop-xiaoye-journalnode-ubuntu3.out ubuntu2: starting journalnode, logging to /home/xiaoye/hadoop/logs/hadoop-xiaoye-journalnode-ubuntu2.out 18/06/09 21:36:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Starting ZK Failover Controllers on NN hosts [ubuntu ubuntu2] ubuntu2: Warning: Permanently added 'ubuntu2,192.168.26.141' (ECDSA) to the list of known hosts. ubuntu: Warning: Permanently added 'ubuntu,192.168.26.140' (ECDSA) to the list of known hosts. ubuntu2: starting zkfc, logging to /home/xiaoye/hadoop/logs/hadoop-xiaoye-zkfc-ubuntu2.out ubuntu: starting zkfc, logging to /home/xiaoye/hadoop/logs/hadoop-xiaoye-zkfc-ubuntu.out xiaoye@ubuntu:~$ jps \2746 NameNode 3252 DFSZKFailoverController 2573 QuorumPeerMain 2861 DataNode 3324 Jps 3060 JournalNode2.3启动spark
在三个节点分别执行以下命令：
xiaoye@ubuntu:~$ ./spark/sbin/start-all.sh org.apache.spark.deploy.master.Master running as process 3359. Stop it first. localhost: Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts. localhost: starting org.apache.spark.deploy.worker.Worker, logging to /home/xiaoye/spark/logs/spark-xiaoye-org.apache.spark.deploy.worker.Worker-1-ubuntu.out好了之后jps分别查看启动的进程：
leader节点：
xiaoye@ubuntu:~$ jps 2746 NameNode 3359 Master 3252 DFSZKFailoverController 3550 Worker 2573 QuorumPeerMain 2861 DataNode 3583 Jps 3060 JournalNode其他两个从节点：
xiaoye@ubuntu2:~$ jps 2488 JournalNode 2748 Worker 2784 Jps 2654 Master 2229 QuorumPeerMain 2377 DataNode xiaoye@ubuntu3:~$ jps 2510 Master 2582 Worker 2298 DataNode 2402 JournalNode 2665 Jps 2198 QuorumPeerMain3，验证是否成功启动
在web上查看：
4，验证HA高可用
动干掉ubuntu上面的Master进程，观察是否会自动进行切换
xiaoye@ubuntu:~$ jps 3656 Jps 2746 NameNode 3359 Master 3252 DFSZKFailoverController 2573 QuorumPeerMain 2861 DataNode 3060 JournalNode xiaoye@ubuntu:~$ kill -9 3359 xiaoye@ubuntu:~$ jps 2746 NameNode 3252 DFSZKFailoverController 2573 QuorumPeerMain 2861 DataNode 3666 Jps 3060 JournalNode查看web：
再看看其他两个节点有没有成功上位成live的：
ubuntu3自动成为live节点说明成功。
5，执行spark程序 on standalone
5.1执行第一个spark程序，执行的是spark自带的案例：
在master下执行：
xiaoye@ubuntu3:~$ /home/xiaoye/spark/bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://ubuntu3:7077 --executor-memory 500m --total-executor-cores 1 /home/xiaoye/spark/examples/jars/spark-examples_2.11-2.3.0.jar 100其中--master spark://ubuntu3:7077是下图的位置的结果：
运行结果：
至于含义我们以后再讨论
5.2 启动spark shell
xiaoye@ubuntu3:~$ /home/xiaoye/spark/bin/spark-shell \ > --master spark://ubuntu3:7077 \ > --executor-memory 500m > --total-executor-cores 1参数说明：
--master spark://ubuntu3:7077 指定Master的地址
--executor-memory 500m:指定每个worker可用内存为500m
--total-executor-cores 1: 指定整个集群使用的cup核数为1个
xiaoye@ubuntu3:~$ xiaoye@ubuntu3:~$ /home/xiaoye/spark/bin/spark-shell \ > --master spark://ubuntu3:7077 \ > --executor-memory 500m --^H^H2018-06-09 23:21:53 WARN Utils:66 - Your hostname, ubuntu3 resolves to a loopback address: 127.0.1.1; using 192.168.26.142 instead (on interface ens33) 2018-06-09 23:21:53 WARN Utils:66 - Set SPARK_LOCAL_IP if you need to bind to another address 2018-06-09 23:21:55 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). Spark context Web UI available at http://ubuntu3:4040 Spark context available as 'sc' (master = spark://ubuntu3:7077, app id = app-20180609232223-0001). Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.3.0 /_/ Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_162) Type in expressions to have them evaluated. Type :help for more information. scala> -- scala>注意：
如果启动spark shell时没有指定master地址，但是也可以正常启动spark shell和执行spark shell中的程序，其实是启动了spark的local模式，该模式仅在本机启动一个进程，没有与集群建立联系。
Spark Shell中已经默认将SparkContext类初始化为对象sc。用户代码如果需要用到，则直接应用sc即可
Spark Shell中已经默认将SparkSQl类初始化为对象spark。用户代码如果需要用到，则直接应用spark即可
5.3在spark shell中编写word count 程序
创建一个文件，随便写点内容：
xiaoye@ubuntu3:~$ cat classes/aa.txt hello world! aa aa d d dg g然后上传到hdfs中：
xiaoye@ubuntu3:~/hadoop$ ./hadoop/ fs -mkdir -p /spark -bash: ./hadoop/: Is a directory xiaoye@ubuntu3:~/hadoop$ hadoop fs -mkdir -p /spark 18/06/09 23:32:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable xiaoye@ubuntu3:~/hadoop$ hadoop fs -ls /spark 18/06/09 23:33:12 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable xiaoye@ubuntu3:~/hadoop$ hadoop fs -put classes/aa.txt /spark 18/06/09 23:34:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable put: `classes/aa.txt': No such file or directory xiaoye@ubuntu3:~/hadoop$ ls bin etc hadoop lib sbin tmp bin-mapreduce1 examples include libexec share cloudera examples-mapreduce1 journal logs src xiaoye@ubuntu3:~/hadoop$ cd .. xiaoye@ubuntu3:~$ hadoop fs -put classes/aa.txt /spark 18/06/09 23:34:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable xiaoye@ubuntu3:~$ hadoop fs -ls /spark 18/06/09 23:34:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Found 1 items -rw-r--r-- 2 xiaoye supergroup 29 2018-06-09 23:34 /spark/aa.txt5.4在spark shell中用scala编写spark程序，按空格分割数据
scala> sc.textFile("/spark/aa.txt").flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_).saveAsTextFile("/spark/out")说明：
sc是SparkContext对象，该对象是提交spark程序的入口
textFile("/spark/aa.txt")是hdfs中读取数据
flatMap(_.split(" "))先map再压平
map((_,1))将单词和1构成元组
reduceByKey(_+_)按照key进行reduce，并将value累加
saveAsTextFile("/spark/out")将结果写入到hdfs中
执行完后，查看hdfs的执行结果：
xiaoye@ubuntu3:~$ hadoop fs -cat /spark/out/p* 18/06/09 23:52:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable (d,2) (aa,2) (hello,1) (world!,1) (dg,1) (g,1)执行结果跟我们相要的结果是一样的。
6，执行spark程序 on Yarn
首先要成功启动zookeeper集群、HDFS集群、YARN集群
启动spark on Yarn
xiaoye@ubuntu3:~$
spark-shell --master yarn --deploy-mode client启动时间比较长，成功启动：
启动的进程如下：
xiaoye@ubuntu3:~$ jps 2298 DataNode 5150 ExecutorLauncher 3950 ResourceManager 2402 JournalNode 5375 Jps 5275 CoarseGrainedExecutorBackend 5274 CoarseGrainedExecutorBackend 4092 NodeManager 2198 QuorumPeerMain 4424 SparkSubmit