一票从、安裝概述
本文將繼續(xù)配置我的虛擬機(jī)漫雕，此文章需要基于hadoop平臺(tái)安裝成功才可spark集群滨嘱，如需參考請(qǐng)點(diǎn)擊鏈接：http://www.reibang.com/p/1bbfbb3968b6，我的虛擬機(jī)的情況也在此篇文章說(shuō)明了浸间，jdk及hadoop是安裝spark集群的依賴(lài)環(huán)境太雨，此處不再贅述了。spark所選版本3.0.3魁蒜，而spark 3.0+是基于scala 2.12版本編譯的囊扳，所以還需要安裝scala2.12。
二兜看、軟件選擇
- spark版本的選擇锥咸，附spark官網(wǎng)的地址，http://spark.apache.org/downloads.html
image.png - scala版本的選擇细移，附scala官網(wǎng)的地址搏予，https://www.scala-lang.org/download/2.12.2.html
image.png
三、scala環(huán)境的安裝弧轧，三臺(tái)機(jī)器都要執(zhí)行以下操作
- 首先上傳scala壓縮包上傳到hadoop用戶(hù)的根目錄
[hadoop@hadoop01 ~]# ll
total 633388
drwxrwxrwx. 11 hadoop hadoop 173 Nov 13 09:08 hadoop-2.10.1
-rw-r--r--. 1 hadoop hadoop 408587111 Nov 12 11:07 hadoop-2.10.1.tar.gz
-rw-r--r--. 1 hadoop hadoop 19596088 Nov 30 17:14 scala-2.12.2.tgz
[hadoop@hadoop01 ~]#
- 非必要步驟雪侥，如后續(xù)上傳文件出現(xiàn)此權(quán)限問(wèn)題，也可使用此方法劣针。當(dāng)用戶(hù)權(quán)限不是hadoop時(shí)校镐，可使用chown命令修改用戶(hù)權(quán)限，此命令需要在root用戶(hù)下執(zhí)行
[hadoop@hadoop01 ~]$ exit
exit
You have new mail in /var/spool/mail/root
[root@hadoop01 ~]# cd /home/hadoop/
[root@hadoop01 hadoop]# ll
total 633388
drwxrwxrwx. 11 hadoop hadoop 173 Nov 13 09:08 hadoop-2.10.1
-rw-r--r--. 1 hadoop hadoop 408587111 Nov 12 11:07 hadoop-2.10.1.tar.gz
-rw-r--r--. 1 hadoop hadoop 19596088 Nov 30 17:14 scala-2.12.2.tgz
[root@hadoop01 hadoop]$ chown -R hadoop:hadoop scala-2.12.2.tgz
[root@hadoop01 hadoop]# su hadoop
- 解壓scala文件
[hadoop@hadoop01 ~]$ tar -zxvf scala-2.12.2.tgz
[hadoop@hadoop01 ~]$ cd scala-2.12.2
[hadoop@hadoop01 scala-2.12.2]$ pwd
/home/hadoop/scala-2.12.2
- 修改scala的環(huán)境變量
[hadoop@hadoop01 scala-2.12.2]$ vim ~/.bashrc
export HADOOP_HOME=/home/hadoop/hadoop-2.10.1
export SCALA_HOME=/home/hadoop/scala-2.12.2
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$SCALA_HOME/bin
- 使環(huán)境變量生效
[hadoop@hadoop01 scala-2.12.2]# source ~/.bashrc
- 驗(yàn)證scala是否安裝成功
[root@hadoop01 scala-2.12.2]# scala -version
Scala code runner version 2.12.2 -- Copyright 2002-2017, LAMP/EPFL and Lightbend, Inc.
三捺典、spark環(huán)境的安裝
- 首先上傳spark壓縮包上傳到hadoop01虛擬機(jī)hadoop用戶(hù)的根目錄
[hadoop@hadoop01 ~]$ ll
total 633388
drwxrwxrwx. 11 hadoop hadoop 173 Nov 13 09:08 hadoop-2.10.1
-rw-r--r--. 1 hadoop hadoop 408587111 Nov 12 11:07 hadoop-2.10.1.tar.gz
drwxrwxr-x. 6 hadoop hadoop 50 Apr 13 2017 scala-2.12.2
-rw-r--r--. 1 hadoop hadoop 19596088 Nov 30 17:14 scala-2.12.2.tgz
-rw-r--r--. 1 hadoop hadoop 220400553 Nov 30 17:14 spark-3.0.3-bin-hadoop2.7.tgz
You have new mail in /var/spool/mail/root
- 解壓spark文件
[hadoop@hadoop01 ~]$ tar -zxvf spark-3.0.3-bin-hadoop2.7.tgz
[hadoop@hadoop01 ~]$ cd spark-3.0.3-bin-hadoop2.7
[hadoop@hadoop01 spark-3.0.3-bin-hadoop2.7]$ pwd
/home/hadoop/spark-3.0.3-bin-hadoop2.7
- 配置slaves文件
[hadoop@hadoop01 spark-3.0.3-bin-hadoop2.7]$ cd conf
[hadoop@hadoop01 conf]$ mv slaves.template slaves
[hadoop@hadoop01 conf]$ vim slaves
#將localhost修改成一下三個(gè)節(jié)點(diǎn)的名稱(chēng)
hadoop01
hadoop02
hadoop03
- 配置spark-env.sh文件
[hadoop@hadoop01 conf]$ mv spark-env.sh.template spark-env.sh
[hadoop@hadoop01 conf]$ vim spark-env.sh
export MASTER=spark://172.16.100.26:7077
export SPARK_MASTER_IP=172.16.100.26
- 配置spark的環(huán)境變量
[hadoop@hadoop01 conf]$ vim ~/.bashrc
export JAVA_HOME=/usr/local/java
export HADOOP_HOME=/home/hadoop/hadoop-2.10.1
export SCALA_HOME=/home/hadoop/scala-2.12.2
export SPARK_HOME=/home/hadoop/spark-3.0.3-bin-hadoop2.7
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$SCALA_HOME/bin:$SPARK_HOME/bin
- 使環(huán)境變量生效
[hadoop@hadoop01 conf]$ source ~/.bashrc
- 將hadoop01節(jié)點(diǎn)的spark軟件拷貝到hadoop02鸟廓、hadoop03節(jié)點(diǎn)的根目錄
[hadoop@hadoop01 ~]$ scp -r spark-3.0.3-bin-hadoop2.7 hadoop@hadoop02:~
[hadoop@hadoop01 ~]$ scp -r spark-3.0.3-bin-hadoop2.7 hadoop@hadoop03:~
- 修改hadoop02、hadoop03節(jié)點(diǎn)的環(huán)境變量襟己，操作命令按照5引谜、6的方法
- 啟動(dòng)spark集群
[hadoop@hadoop01 sbin]$ ./stop-all.sh
hadoop02: stopping org.apache.spark.deploy.worker.Worker
hadoop03: stopping org.apache.spark.deploy.worker.Worker
hadoop01: no org.apache.spark.deploy.worker.Worker to stop
stopping org.apache.spark.deploy.master.Master
[hadoop@hadoop01 sbin]$ ./start-all.sh
starting org.apache.spark.deploy.master.Master, logging to /home/hadoop/spark-3.0.3-bin-hadoop2.7/logs/spark-hadoop-org.apache.spark.deploy.master.Master-1-hadoop01.out
hadoop01: starting org.apache.spark.deploy.worker.Worker, logging to /home/hadoop/spark-3.0.3-bin-hadoop2.7/logs/spark-hadoop-org.apache.spark.deploy.worker.Worker-1-hadoop01.out
hadoop03: starting org.apache.spark.deploy.worker.Worker, logging to /home/hadoop/spark-3.0.3-bin-hadoop2.7/logs/spark-hadoop-org.apache.spark.deploy.worker.Worker-1-hadoop03.out
hadoop02: starting org.apache.spark.deploy.worker.Worker, logging to /home/hadoop/spark-3.0.3-bin-hadoop2.7/logs/spark-hadoop-org.apache.spark.deploy.worker.Worker-1-hadoop02.out
-
在網(wǎng)頁(yè)上通過(guò)ui界面驗(yàn)證啟動(dòng)結(jié)果，http://172.16.100.26:8080
image.png
四擎浴、我遇到的問(wèn)題
- 在安裝jdk時(shí)员咽，將環(huán)境變量配置在/etc/profile中，想要把jdk當(dāng)成全局變量去使用贮预，所以我在3.5步驟去配置當(dāng)前用戶(hù)的環(huán)境變量時(shí)贝室，沒(méi)有設(shè)置JAVA_HOME，于是啟動(dòng)報(bào)了如下錯(cuò)誤：
[hadoop@hadoop01 sbin]$ ./start-all.sh
starting org.apache.spark.deploy.master.Master, logging to /home/hadoop/spark-3.0.3-bin-hadoop2.7/logs/spark-hadoop-org.apache.spark.deploy.master.Master-1-hadoop01.out
hadoop03: starting org.apache.spark.deploy.worker.Worker, logging to /home/hadoop/spark-3.0.3-bin-hadoop2.7/logs/spark-hadoop-org.apache.spark.deploy.worker.Worker-1-hadoop03.out
hadoop02: starting org.apache.spark.deploy.worker.Worker, logging to /home/hadoop/spark-3.0.3-bin-hadoop2.7/logs/spark-hadoop-org.apache.spark.deploy.worker.Worker-1-hadoop02.out
hadoop01: starting org.apache.spark.deploy.worker.Worker, logging to /home/hadoop/spark-3.0.3-bin-hadoop2.7/logs/spark-hadoop-org.apache.spark.deploy.worker.Worker-1-hadoop01.out
hadoop01: failed to launch: nice -n 0 /home/hadoop/spark-3.0.3-bin-hadoop2.7/bin/spark-class org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://hadoop01:7077
hadoop01: JAVA_HOME is not set
hadoop01: full log in /home/hadoop/spark-3.0.3-bin-hadoop2.7/logs/spark-hadoop-org.apache.spark.deploy.worker.Worker-1-hadoop01.out
很明顯仿吞，是由于讀取不到j(luò)ava的環(huán)境變量滑频，于是我在當(dāng)前目錄的.bashrc文件中配置了，成功啟動(dòng)唤冈，想不通為什么會(huì)這樣峡迷，全局變量中設(shè)置的環(huán)境變量怎么就不生效吶？姑且把它當(dāng)成spark的遺留小問(wèn)題吧，等我想起來(lái)再來(lái)探尋答案；娓恪Ｍ堋！
