Hadoop的安裝部署
下載的安裝包最好放到opt去解壓安裝，因?yàn)楹竺鎑ata有默認(rèn)在opt創(chuàng)建胳嘲，如果沒有在opt，要把opt給bxwl授權(quán)，授權(quán)都需要切換到root用戶? chmod 777 -R /opt ，否則sbin/start-dfs.sh 會(huì)報(bào)錯(cuò)Cannot set priority of datanode process XX肃弟；報(bào)錯(cuò)詳情可格式化查看hdfs namenode -format
[bxwl@snode028 bin]$ vim /etc/profile.d/bxwl.sh
export JAVA_HOME=/opt/jdk1.8.0_291
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export HADOOP_HOME=/opt/hadoop-3.3.2
#export HADOOP_CONF_DIR=/opt/hadoop-3.3.2/etc
export PATH=${JAVA_HOME}/bin:${HADOOP_HOME}/bin:$PATH
2零蓉、配置集群服務(wù)器節(jié)點(diǎn)workers
[bxwl@snode028 ~]$ cd /opt/hadoop-3.3.2/etc/hadoop
[bxwl@snode028 hadoop]$ vim workers
snode028
snode029
snode030
3笤受、配置core-site.xml
指定NameNode的地址、指定Hadoop數(shù)據(jù)的存儲(chǔ)目錄敌蜂、配置HDFS網(wǎng)頁登錄使用的靜態(tài)用戶為bxwl（不能是root）
[bxwl@snode028 hadoop]$ vim core-site.xml
... <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://snode028:8020</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/opt/hadoop-3.3.2/data</value> </property> <property> <name>hadoop.http.staticuser.user</name> <value>bxwl</value> </property> </configuration>
4箩兽、配置hdfs-site.xml
指定NameNode的 http地址、指定secondary NameNode的http地址章喉、在webhdfs后臺(tái)系統(tǒng)能查看文件內(nèi)容
[bxwl@snode028 hadoop]$ vim hdfs-site.xml
... <configuration> <property> <name>dfs.namenode.http-address</name> <value>snode028:9870</value> </property> <property> <name>dfs.namenode.secondary.http-address</name> <value>snode029:9868</value> </property> <property> <name>dfs.webhdfs.enabled</name> <value>true</value> </property> </configuration>
指定MapReduce程序運(yùn)行在Yarn上、Job歷史服務(wù)器地址秸脱、Job歷史服務(wù)器Web端地址
[bxwl@snode028 hadoop]$ vim mapred-site.xml
... <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>snode028:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>snode028:19888</value> </property> </configuration>
6落包、配置yarn-site.xml
指定MR走shuffle、指定ResourceManager的地址摊唇、環(huán)境變量的繼承咐蝇、開啟日志聚集功能、設(shè)置日志聚集服務(wù)器地址巷查、設(shè)置日志保留時(shí)間為7天
[bxwl@snode028 hadoop]$ vim yarn-site.xml
... yarn.nodemanager.aux-services mapreduce_shuffle yarn.resourcemanager.hostname snode030 yarn.nodemanager.env-whitelist JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME yarn.log-aggregation-enable true yarn.log.server.url http://snode028:19888/jobhistory/logs yarn.log-aggregation.retain-seconds 604800
7有序、格式化 NameNode (第一次啟動(dòng)時(shí)需要)
[bxwl@snode028 hadoop-3.3.2]$ hdfs namenode -format
WARNING: /opt/hadoop-3.3.2/logs does not exist. Creating.
2022-04-15 18:44:40,635 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:? host = snode028/192.168.100.28
STARTUP_MSG:? args = [-format]
STARTUP_MSG:? version = 3.3.2
STARTUP_MSG:? ...
[bxwl@snode028 hadoop-3.3.2]$ sbin/start-dfs.sh
Starting namenodes on [snode028]
Starting datanodes
snode030: WARNING: /opt/hadoop-3.3.2/logs does not exist. Creating.
snode029: WARNING: /opt/hadoop-3.3.2/logs does not exist. Creating.
Starting secondary namenodes [snode029]
瀏覽器直接訪問：http://192.168.100.28:9870
9岛请、啟動(dòng)Yarn
// Yarn配置在snode030旭寿，所以要到snode030上啟動(dòng)
[bxwl@snode028 hadoop-3.3.2]$ ssh snode030
[bxwl@snode030 ~]$ cd /opt/hadoop-3.3.2
[bxwl@snode030 hadoop-3.3.2]$ sbin/start-yarn.sh
Starting resourcemanager
Starting nodemanagers
[bxwl@snode030 hadoop-3.3.2]$ ~/bin/jpsall
---------- snode028 jps ------------
6672 DataNode
6521 NameNode
7003 NodeManager
6029 QuorumPeerMain
7101 Jps
---------- snode029 jps ------------
6146 SecondaryNameNode
6306 NodeManager
6036 DataNode
5750 QuorumPeerMain
6406 Jps
---------- snode030 jps ------------
6195 NodeManager
6070 ResourceManager
5595 QuorumPeerMain
5837 DataNode
6527 Jps
[bxwl@snode030 hadoop-3.3.2]$
瀏覽器直接訪問：http://192.168.100.30:8088
10、編寫hadoop 啟動(dòng)關(guān)閉的腳本
[bxwl@snode028 bin]$ vim hadoop.sh
#!/bin/bash
case $1 in
"start"){
? echo ----------HDFS 啟動(dòng)------------
? ssh snode028 "/opt/hadoop-3.3.2/sbin/start-dfs.sh"
? echo ---------- Yarn 啟動(dòng)------------
? ssh snode030 "/opt/hadoop-3.3.2/sbin/start-yarn.sh"
? echo ---------- Job歷史服務(wù)器 啟動(dòng)------------
? ssh snode028 "/opt/hadoop-3.3.2/bin/mapred --daemon start historyserver"
};;
"stop"){
? echo ---------- Job歷史服務(wù)器 關(guān)閉------------
? ssh snode028 "/opt/hadoop-3.3.2/bin/mapred --daemon stop historyserver"
? echo ---------- Yarn $i 關(guān)閉------------
? ssh snode030 "/opt/hadoop-3.3.2/sbin/stop-yarn.sh"
? echo ----------HDFS $i 關(guān)閉------------
? ssh snode028 "/opt/hadoop-3.3.2/sbin/stop-dfs.sh"
};;
esac
// 文件權(quán)限
[bxwl@snode028 bin]$ chmod +x hadoop.sh
// 關(guān)閉
[bxwl@snode028 bin]$ hadoop.sh stop
---------- Job歷史服務(wù)器 關(guān)閉------------
---------- Yarn 關(guān)閉------------
Stopping nodemanagers
Stopping resourcemanager
----------HDFS 關(guān)閉------------
Stopping namenodes on [snode028]
Stopping datanodes
Stopping secondary namenodes [snode029]
// 啟動(dòng)
[bxwl@snode028 bin]$ hadoop.sh start
----------HDFS 啟動(dòng)------------
Starting namenodes on [snode028]
Starting datanodes
Starting secondary namenodes [snode029]
---------- Yarn 啟動(dòng)------------
Starting resourcemanager
Starting nodemanagers
---------- Job歷史服務(wù)器 啟動(dòng)------------
