前提：搭建Hadoop-HA + ZooKeeper環(huán)境
| node01 | node02 | node03 | node04 |
|---|---|---|---|
| NameNode01 | NameNode02 | NameNode03 | |
| DataNode01 | DataNode02 | DataNode03 | |
| JournalNode01 | JournalNode02 | JournalNode03 | |
| ZooKeeper01 | ZooKeeper02 | ZooKeeper03 | |
| ZooKeeperFailoverController01 | ZooKeeperFailoverController02 | ZooKeeperFailoverController03 | |
| ResourceManager01 | ResourceManager02 | ||
| NodeManager01 | NodeManager02 | NodeManager03 |
- 配置node01、node02、node03腕侄、node04上的Yarn
在node01上修改
/opt/hadoop/hadoop-3.1.1/etc/hadoop/yarn-site.xml：
vim /opt/hadoop/hadoop-3.1.1/etc/hadoop/yarn-site.xml
添加：<configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.nodemanager.env-whitelist</name> <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value> </property> <property> <name>yarn.resourcemanager.ha.enabled</name> <value>true</value> </property> <property> <name>yarn.resourcemanager.cluster-id</name> <value>yarnResourceManagerHACluster</value> </property> <property> <name>yarn.resourcemanager.ha.rm-ids</name> <value>RM01,RM02</value> </property> <property> <name>yarn.resourcemanager.hostname.RM01</name> <value>node03</value> </property> <property> <name>yarn.resourcemanager.hostname.RM02</name> <value>node04</value> </property> <property> <name>yarn.resourcemanager.webapp.address.RM01</name> <value>node03:8088</value> </property> <property> <name>yarn.resourcemanager.webapp.address.RM02</name> <value>node04:8088</value> </property> <property> <name>yarn.resourcemanager.zk-address</name> <value>node02:2181,node03:2181,node04:2181</value> </property> </configuration>在node01上修改
/opt/hadoop/hadoop-3.1.1/etc/hadoop/mapred-site.xml：
vim /opt/hadoop/hadoop-3.1.1/etc/hadoop/mapred-site.xml
添加：<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.application.classpath</name> <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value> </property> </configuration>將node01上的
/opt/hadoop/hadoop-3.1.1/etc/hadoop/yarn-site.xml殊校、/opt/hadoop/hadoop-3.1.1/etc/hadoop/mapred-site.xml拷貝到node02、node03晶默、node04：
scp /opt/hadoop/hadoop-3.1.1/etc/hadoop/yarn-site.xml /opt/hadoop/hadoop-3.1.1/etc/hadoop/mapred-site.xml node02:/opt/hadoop/hadoop-3.1.1/etc/hadoop/ && scp /opt/hadoop/hadoop-3.1.1/etc/hadoop/yarn-site.xml /opt/hadoop/hadoop-3.1.1/etc/hadoop/mapred-site.xml node03:/opt/hadoop/hadoop-3.1.1/etc/hadoop/ && scp /opt/hadoop/hadoop-3.1.1/etc/hadoop/yarn-site.xml /opt/hadoop/hadoop-3.1.1/etc/hadoop/mapred-site.xml node04:/opt/hadoop/hadoop-3.1.1/etc/hadoop/
- 配置node01谨娜、node02、node03磺陡、node04上的環(huán)境變量
在node01上修改
/etc/profile：
vim /etc/profile
添加：export HADOOP_MAPRED_HOME=/opt/hadoop/hadoop-3.1.1 export HADOOP_COMMON_HOME=/opt/hadoop/hadoop-3.1.1 export HADOOP_HDFS_HOME=/opt/hadoop/hadoop-3.1.1 export HADOOP_CONF_DIR=/opt/hadoop/hadoop-3.1.1/etc/hadoop export HADOOP_YARN_HOME=/opt/hadoop/hadoop-3.1.1在node02上修改
/etc/profile
vim /etc/profile
添加：export YARN_NODEMANAGER_USER=root export HADOOP_MAPRED_HOME=/opt/hadoop/hadoop-3.1.1 export HADOOP_COMMON_HOME=/opt/hadoop/hadoop-3.1.1 export HADOOP_HDFS_HOME=/opt/hadoop/hadoop-3.1.1 export HADOOP_CONF_DIR=/opt/hadoop/hadoop-3.1.1/etc/hadoop export HADOOP_YARN_HOME=/opt/hadoop/hadoop-3.1.1在node03上修改
/etc/profile
vim /etc/profile
添加：export YARN_NODEMANAGER_USER=root export YARN_RESOURCEMANAGER_USER=root export HADOOP_MAPRED_HOME=/opt/hadoop/hadoop-3.1.1 export HADOOP_COMMON_HOME=/opt/hadoop/hadoop-3.1.1 export HADOOP_HDFS_HOME=/opt/hadoop/hadoop-3.1.1 export HADOOP_CONF_DIR=/opt/hadoop/hadoop-3.1.1/etc/hadoop export HADOOP_YARN_HOME=/opt/hadoop/hadoop-3.1.1在node04上修改
/etc/profile
vim /etc/profile
添加：export YARN_NODEMANAGER_USER=root export YARN_RESOURCEMANAGER_USER=root export HADOOP_MAPRED_HOME=/opt/hadoop/hadoop-3.1.1 export HADOOP_COMMON_HOME=/opt/hadoop/hadoop-3.1.1 export HADOOP_HDFS_HOME=/opt/hadoop/hadoop-3.1.1 export HADOOP_CONF_DIR=/opt/hadoop/hadoop-3.1.1/etc/hadoop export HADOOP_YARN_HOME=/opt/hadoop/hadoop-3.1.1在node01趴梢、node02、node03币他、node04上運(yùn)行：
. /etc/profile
- 啟動(dòng)ZooKeeper
在node02坞靶、node03、node04上運(yùn)行：
zkServer.sh start
- 啟動(dòng)ResourceManager和NodeManager
在node3/node4上運(yùn)行：
start-yarn.sh
- 啟動(dòng)Hadoop
在node01/node02/node03/node04上運(yùn)行：
start-dfs.sh
- 查看進(jìn)程
在node01蝴悉、node02彰阴、node03、node04上運(yùn)行：
jps
- 訪問(wèn)網(wǎng)頁(yè)
NameNode01：http://192.168.163.191:9870
NameNode02：http://192.168.163.192:9870
NameNode03：http://192.168.163.193:9870
DataNode01：http://192.168.163.192:9864
DataNode02：http://192.168.163.193:9864
DataNode03：http://192.168.163.194:9864
ResourceManager01：http://192.168.163.193:8088
ResourceManager02：http://192.168.163.194:8088
