環(huán)境準(zhǔn)備
| 主機(jī)名稱(chēng) | IP地址 |
|---|---|
| s101 | 192.168.200.101 |
| s102 | 192.168.200.102 |
| s103 | 192.168.200.103 |
| s104 | 192.168.200.104 |
- 操作系統(tǒng)信息
[root@hadoop-master ~]# uname -r
2.6.32-358.el6.x86_64
[root@hadoop-master ~]# uname -m
x86_64
[root@hadoop-master ~]# cat /etc/redhat-release
CentOS release 6.4 (Final)
[root@hadoop-master ~]# getenforce
Disabled
- 所有主機(jī)/etc/hosts文件添加[s101~s104]
192.168.200.101 s101
192.168.200.102 s102
192.168.200.103 s103
192.168.200.104 s104
配置root免密碼
登陸到s101服務(wù)器滤灯，并切換到root用戶(hù)嗜桌，su - root
- 在s101主機(jī)上生成密鑰對(duì)
[root@s101 ~]# ssh-keygen
一直回車(chē)
- 配置本機(jī)登陸面密碼
[root@s101 ~]#cd /root/.ssh/
[root@s101 .ssh]#cp id_rsa.pub authorized_keys
測(cè)試本機(jī)是否免密碼成功蒿偎，如果直登陸上，則說(shuō)明配置成功
[root@s101 ~]ssh s101
- 將s101的公鑰復(fù)制到到s102~104主機(jī)上
[root@s101 ~]ssh-copy-id -i /root/.ssh/id_rsa.pub root@s102
輸入yes晤郑，然后輸入s102的root密碼负蚊，同理s103和s104
[root@s101 ~]ssh-copy-id -i /root/.ssh/id_rsa.pub root@s103
[root@s101 ~]ssh-copy-id -i /root/.ssh/id_rsa.pub root@s104
分別測(cè)試s102~s104是否免密碼成功，如果直登陸上浆兰，則說(shuō)明配置成功
[root@s101 ~]ssh s102
創(chuàng)建hadoop用戶(hù)磕仅，配置hadoop用戶(hù)免密碼登陸
- 創(chuàng)建用戶(hù)[s101~s104]
useradd hadoop && echo 123456 | passwd --stdin hadoop
- 配置免密碼登陸【原理同root配置，此處簡(jiǎn)寫(xiě)】
在s101上生成密鑰對(duì)[s101]
[hadoop@s101 ~]$ ssh-keygen
hadoop@s101 ~]$ cd /home/hadoop/.ssh/
[hadoop@s101 .ssh]$ cp id_rsa.pub authorized_key
[hadoop@s101 .ssh]$ ssh s101
[hadoop@s101 .ssh]$ ssh-copy-id -i /home/hadoop/.ssh/id_rsa.pub hadoop@s102
[hadoop@s101 .ssh]$ ssh s102
[hadoop@s101 .ssh]$ ssh-copy-id -i /home/hadoop/.ssh/id_rsa.pub hadoop@s103
[hadoop@s101 .ssh]$ ssh s103
[hadoop@s101 .ssh]$ ssh-copy-id -i /home/hadoop/.ssh/id_rsa.pub hadoop@s104
[hadoop@s101 .ssh]$ ssh s104
給Hadoop用戶(hù)安裝配置JDK
- 解壓安裝
[hadoop@s101 app]$ tar zxf /app/jdk-8u144-linux-x64.tar.gz -C /app/
[hadoop@s101 app]$ ln -s /app/jdk1.8.0_144 /app/jdk
- 配置hadoop用戶(hù)的java環(huán)境變量
[hadoop@s101 app]$echo -e '##################JAVA環(huán)境變量配置#############\nexport JAVA_HOME=/app/jdk\nexport JRE_HOME=$JAVA_HOME/jre\nexport CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH\nexport PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH' >> ~/.bash_profile && source ~/.bash_profile&&tail -5 ~/.bash_profile
- 檢查s101是否配置成功
[hadoop@s101 app]$ java -version
安裝Hadoop
- 準(zhǔn)備Hadoop安裝目錄
[root@s101 ~]# mkdir /app
[root@s101 ~]# chown -R hadoop:hadoop /app
[root@s101 ~]# ll -d /app/
drwxr-xr-x 2 hadoop hadoop 4096 Nov 5 15:42 /app/
- 下載安裝簸呈，首先在s101上安裝榕订，然后發(fā)送到其他機(jī)器
[root@s101 home]# su - hadoop
[hadoop@s101 ~]$ cd /app/
[hadoop@s101 app]$ wget --no-check-certificate https://mirrors.cnnic.cn/apache/hadoop/common/hadoop-2.9.0/hadoop-2.9.0.tar.gz
[hadoop@s101 app]$ tar zxf /app/hadoop-2.9.0.tar.gz -C /app/
[hadoop@s101 app]$ ln -s /app/hadoop-2.9.0 /app/hadoop
- 配置hadoop環(huán)境變量
echo -e '######Hadoop環(huán)境變量配置######\nexport HADOOP_HOME=/app/hadoop\nexport PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH' >> /home/hadoop/.bash_profile && source /home/hadoop/.bash_profile&&tail -3 /home/hadoop/.bash_profile
配置hadoop集群配置相關(guān)文件
- [hadoop@s101 ~]$ vim /app/hadoop/etc/hadoop/hadoop-env.sh
export JAVA_HOME=/app/jdk
- [hadoop@s101 hadoop]$ vim /app/hadoop/etc/hadoop/slaves
s102
s103
s104
- [hadoop@s101 ~]$ vim /app/hadoop/etc/hadoop/core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://s101:9000</value>
</property>
</configuration>
- [hadoop@s101 ~]$vim /app/hadoop/etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
</configuration>
- hadoop@s101 ~]$ vim /app/hadoop/etc/hadoop/mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
- [hadoop@s101 hadoop]$ vim /app/hadoop/etc/hadoop/yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>s101</value>
</property>
</configuration>
分發(fā)文件給s102-s104
- hadoop和jdk
[hadoop@s101 /]$ rsync -lr app hadoop@s102:/
[hadoop@s101 /]$ rsync -lr app hadoop@s103:/
[hadoop@s101 /]$ rsync -lr app hadoop@s104:/
- hadoop環(huán)境變量
[hadoop@s101 /]$ cd /home/hadoop/
[hadoop@s101 ~]$ rsync -lr .bash_profile hadoop@s102:/home/hadoop/
[hadoop@s101 ~]$ rsync -lr .bash_profile hadoop@s103:/home/hadoop/
[hadoop@s101 ~]$ rsync -lr .bash_profile hadoop@s104:/home/hadoop/
- 檢查s102-s104服務(wù)器上對(duì)應(yīng)文件是否分發(fā)成功
[hadoop@s101 /home/hadoop]$ll /app/
[hadoop@s101 /home/hadoop]$cat /home/hadoop/.bash_profile
啟動(dòng)hadoop集群
格式化文件系統(tǒng)
[hadoop@s101 /home/hadoop]$hdfs namenode -format
啟動(dòng)集群
執(zhí)行start-dfs.sh
[hadoop@s101 /home/hadoop]$start-dfs.sh
Starting namenodes on [s101]
s101: starting namenode, logging to /app/hadoop-2.9.0/logs/hadoop-hadoop-namenode-s101.out
s103: starting datanode, logging to /app/hadoop-2.9.0/logs/hadoop-hadoop-datanode-s103.out
s104: starting datanode, logging to /app/hadoop-2.9.0/logs/hadoop-hadoop-datanode-s104.out
s102: starting datanode, logging to /app/hadoop-2.9.0/logs/hadoop-hadoop-datanode-s102.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /app/hadoop-2.9.0/logs/hadoop-hadoop-secondarynamenode-s101.out
- 通過(guò)jps查看炫加，s101存在進(jìn)程N(yùn)ameNode和SecondaryNameNode
[hadoop@s101 /home/hadoop]$jps
24392 NameNode
24589 SecondaryNameNode
- s102~s104存在DataNode
[hadoop@s102 /home/hadoop]$jps
3969 DataNode
[hadoop@s103 /home/hadoop]$jps
3896 DataNode
[hadoop@s104 /home/hadoop]$jps
3936 DataNode
執(zhí)行start-yarn.sh
[hadoop@s101 /home/hadoop]$start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /app/hadoop-2.9.0/logs/yarn-hadoop-resourcemanager-s101.out
s104: starting nodemanager, logging to /app/hadoop-2.9.0/logs/yarn-hadoop-nodemanager-s104.out
s103: starting nodemanager, logging to /app/hadoop-2.9.0/logs/yarn-hadoop-nodemanager-s103.out
s102: starting nodemanager, logging to /app/hadoop-2.9.0/logs/yarn-hadoop-nodemanager-s102.out
- 通過(guò)jps查看哀托，s101存在進(jìn)程N(yùn)ameNode、SecondaryNameNode和ResourceManager（新增）
[hadoop@s101 /home/hadoop]$jps
24392 NameNode
24589 SecondaryNameNode
24797 ResourceManager
- s102~s104存在DataNode和NodeManager（新增）
[hadoop@s102 /home/hadoop]$jps
3969 DataNode
4105 NodeManager
[hadoop@s103 /home/hadoop]$jps
4032 NodeManager
3896 DataNode
[hadoop@s104 /home/hadoop]$jps
3936 DataNode
4072 NodeManager
- 訪問(wèn) http://192.168.200.101:50070/dfshealth.html#tab-overview
image.png
