Hadoop部署的三種模式：
- 單機(jī)模式（standalone）
一般不建議安裝悼凑，網(wǎng)絡(luò)上很少這方面資料
單機(jī)模式是Hadoop的默認(rèn)模式嘁酿。當(dāng)首次解壓Hadoop的源碼包時(shí)徙硅，Hadoop無法了解硬件安裝環(huán)境即碗，便保守地選擇了最小配置。
在這種默認(rèn)模式下所有3個(gè)XML文件均為空涝桅。當(dāng)配置文件為空時(shí)拜姿，Hadoop會(huì)完全運(yùn)行在本地。
因?yàn)椴恍枰c其他節(jié)點(diǎn)交互冯遂，單機(jī)模式就不使用HDFS蕊肥，也不加載任何Hadoop的守護(hù)進(jìn)程。
該模式主要用于開發(fā)調(diào)試MapReduce程序的應(yīng)用邏輯蛤肌。
-
偽分布模式（Pseudo-Distributed Mode）
偽分布模式在“單節(jié)點(diǎn)集群”上運(yùn)行Hadoop壁却，其中所有的守護(hù)進(jìn)程都運(yùn)行在同一臺(tái)機(jī)器上。 該模式在單機(jī)模式之上增加了代碼調(diào)試功能裸准，允許你檢查內(nèi)存使用情況展东，HDFS輸入輸出，以及其他的守護(hù)進(jìn)程交互炒俱。 比如namenode盐肃，datanode，secondarynamenode权悟，jobtracer砸王，tasktracer這5個(gè)進(jìn)程，都能在集群上看到峦阁。 -
全分布模式（Fully Distributed Mode）
Hadoop守護(hù)進(jìn)程運(yùn)行在一個(gè)集群上谦铃。 意思是說master上看到namenode,jobtracer，secondarynamenode可以安裝在master節(jié)點(diǎn)榔昔，也可以單獨(dú)安裝驹闰。slave節(jié)點(diǎn)能看到datanode和tasktracer
Hadoop偽分布模式部署
環(huán)境要求java、ssh
-
添加hadoop用戶
[root@hadoop-01 ~]# useradd hadoop [root@hadoop-01 ~]# vi /etc/sudoers # 找到root ALL=(ALL) ALL撒会，添加 hadoop ALL=(ALL) NOPASSWD:ALL -
上傳并解壓
[root@hadoop-01 software]# rz #上傳hadoop-2.8.1.tar.gz [root@hadoop-01 software]# tar -xzvf hadoop-2.8.1.tar.gz -
軟連接
[root@hadoop-01 software]# ln -s /opt/software/hadoop-2.8.1 /opt/software/hadoop -
設(shè)置環(huán)境變量
[root@hadoop-01 software]# vi /etc/profile export HADOOP_HOME=/opt/software/hadoop export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PROTOC_HOME/bin:$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH [root@hadoop-01 software]# source /etc/profile -
設(shè)置用戶用戶組
[root@hadoop-01 software]# chown -R hadoop:hadoop hadoop [root@hadoop-01 software]# chown -R hadoop:hadoop hadoop/* [root@hadoop-01 software]# chown -R hadoop:hadoop hadoop-2.8.1 [root@hadoop-01 software]# cd hadoop [root@hadoop-01 hadoop]# rm -f *.txt -
切換用戶hadoop
[root@hadoop-01 software]# su - hadoop [root@hadoop-01 hadoop]# ll total 32 drwxrwxr-x. 2 hadoop hadoop 4096 Jun 2 14:24 bin drwxrwxr-x. 3 hadoop hadoop 4096 Jun 2 14:24 etc drwxrwxr-x. 2 hadoop hadoop 4096 Jun 2 14:24 include drwxrwxr-x. 3 hadoop hadoop 4096 Jun 2 14:24 lib drwxrwxr-x. 2 hadoop hadoop 4096 Aug 20 13:59 libexec drwxr-xr-x. 2 hadoop hadoop 4096 Aug 20 13:59 logs drwxrwxr-x. 2 hadoop hadoop 4096 Jun 2 14:24 sbin drwxrwxr-x. 4 hadoop hadoop 4096 Jun 2 14:24 share # bin: 可執(zhí)行文件 # etc: 配置文件 # sbin: shell腳本嘹朗，啟動(dòng)關(guān)閉hdfs,yarn等 -
配置文件
[hadoop@hadoop-01 ~]# cd /opt/software/hadoop [hadoop@hadoop-01 hadoop]# vi etc/hadoop/core-site.xml <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://192.168.137.130:9000</value> # 配置自己機(jī)器的IP </property> </configuration> [hadoop@hadoop-01 hadoop]# vi etc/hadoop/hdfs-site.xml <configuration> <property> <name>dfs.replication</name> <value>1</value> </property> </configuration> -
配置hadoop用戶的ssh信任關(guān)系
# 公鑰/密鑰 配置無密碼登錄 [hadoop@hadoop-01 ~]# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa [hadoop@hadoop-01 ~]# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys [hadoop@hadoop-01 ~]# chmod 0600 ~/.ssh/authorized_keys # 查看日期，看是否配置成功 [hadoop@hadoop-01 ~]# ssh hadoop-01 date The authenticity of host 'hadoop-01 (192.168.137.130)' can't be established. RSA key fingerprint is 09:f6:4a:f1:a0:bd:79:fd:34:e7:75:94:0b:3c:83:5a. Are you sure you want to continue connecting (yes/no)? yes # 第一次回車輸入yes Warning: Permanently added 'hadoop-01,192.168.137.130' (RSA) to the list of known hosts. Sun Aug 20 14:22:28 CST 2017 [hadoop@hadoop-01 ~]# ssh hadoop-01 date #不需要回車輸入yes,即OK Sun Aug 20 14:22:29 CST 2017 [hadoop@hadoop-01 ~]# ssh localhost date The authenticity of host 'hadoop-01 (192.168.137.130)' can't be established. RSA key fingerprint is 09:f6:4a:f1:a0:bd:79:fd:34:e7:75:94:0b:3c:83:5a. Are you sure you want to continue connecting (yes/no)? yes # 第一次回車輸入yes Warning: Permanently added 'hadoop-01,192.168.137.130' (RSA) to the list of known hosts. Sun Aug 20 14:22:28 CST 2017 [hadoop@hadoop-01 ~]# ssh localhost date #不需要回車輸入yes,即OK Sun Aug 20 14:22:29 CST 2017 -
格式化和啟動(dòng)
[hadoop@hadoop-01 hadoop]# bin/hdfs namenode -format [hadoop@hadoop-01 hadoop]# sbin/start-dfs.sh ERROR: hadoop-01: Error: JAVA_HOME is not set and could not be found. localhost: Error: JAVA_HOME is not set and could not be found. 解決方法:添加環(huán)境變量 [hadoop@hadoop-01 hadoop]# vi etc/hadoop/hadoop-env.sh # 將export JAVA_HOME=${JAVA_HOME}改為 export JAVA_HOME=/usr/java/jdk1.8.0_45 [hadoop@hadoop-01 hadoop]# sbin/start-dfs.sh ERROR: mkdir: cannot create directory `/opt/software/hadoop-2.8.1/logs': Permission denied 解決方法: [hadoop@hadoop-01 hadoop]# exit [root@hadoop-01 hadoop]# cd ../ [root@hadoop-01 software]# chown -R hadoop:hadoop hadoop-2.8.1 [root@hadoop-01 software]# su - hadoop [root@hadoop-01 ~]# cd /opt/software/hadoop # 繼續(xù)啟動(dòng) [hadoop@hadoop-01 hadoop]# sbin/start-dfs.sh -
檢查是否成功
[hadoop@hadoop-01 hadoop]# jps 19536 DataNode 19440 NameNode 19876 Jps 19740 SecondaryNameNode
訪問： http://192.168.137.130:50070
-
修改dfs啟動(dòng)的進(jìn)程诵肛，以hadoop-01啟動(dòng)
啟動(dòng)的三個(gè)進(jìn)程： namenode: hadoop-01 bin/hdfs getconf -namenodes datanode: localhost datanodes (using default slaves file) etc/hadoop/slaves secondarynamenode: 0.0.0.0 [hadoop@hadoop-01 ~]# cd /opt/software/hadoop [hadoop@hadoop-01 hadoop]# echo "hadoop-01" > ./etc/hadoop/slaves [hadoop@hadoop-01 hadoop]# cat ./etc/hadoop/slaves hadoop-01 [hadoop@hadoop-01 hadoop]# vi ./etc/hadoop/hdfs-site.xml <property> <name>dfs.namenode.secondary.http-address</name> <value>hadoop-01:50090</value> </property> <property> <name>dfs.namenode.secondary.https-address</name> <value>hadoop-01:50091</value> </property> # 重啟 [hadoop@hadoop-01 hadoop]# sbin/stop-dfs.sh [hadoop@hadoop-01 hadoop]# sbin/start-dfs.sh
