環(huán)境準(zhǔn)備
| 主機(jī)名稱(chēng) | IP地址 |
|---|---|
| hadoop-master | 192.168.200.100 |
echo "192.168.200.100 hadoop-master">>/etc/hosts
其他信息：
[root@hadoop-master ~]# uname -r
2.6.32-358.el6.x86_64
[root@hadoop-master ~]# uname -m
x86_64
[root@hadoop-master ~]# cat /etc/redhat-release
CentOS release 6.4 (Final)
[root@hadoop-master ~]# getenforce
Disabled
一泽铛、安裝JDK
請(qǐng)參考：Linux下快速安裝JDK
二聚凹、安裝hadoop
- 創(chuàng)建用戶(hù)
useradd hadoop && echo 123456 | passwd --stdin hadoop
- 下載
wget --no-check-certificate https://mirrors.cnnic.cn/apache/hadoop/common/hadoop-2.9.0/hadoop-2.9.0.tar.gz
- 解壓安裝
mkdir /application && tar zxf /usr/local/src/hadoop-2.9.0.tar.gz -C /application/&&ln -s /application/hadoop-2.9.0 /application/hadoop&&chown -R hadoop:hadoop /application/hadoop*&&ll /application/
- 配置Hadoop環(huán)境變量
##################Hadoop環(huán)境變量配置#############
export HADOOP_HOME=/application/hadoop
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
- 測(cè)試
hadoop version
三逼龟、配置Hadoop
3.1 單機(jī)模式（standalone）
- 不需要任何配置臂外，不需要啟用獨(dú)立的hadoop進(jìn)程伴澄，常用語(yǔ)開(kāi)發(fā)和調(diào)試。
測(cè)試
hdfs dfs -ls /
3.2 偽分布模式
- 偽分布模式是在一臺(tái)機(jī)器上模擬完全分布式账劲，只有一個(gè)副本
修改配置文件
- vim /application/hadoop/etc/hadoop/hadoop-env.sh
export JAVA_HOME=/usr/local/jdk
- vim /application/hadoop/etc/hadoop/core-site.xml
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/application/hadoop/tmp</value>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop-master:9000</value>
</property>
</configuration>
- vim /application/hadoop/etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/application/hadoop/tmp/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/application/hadoop/tmp/dfs/data</value>
</property>
</configuration>
- cp /application/hadoop/etc/hadoop/mapred-site.xml.template /application/hadoop/etc/hadoop/mapred-site.xml
- vim /application/hadoop/etc/hadoop/mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
- vim /application/hadoop/etc/hadoop/yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
配置SSH免密登陸
- 配置本機(jī)免密碼
檢查是否安裝了ssh相關(guān)軟件包（openssh-server openssh-clients openssh）
yum list installed | grep ssh
檢查是否啟動(dòng)了sshd進(jìn)程
ps -Af | grep sshd
su - hadoop
在客戶(hù)端生成密鑰對(duì)
ssh-keygen -t dsa
將公鑰追加到服務(wù)端的認(rèn)證庫(kù)（本機(jī)）
cat .ssh/id_dsa.pub >> .ssh/authorized_keys
檢查authorized_keys的權(quán)限是否為644戳护，不是的話(huà)，需要修改為644
- 測(cè)試(如果第一次需要密碼瀑焦，第二次不需要即可)
ssh hadoop-master
格式化文件系統(tǒng)
hdfs namenode -format
成功字樣：has been successfully formatted
Existing with status 0
啟動(dòng)
[hadoop@hadoop-master ~]start-dfs.sh
[hadoop@hadoop-master ~]$ jps
8880 Jps
8593 DataNode
8769 SecondaryNameNode
8454 NameNode
[hadoop@hadoop-master ~]$ start-yarn.sh
[hadoop@hadoop-master ~]$ jps
8593 DataNode
9041 NodeManager
8769 SecondaryNameNode
8454 NameNode
9338 Jps
8939 ResourceManager
測(cè)試
瀏覽器訪問(wèn)：
http://192.168.200.100:50070
測(cè)試往HDFS寫(xiě)文件：
echo "123456" >>/home/hadoop/1.txt
bin/hdfs dfs -appendToFile /home/hadoop/1.txt hdfs://hadoop-master:9000/1.txt
bin/hdfs dfs -ls hdfs://hadoop-master:9000/
四腌且、三種模式共存
配置hadoop，讓其三種模式共存榛瓮，方便切換
- 分別復(fù)制${HADOOP_HOME}/etc/hadoop文件夾3次铺董，分別更名為local、pseudo禀晓、full
- 刪除${HADOOP_HOME}/etc/hadoop文件夾
- 創(chuàng)建對(duì)應(yīng)軟連接ln -s
