設置固定IP地址及網(wǎng)關

• 設置IP

# vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
HWADDR=08:00:27:BD:9D:B5  #不用改
TYPE=Ethernet
UUID=53e4e4b6-9724-43ab-9da7-68792e611031 #不用改
ONBOOT=yes  #開機啟動
NM_CONTROLLED=yes
BOOTPROTO=static  #靜態(tài)IP
IPADDR=192.168.30.50  #IP地址
NETMASK=255.255.255.0 #子網(wǎng)掩碼

• 設置網(wǎng)關

# vi /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=Hadoop.Master
GATEWAY=192.168.30.1 #網(wǎng)關

• 設置DNS

vi /etc/resolv.conf
nameserver xxx.xxx.xxx.xxx #根據(jù)實際情況設置
nameserver 114.114.114.114 #可以設置多個

• 重啟網(wǎng)卡

service network restart

• 測試網(wǎng)絡

ping www.baidu.com #ping不通一般是DNS問題

• 設置主機名對應IP地址

vi /etc/hosts
#添加如下內(nèi)容
192.168.30.50 Hadoop.Master

添加Hadoop用戶

• 添加用戶組

groupadd hadoop

• 添加用戶并分配用戶組

useradd -g hadoop hadoop

• 修改用戶密碼

passwd hadoop

關閉服務

• 關閉防火墻

service iptables stop #關閉防火墻服務
chkconfig iptables off #關閉防火墻開機啟動
service ip6tables stop
chkconfig ip6tables off

• 關閉SELinux

vi /etc/sysconfig/selinux
#修改如下內(nèi)容
SELINUX=enforcing -> SELINUX=disabled
#再執(zhí)行如下命令
setenforce 0
getenforce

Required Software

• 查看ssh與rsync安裝狀態(tài)

rpm -qa|grep openssh
rpm -qa|grep rsync

• 安裝ssh與rsync

yum -y install ssh
yum -y install rsync

• 切換hadoop用戶

 su - hadoop

• Setup passphraseless ssh

#use dsa
$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
#use rsa
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
#將id_dsa.pub追加到授權的key中
#選擇 dsa 或 rsa 二選一
$ chmod 0600 ~/.ssh/authorized_keys
#權限的設置非常重要精绎，因為不安全的設置安全設置屉佳，會讓你不能使用RSA功能

• 測試ssh連接

ssh localhost
#如果不需要輸入密碼掺炭，則是成功

Java安裝與配置

• 下載地址

http://www.oracle.com/technetwork/java/javase/downloads/java-archive-downloads-javase7-521261.html
注：我這里使用的是：jdk-8u91-linux-x64.tar.gz

• 切換至root用戶

sudo su -

• 將壓縮包解壓至/usr/local/jdk1.8.0_91 目錄

tar zxvf /home/hadoop/jdk-8u91-linux-x64.tar.gz -C /usr/local/jdk1.8.0_91

• 設置環(huán)境變量

vi /etc/profile
#追加如下內(nèi)容
export JAVA_HOME=/usr/local/jdk1.8.0_91
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin

• 使環(huán)境變量生效

source /etc/profile
java -version

Hadoop安裝與配置

• 下載地址

http://hadoop.apache.org/releases.html
注:我下載的是hadoop-2.7.2.tar.gz

• 將壓縮包解壓至/usr/local目錄

tar zxvf /home/hadoop/hadoop-2.7.2.tar.gz -C /usr/local/

• 創(chuàng)建hadoop數(shù)據(jù)目錄

mkdir /usr/local/hadoop-2.7.2/tmp

• 將hadoop文件夾授權給hadoop用戶

chown -R hadoop:hadoop /usr/local/hadoop-2.7.2/

• 設置環(huán)境變量

vim /etc/profile
#追加如下內(nèi)容
export HADOOP_HOME=/usr/local/hadoop-2.7.2
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

• 使環(huán)境變量生效

source /etc/profile
#測試環(huán)境變量設置
hadoop version

配置HDFS

• 切換至Hadoop用戶

su - hadoop

• 修改hadoop-env.sh

cd /usr/local/hadoop-2.7.2/etc/hadoop/
vi hadoop-env.sh 
#追加如下內(nèi)容
export JAVA_HOME=/usr/local/jdk1.8.0_91

• 修改core-site.xml

vi core-site.xml 
#添加如下內(nèi)容
<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://hadoop:10080</value>
        </property>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/usr/local/hadoop-2.7.2/tmp/</value>
                <description>A base for other temporary directories.</description>
        </property>
</configuration>

• 修改hdfs-site.xml

vi hdfs-site.xml
#添加如下內(nèi)容
<configuration>
   <property>
        <name>dfs.replication</name>
        <value>1</value>
   </property>
   <property>
        <name>dfs.datanode.http.address</name>
        <value>0.0.0.0:10675</value>
   </property>
   <property>
        <name>dfs.namenode.http-address</name>
        <value>0.0.0.0:10670</value>
   </property>
</configuration>

• 格式化hdfs

hdfs namenode -format
#注：出現(xiàn)Exiting with status 0即為成功

• 啟動hdfs

start-dfs.sh
#停止命令
stop-dfs.sh
#注：輸出如下內(nèi)容
15/09/21 18:09:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [Hadoop.Master]
Hadoop.Master: starting namenode, logging to /usr/hadoop/logs/hadoop-hadoop-namenode-Hadoop.Master.out
Hadoop.Master: starting datanode, logging to /usr/hadoop/logs/hadoop-hadoop-datanode-Hadoop.Master.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
RSA key fingerprint is b5:96:b2:68:e6:63:1a:3c:7d:08:67:4b:ae:80:e2:e3.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (RSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /usr/hadoop/logs/hadoop-hadoop-secondarynamenode-Hadoop.Master.out
15/09/21 18:09:45 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicab

• 查看進程

jps
#注：輸出類似如下內(nèi)容
[hadoop@CDVM-213018144 hadoop]$ jps
1155 NameNode
1492 SecondaryNameNode
1307 DataNode
6142 Jps

• 使用web查看Hadoop運行狀態(tài)

http://10.213.18.144:10670/
#默認是50070

在HDFS上運行WordCount

• 創(chuàng)建HDFS用戶目錄

hdfs dfs -mkdir /user
hdfs dfs -mkdir /user/hadoop #根據(jù)自己的情況調(diào)整/user/<username>

• 復制輸入文件（要處理的文件）到HDFS上

hdfs dfs -put /usr/local/hadoop-2.7.2/etc/hadoop input

• 查看我們復制到HDFS上的文件

hdfs dfs -ls input

• 運行單詞檢索（grep）程序

hadoop jar /usr/local/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar grep input output 'dfs[a-z.]+'
#WordCount
#hadoop jar /usr/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar wordcount input output
#說明：output文件夾如已經(jīng)存在則需要刪除或指定其他文件夾。

• 查看運行結(jié)果

hdfs dfs -cat output/*

配置YARN

• 修改mapred-site.xml

cd /usr/local/hadoop-2.7.2/etc/hadoop/
cp mapred-site.xml.template mapred-site.xml
vim mapred-site.xml
#添加如下內(nèi)容
<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

• 修改yarn-site.xml

vi yarn-site.xml
#添加如下內(nèi)容
<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
</configuration>

• 啟動YARN

start-yarn.sh
#停止yarn stop-yarn.sh

• 查看當前java進程

[hadoop@CDVM-213018144 hadoop]$ jps
#輸出如下
1155 NameNode
1492 SecondaryNameNode
1307 DataNode
21979 Jps
21551 ResourceManager
21679 NodeManager

運行你的mapReduce程序
配置好如上配置再運行mapReduce程序時即是yarn中運行, 使用web

• 查看Yarn運行狀態(tài)

http://你的服務器ip地址:8088/

HDFS常用命令

• 創(chuàng)建HDFS文件夾

在根目錄創(chuàng)建input文件夾
hdfs dfs -mkdir -p /input
在用戶目錄創(chuàng)建input文件夾
說明：如果不指定“/目錄”卓研，則默認在用戶目錄創(chuàng)建文件夾
hdfs dfs -mkdir -p input
#hdfs dfs -mkdir -p /user/hadoop/input

• 查看HDFS文件夾

查看HDFS根文件夾
hdfs  dfs  -ls /
查看HDFS用戶目錄文件夾
hdfs  dfs  -ls
查看HDFS用戶目錄文件夾下input文件夾
hdfs  dfs  -ls input
#等同與 hdfs  dfs  -ls /user/hadoop/input
復制文件到HDFS
hdfs dfs -put /usr/hadoop/etc/hadoop input
刪除文件夾
hdfs  dfs  -rm -r input

參考資料

http://baike.xsoftlab.net/view/292.html
http://www.powerxing.com/install-hadoop/
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html
http://www.cnblogs.com/zx247549135/p/4381800.html
http://www.zhixing123.cn/ubuntu/40649.html
http://stackoverflow.com/questions/24119486/hadoop-namenode-recovery-from-metadata-backup
http://blog.csdn.net/amber_amber/article/details/46896719