hadoop 完全分布式HA高可用自動切換集群環(huán)境搭建

參考了很多博客，根據(jù)一些博客綜合之后自己搭建成功了邪铲，寫了這篇文章記錄一下過程，希望給各位朋友一些幫助

環(huán)境準備

自己準備虛擬機環(huán)境配置好虛擬機[vm搭建] (http://note.youdao.com/noteshare?id=1cd5582fbe7351bb04e45f9e740ecd6f)

linux系統(tǒng)：CentOS-7-x86_64-Minimal-1810
jdk版本：jdk8+
zookeeper: zookeeper-3.4.14
hadoop: hadoop-2.7.7

jdk-8u231
zookeeper-3.4.14
hadoop-2.7.7

卸載自帶jdk叹侄，使用jdk1.8+以上

rpm -qa | grep java
使用rpm進行卸載
rpm -e java-xxx
rpm -e --nodeps java-xxx   #強制卸載

一艺沼、分布式集群規(guī)劃

節(jié)點名稱	IP地址	NAMENODE(NN)	DATANODE(DN)	JJN	ZKFC	ZK
hadoop1	172.16.0.161	namenode 1	datanode1	journalnode	zkfc	zookeeper
hadoop2	172.16.0.162	namenode 2	datanode2	journalnode	zkfc	zookeeper
hadoop3	172.16.0.163		datanode3	journalnode		zookeeper

二、網(wǎng)絡IP規(guī)劃

2.1 修改主機名

以centos7為例修改主機名

vi /etc/hostname

在三臺機器上分別執(zhí)行
hostnamectl set-hostname hadoop1
hostnamectl set-hostname hadoop2
hostnamectl set-hostname hadoop3

補充：centos6修改主機名
vi /etc/sysconfig/network  
將里面原來的信息修改如下:
NETWORKING=yes
HOSTNAME=hadoop1

2.2 修改相對應的hosts

查看ip  ip addr

vi /etc/hosts 添加主機映射
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6


172.16.0.161 hadoop1   #由于我的虛擬機ip被占用所有改成了172.16.0.170
172.16.0.162 hadoop2
172.16.0.163 hadoop3

重啟系統(tǒng) reboot

重啟網(wǎng)路  service network start

三骚露、關(guān)閉防火墻

Centos7系統(tǒng)默認防火墻不是iptables,而是firewall,那就得使用以下方式關(guān)閉防火墻了蹬挤。

systemctl stop firewalld.service            #停止firewall
systemctl disable firewalld.service        #禁止firewall開機啟動

補充：Centos6關(guān)閉防火墻
service iptables status             #查看防火墻狀態(tài)
service iptables stop               #關(guān)閉防火墻，但是重啟后會恢復原來狀態(tài)
chkconfig iptables --list           #查看系統(tǒng)中防火墻的自動
chkconfig iptables off              #關(guān)閉防火墻自啟動
chkconfig iptables --list           #再次查看防火墻自啟動的情況棘幸，所有啟動狀態(tài)都變成額off

四焰扳、設(shè)置ssh免密登錄

關(guān)于ssh免密碼的設(shè)置，要求每兩臺主機之間設(shè)置免密碼误续，自己的主機與自己的主機之間也要求設(shè)置免密碼吨悍。 這項操作可以在root用戶下執(zhí)行，執(zhí)行完畢公鑰在/home/root/.ssh/id_rsa.pub

[root@hadoop1 ~]# ssh-keygen -t rsa
[root@hadoop1 ~]# ssh-copy-id hadoop1
[root@hadoop1 ~]# ssh-copy-id hadoop2
[root@hadoop1 ~]# ssh-copy-id hadoop3

免密登錄有時候沒有成功遇到permission錯誤蹋嵌，可能是ids_rsa權(quán)限給大了,降低下就可以了

問題：Permissions 0644 for '/root/.ssh/id_rsa' are too open.
It is required that your private key files are NOT accessible by others.
This private key will be ignored.
Load key "/root/.ssh/id_rsa": bad permissions
root@192.168.108.130's password:
Permission denied, please try again.
沒有改錯育瓜，我是用了命令 chmod 600 /root/.ssh/id_rsa ,而且我看到權(quán)限也降低了，但是再次執(zhí)行腳本還是出來上面這個錯誤栽烂，我輸入root密碼后就卡住了

修改本機 /root/.ssh/id_rsa
chmod 400 /root/.ssh/id_rsa

五躏仇、安裝jdk

安裝軟件全部安裝到/usr/local/hadoop目錄下，統(tǒng)一管理

在/usr/local下創(chuàng)建目錄 hadoop , 并賦予權(quán)限
mkdir hadoop
sudo chmod -R 777 /usr/local/hadoop
解壓jdk-8u231
tar -zxvf jdk-8u231-linux-x64.tar.gz
配置環(huán)境變量vi /etc/profile

export JAVA_HOME=/usr/local/hadoop/jdk1.8.0_211
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin

刷新配置腺办，使其生效
source /etc/profile

六焰手、安裝zookeeper

在/usr/local/hadoop/ 目錄下解壓zookeeper
tar -zxvf zookeeper-3.4.14.tar.gz
配置環(huán)境變量vi /etc/profile

export ZOOKEEPER_HOME=/usr/local/hadoop/zookeeper-3.4.14/
export PATH=$PATH:$ZOOKEEPER_HOME/bin

刷新配置，使其生效
source /etc/profile
zookeeper集群安裝
4.1 進入/usr/local/hadoop/zookeeper-3.4.14/conf目錄下 cp zoo_sample.cfg zoo.cfg
4.2 修改zoo.cfg
注意：需要在相對應的目錄下創(chuàng)建myid

例如在/data/zookeeper/ 目錄下怀喉，命令如下：
cd /data/zookeeper
echo 1 > myid

#1书妻，2，3分別表示dataDir目錄（/data/zookeeper/myid）中的內(nèi)容
server.1=hadoop1:2888:3888
server.2=hadoop2:2888:3888
server.3=hadoop3:2888:3888

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
# 此處填寫自己的目錄位置
dataDir=/data/zookeeper
dataLogDir=/logs/zookeeper
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

七磺送、安裝hadoop

在/usr/local/hadoop/ 目錄下解壓hadoop-2.7.7
tar -zxvf hadoop-2.7.7.tar.gz
配置環(huán)境變量vi /etc/profile

export HADOOP_HOME=/usr/local/hadoop/hadoop-2.7.7
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin

刷新配置驻子，使其生效
source /etc/profile
修改hadoop配置文件,進入/usr/local/hadoop/hadoop-2.7.7/etc/hadoop
4.1 修改 hadoop-env.sh 中的JAVA_HOME灿意，設(shè)置成絕對路徑

#export JAVA_HOME=${JAVA_HOME}
export JAVA_HOME=/usr/local/hadoop/jdk1.8.0_211

注意：配置中的注釋在搭建環(huán)境中最好去掉，以免出現(xiàn)莫名其妙的問題崇呵，此處只是為了便于了解屬性的含義
4.2 core-site.xml

<configuration>
<!-- 指定文件系統(tǒng)的主節(jié)點-->
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://mycluster</value>
  </property>
<!-- hadoop日志路徑-->
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/data/hadoop/tmp</value>
  </property>

  <property>
    <name>io.file.buffer.size</name>
    <value>4096</value>
  </property>
<!-- 指定可以在任何IP訪問-->
  <property>
    <name>hadoop.proxuuser.hduser.hosts</name>
    <value>*</value>
  </property>
<!-- 指定所有用戶可以訪問 -->
  <property>
    <name>hadoop.proxyuser.hduser.groups</name>
    <value>*</value>
  </property>
<!-- zookeeper集群地址 -->
  <property>
    <name>ha.zookeeper.quorum</name>
    <value>hadoop1:2181,hadoop2:2181,hadoop3:2181</value>
  </property>
</configuration>

4.3 hdfs-site.xml

<configuration>
<!-- HA配置-->
<!-- 指定hdfs的集群名為mycluster -->
  <property>
    <name>dfs.nameservices</name>
    <value>mycluster</value>
  </property>

  <property>
    <name>dfs.ha.namenodes.mycluster</name>
    <value>nn1,nn2</value>
  </property>
<!-- namenode1 RPC端口-->
  <property>
    <name>dfs.namenode.rpc-address.mycluster.nn1</name>
    <value>hadoop1:9000</value>
  </property>
<!-- namenode2 RPC端口-->
  <property>
    <name>dfs.namenode.rpc-address.mycluster.nn2</name>
    <value>hadoop2:9000</value>
  </property>
<!-- namenode1 HTTP端口-->
  <property>
    <name>dfs.namenode.http-address.mycluster.nn1</name>
    <value>hadoop1:50070</value>
  </property>
<!-- namenode2 HTTP端口-->
  <property>
    <name>dfs.namenode.http-address.mycluster.nn2</name>
    <value>hadoop2:50070</value>
  </property>
<!-- HA故障切換 -->
  <property>
    <name>dfs.ha.automic-failover.enabled.cluster</name>
    <value>true</value>
  </property>
<!-- journalnode配置-->
  <property>
    <name>dfs.namenode.shared.edits.dir</name>
  <value>qjournal://hadoop1:8485;hadoop2:8485;hadoop3:8485/mycluster</value>
  </property>
  <property>
    <name>dfs.namenode.edits.journal-plugin.qjournal </name>
    <value>org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager</value>
  </property>
<!-- 發(fā)生failover時缤剧，Standby的節(jié)點要執(zhí)行一系列方法把原來的Active節(jié)點中不健康的NameNode服務殺掉，這個叫fence過程域慷。sshfence會公國ssh遠程調(diào)用fuser命令去找到Active節(jié)點的NameNode服務并殺死它-->
  <property>
    <name>dfs.ha.fencing.methods</name>
    <value>
sshfence
shell(/bin/true)
    </value>
  </property>
<!-- SSH私鑰 -->
  <property>
    <name>dfs.ha.fencing.ssh.private-key-files</name>
    <value>/root/.ssh/id_rsa</value>
  </property>
<!-- JournalNode 文件存儲地址-->
  <property>
    <name>dfs.journalnode.edits.dir</name>
    <value>/data/hadoop/ha/jn</value>
  </property>
  <property>
    <name>dfs.permissions.enable</name>
    <value>false</value>
  </property>
<!-- 負責故障切換實現(xiàn)類 -->
  <property>
    <name>dfs.client.failover.proxy.provider.mycluster</name>
    <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
  </property>
  <property>
    <name>dfs.ha.automatic-failover.enabled</name>
    <value>true</value>
  </property>

  <property>
    <name>dfs.namenode.name.dir.restore</name>
    <value>true</value>
  </property>

  <property>
    <name>dfs.namenode.name.dir</name>
    <value>file:///data/hadoop/dfsdata/name</value>
  </property>

  <property>
    <name>dfs.blocksize</name>
    <value>67108864</value>
  </property>

  <property>
    <name>dfs.datanode.data.dir</name>
    <value>file:///data/hadoop/dfsdata/data</value>
  </property>

  <property>
    <name>dfs.replication</name>
    <value>3</value>
  </property>
<!-- 指定web可以方位hdfs目錄 -->
  <property>
    <name>dfs.webhdfs.enabled</name>
    <value>true</value>
  </property>
</configuration>

4.4 mapred-site.xml

拷貝mapred-queues.xml.template 為 mapred-site.xml
cp mapred-queues.xml.template mapred-site.xml

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
</configuration>

4.5 yarn-site.xml

<!-- resourcemanager 失聯(lián)后重新鏈接的時間 -->
<configuration>
  <property>
    <name>yarn.resourcemanager.connect.retry-interval.ms</name>
    <value>2000</value>
  </property>
<!-- 開啟resourcemanager HA荒辕，默認為false -->
  <property>
    <name>yarn.resourcemanager.ha.enabled</name>
    <value>true</value>
  </property>
<!-- 開啟resourcemanager 命名 -->
  <property>
    <name>yarn.resourcemanager.ha.rm-ids</name>
    <value>rm1,rm2</value>
  </property>
  <property>
    <name>yarn.resourcemanager.hostname.rm1</name>
    <value>hadoop2</value>
  </property>

  <property>
    <name>yarn.resourcemanager.hostname.rm2</name>
    <value>hadoop3</value>
  </property>
<!-- 開啟resourcemanager故障自動切換，指定機器-->
  <property>
    <name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
    <value>true</value>
  </property>
    <!--在 hadoop1 上配置 rm1,在 hadoop2 上配置 rm2, 注意：一般都喜歡把配置好的文件遠程復制到其它機器上犹褒，但這個在 YARN 的另一個機器上一定要修改抵窒，其他機器上不配置此項-->
  <property>
    <name>yarn.resourcemanager.ha.id</name>
    <value>rm1</value>
  </property>
<!-- 開啟resourcemanager故障自動恢復-->
  <property>
    <name>yarn.resourcemanager.recovery.enabled</name>
    <value>true</value>
  </property>
<!-- 用戶持久存儲的類-->
  <property>
    <name>yarn.resourcemanager.store.class</name>
    <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
  </property>
<!-- zookeeper集群地址-->
  <property>
    <name>yarn.resourcemanager.zk-address</name>
    <value>hadoop1:2181,hadoop2:2181,hadoop3:2181</value>
  </property>
<!-- 失聯(lián)等待鏈接時間-->
  <property>
    <name>yarn.app.mapreduce.am.scheduler.connection.wait.interval-ms</name>
    <value>5000</value>
  </property>
<!-- 集群ID -->
  <property>
    <name>yarn.resourcemanager.cluster-id</name>
    <value>mycluster</value>
  </property>
<!-- 開啟resourcemanager故障自動恢復-->
  <property>
    <name>yarn.resourcemanager.address.rm1</name>
    <value>hadoop1:8132</value>
  </property>

  <property>
    <name>yarn.resourcemanager.scheduler.address.rm1</name>
    <value>hadoop1:8130</value>
  </property>

    <!-- RM 的網(wǎng)頁接口地址：端口-->
  <property>
    <name>yarn.resourcemanager.webapp.address.rm1</name>
    <value>hadoop1:8088</value>
  </property>

  <property>
    <name>yarn.resourcemanager.resource-tracker.address.rm1</name>
    <value>hadoop1:8131</value>
  </property>

<!-- RM 管理接口地址：端口-->
  <property>
    <name>yarn.resourcemanager.admin.address.rm1</name>
    <value>hadoop1:8033</value>
  </property>

  <property>
    <name>yarn.resourcemanager.ha.admin.address.rm1</name>
    <value>hadoop1:23142</value>
  </property>

  <property>
    <name>yarn.resourcemanager.address.rm2</name>
    <value>hadoop2:8132</value>
  </property>

  <property>
    <name>yarn.resourcemanager.scheduler.address.rm2</name>
    <value>hadoop2:8130</value>
  </property>


  <property>
    <name>yarn.resourcemanager.webapp.address.rm2</name>
    <value>hadoop2:8088</value>
  </property>

  <property>
    <name>yarn.resourcemanager.resource-tracker.address.rm2</name>
    <value>hadoop2:8131</value>
  </property>

  <property>
    <name>yarn.resourcemanager.admin.address.rm2</name>
    <value>hadoop2:8033</value>
  </property>

  <property>
    <name>yarn.resourcemanager.ha.admin.address.rm2</name>
    <value>hadoop2:23142</value>
  </property>

  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>

  <property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>

  <property>
    <name>yarn.nodemanager.local-dirs</name>
    <value>/data/hadoop/dfsdata/yarn/local</value>
  </property>

  <property>
    <name>yarn.nodemanager.log-dirs</name>
    <value>/data/hadoop/dfsdata/logs</value>
  </property>

  <property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>1024</value>
    <discription>每個節(jié)點可用內(nèi)存,單位 MB</discription>
  </property>

  <property>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>258</value>
    <discription>單個任務可申請最少內(nèi)存，默認 1024MB</discription>
  </property>

  <property>
    <name>yarn.scheduler.maximum-allocation-mb</name>
    <value>512</value>
    <discription>單個任務可申請最大內(nèi)存叠骑，默認 8192MB</discription>
  </property>

  <property>
    <name>yarn.nodemanager.webapp.address</name>
    <value>0.0.0.0:8042</value>
  </property>
</configuration>

4.6 修改slaves文件

vi slaves
在其中添加

hadoop1
hadoop2
hadoop3

八李皇、每個節(jié)點創(chuàng)建相對應的目錄，分發(fā)hadoop所需文件到其他節(jié)點

創(chuàng)建目錄, 并賦予讀寫權(quán)限

mkdir -p /data/zookeeper
mkdir -p /logs/zookeeper
mkdir -p /data/hadoop/dfsdata/name
mkdir -p /data/hadoop/dfsdata/data
mkdir -p /data/hadoop/dfsdata/logs
mkdir -p /data/hadoop/dfsdata/yarn/local
mkdir -p /data/hadoop/ha/jn

sudo chmod -R 777 /data

分發(fā)文件, 進入/usr/local目錄下

scp -r hadoop root@hadoop2:/usr/local
scp -r hadoop root@hadoop3:/usr/local

九宙枷、啟動

啟動zookeeper集群, 按順序啟動

[root@hadoop1 zookeeper-3.4.14]# ./bin/zkServer.sh start

[root@hadoop2 zookeeper-3.4.14]# ./bin/zkServer.sh start

[root@hadoop3 zookeeper-3.4.14]# ./bin/zkServer.sh start

初始化hadoop掉房，進入/usr/local/hadoop/hadoop-2.7.7
2.1 格式化zk集群

/bin/hdfs zkfc -formatZK

2.2 開啟journalnode進程，啟動journalnode集群慰丛，在hadoop1卓囚，hadoop2，hadoop3上執(zhí)行

[root@hadoop1 hadoop-2.7.7]# ./sbin/hadoop-daemon.sh start journalnode

[root@hadoop2 hadoop-2.7.7]# ./sbin/hadoop-daemon.sh start journalnode

[root@hadoop3 hadoop-2.7.7]# ./sbin/hadoop-daemon.sh start journalnode

2.3 在namenode1上執(zhí)行格式化namenode

[root@hadoop1 hadoop-2.7.7]# ./bin/hadoop namenode -format

2.4 啟動datanode诅病，在hadoop1哪亿，hadoop2，hadoop3上執(zhí)行

[root@hadoop1 hadoop-2.7.7]# ./sbin/hadoop-daemon.sh start datanode
[root@hadoop2 hadoop-2.7.7]# ./sbin/hadoop-daemon.sh start datanode
[root@hadoop3 hadoop-2.7.7]# ./sbin/hadoop-daemon.sh start datanode

2.5 啟動namenode
2.5.1 namenode1

[root@hadoop1 hadoop-2.7.7]# ./sbin/hadoop-daemon.sh start namenode

2.5.2 namenode2

[root@hadoop2 hadoop-2.7.7]# ./bin/hdfs namenode -bootstrapStandby
[root@hadoop2 hadoop-2.7.7]# ./sbin/hadoop-daemon.sh start namenode

此時namenode1和namenode2同時處于 standby狀態(tài)
http://172.16.0.170:50070/dfshealth.html#tab-overview

image.png

http://172.16.0.164:50070/dfshealth.html#tab-overview

image.png

2.6啟動zkfc服務

在namenode1和namenode2上同時執(zhí)行以下命令：
[root@hadoop1 hadoop-2.7.7]# ./sbin/hadoop-daemon.sh start zkfc
[root@hadoop2 hadoop-2.7.7]# ./sbin/hadoop-daemon.sh start zkfc

啟動zkfc服務后贤笆，namenode1和namenode2會自動選舉出active節(jié)點

image.png

十蝇棉、驗證

在/data目錄下創(chuàng)建文件hello.txt

[root@hadoop1 hadoop-2.7.7]# cd /data
[root@hadoop1 hadoop-2.7.7]# echo hello world > hello.txt
[root@hadoop1 hadoop-2.7.7]# cd /usr/local/hadoop/hadoop-2.7.7
[root@hadoop1 hadoop-2.7.7]# ./bin/hdfs dfs -mkdir /test
[root@hadoop1 hadoop-2.7.7]# ./bin/hdfs dfs -put /data/hello.txt /test
[root@hadoop1 hadoop-2.7.7]# ./bin/hdfs dfs -cat /test/hello.txt

HA故障自動切換
[root@hadoop1 hadoop-2.7.7]# jps
[root@hadoop1 hadoop-2.7.7]# kill -9 pid #namenode pid

通過頁面查看節(jié)點狀態(tài)
http://172.16.0.170:50070/dfshealth.html#tab-overview
已經(jīng)訪問不了了

image.png