前言：Hadoop NameNode節(jié)點維護了整個HDFS集群所有的元數(shù)據(jù)信息疹蛉，一旦NameNode發(fā)生腦裂，或者服務(wù)不可用力麸，整個HDFS集群都將處于不可用狀態(tài)可款。對于線上生產(chǎn)環(huán)境，將會造成不可估量的損失克蚂。因此闺鲸，對于NameNode節(jié)點的狀態(tài)監(jiān)控不可或缺，一旦發(fā)生腦裂或者服務(wù)異常停止埃叭，都能夠及時通知運維介入摸恍。當(dāng)然，NameNode本身有Fencing機制游盲，Hadoop默認(rèn)Fencing機制不做任何動作，這就需要開發(fā)人員自主配置Fencing機制需要執(zhí)行的操作蛮粮。

本文要旨：① 如何監(jiān)控NameNode的狀態(tài)切換益缎； ② 自主配置Fencing機制；

集群環(huán)境：Ambari 2.2.1然想， HDP 2.3.4莺奔，Hadoop 2.7.1

【NameNode狀態(tài)監(jiān)控】

① 訪問url獲取集群Namenode狀態(tài)：

http://${hostname}:50070/jmx?qry=Hadoop:service=NameNode,name=NameNodeStatus

返回如下結(jié)果：

   {
  "beans" : [ {
    "name" : "Hadoop:service=NameNode,name=NameNodeStatus",
    "modelerType" : "org.apache.hadoop.hdfs.server.namenode.NameNode",
    "State" : "active",
    "NNRole" : "NameNode",
    "HostAndPort" : "hostname1:8020",
    "SecurityEnabled" : false,
    "LastHATransitionTime" : 1519300091101,
    "BytesWithFutureGenerationStamps" : 0
  } ]
}

② 依次訪問兩個namenode節(jié)點，返回信息通過python腳本解析变泄，獲取節(jié)點狀態(tài):

# -*- coding: utf-8 -*-
import json
import urllib
import sys

def get_namenode_status(hostname):
    jmxport = 50070
    url = "http://{0}:{1}/jmx?qry=Hadoop:service=NameNode,name=NameNodeStatus".format(hostname, jmxport)
    try:
        response = urllib.urlopen(url)
        jsonString = json.load(response)
        nnstatus = jsonString.get('beans', [{}])[0].get('State', '')
        if 'active' == nnstatus:
            return 'active'
        elif 'standby' == nnstatus:
            return 'standby'
        else:
            return
    except:
        return

if __name__ == "__main__":
    hostname = (sys.argv[1] if len(sys.argv) > 1 else None)
    if hostname:
        print get_namenode_status(hostname)
    else:
        print None

腳本返回三種狀態(tài)：active(主)令哟，standby(備)，None(服務(wù)異常妨蛹，無法獲取狀態(tài))

③ 通過shell腳本根據(jù)返回狀態(tài)判斷是否報警：

#!/bin/bash

bin=`dirname "$0"`
code_path=`cd "$bin";pwd`
file_name="last_ha_status"

export monitor_info=""

nn1=`python $code_path/namenodestatus.py hostname1`
nn2=`python $code_path/namenodestatus.py hostname2`

last_status_hdfs=`cat /tmp/$file_name`
current_status_hdfs=$nn1,$nn2

if [[ "standby" == $nn1 ]] && [[ "standby" == $nn2 ]]; then
        monitor_info="hostname1:"$nn1" hostname2:"$nn2" 上次HA狀態(tài)信息:"$last_status_hdfs" 當(dāng)前HA狀態(tài)信息:"$current_status_hdfs
        echo $nn1,$nn2 > /tmp/$file_name
        #todo 調(diào)用報警接口屏富，打印monitor_info內(nèi)容
        exit 1
elif [[ "active" == $nn1 ]] && [[ "active" == $nn2 ]]; then
        monitor_info="hostname1:"$nn1" hostname2:"$nn2" 上次HA狀態(tài)信息:"$last_status_hdfs" 當(dāng)前HA狀態(tài)信息:"$current_status_hdfs
        echo $nn1,$nn2 > /tmp/$file_name
        #todo 調(diào)用報警接口，打印monitor_info內(nèi)容
        exit 1
elif [[ "None" == $nn1 ]] || [[ "None" == $nn2 ]]; then
        monitor_info="hostname1:"$nn1" hostname2:"$nn2" 上次HA狀態(tài)信息:"$last_status_hdfs" 當(dāng)前HA狀態(tài)信息:"$current_status_hdfs
        echo $nn1,$nn2 > /tmp/$file_name
        #todo 調(diào)用報警接口蛙卤，打印monitor_info內(nèi)容
        exit 1
else
        #狀態(tài)正常狠半，將正常狀態(tài)寫入本地文件
        echo $nn1,$nn2 > /tmp/$file_name
        exit 0
fi

報警閾值：
① 兩個NameNode都處于standby狀態(tài)；
② 兩個NameNode都處于active狀態(tài)颤难；
③ 任何一個NameNode服務(wù)停止神年；

NameNode的主備狀態(tài)獲取途徑除了通過訪問50070獲取JMX監(jiān)控信息外，還可以通過hdfs haadmin命令獲刃朽汀：

nn1=`ssh hdfs@hostname1 hdfs haadmin -Dipc.client.connect.max.retries.on.timeouts=3 -Dipc.client.connect.timeout=3000 -GetServiceState nn1`
nn2=`ssh hdfs@hostname2 hdfs haadmin -Dipc.client.connect.max.retries.on.timeouts=3 -Dipc.client.connect.timeout=3000 -GetServiceState nn2`

備注①：執(zhí)行監(jiān)控腳本的用戶A需要設(shè)置訪問hdfs用戶的免密碼登錄已日，當(dāng)然也可以開通用戶A訪問hdfs的權(quán)限；
備注②：hdfs core-site.xml默認(rèn)配置定義了socket連接最大重試次數(shù)為45栅屏，以及每次等待連接時間為20s飘千。這種情況下 如果NameNode節(jié)點異常堂鲜，需要等待45 * 20s才能返回相應(yīng)結(jié)果。對于監(jiān)控腳本占婉，這么長的等待時間顯然不合理泡嘴，因此需要我們自定義最長允許的等待時間。

<property>
  <name>ipc.client.connect.timeout</name>
  <value>20000</value>
  <description>Indicates the number of milliseconds a client will wait for the socket to establish a server connection.
  </description>
</property>

<property>
  <name>ipc.client.connect.max.retries.on.timeouts</name>
  <value>45</value>
  <description>Indicates the number of retries a client will make on socket timeout to establish a server connection.
  </description>
</property>

【自定義Fencing機制】

參考官方文檔：https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html
在分布式系統(tǒng)中腦裂又稱為雙主現(xiàn)象逆济，由于Zookeeper的“假死”酌予，長時間的垃圾回收或其它原因都可能導(dǎo)致雙Active NameNode現(xiàn)象，此時兩個NameNode都可以對外提供服務(wù)奖慌，無法保證數(shù)據(jù)一致性抛虫。對于生產(chǎn)環(huán)境，這種情況的出現(xiàn)是毀滅性的简僧，必須通過自帶的隔離（Fencing）機制預(yù)防這種現(xiàn)象的出現(xiàn)建椰。

Hadoop目前主要提供兩種隔離措施：
sshfence：SSH to the Active NameNode and kill the process；
shellfence：run an arbitrary shell command to fence the Active NameNode岛马；

只有在成功地執(zhí)行完成fencing之后棉姐，選主成功的ActiveStandbyElector才會回調(diào)ZKFC的becomeActive方法將對應(yīng)的NameNode切換為Active，開始對外提供服務(wù)啦逆。

2018-02-22 18:57:11,865 INFO  ha.NodeFencer (NodeFencer.java:fence(91)) - ====== Beginning Service Fencing Process... ======
2018-02-22 18:57:11,865 INFO  ha.NodeFencer (NodeFencer.java:fence(94)) - Trying method 1/1: org.apache.hadoop.ha.ShellCommandFencer(/usr/custom/hdfs/restart_namenode.sh)
2018-02-22 18:57:11,871 INFO  ha.ShellCommandFencer (ShellCommandFencer.java:tryFence(99)) - Launched fencing command '/usr/custom/hdfs/restart_namenode.sh' with pid 21772
2018-02-22 18:57:11,894 WARN  ha.ShellCommandFencer (StreamPumper.java:pump(88)) - [PID 21772] /usr/cus...menode.sh:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2018-02-22 18:57:11,894 WARN  ha.ShellCommandFencer (StreamPumper.java:pump(88)) - [PID 21772] /usr/cus...menode.sh:                                  Dload  Upload   Total   Spent    Left  Speed
2018-02-22 18:57:11,894 WARN  ha.ShellCommandFencer (StreamPumper.java:pump(88)) - [PID 21772] /usr/cus...menode.sh:
2018-02-22 18:57:11,965 INFO  ha.ShellCommandFencer (StreamPumper.java:pump(86)) - [PID 21772] /usr/cus...menode.sh: {
2018-02-22 18:57:11,965 INFO  ha.ShellCommandFencer (StreamPumper.java:pump(86)) - [PID 21772] /usr/cus...menode.sh:   "href" : "http://hostname1:8080/api/v1/clusters/cbas_cluster/requests/5363",
2018-02-22 18:57:11,965 INFO  ha.ShellCommandFencer (StreamPumper.java:pump(86)) - [PID 21772] /usr/cus...menode.sh:   "Requests" : {
2018-02-22 18:57:11,965 INFO  ha.ShellCommandFencer (StreamPumper.java:pump(86)) - [PID 21772] /usr/cus...menode.sh:     "id" : 5363,
2018-02-22 18:57:11,965 INFO  ha.ShellCommandFencer (StreamPumper.java:pump(86)) - [PID 21772] /usr/cus...menode.sh:     "status" : "Accepted"
2018-02-22 18:57:11,965 WARN  ha.ShellCommandFencer (StreamPumper.java:pump(88)) - [PID 21772] /usr/cus...menode.sh:   0     0    0     0    0   346      0   343k --:--:-- --:--:-- --:--:--  343k
2018-02-22 18:57:11,965 INFO  ha.ShellCommandFencer (StreamPumper.java:pump(86)) - [PID 21772] /usr/cus...menode.sh:   }
2018-02-22 18:57:11,966 INFO  ha.ShellCommandFencer (StreamPumper.java:pump(86)) - [PID 21772] /usr/cus...menode.sh: }
2018-02-22 18:57:11,965 WARN  ha.ShellCommandFencer (StreamPumper.java:pump(88)) - [PID 21772] /usr/cus...menode.sh: 495   149  149   149    0   346   2073   4815 --:--:-- --:--:-- --:--:--     0
2018-02-22 18:57:11,967 INFO  ha.NodeFencer (NodeFencer.java:fence(98)) - ====== Fencing successful by method org.apache.hadoop.ha.ShellCommandFencer(/usr/custom/hdfs/restart_namenode.sh) ======
2018-02-22 18:57:11,967 INFO  ha.ActiveStandbyElector (ActiveStandbyElector.java:writeBreadCrumbNode(878)) - Writing znode /hadoop-ha/cbas/ActiveBreadCrumb to indicate that the local node is the most recent active...
2018-02-22 18:57:11,969 INFO  ha.ZKFailoverController (ZKFailoverController.java:becomeActive(380)) - Trying to make NameNode at hostname1/xx.x.xx.xxx:8021 active...
2018-02-22 18:57:13,710 INFO  ha.ZKFailoverController (ZKFailoverController.java:becomeActive(387)) - Successfully transitioned NameNode at hostname1/xx.x.xx.xxx:8021 to active state

本章只介紹shellfence機制伞矩，因為作者所使用集群通過Ambari管理，因此可以調(diào)用ambari自帶的Rest API達(dá)到重啟NameNode節(jié)點的效果夏志。
Ambari Rest API官網(wǎng)文檔：https://cwiki.apache.org/confluence/display/AMBARI/Restart+Host+Components

#!/bin/bash
curl -u admin:${admin_password} -H 'X-Requested-By: ambari' -X POST -d '
{
   "RequestInfo":{
      "command":"RESTART",
      "context":"Restart NameNode",
      "operation_level":{
         "level":"HOST",
         "cluster_name":"${cluster_name}"
      }
   },
   "Requests/resource_filters":[
      {
         "service_name":"HDFS",
         "component_name":"NAMENODE",
         "hosts":"${hostname1}"
      }
   ]
}' http://${ambari_hostname}:8080/api/v1/clusters/${cluster_name}/requests

① 將腳本存放在兩臺NameNode所在服務(wù)器上：/usr/custom/hdfs/restart_namenode.sh乃坤，注意腳本的可執(zhí)行權(quán)限，并修改相應(yīng)變量以達(dá)到遠(yuǎn)程重啟另外一個NameNode服務(wù)的目的沟蔑；
② 修改hdfs配置文件：
Amabri入口：Ambari UI => HDFS => Configs => Advanced => Custom hdfs-site

<property>
      <name>dfs.ha.fencing.methods</name>
      <value>shell(/usr/custom/hdfs/restart_namenode.sh)</value>
    </property>

結(jié)束語：對于生產(chǎn)環(huán)境的大數(shù)據(jù)平臺建設(shè)湿诊，NameNode HA模式的狀態(tài)切換都應(yīng)積極跟進，查找狀態(tài)發(fā)生切換的根本原因瘦材，避免或者是減少切換的頻率厅须。狀態(tài)監(jiān)控和自定義Fencing機制都只是NameNode異常發(fā)生后的處理措施。即使發(fā)生狀態(tài)切換食棕，也應(yīng)該實現(xiàn)自動化九杂，避免人工干預(yù)，用戶無感知宣蠕。

博客主頁：http://www.reibang.com/u/e97bb429f278