hmaster啟動(dòng)異常解決記錄

環(huán)境背景
在單臺(tái)云服務(wù)器上使用docker搭了一套簡(jiǎn)單的大數(shù)據(jù)開(kāi)發(fā)測(cè)試環(huán)境凸郑,1個(gè)master,2個(gè)slave,服務(wù)啟動(dòng)使用docker-compose簡(jiǎn)單編排诲祸。

hmaster啟動(dòng)失敗而昨,查看日志出現(xiàn)如下異常信息

2018-08-19 13:49:29,121 FATAL [78e7a081d8b6:16000.activeMasterManager] master.HMaster: Unhandled exception. Starting shutdown.
org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-914767151-172.18.0.4-1533896696236:blk_1073741825_1001 file=/hbase/hbase.version
        at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:976)
        at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:632)
        at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:874)
        at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:926)
        at java.io.DataInputStream.read(DataInputStream.java:149)
        at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:200)
        at org.apache.hadoop.hbase.util.FSUtils.getVersion(FSUtils.java:608)
        at org.apache.hadoop.hbase.util.FSUtils.checkVersion(FSUtils.java:691)
        at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:509)
        at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:166)
        at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:141)
        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:741)
        at org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:205)
        at org.apache.hadoop.hbase.master.HMaster$2.run(HMaster.java:2023)
        at java.lang.Thread.run(Thread.java:748)

查看hdfs上的文件塊信息配紫,發(fā)現(xiàn)有CORRUPT的情況

[root@78e7a081d8b6 logs]# hdfs fsck / -files -blocks
18/08/19 13:52:54 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Connecting to namenode via http://master:50070/fsck?ugi=root&files=1&blocks=1&path=%2F
FSCK started by root (auth:SIMPLE) from /172.18.0.3 for path / at Sun Aug 19 13:52:55 UTC 2018
/ <dir>
/hbase <dir>
/hbase/.tmp <dir>
/hbase/MasterProcWALs <dir>
/hbase/MasterProcWALs/state-00000000000000000015.log 0 bytes, 0 block(s):  OK

/hbase/WALs <dir>
/hbase/WALs/zhaocan_slave1_1.zhaocan_default,16020,1534214426421 <dir>
/hbase/WALs/zhaocan_slave2_1.zhaocan_default,16020,1534214426422 <dir>
/hbase/archive <dir>
/hbase/corrupt <dir>
/hbase/data <dir>
/hbase/data/default <dir>
/hbase/data/default/soy_test1 <dir>
/hbase/data/default/soy_test1/.tabledesc <dir>
/hbase/data/default/soy_test1/.tabledesc/.tableinfo.0000000001 289 bytes, 1 block(s): 
/hbase/data/default/soy_test1/.tabledesc/.tableinfo.0000000001: CORRUPT blockpool BP-914767151-172.18.0.4-1533896696236 block blk_1073741857
 MISSING 1 blocks of total size 289 B
0. BP-914767151-172.18.0.4-1533896696236:blk_1073741857_1033 len=289 MISSING!

/hbase/data/default/soy_test1/.tmp <dir>
/hbase/data/default/soy_test1/e332a1b73e8bcac9b69e446999e834fb <dir>
/hbase/data/default/soy_test1/e332a1b73e8bcac9b69e446999e834fb/.regioninfo 44 bytes, 1 block(s): 
/hbase/data/default/soy_test1/e332a1b73e8bcac9b69e446999e834fb/.regioninfo: CORRUPT blockpool BP-914767151-172.18.0.4-1533896696236 block blk_1073741858
 MISSING 1 blocks of total size 44 B
0. BP-914767151-172.18.0.4-1533896696236:blk_1073741858_1034 len=44 MISSING!

/hbase/data/default/soy_test1/e332a1b73e8bcac9b69e446999e834fb/data <dir>
/hbase/data/default/soy_test1/e332a1b73e8bcac9b69e446999e834fb/recovered.edits <dir>
/hbase/data/default/soy_test1/e332a1b73e8bcac9b69e446999e834fb/recovered.edits/4.seqid 0 bytes, 0 block(s):  OK

/hbase/data/default/t1 <dir>
/hbase/data/default/t1/.tabledesc <dir>
/hbase/data/default/t1/.tabledesc/.tableinfo.0000000001 766 bytes, 1 block(s): 
/hbase/data/default/t1/.tabledesc/.tableinfo.0000000001: CORRUPT blockpool BP-914767151-172.18.0.4-1533896696236 block blk_1073741853
 MISSING 1 blocks of total size 766 B
0. BP-914767151-172.18.0.4-1533896696236:blk_1073741853_1029 len=766 MISSING!

/hbase/data/default/t1/.tmp <dir>
/hbase/data/default/t1/9384715a6b039dd6db92d729703a01d8 <dir>
/hbase/data/default/t1/9384715a6b039dd6db92d729703a01d8/.regioninfo 37 bytes, 1 block(s): 
/hbase/data/default/t1/9384715a6b039dd6db92d729703a01d8/.regioninfo: CORRUPT blockpool BP-914767151-172.18.0.4-1533896696236 block blk_1073741854
 MISSING 1 blocks of total size 37 B
0. BP-914767151-172.18.0.4-1533896696236:blk_1073741854_1030 len=37 MISSING!

/hbase/data/default/t1/9384715a6b039dd6db92d729703a01d8/f1 <dir>
/hbase/data/default/t1/9384715a6b039dd6db92d729703a01d8/f1/37aeca77a262451f89bc9745de640e31 4925 bytes, 1 block(s): 
/hbase/data/default/t1/9384715a6b039dd6db92d729703a01d8/f1/37aeca77a262451f89bc9745de640e31: CORRUPT blockpool BP-914767151-172.18.0.4-1533896696236 block blk_1073741859
 MISSING 1 blocks of total size 4925 B
0. BP-914767151-172.18.0.4-1533896696236:blk_1073741859_1035 len=4925 MISSING!

/hbase/data/default/t1/9384715a6b039dd6db92d729703a01d8/f2 <dir>
/hbase/data/default/t1/9384715a6b039dd6db92d729703a01d8/f3 <dir>
/hbase/data/default/t1/9384715a6b039dd6db92d729703a01d8/recovered.edits <dir>
/hbase/data/default/t1/9384715a6b039dd6db92d729703a01d8/recovered.edits/14.seqid 0 bytes, 0 block(s):  OK

/hbase/data/hbase <dir>
/hbase/data/hbase/meta <dir>
/hbase/data/hbase/meta/.tabledesc <dir>
/hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001 397 bytes, 1 block(s): 
/hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001: CORRUPT blockpool BP-914767151-172.18.0.4-1533896696236 block blk_1073741828
 MISSING 1 blocks of total size 397 B
0. BP-914767151-172.18.0.4-1533896696236:blk_1073741828_1004 len=397 MISSING!

/hbase/data/hbase/meta/.tmp <dir>
/hbase/data/hbase/meta/1588230740 <dir>
/hbase/data/hbase/meta/1588230740/.regioninfo 32 bytes, 1 block(s): 
/hbase/data/hbase/meta/1588230740/.regioninfo: CORRUPT blockpool BP-914767151-172.18.0.4-1533896696236 block blk_1073741827
 MISSING 1 blocks of total size 32 B
0. BP-914767151-172.18.0.4-1533896696236:blk_1073741827_1003 len=32 MISSING!

/hbase/data/hbase/meta/1588230740/.tmp <dir>
/hbase/data/hbase/meta/1588230740/info <dir>
/hbase/data/hbase/meta/1588230740/info/b4a2d55521614033bd88eeeb82c3fd4d 9013 bytes, 1 block(s): 
/hbase/data/hbase/meta/1588230740/info/b4a2d55521614033bd88eeeb82c3fd4d: CORRUPT blockpool BP-914767151-172.18.0.4-1533896696236 block blk_1073741928
 MISSING 1 blocks of total size 9013 B
0. BP-914767151-172.18.0.4-1533896696236:blk_1073741928_1110 len=9013 MISSING!

/hbase/data/hbase/meta/1588230740/recovered.edits <dir>
/hbase/data/hbase/meta/1588230740/recovered.edits/31.seqid 0 bytes, 0 block(s):  OK

/hbase/data/hbase/namespace <dir>
/hbase/data/hbase/namespace/.tabledesc <dir>
/hbase/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 312 bytes, 1 block(s): 
/hbase/data/hbase/namespace/.tabledesc/.tableinfo.0000000001: CORRUPT blockpool BP-914767151-172.18.0.4-1533896696236 block blk_1073741834
 MISSING 1 blocks of total size 312 B
0. BP-914767151-172.18.0.4-1533896696236:blk_1073741834_1010 len=312 MISSING!

/hbase/data/hbase/namespace/.tmp <dir>
/hbase/data/hbase/namespace/23dc1dbbc536758979b2bcdfb7b6d556 <dir>
/hbase/data/hbase/namespace/23dc1dbbc536758979b2bcdfb7b6d556/.regioninfo 42 bytes, 1 block(s): 
/hbase/data/hbase/namespace/23dc1dbbc536758979b2bcdfb7b6d556/.regioninfo: CORRUPT blockpool BP-914767151-172.18.0.4-1533896696236 block blk_1073741835
 MISSING 1 blocks of total size 42 B
0. BP-914767151-172.18.0.4-1533896696236:blk_1073741835_1011 len=42 MISSING!

/hbase/data/hbase/namespace/23dc1dbbc536758979b2bcdfb7b6d556/info <dir>
/hbase/data/hbase/namespace/23dc1dbbc536758979b2bcdfb7b6d556/info/f642f548417241e3a1fbbe34103507a2 4963 bytes, 1 block(s): 
/hbase/data/hbase/namespace/23dc1dbbc536758979b2bcdfb7b6d556/info/f642f548417241e3a1fbbe34103507a2: CORRUPT blockpool BP-914767151-172.18.0.4-1533896696236 block blk_1073741850
 MISSING 1 blocks of total size 4963 B
0. BP-914767151-172.18.0.4-1533896696236:blk_1073741850_1026 len=4963 MISSING!

/hbase/data/hbase/namespace/23dc1dbbc536758979b2bcdfb7b6d556/recovered.edits <dir>
/hbase/data/hbase/namespace/23dc1dbbc536758979b2bcdfb7b6d556/recovered.edits/16.seqid 0 bytes, 0 block(s):  OK

/hbase/hbase.id 42 bytes, 1 block(s): 
/hbase/hbase.id: CORRUPT blockpool BP-914767151-172.18.0.4-1533896696236 block blk_1073741826
 MISSING 1 blocks of total size 42 B
0. BP-914767151-172.18.0.4-1533896696236:blk_1073741826_1002 len=42 MISSING!

/hbase/hbase.version 7 bytes, 1 block(s): 
/hbase/hbase.version: CORRUPT blockpool BP-914767151-172.18.0.4-1533896696236 block blk_1073741825
 MISSING 1 blocks of total size 7 B
0. BP-914767151-172.18.0.4-1533896696236:blk_1073741825_1001 len=7 MISSING!

/hbase/oldWALs <dir>
/root <dir>
/root/hive <dir>
/root/hive/root <dir>
/root/hive/root/16e2f965-2d21-4504-992e-4d64079f5a51 <dir>
/root/hive/root/16e2f965-2d21-4504-992e-4d64079f5a51/_tmp_space.db <dir>
/root/hive/root/3a835fae-a5d9-4cfd-8f14-b2fc1b35e984 <dir>
/root/hive/root/3a835fae-a5d9-4cfd-8f14-b2fc1b35e984/_tmp_space.db <dir>
/root/hive/root/4e754087-66f8-471e-9717-5ecc7bebc29b <dir>
/root/hive/root/4e754087-66f8-471e-9717-5ecc7bebc29b/_tmp_space.db <dir>
/root/hive/root/e6fe8961-9813-4e26-9366-b323d5a16479 <dir>
/root/hive/root/e6fe8961-9813-4e26-9366-b323d5a16479/_tmp_space.db <dir>
/root/hive/warehouse <dir>
/root/hive/warehouse/t1 <dir>
/root/hive/warehouse/testdb.db <dir>
/root/hive/warehouse/testdb.db/soy_test1 <dir>
/test <dir>
Status: CORRUPT
 Total size:    20869 B (Total open files size: 249 B)
 Total dirs:    56
 Total files:   18
 Total symlinks:        0 (Files currently being written: 3)
 Total blocks (validated):  13 (avg. block size 1605 B) (Total open file blocks (not validated): 3)
  ********************************
  UNDER MIN REPL'D BLOCKS:  13 (100.0 %)
  dfs.namenode.replication.min: 1
  CORRUPT FILES:    13
  MISSING BLOCKS:   13
  MISSING SIZE:     20869 B
  CORRUPT BLOCKS:   13
  ********************************
 Minimally replicated blocks:   0 (0.0 %)
 Over-replicated blocks:    0 (0.0 %)
 Under-replicated blocks:   0 (0.0 %)
 Mis-replicated blocks:     0 (0.0 %)
 Default replication factor:    3
 Average block replication: 0.0
 Corrupt blocks:        13
 Missing replicas:      0
 Number of data-nodes:      2
 Number of racks:       1
FSCK ended at Sun Aug 19 13:52:55 UTC 2018 in 8 milliseconds


The filesystem under path '/' is CORRUPT

由于是開(kāi)發(fā)環(huán)境,直接執(zhí)行問(wèn)題塊的刪除操作氛魁,然后再次啟動(dòng)hbase

[root@78e7a081d8b6 logs]# hdfs fsck -delete
18/08/19 13:54:54 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Connecting to namenode via http://master:50070/fsck?ugi=root&delete=1&path=%2F
FSCK started by root (auth:SIMPLE) from /172.18.0.3 for path / at Sun Aug 19 13:54:55 UTC 2018
.....Status: HEALTHY
 Total size:    0 B (Total open files size: 249 B)
 Total dirs:    56
 Total files:   5
 Total symlinks:        0 (Files currently being written: 3)
 Total blocks (validated):  0 (Total open file blocks (not validated): 3)
 Minimally replicated blocks:   0
 Over-replicated blocks:    0
 Under-replicated blocks:   0
 Mis-replicated blocks:     0
 Default replication factor:    3
 Average block replication: 0.0
 Corrupt blocks:        0
 Missing replicas:      0
 Number of data-nodes:      2
 Number of racks:       1
FSCK ended at Sun Aug 19 13:54:55 UTC 2018 in 4 milliseconds


The filesystem under path '/' is HEALTHY

此時(shí)出現(xiàn)如下異常秀存,可以直接使用hdfs -rmr /hbase刪除整個(gè)目錄的數(shù)據(jù)

2018-08-19 13:56:13,652 FATAL [78e7a081d8b6:16000.activeMasterManager] master.HMaster: Failed to become active master
org.apache.hadoop.hbase.util.FileSystemVersionException: HBase file layout needs to be upgraded. You have version null and I want version 8. Consult http://hbase.apache.org/book.html for further information about upgrading HBase. Is your hbase.rootdir valid? If so, you may need to run 'hbase hbck -fixVersionFile'.
        at org.apache.hadoop.hbase.util.FSUtils.checkVersion(FSUtils.java:712)
        at org.apache.hadoop.hbase.master.MasterFileSystem.checkRootDir(MasterFileSystem.java:509)
        at org.apache.hadoop.hbase.master.MasterFileSystem.createInitialFileSystemLayout(MasterFileSystem.java:166)
        at org.apache.hadoop.hbase.master.MasterFileSystem.<init>(MasterFileSystem.java:141)
        at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:741)
        at org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:205)
        at org.apache.hadoop.hbase.master.HMaster$2.run(HMaster.java:2023)
        at java.lang.Thread.run(Thread.java:748)
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
  • 序言:七十年代末或链,一起剝皮案震驚了整個(gè)濱河市,隨后出現(xiàn)的幾起案子祈纯,更是在濱河造成了極大的恐慌叼耙,老刑警劉巖,帶你破解...
    沈念sama閱讀 207,113評(píng)論 6 481
  • 序言:濱河連續(xù)發(fā)生了三起死亡事件簇爆,死亡現(xiàn)場(chǎng)離奇詭異倾贰,居然都是意外死亡匆浙,警方通過(guò)查閱死者的電腦和手機(jī),發(fā)現(xiàn)死者居然都...
    沈念sama閱讀 88,644評(píng)論 2 381
  • 文/潘曉璐 我一進(jìn)店門(mén)挑庶,熙熙樓的掌柜王于貴愁眉苦臉地迎上來(lái)软能,“玉大人,你說(shuō)我怎么就攤上這事凳枝“虾耍” “怎么了?”我有些...
    開(kāi)封第一講書(shū)人閱讀 153,340評(píng)論 0 344
  • 文/不壞的土叔 我叫張陵蹋订,是天一觀的道長(zhǎng)露戒。 經(jīng)常有香客問(wèn)我,道長(zhǎng)智什,這世上最難降的妖魔是什么? 我笑而不...
    開(kāi)封第一講書(shū)人閱讀 55,449評(píng)論 1 279
  • 正文 為了忘掉前任悦屏,我火速辦了婚禮,結(jié)果婚禮上础爬,老公的妹妹穿的比我還像新娘。我一直安慰自己叫搁,他們只是感情好供炎,可當(dāng)我...
    茶點(diǎn)故事閱讀 64,445評(píng)論 5 374
  • 文/花漫 我一把揭開(kāi)白布音诫。 她就那樣靜靜地躺著,像睡著了一般梨撞。 火紅的嫁衣襯著肌膚如雪香罐。 梳的紋絲不亂的頭發(fā)上,一...
    開(kāi)封第一講書(shū)人閱讀 49,166評(píng)論 1 284
  • 那天港粱,我揣著相機(jī)與錄音旦签,去河邊找鬼。 笑死咪惠,一個(gè)胖子當(dāng)著我的面吹牛淋淀,可吹牛的內(nèi)容都是我干的覆醇。 我是一名探鬼主播炭臭,決...
    沈念sama閱讀 38,442評(píng)論 3 401
  • 文/蒼蘭香墨 我猛地睜開(kāi)眼鞋仍,長(zhǎng)吁一口氣:“原來(lái)是場(chǎng)噩夢(mèng)啊……” “哼!你這毒婦竟也來(lái)了落午?” 一聲冷哼從身側(cè)響起肚豺,我...
    開(kāi)封第一講書(shū)人閱讀 37,105評(píng)論 0 261
  • 序言:老撾萬(wàn)榮一對(duì)情侶失蹤,失蹤者是張志新(化名)和其女友劉穎梗劫,沒(méi)想到半個(gè)月后截碴,有當(dāng)?shù)厝嗽跇?shù)林里發(fā)現(xiàn)了一具尸體,經(jīng)...
    沈念sama閱讀 43,601評(píng)論 1 300
  • 正文 獨(dú)居荒郊野嶺守林人離奇死亡走哺,尸身上長(zhǎng)有42處帶血的膿包…… 初始之章·張勛 以下內(nèi)容為張勛視角 年9月15日...
    茶點(diǎn)故事閱讀 36,066評(píng)論 2 325
  • 正文 我和宋清朗相戀三年割坠,在試婚紗的時(shí)候發(fā)現(xiàn)自己被綠了妒牙。 大學(xué)時(shí)的朋友給我發(fā)了我未婚夫和他白月光在一起吃飯的照片。...
    茶點(diǎn)故事閱讀 38,161評(píng)論 1 334
  • 序言:一個(gè)原本活蹦亂跳的男人離奇死亡敢朱,死狀恐怖拴签,靈堂內(nèi)的尸體忽然破棺而出旗们,到底是詐尸還是另有隱情,我是刑警寧澤上渴,帶...
    沈念sama閱讀 33,792評(píng)論 4 323
  • 正文 年R本政府宣布,位于F島的核電站曹阔,受9級(jí)特大地震影響赃份,放射性物質(zhì)發(fā)生泄漏。R本人自食惡果不足惜抓韩,卻給世界環(huán)境...
    茶點(diǎn)故事閱讀 39,351評(píng)論 3 307
  • 文/蒙蒙 一园蝠、第九天 我趴在偏房一處隱蔽的房頂上張望。 院中可真熱鬧,春花似錦怠蹂、人聲如沸。這莊子的主人今日做“春日...
    開(kāi)封第一講書(shū)人閱讀 30,352評(píng)論 0 19
  • 文/蒼蘭香墨 我抬頭看了看天上的太陽(yáng)。三九已至揩魂,卻和暖如春炮温,著一層夾襖步出監(jiān)牢的瞬間,已是汗流浹背倦挂。 一陣腳步聲響...
    開(kāi)封第一講書(shū)人閱讀 31,584評(píng)論 1 261
  • 我被黑心中介騙來(lái)泰國(guó)打工担巩, 沒(méi)想到剛下飛機(jī)就差點(diǎn)兒被人妖公主榨干…… 1. 我叫王不留涛癌,地道東北人窥浪。 一個(gè)月前我還...
    沈念sama閱讀 45,618評(píng)論 2 355
  • 正文 我出身青樓笛丙,卻偏偏與公主長(zhǎng)得像,于是被迫代替她去往敵國(guó)和親骨稿。 傳聞我的和親對(duì)象是個(gè)殘疾皇子姜钳,可洞房花燭夜當(dāng)晚...
    茶點(diǎn)故事閱讀 42,916評(píng)論 2 344

推薦閱讀更多精彩內(nèi)容

  • 入門(mén)指南 1. 簡(jiǎn)介 Quickstart會(huì)讓你啟動(dòng)和運(yùn)行一個(gè)單節(jié)點(diǎn)單機(jī)HBase哥桥。 2. 快速啟動(dòng) – 單點(diǎn)HB...
    和心數(shù)據(jù)閱讀 4,522評(píng)論 1 41
  • 前言 在本節(jié)中拟糕,你將首先學(xué)習(xí)單節(jié)點(diǎn)、獨(dú)立的HBase的設(shè)置送滞,并且學(xué)會(huì)運(yùn)行單節(jié)點(diǎn)、獨(dú)立的HBase實(shí)例边涕! 在一個(gè)獨(dú)立...
    步閑閱讀 1,859評(píng)論 1 3
  • Zookeeper用于集群主備切換褂微。 YARN讓集群具備更好的擴(kuò)展性。 Spark沒(méi)有存儲(chǔ)能力式撼。 Spark的Ma...
    Yobhel閱讀 7,246評(píng)論 0 34
  • 本文首先簡(jiǎn)單介紹了HBase,然后重點(diǎn)講述了HBase的高并發(fā)和實(shí)時(shí)處理數(shù)據(jù) 端衰、HBase數(shù)據(jù)模型甘改、HBase物理...
    達(dá)微閱讀 2,726評(píng)論 1 13
  • 法治的發(fā)展映射到個(gè)人表現(xiàn)在權(quán)利義務(wù)意識(shí)的增強(qiáng)忘嫉,在當(dāng)今權(quán)利意識(shí)較強(qiáng)的生活環(huán)境里案腺,不同的季節(jié)你享有的權(quán)利會(huì)有...
    c62ab6e84380閱讀 705評(píng)論 3 4