HMaster無法啟動(dòng)（報(bào)錯(cuò)： Cannot recoverLease）

一宪卿、報(bào)錯(cuò)：

關(guān)鍵字：

Cannot recoverLease

Failed to recover lease

master.HMaster: Failed to become active master

Cannot obtain block length for LocatedBlock

報(bào)錯(cuò)堆棧：

2019-10-31 18:34:48,318 INFO [cloud1:16000.activeMasterManager] balancer.StochasticLoadBalancer: loading config

2019-10-31 18:34:48,389 INFO? [cloud1:16000.activeMasterManager] master.HMaster: Server active/primary master=cloud1,16000,1572518079864, sessionid=0x26e215f68470001, setting cluster-up flag (Was=false)

2019-10-31 18:34:48,392 INFO? [master/cloud1/172.18.49.221:16000] regionserver.HRegionServer: ClusterId : 6f9b6fa5-d0aa-426e-bd0c-e7b12d437a2d

2019-10-31 18:34:48,438 INFO? [cloud1:16000.activeMasterManager] procedure.ZKProcedureUtil: Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort

2019-10-31 18:34:48,484 INFO? [cloud1:16000.activeMasterManager] procedure.ZKProcedureUtil: Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort

2019-10-31 18:34:48,533 INFO? [cloud1:16000.activeMasterManager] master.MasterCoprocessorHost: System coprocessor loading is enabled

2019-10-31 18:34:48,555 INFO? [cloud1:16000.activeMasterManager] procedure2.ProcedureExecutor: Starting procedure executor threads=13

2019-10-31 18:34:48,556 INFO? [cloud1:16000.activeMasterManager] wal.WALProcedureStore: Starting WAL Procedure Store lease recovery

2019-10-31 18:34:48,559 INFO? [cloud1:16000.activeMasterManager] util.FSHDFSUtils: Recover lease on dfs file hdfs://mycluster/hbase/MasterProcWALs/state-00000000000000000001.log

2019-10-31 18:34:48,588 INFO? [cloud1:16000.activeMasterManager] util.FSHDFSUtils: Failed to recover lease, attempt=0 on file=hdfs://mycluster/hbase/MasterProcWALs/state-00000000000000000001.log after 29ms

2019-10-31 18:34:52,608 INFO? [cloud1:16000.activeMasterManager] util.FSHDFSUtils: Failed to recover lease, attempt=1 on file=hdfs://mycluster/hbase/MasterProcWALs/state-00000000000000000001.log after 4049ms

2019-10-31 18:35:56,728 INFO? [cloud1:16000.activeMasterManager] util.FSHDFSUtils: Failed to recover lease, attempt=2 on file=hdfs://mycluster/hbase/MasterProcWALs/state-00000000000000000001.log after 68169ms

2019-10-31 18:38:05,209 INFO? [cloud1:16000.activeMasterManager] util.FSHDFSUtils: Failed to recover lease, attempt=3 on file=hdfs://mycluster/hbase/MasterProcWALs/state-00000000000000000001.log after 196650ms

2019-10-31 18:41:17,427 INFO? [cloud1:16000.activeMasterManager] util.FSHDFSUtils: Failed to recover lease, attempt=4 on file=hdfs://mycluster/hbase/MasterProcWALs/state-00000000000000000001.log after 388868ms

2019-10-31 18:45:33,913 INFO? [cloud1:16000.activeMasterManager] util.FSHDFSUtils: Failed to recover lease, attempt=5 on file=hdfs://mycluster/hbase/MasterProcWALs/state-00000000000000000001.log after 645354ms

2019-10-31 18:49:46,673 ERROR [ActiveMasterInitializationMonitor-1572518086672] master.HMaster: Master failed to complete initialization after 900000ms. Please consider submitting a bug report including a thread dump of this process.

2019-10-31 18:50:54,185 INFO? [cloud1:16000.activeMasterManager] util.FSHDFSUtils: Failed to recover lease, attempt=6 on file=hdfs://mycluster/hbase/MasterProcWALs/state-00000000000000000001.log after 965626ms

2019-10-31 18:50:54,185 WARN? [cloud1:16000.activeMasterManager] util.FSHDFSUtils: Cannot recoverLease after trying for 900000ms (hbase.lease.recovery.timeout); continuing, but may be DATALOSS!!!; attempt=6 on file=hdfs://mycluster/hbase/MasterProcWALs/state-00000000000000000001.log after 965626ms

2019-10-31 18:50:54,450 ERROR [cloud1:16000.activeMasterManager] wal.WALProcedureStore: Unable to read state log: FileStatus{path=hdfs://mycluster/hbase/MasterProcWALs/state-00000000000000000001.log; isDirectory=false; length=132; replication=3; blocksize=67108864; modification_time=1572514780827; access_time=1572514780827; owner=cloud; group=supergroup; permission=rw-r--r--; isSymlink=false}

java.io.IOException: Cannot obtain block length for LocatedBlock{BP-2130371621-192.168.120.226-1506852297811:blk_1075210355_1469774; getBlockSize()=132; corrupt=false; offset=0; locs=[172.18.49.224:50010, 172.18.49.222:50010, 172.18.49.223:50010]}

? ? ? ? at org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:354)

? ? ? ? at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:298)

? ? ? ? at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:235)

? ? ? ? at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:228)

? ? ? ? at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1318)

? ? ? ? at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:293)

? ? ? ? at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:289)

? ? ? ? at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)

? ? ? ? at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:289)

? ? ? ? at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:764)

? ? ? ? at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFile.open(ProcedureWALFile.java:67)

? ? ? ? at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.initOldLog(WALProcedureStore.java:961)

? ? ? ? at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.initOldLogs(WALProcedureStore.java:924)

? ? ? ? at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:262)

? ? ? ? at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:511)

? ? ? ? at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1224)

? ? ? ? at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1136)

? ? ? ? at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:725)

? ? ? ? at org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:189)

? ? ? ? at org.apache.hadoop.hbase.master.HMaster$2.run(HMaster.java:1803)

? ? ? ? at java.lang.Thread.run(Thread.java:745)

2019-10-31 18:50:54,452 FATAL [cloud1:16000.activeMasterManager] master.HMaster: Failed to become active master

java.io.IOException: Unable to read state log: FileStatus{path=hdfs://mycluster/hbase/MasterProcWALs/state-00000000000000000001.log; isDirectory=false; length=132; replication=3; blocksize=67108864; modification_time=1572514780827; access_time=1572514780827; owner=cloud; group=supergroup; permission=rw-r--r--; isSymlink=false}

? ? ? ? at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.initOldLog(WALProcedureStore.java:969)