Hadoop服務(wù)好一陣沒管了娘汞，今天上線發(fā)現(xiàn)從節(jié)點已宕機锈死。
重啟服務(wù)后蝙搔，有個從節(jié)點只運行了DataNode而沒有NodeManager進程钓辆，查看日志報錯如下剪验。
2020-03-14 11:26:37,654 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.net.BindException: Problem binding to [0.0.0.0:8040] java.net.BindException: Address already in use; For more details see: http://wiki.apache.org/hadoop/BindException
at org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:138)
at org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:65)
at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(YarnRPC.java:54)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.createServer(ResourceLocalizationService.java:412)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.serviceStart(ResourceLocalizationService.java:388)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStart(ContainerManagerImpl.java:668)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:937)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:1016)
Caused by: java.net.BindException: Problem binding to [0.0.0.0:8040] java.net.BindException: Address already in use; For more details see: http://wiki.apache.org/hadoop/BindException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:736)
at org.apache.hadoop.ipc.Server.bind(Server.java:621)
at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:1185)
at org.apache.hadoop.ipc.Server.<init>(Server.java:3067)
at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:1005)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.<init>(ProtobufRpcEngine.java:426)
at org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:347)
at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:846)
at org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.createServer(RpcServerFactoryPBImpl.java:172)
at org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:131)
... 12 more
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:220)
敲重點：Problem binding to [0.0.0.0:8040]......Address already in use
$ netstat anp | grep 8040 查看該端口并未被占用哼绑。如果被占用了就kill -9 {PROCESS_ID}
$ ./sbin/start-yarn.sh 在該節(jié)點單獨重啟服務(wù)，啟動成功碉咆。
問題解決了，追蹤溯源了一下蛀恩，發(fā)現(xiàn)該節(jié)點經(jīng)常出現(xiàn)問題pdsh@blockchain-004: blockchain-003: ssh exited with exit code 1疫铜。
推測：節(jié)點與master的ssh連接斷開了，但節(jié)點本地的NodeMagager進程當時仍在運行双谆；此時Master重連上該點的SSH之后壳咕，嘗試重啟該節(jié)點的NodeMagager服務(wù)，造成服務(wù)沖突出錯顽馋。而當我發(fā)現(xiàn)時谓厘，之前的NodeManager也已經(jīng)自動結(jié)束了，造成了“謊報軍情”的假象寸谜。
