kubelet無(wú)法啟動(dòng)排查

問(wèn)題描述

使用ansible安裝Kubernetes安拟，最后出現(xiàn)如所示報(bào)錯(cuò)，提示kubelet啟動(dòng)異常

TASK [kube-node : 輪詢(xún)等待kube-proxy啟動(dòng)] ****************************************************************************
changed: [172.16.255.136]
changed: [172.16.255.132]
FAILED - RETRYING: 輪詢(xún)等待kubelet啟動(dòng) (4 retries left).
FAILED - RETRYING: 輪詢(xún)等待kubelet啟動(dòng) (4 retries left).
FAILED - RETRYING: 輪詢(xún)等待kubelet啟動(dòng) (3 retries left).
FAILED - RETRYING: 輪詢(xún)等待kubelet啟動(dòng) (3 retries left).
FAILED - RETRYING: 輪詢(xún)等待kubelet啟動(dòng) (2 retries left).
FAILED - RETRYING: 輪詢(xún)等待kubelet啟動(dòng) (2 retries left).
FAILED - RETRYING: 輪詢(xún)等待kubelet啟動(dòng) (1 retries left).
FAILED - RETRYING: 輪詢(xún)等待kubelet啟動(dòng) (1 retries left).

TASK [kube-node : 輪詢(xún)等待kubelet啟動(dòng)] *******************************************************************************
fatal: [172.16.255.136]: FAILED! => {"attempts": 4, "changed": true, "cmd": "systemctl is-active kubelet.service", "delta": "0:00:00.007244", "end": "2022-12-07 23:09:47.327728", "msg": "non-zero return code", "rc": 3, "start": "2022-12-07 23:09:47.320484", "stderr": "", "stderr_lines": [], "stdout": "activating", "stdout_lines": ["activating"]}
fatal: [172.16.255.132]: FAILED! => {"attempts": 4, "changed": true, "cmd": "systemctl is-active kubelet.service", "delta": "0:00:00.003806", "end": "2022-12-07 23:09:47.492355", "msg": "non-zero return code", "rc": 3, "start": "2022-12-07 23:09:47.488549", "stderr": "", "stderr_lines": [], "stdout": "activating", "stdout_lines": ["activating"]}

問(wèn)題排查

檢查kubelet狀態(tài)泽示，顯示沒(méi)啟動(dòng)成功

root@iZ2vc2h2j9l2p8zqnwy6zoZ:~# systemctl status kubelet
● kubelet.service - Kubernetes Kubelet
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Wed 2022-12-07 23:14:26 CST; 3s ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
  Process: 30668 ExecStart=/usr/bin/kubelet --config=/var/lib/kubelet/config.yaml --container-runtime-endpoint=unix:///run/containerd/containerd.sock --hostname-override=172.16.255.132 --kubeconfig=/etc/kubernetes/kubelet.kubeconfig --root-dir=/var/lib/kubelet --v=2 (code=exited, status=1/FAILURE)
 Main PID: 30668 (code=exited, status=1/FAILURE)

使用journalctl -u kubelet --no-pager 查看啟動(dòng)報(bào)錯(cuò)日志

Dec 07 23:50:21 iZ2vc2h2j9l2p8zqnwy6zoZ kubelet[24786]: E1207 23:50:21.347929   24786 remote_runtime.go:168] "Version from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
Dec 07 23:50:21 iZ2vc2h2j9l2p8zqnwy6zoZ kubelet[24786]: E1207 23:50:21.348041   24786 kuberuntime_manager.go:225] "Get runtime version failed" err="get remote runtime typed version failed: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"
Dec 07 23:50:21 iZ2vc2h2j9l2p8zqnwy6zoZ kubelet[24786]: Error: failed to run Kubelet: failed to create kubelet: get remote runtime typed version failed: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService

將報(bào)錯(cuò)在網(wǎng)上搜索看到有人說(shuō)是containerd的問(wèn)題，確認(rèn)containerd狀態(tài)，

root@iZ2vc2h2j9l2p8zqnwy6zoZ:~# systemctl status containerd
● containerd.service - containerd container runtime
   Loaded: loaded (/lib/systemd/system/containerd.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2022-12-06 01:18:16 CST; 1 day 22h ago
     Docs: https://containerd.io
 Main PID: 3917 (containerd)
    Tasks: 34
   CGroup: /system.slice/containerd.service
           ├─ 3917 /usr/bin/containerd
           ├─11401 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 9e95bf63c2a1e18bf6ddc2b79b840931e443ebda1d6a03b0f3d2c4c4f3b16ecd -address /run/containerd/containerd.sock
           └─14318 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 1add85e11ac29505b1155c558b118d5cc2e3fa1e347414e1c4816bdc43bb7427 -address /run/containerd/containerd.sock

Dec 07 22:43:38 iZ2vc2h2j9l2p8zqnwy6zoZ containerd[3917]: time="2022-12-07T22:43:38.155622905+08:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Dec 07 22:43:38 iZ2vc2h2j9l2p8zqnwy6zoZ containerd[3917]: time="2022-12-07T22:43:38.155722663+08:00" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/d8021addb602a9977f6115ae6d305c30fb23690f68553e5cca512a3fc63a0bd2 pid=12434 runtime=io.containerd.runc.v2
Dec 07 22:43:38 iZ2vc2h2j9l2p8zqnwy6zoZ containerd[3917]: time="2022-12-07T22:43:38.600183434+08:00" level=info msg="shim disconnected" id=d8021addb602a9977f6115ae6d305c30fb23690f68553e5cca512a3fc63a0bd2
Dec 07 22:43:38 iZ2vc2h2j9l2p8zqnwy6zoZ containerd[3917]: time="2022-12-07T22:43:38.600230978+08:00" level=warning msg="cleaning up after shim disconnected" id=d8021addb602a9977f6115ae6d305c30fb23690f68553e5cca512a3fc63a0bd2 namespace=moby
Dec 07 22:43:38 iZ2vc2h2j9l2p8zqnwy6zoZ containerd[3917]: time="2022-12-07T22:43:38.600245849+08:00" level=info msg="cleaning up dead shim"
Dec 07 22:43:38 iZ2vc2h2j9l2p8zqnwy6zoZ containerd[3917]: time="2022-12-07T22:43:38.605221254+08:00" level=warning msg="cleanup warnings time=\"2022-12-07T22:43:38+08:00\" level=info msg=\"starting signal loop\" namespace=moby pid=12610 runtime=io.containerd.runc.v2\n"
Dec 07 22:52:39 iZ2vc2h2j9l2p8zqnwy6zoZ containerd[3917]: time="2022-12-07T22:52:39.669074691+08:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Dec 07 22:52:39 iZ2vc2h2j9l2p8zqnwy6zoZ containerd[3917]: time="2022-12-07T22:52:39.669118722+08:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Dec 07 22:52:39 iZ2vc2h2j9l2p8zqnwy6zoZ containerd[3917]: time="2022-12-07T22:52:39.669135217+08:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Dec 07 22:52:39 iZ2vc2h2j9l2p8zqnwy6zoZ containerd[3917]: time="2022-12-07T22:52:39.669236032+08:00" level=info msg="starting signal loop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/1add85e11ac29505b1155c558b118d5cc2e3fa1e347414e1c4816bdc43bb7427 pid=14318 runtime=io.containerd.runc.v2

看有人說(shuō)和配置文件/etc/containerd/config.toml中的disabled_plugins = ["cri"]有關(guān)涂乌，詳情參見(jiàn)https://github.com/containerd/containerd/issues/4581
移除/etc/containerd/config.toml配置文件

root@iZ2vc2h2j9l2p8zqnwy6zoZ:~# grep "disabled_plugins" /etc/containerd/config.toml 
disabled_plugins = ["cri"]
root@iZ2vc2h2j9l2p8zqnwy6zoZ:~# 
root@iZ2vc2h2j9l2p8zqnwy6zoZ:~# mv /etc/containerd/config.toml /tmp/

重啟 containerd和kubelet，kubelet啟動(dòng)成功

root@iZ2vc2h2j9l2p8zqnwy6zoZ:~# systemctl restart containerd
root@iZ2vc2h2j9l2p8zqnwy6zoZ:~# systemctl status containerd
● containerd.service - containerd container runtime
   Loaded: loaded (/lib/systemd/system/containerd.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2022-12-08 00:23:37 CST; 10s ago
     Docs: https://containerd.io
  Process: 16220 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
 Main PID: 16226 (containerd)
    Tasks: 33
   CGroup: /system.slice/containerd.service
           ├─16226 /usr/bin/containerd
           ├─16419 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 1add85e11ac29505b1155c558b118d5cc2e3fa1e347414e1c4816bdc43bb7427 -address /run/containe
           └─16428 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 9e95bf63c2a1e18bf6ddc2b79b840931e443ebda1d6a03b0f3d2c4c4f3b16ecd -address /run/containe

Dec 08 00:23:37 iZ2vc2h2j9l2p8zqnwy6zoZ systemd[1]: Started containerd container runtime.
Dec 08 00:23:37 iZ2vc2h2j9l2p8zqnwy6zoZ containerd[16226]: time="2022-12-08T00:23:37.930877956+08:00" level=info msg="loading plugin \"io.containerd.event.v1.p
Dec 08 00:23:37 iZ2vc2h2j9l2p8zqnwy6zoZ containerd[16226]: time="2022-12-08T00:23:37.930929990+08:00" level=info msg="loading plugin \"io.containerd.internal.v
Dec 08 00:23:37 iZ2vc2h2j9l2p8zqnwy6zoZ containerd[16226]: time="2022-12-08T00:23:37.930942089+08:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.t
Dec 08 00:23:37 iZ2vc2h2j9l2p8zqnwy6zoZ containerd[16226]: time="2022-12-08T00:23:37.931105240+08:00" level=info msg="starting signal loop" namespace=moby path
Dec 08 00:23:37 iZ2vc2h2j9l2p8zqnwy6zoZ containerd[16226]: time="2022-12-08T00:23:37.932520723+08:00" level=info msg="loading plugin \"io.containerd.event.v1.p
Dec 08 00:23:37 iZ2vc2h2j9l2p8zqnwy6zoZ containerd[16226]: time="2022-12-08T00:23:37.932563830+08:00" level=info msg="loading plugin \"io.containerd.internal.v
Dec 08 00:23:37 iZ2vc2h2j9l2p8zqnwy6zoZ containerd[16226]: time="2022-12-08T00:23:37.932582235+08:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.t
Dec 08 00:23:37 iZ2vc2h2j9l2p8zqnwy6zoZ containerd[16226]: time="2022-12-08T00:23:37.932692186+08:00" level=info msg="starting signal loop" namespace=moby path
Dec 08 00:23:41 iZ2vc2h2j9l2p8zqnwy6zoZ containerd[16226]: time="2022-12-08T00:23:41.811347380+08:00" level=info msg="No cni config template is specified, wait

root@iZ2vc2h2j9l2p8zqnwy6zoZ:~# 
root@iZ2vc2h2j9l2p8zqnwy6zoZ:~# systemctl restart kubelet
root@iZ2vc2h2j9l2p8zqnwy6zoZ:~# systemctl status kubelet
● kubelet.service - Kubernetes Kubelet
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2022-12-08 00:24:06 CST; 10s ago
     Docs: https://github.com/GoogleCloudPlatform/kubernetes
 Main PID: 16852 (kubelet)
    Tasks: 15 (limit: 4915)
   CGroup: /system.slice/kubelet.service
           └─16852 /usr/bin/kubelet --config=/var/lib/kubelet/config.yaml --container-runtime-endpoint=unix:///run/containerd/containerd.sock --hostname-overri

Dec 08 00:24:06 iZ2vc2h2j9l2p8zqnwy6zoZ kubelet[16852]: I1208 00:24:06.371586   16852 kubelet_network.go:60] "Updating Pod CIDR" originalPodCIDR="" newPodCIDR=
Dec 08 00:24:06 iZ2vc2h2j9l2p8zqnwy6zoZ kubelet[16852]: I1208 00:24:06.371664   16852 kubelet_node_status.go:563] "Recording event message for node" node="172.
Dec 08 00:24:06 iZ2vc2h2j9l2p8zqnwy6zoZ kubelet[16852]: I1208 00:24:06.371756   16852 kubelet_node_status.go:563] "Recording event message for node" node="172.
Dec 08 00:24:06 iZ2vc2h2j9l2p8zqnwy6zoZ kubelet[16852]: I1208 00:24:06.371770   16852 kubelet_node_status.go:563] "Recording event message for node" node="172.
Dec 08 00:24:06 iZ2vc2h2j9l2p8zqnwy6zoZ kubelet[16852]: I1208 00:24:06.371788   16852 kubelet_node_status.go:70] "Attempting to register node" node="172.16.255
Dec 08 00:24:06 iZ2vc2h2j9l2p8zqnwy6zoZ kubelet[16852]: I1208 00:24:06.378384   16852 kubelet_node_status.go:108] "Node was previously registered" node="172.16
Dec 08 00:24:06 iZ2vc2h2j9l2p8zqnwy6zoZ kubelet[16852]: I1208 00:24:06.378430   16852 kubelet_node_status.go:73] "Successfully registered node" node="172.16.25
Dec 08 00:24:07 iZ2vc2h2j9l2p8zqnwy6zoZ kubelet[16852]: I1208 00:24:07.266364   16852 apiserver.go:52] "Watching apiserver"
Dec 08 00:24:07 iZ2vc2h2j9l2p8zqnwy6zoZ kubelet[16852]: I1208 00:24:07.269182   16852 kubelet.go:2072] "SyncLoop ADD" source="api" pods=[]
Dec 08 00:24:07 iZ2vc2h2j9l2p8zqnwy6zoZ kubelet[16852]: I1208 00:24:07.274953   16852 reconciler.go:157] "Reconciler: start to sync state"

問(wèn)題原因

參見(jiàn)https://github.com/containerd/containerd/issues/4581

解決辦法

mv /etc/containerd/config.toml /tmp
systemctl restart containerd
systemctl restart kubelet 或者 kubeadm init

最后編輯于：2022.12.10 00:50:08

?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者

人面猴
序言：七十年代末英岭，一起剝皮案震驚了整個(gè)濱河市湾盒，隨后出現(xiàn)的幾起案子，更是在濱河造成了極大的恐慌诅妹，老刑警劉巖罚勾，帶你破解...
沈念sama閱讀 218,386評(píng)論 6贊 506
死咒
序言：濱河連續(xù)發(fā)生了三起死亡事件毅人，死亡現(xiàn)場(chǎng)離奇詭異，居然都是意外死亡尖殃，警方通過(guò)查閱死者的電腦和手機(jī)丈莺，發(fā)現(xiàn)死者居然都...
沈念sama閱讀 93,142評(píng)論 3贊 394
救了他兩次的神仙讓他今天三更去死
文/潘曉璐我一進(jìn)店門(mén)，熙熙樓的掌柜王于貴愁眉苦臉地迎上來(lái)分衫，“玉大人场刑，你說(shuō)我怎么就攤上這事◎秸剑” “怎么了牵现？”我有些...
開(kāi)封第一講書(shū)人閱讀 164,704評(píng)論 0贊 353
道士緝兇錄：失蹤的賣(mài)姜人
文/不壞的土叔我叫張陵，是天一觀的道長(zhǎng)邀桑。經(jīng)常有香客問(wèn)我瞎疼，道長(zhǎng)，這世上最難降的妖魔是什么壁畸？我笑而不...
開(kāi)封第一講書(shū)人閱讀 58,702評(píng)論 1贊 294
?港島之戀（遺憾婚禮）
正文為了忘掉前任贼急，我火速辦了婚禮，結(jié)果婚禮上捏萍，老公的妹妹穿的比我還像新娘太抓。我一直安慰自己，他們只是感情好令杈，可當(dāng)我...
茶點(diǎn)故事閱讀 67,716評(píng)論 6贊 392
惡毒庶女頂嫁案：這布局不是一般人想出來(lái)的
文/花漫我一把揭開(kāi)白布走敌。她就那樣靜靜地躺著，像睡著了一般逗噩。火紅的嫁衣襯著肌膚如雪掉丽。梳的紋絲不亂的頭發(fā)上，一...
開(kāi)封第一講書(shū)人閱讀 51,573評(píng)論 1贊 305
城市分裂傳說(shuō)
那天异雁，我揣著相機(jī)與錄音捶障，去河邊找鬼。笑死纲刀，一個(gè)胖子當(dāng)著我的面吹牛项炼，可吹牛的內(nèi)容都是我干的。我是一名探鬼主播示绊，決...
沈念sama閱讀 40,314評(píng)論 3贊 418
雙鴛鴦連環(huán)套：你想象不到人心有多黑
文/蒼蘭香墨我猛地睜開(kāi)眼芥挣，長(zhǎng)吁一口氣：“原來(lái)是場(chǎng)噩夢(mèng)啊……” “哼！你這毒婦竟也來(lái)了耻台？” 一聲冷哼從身側(cè)響起空免，我...
開(kāi)封第一講書(shū)人閱讀 39,230評(píng)論 0贊 276
萬(wàn)榮殺人案實(shí)錄
序言：老撾萬(wàn)榮一對(duì)情侶失蹤，失蹤者是張志新（化名）和其女友劉穎盆耽，沒(méi)想到半個(gè)月后蹋砚，有當(dāng)?shù)厝嗽跇?shù)林里發(fā)現(xiàn)了一具尸體扼菠，經(jīng)...
沈念sama閱讀 45,680評(píng)論 1贊 314
?護(hù)林員之死
正文獨(dú)居荒郊野嶺守林人離奇死亡，尸身上長(zhǎng)有42處帶血的膿包…… 初始之章·張勛以下內(nèi)容為張勛視角年9月15日...
茶點(diǎn)故事閱讀 37,873評(píng)論 3贊 336
?白月光啟示錄
正文我和宋清朗相戀三年坝咐，在試婚紗的時(shí)候發(fā)現(xiàn)自己被綠了循榆。大學(xué)時(shí)的朋友給我發(fā)了我未婚夫和他白月光在一起吃飯的照片。...
茶點(diǎn)故事閱讀 39,991評(píng)論 1贊 348
活死人
序言：一個(gè)原本活蹦亂跳的男人離奇死亡墨坚，死狀恐怖秧饮，靈堂內(nèi)的尸體忽然破棺而出，到底是詐尸還是另有隱情泽篮，我是刑警寧澤盗尸，帶...
沈念sama閱讀 35,706評(píng)論 5贊 346
?日本核電站爆炸內(nèi)幕
正文年R本政府宣布，位于F島的核電站帽撑，受9級(jí)特大地震影響泼各，放射性物質(zhì)發(fā)生泄漏。R本人自食惡果不足惜亏拉，卻給世界環(huán)境...
茶點(diǎn)故事閱讀 41,329評(píng)論 3贊 330
男人毒藥：我在死后第九天來(lái)索命
文/蒙蒙一扣蜻、第九天我趴在偏房一處隱蔽的房頂上張望。院中可真熱鬧及塘，春花似錦莽使、人聲如沸。這莊子的主人今日做“春日...
開(kāi)封第一講書(shū)人閱讀 31,910評(píng)論 0贊 22
一樁弒父案芳肌，背后竟有這般陰謀
文/蒼蘭香墨我抬頭看了看天上的太陽(yáng)。三九已至味咳，卻和暖如春庇勃，著一層夾襖步出監(jiān)牢的瞬間檬嘀，已是汗流浹背槽驶。一陣腳步聲響...
開(kāi)封第一講書(shū)人閱讀 33,038評(píng)論 1贊 270
情欲美人皮
我被黑心中介騙來(lái)泰國(guó)打工斥黑，沒(méi)想到剛下飛機(jī)就差點(diǎn)兒被人妖公主榨干…… 1. 我叫王不留硕舆，地道東北人眼坏。一個(gè)月前我還...
沈念sama閱讀 48,158評(píng)論 3贊 370
代替公主和親
正文我出身青樓誓焦，卻偏偏與公主長(zhǎng)得像团甲，于是被迫代替她去往敵國(guó)和親现拒。傳聞我的和親對(duì)象是個(gè)殘疾皇子痴晦，可洞房花燭夜當(dāng)晚...
茶點(diǎn)故事閱讀 44,941評(píng)論 2贊 355

kubelet無(wú)法啟動(dòng)排查

問(wèn)題描述

問(wèn)題排查

問(wèn)題原因

解決辦法

推薦閱讀更多精彩內(nèi)容