很久沒開的k8s測試環(huán)境设凹，今天打開發(fā)現(xiàn)在master節(jié)點(diǎn)查看node發(fā)現(xiàn)node2 notready 狀態(tài)
在node2節(jié)點(diǎn)查看發(fā)現(xiàn)kubelet停止運(yùn)行了
kubelet報錯：
part of the existing bootstrap client certificate is expired: 2022-06-04
通過查看/etc/kubernetes/kubelet.conf 發(fā)現(xiàn)證書路徑/var/lib/kubelet/pki/kubelet-client-current.pem
cat /etc/kubernetes/kubelet.conf
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1......UtLS0tLQo=
server: https://192.168.100.201:6443
name: default-cluster
contexts:
- context:
cluster: default-cluster
namespace: default
user: default-auth
name: default-context
current-context: default-context
kind: Config
preferences: {}
users:
- name: default-auth
user:
client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
client-key: /var/lib/kubelet/pki/kubelet-client-current.pem
然后切換到/var/lib/kubelet/pki/ 路徑下查看證書日期
cd /var/lib/kubelet/pki
ll
總用量 20
-rw------- 1 root root 1061 9月 14 2020 kubelet-client-2020-09-14-18-00-01.pem
-rw------- 1 root root 1061 6月 4 2021 kubelet-client-2021-06-04-19-03-23.pem
-rw------- 1 root root 1066 6月 10 11:00 kubelet-client-2022-06-10-11-00-15.pem
lrwxrwxrwx 1 root root 59 6月 10 11:00 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2021-06-04-19-03-23.pem
-rw-r--r-- 1 root root 2144 9月 14 2020 kubelet.crt
-rw------- 1 root root 1679 9月 14 2020 kubelet.key
可以看出kubelet-client-current.pem指向的是kubelet-client-2021-06-04-19-03-23.pem 現(xiàn)在是2022-06-10 所以證書已經(jīng)過期了。
在node2上查看證書有效期
# openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -text | grep Not
Not Before: Jun 4 10:58:23 2021 GMT
Not After : Jun 4 10:58:23 2022 GMT
由于我的 master節(jié)點(diǎn)和node1節(jié)點(diǎn)都正常；
我可以用之前的kubeadm.yaml配置文件重新生成下證書
#備份之前的證書
# cp -rp /etc/kubernetes /etc/kubernetes.bak
#生成新的證書
# kubeadm alpha certs renew all --config=kubeadm.yaml
W0610 09:24:36.851093 26346 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
#備份之前的配置文件
# mkdir /root/backconf
# mv /etc/kubernetes/*.conf /root/backconf/
# ll backconf/
總用量 32
-rw------- 1 root root 5451 6月 10 09:24 admin.conf
-rw------- 1 root root 5491 6月 10 09:24 controller-manager.conf
-rw------- 1 root root 5463 9月 1 2021 kubelet.conf
-rw------- 1 root root 5439 6月 10 09:24 scheduler.conf
#重新生成配置文件
# kubeadm init phase kubeconfig all --config kubeadm.yaml
W0610 09:26:59.426236 27497 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
# ll /etc/kubernetes/
總用量 52
-rw------- 1 root root 5451 6月 10 09:27 admin.conf
-rw-r--r-- 1 root root 1025 3月 23 2021 ca.crt
-rw-r--r-- 1 root root 3117 3月 23 2021 cert.pfx
-rw-r--r-- 1 root root 1082 3月 23 2021 client.crt
-rw-r--r-- 1 root root 1679 3月 23 2021 client.key
-rw------- 1 root root 5487 6月 10 09:27 controller-manager.conf
-rw------- 1 root root 5459 6月 10 09:27 kubelet.conf
drwxr-xr-x 2 root root 113 10月 6 2021 manifests
drwxr-xr-x 3 root root 4096 9月 14 2020 pki
-rw------- 1 root root 5439 6月 10 09:27 scheduler.conf
# 將新生成的admin.conf文件覆蓋掉.kube/config文件:
mv $HOME/.kube/config $HOME/.kube/config.old
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
chmod 644 $HOME/.kube/config
# 重啟kube-apiserver,kube-controller,kube-scheduler,etcd這4個容器：(一定要ps -a要不有可能服務(wù)容器沒啟動)
# docker ps -a | grep -v pause | grep -E "etcd|scheduler|controller|apiserver" | awk '{print $1}' | awk '{print "docker","restart",$1}' | bash
# 各節(jié)點(diǎn)重啟kubelet或相關(guān)組件:
systemctl restart kubelet
master節(jié)點(diǎn)就更新完成了护赊，然后獲取token在更新slave節(jié)點(diǎn)時要用
# kubeadm token create --print-join-command
W0610 09:40:30.975578 2435 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
kubeadm join 192.168.100.201:6443 --token 6co5f1.g8wnog41jopfchp8 --discovery-token-ca-cert-hash sha256:8adf630dbe900681db88950f0877faa7be4308f6fd837029ab7e9e41dd0eafd6
# kubeadm token list
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
6co5f1.g8wnog41jopfchp8 23h 2022-06-11T09:40:31+08:00 authentication,signing <none> system:bootstrappers:kubeadm:default-node-token
node節(jié)點(diǎn)添加進(jìn)集群（需刪除原先kubelet配置文件，否則加入失斝蚶鳌）
先備份下配置文件的存放目錄
cp -r /etc/kubernetes /etc/kubernetes.bak
# ll /etc/kubernetes*
/etc/kubernetes:
總用量 4
-rw------- 1 root root 1856 9月 14 2020 kubelet.conf
drwxr-xr-x 2 root root 6 4月 9 2020 manifests
drwxr-xr-x 2 root root 20 9月 14 2020 pki
/etc/kubernetes.bak:
總用量 4
-rw------- 1 root root 1856 6月 10 10:58 kubelet.conf
drwxr-xr-x 2 root root 6 6月 10 10:58 manifests
drwxr-xr-x 2 root root 20 6月 10 10:58 pki
然后刪除舊的kubelet配置文件
# rm -rf /etc/kubernetes/kubelet.conf
# rm -rf /etc/kubernetes/pki/ca.crt
# rm -rf /etc/kubernetes/bootstrap-kubelet.conf #這個文件我沒有
# systemctl stop kubelet
# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /usr/lib/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: inactive (dead) since 五 2022-06-10 09:38:04 CST; 1h 20min ago
Docs: https://kubernetes.io/docs/
Process: 31448 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=0/SUCCESS)
Main PID: 31448 (code=exited, status=0/SUCCESS)
6月 10 09:37:59 node2 kubelet[31448]: E0610 09:37:59.469934 31448 reflector.go:178] object-"loki"/"loki": Failed to list *v1.Secret: secrets "loki" is forb...this object
6月 10 09:37:59 node2 kubelet[31448]: W0610 09:37:59.676710 31448 status_manager.go:572] Failed to update status for pod "loki-0_loki(e0ea4379-7e48-4107-83...\"Initializ
6月 10 09:38:00 node2 kubelet[31448]: W0610 09:38:00.077588 31448 status_manager.go:572] Failed to update status for pod "sentinel-0_default(49b3d865-37ae-...type\":\"In
6月 10 09:38:00 node2 kubelet[31448]: W0610 09:38:00.476110 31448 status_manager.go:572] Failed to update status for pod "usercenter-deployment-7bf4744f58-...ementOrder/
6月 10 09:38:00 node2 kubelet[31448]: W0610 09:38:00.877862 31448 status_manager.go:572] Failed to update status for pod "getaway-deployment-6595fb8444-ztf...ntOrder/con
6月 10 09:38:02 node2 kubelet[31448]: I0610 09:38:02.721843 31448 kubelet_node_status.go:294] Setting node annotation to enable volume controller attach/detach
6月 10 09:38:02 node2 kubelet[31448]: I0610 09:38:02.849726 31448 kubelet_node_status.go:70] Attempting to register node node2
6月 10 09:38:02 node2 kubelet[31448]: E0610 09:38:02.859581 31448 kubelet_node_status.go:92] Unable to register node "node2" with API server: nodes "node2"...ode "node2"
6月 10 09:38:04 node2 systemd[1]: Stopping kubelet: The Kubernetes Node Agent...
6月 10 09:38:04 node2 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Hint: Some lines were ellipsized, use -l to show in full.
node2重新加入集群
# kubeadm join 192.168.100.201:6443 --token 6co5f1.g8wnog41jopfchp8 --discovery-token-ca-cert-hash sha256:8adf630dbe900681db88950f0877faa7be4308f6fd837029ab7e9e41dd0eafd6
W0610 11:00:11.849573 5754 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.18" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
驗(yàn)證結(jié)果
[root@master ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready master 633d v1.18.1
node1 Ready <none> 633d v1.18.1
node2 Ready <none> 633d v1.18.1
[root@master ~]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-7ff77c879f-629sv 1/1 Running 15 633d
coredns-7ff77c879f-hk25m 1/1 Running 15 633d
default-http-backend-55fb564b-rrddj 1/1 Running 3 146d
etcd-master 1/1 Running 15 633d
kube-apiserver-master 1/1 Running 8 386d
kube-controller-manager-master 1/1 Running 7 281d
kube-flannel-ds-amd64-g885t 1/1 Running 15 633d
kube-flannel-ds-amd64-nm5xp 1/1 Running 14 633d
kube-flannel-ds-amd64-zd56s 1/1 Running 15 633d
kube-proxy-rdf9s 1/1 Running 16 633d
kube-proxy-rsm5n 1/1 Running 14 633d
kube-proxy-wc7zr 1/1 Running 15 633d
kube-scheduler-master 1/1 Running 17 633d
kube-state-metrics-99d76dd5d-srlvt 1/1 Running 8 300d
metrics-server-7b75fd6bfb-4prml 1/1 Running 9 386d
nginx-ingress-controller-5cf88d6db5-mqp8c 1/1 Running 3 146d
