k8s-證書到期輪換

場景：kubernetes(1.1.2版本)默認證書有效期是一年是目，到期后需要重新生成

方案一：集群處于不可用狀態(tài)渠欺，手動更新證書和配置文件

重新生成證書

在所有控制平面執(zhí)行以下命令，備份或刪除過期證書

mkdir -p /etc/kubernetes/pki.bak/etcd
cd /etc/kubernetes/pki
mv apiserver* front-proxy-client* /etc/kubernetes/pki.bak
cd /etc/kubernetes/pki/etcd/
mv healthcheck-client* peer* server* /etc/kubernetes/pki.bak/etcd

可以選擇保留ca證書跳纳，ca證書有效期一般為10年面氓，可以重用，不需要刪除

如果選擇刪除所有證書霹疫，這樣的話就需要手動將第一個控制平面新生成的ca證書copy到其它控制平面的對應(yīng)目錄中

ca證書如下:

/etc/kubernetes/pki/ca.crt
/etc/kubernetes/pki/ca.key
/etc/kubernetes/pki/front-proxy-ca.crt
/etc/kubernetes/pki/front-proxy-ca.key
/etc/kubernetes/pki/sa.key
/etc/kubernetes/pki/sa.pub
/etc/kubernetes/pki/etcd/ca.crt
/etc/kubernetes/pki/etcd/ca.key

執(zhí)行以下命令生成新證書

kubeadm alpha phase certs all --config=kubeadm.yaml

其中kubeadm.yaml為初始化集群時使用的配置文件，具體配置如下
（kubeadm config view）

kind: ClusterConfiguration
apiVersion: kubeadm.k8s.io/v1beta1
controlPlaneEndpoint: "192.168.100.170:6443"
apiServer:
  certSANS:
  - master1
  - master2
  - master3
  - 192.168.100.170
  - 192.168.100.171
  - 192.168.100.181
  - 192.168.100.191

重新生成配置文件

在每個控制平面節(jié)點執(zhí)行以下命令综芥，備份并重新生成配置文件丽蝎，包括admin.conf、controller-manager.conf膀藐、kubelet.conf屠阻、scheduler.conf

cd /etc/kubernetes
mkdir conf.bak 
mv *.conf conf.bak/
kubeadm alpha phase kubeconfig all --config kubeadm.yaml

在每個worker節(jié)點執(zhí)行以下命令，備份重新生成kubelet的配置文件

cd /etc/kubernetes
mkdir conf.bak 
mv *.conf conf.bak/
kubeadm alpha kubeconfig user --org system:nodes --client-name system:node:$(hostname) --config kubeadm.yaml > /etc/kubernetes/kubelet.conf

注意额各，一般來說worker節(jié)點上只有ca.crt證書国觉，沒有ca.key，生成配置文件時會報錯虾啦，這時需要從控制平面將ca.key拷貝過來麻诀，配置文件生成完畢后即可刪除。上述命令在某版本執(zhí)行時報錯可以改寫為：kubeadm alpha kubeconfig user --org system:nodes --client-name system:node:$(hostname) --apiserver-advertise-address 192.168.100.170 > /etc/kubernetes/kubelet.conf

在每個worker節(jié)點執(zhí)行以下命令傲醉，備份或刪除過期的證書（kubelet啟動時會自動生成pki目錄及證書）蝇闭，然后重啟kubelet。

mv /var/lib/kubelet/pki /var/lib/kubelet/pki.bak
systemctl daemon-reload && systemctl restart kubelet

注意：命令執(zhí)行過程中可能有配置文件版本與kubeadm不配套的情況硬毕，這時候按提示升級配置文件即可呻引，如下

kubeadm config migrate --old-config old.yaml --new-config new.yaml

最后重啟管理部件pod，包括etcd吐咳，apiserver逻悠，controller-manager，scheduler韭脊。如果集群不可用蹂风，可以到對應(yīng)節(jié)點上通過docker rm 的方式刪除容器，kubelet會自動拉起新的pod

踩坑：
etcd證書使用本地外部yaml生成
kubeadm init phase certs all --config=/home/kubernetes/config_1.13_local_etcd.yaml

當出現(xiàn) Unable to connect to the server: x509: certificate has expired or is not yet valid

image.png

mv ~/.kube/config ~/.kube/config.bak && cp /etc/kubernetes/admin.conf ~/.kube/config

方案二：（未實踐）如果集群正常使用中乾蓬，則可以使用kubeadm進行證書輪換

檢查證書是否過期

root@master1:~/.kube# kubeadm alpha certs check-expiration
CERTIFICATE                EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
admin.conf                 Dec 30, 2020 02:43 UTC   364d            no      
apiserver                  Dec 30, 2020 02:25 UTC   364d            no      
apiserver-etcd-client      Dec 30, 2020 02:25 UTC   364d            no      
apiserver-kubelet-client   Dec 30, 2020 02:25 UTC   364d            no      
controller-manager.conf    Jul 07, 2020 08:16 UTC   189d            no      
etcd-healthcheck-client    Jul 04, 2020 09:26 UTC   186d            no      
etcd-peer                  Jul 04, 2020 09:26 UTC   186d            no      
etcd-server                Jul 04, 2020 09:26 UTC   186d            no      
front-proxy-client         Dec 30, 2020 02:25 UTC   364d            no      
scheduler.conf             Jul 07, 2020 08:16 UTC   189d            no

證書輪換

# Step 1): Backup old certs and kubeconfigs
mkdir /etc/kubernetes.bak
cp -r /etc/kubernetes/pki/ /etc/kubernetes.bak
cp /etc/kubernetes/*.conf /etc/kubernetes.bak

# Step 2): Renew all certs
kubeadm alpha certs renew all --config kubeadm.yaml

# Step 3): Renew all kubeconfigs
kubeadm alpha kubeconfig user --client-name=admin
kubeadm alpha kubeconfig user --org system:masters --client-name kubernetes-admin  > /etc/kubernetes/admin.conf
kubeadm alpha kubeconfig user --client-name system:kube-controller-manager > /etc/kubernetes/controller-manager.conf
kubeadm alpha kubeconfig user --org system:nodes --client-name system:node:$(hostname) > /etc/kubernetes/kubelet.conf
kubeadm alpha kubeconfig user --client-name system:kube-scheduler > /etc/kubernetes/scheduler.conf

# Another way to renew kubeconfigs
# kubeadm init phase kubeconfig all --config kubeadm.yaml

# Step 4): Copy certs/kubeconfigs and restart Kubernetes services