Note: 本文基于使用kubeadm搭建的kubernetes集群進(jìn)行講解
第一步:獲取證書簽發(fā)信息
-
方式一:通過原有證書進(jìn)行獲取相關(guān)信息
-
獲取apiserver簽發(fā)信息
$ openssl x509 -noout -text -in /etc/kubernetes/pki/apiserver.crt ...... X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Extended Key Usage: TLS Web Server Authentication X509v3 Subject Alternative Name: DNS:localhost, DNS:node1, DNS:node2, DNS:node3, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:127.0.0.1, IP Address:10.96.0.1, IP Address:192.168.12.10, IP Address:192.168.12.11, IP Address:192.168.12.12, IP Address:192.168.12.13 ......
-
從上面輸出信息可知簽發(fā)的DNS和IP詳情
DNS.1 = localhost # 各 master 節(jié)點 nodeName DNS.2 = node1 DNS.3 = node2 DNS.4 = node3 # kubenertes svc 地址 DNS.5 = kubernetes DNS.6 = kubernetes.default DNS.7 = kubernetes.default.svc # 帶 dns domain 的 kubenertes svc 地址 DNS.8 = kubernetes.default.svc.cluster.local IP.6 = 127.0.0.1 # kubenertes svc 的 clusterip IP.5 = 10.96.0.1 # api-server VIP 地址 IP.4 = 192.168.12.10 # 各 master 節(jié)點 IP IP.1 = 192.168.12.11 IP.2 = 192.168.12.12 IP.3 = 192.168.12.13
-
-
方式二:自行統(tǒng)計各項信息
-
在創(chuàng)建證書之前需要獲取到以下信息盟广,在簽發(fā)證書的時候會用到它:
- 各master節(jié)點IP
- 各master節(jié)點nodeName
- 如果有設(shè)置apiserver負(fù)載均衡則需要VIP丰歌,否則請忽略
- kubernetes集群dns domain
- kubenertes.default.svc的clusterip
-
本文以下面集群信息為例:
-
節(jié)點信息:
nodeName ip relo slb 192.168.12.10 slb node1 192.168.12.11 master node2 192.168.12.12 master node3 192.168.12.13 master node4 192.168.12.14 worker node5 192.168.12.15 worker -
獲取kubernetes dns domain:
# CoreDNS 查看方法 $ kubectl get cm -n kube-system coredns -o yaml | grep kubernetes kubernetes cluster.local in-addr.arpa ip6.arpa { # 由以上輸出可知dns domain為 cluster.local # kube-dns 查看方法 $ kubectl get deployment -n kube-system kube-dns -o yaml | grep domain - --domain=cluster.local. # 由以上輸出可知dns domain為 cluster.local
-
kubernetes apiserver clusterip
$ kubectl get svc kubernetes -n default NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 280d # 由結(jié)果可知apiserver clusterip
-
-
同理獲取etcd集群證書簽發(fā)的DNS和IP詳情深夯,默認(rèn)etcd證書路徑為
/etc/kubernetes/pki/etcd
第二步:創(chuàng)建證書
Note: 我們只需要在一個節(jié)點上進(jìn)行證書生成本冲,生成的證書分發(fā)到其他節(jié)點即可
-
創(chuàng)建CA服務(wù)端證書簽名請求配置文件
openssl.conf
,內(nèi)容如下王浴,注意替換alt_names_cluster
拥知、alt_names_etcd
域中的值[ req ] default_bits = 2048 default_md = sha256 distinguished_name = req_distinguished_name [req_distinguished_name] [ v3_ca ] basicConstraints = critical, CA:TRUE keyUsage = critical, digitalSignature, keyEncipherment, keyCertSign [ v3_req_server ] basicConstraints = CA:FALSE keyUsage = critical, digitalSignature, keyEncipherment extendedKeyUsage = serverAuth [ v3_req_client ] basicConstraints = CA:FALSE keyUsage = critical, digitalSignature, keyEncipherment extendedKeyUsage = clientAuth [ v3_req_apiserver ] basicConstraints = CA:FALSE keyUsage = critical, digitalSignature, keyEncipherment extendedKeyUsage = serverAuth subjectAltName = @alt_names_cluster [ v3_req_etcd ] basicConstraints = CA:FALSE keyUsage = critical, digitalSignature, keyEncipherment extendedKeyUsage = serverAuth, clientAuth subjectAltName = @alt_names_etcd [ alt_names_cluster ] DNS.1 = localhost DNS.2 = node1 DNS.3 = node2 DNS.4 = node3 DNS.5 = kubernetes DNS.6 = kubernetes.default DNS.7 = kubernetes.default.svc DNS.8 = kubernetes.default.svc.cluster.local IP.1 = 127.0.0.1 IP.2 = 10.96.0.1 IP.3 = 192.168.12.10 IP.4 = 192.168.12.11 IP.5 = 192.168.12.12 IP.6 = 192.168.12.13 [ alt_names_etcd ] DNS.1 = localhost DNS.2 = node1 DNS.3 = node2 DNS.4 = node3 IP.1 = 127.0.0.1 IP.2 = 0:0:0:0:0:0:0:1 IP.3 = 192.168.12.11 IP.4 = 192.168.12.12 IP.5 = 192.168.12.13
-
創(chuàng)建集群 key 與 CA
-
將要創(chuàng)建的 CA
路徑 Common Name 描述 ca.crt,key kubernetes Kubernetes general CA etcd/ca.crt,key kubernetes For all etcd-related functions front-proxy-ca.crt,key kubernetes For the front-end proxy -
要注意 CA 中 CN(Common Name) 與 O(Organization) 等內(nèi)容是會影響Kubernetes組件認(rèn)證的
- CA (Certificate Authority) 是自簽名的根證書,用來簽名后續(xù)創(chuàng)建的其它證書
- CN (Common Name), apiserver 會從證書中提取該字段作為請求的用戶名 (User Name)
- O (Organization), apiserver 會從證書中提取該字段作為請求用戶所屬的組 (Group)
一般CA根證書有效期為10年岩饼,若舉例過期時間還長荚虚,可跳過本節(jié)操作,命令中的
3650
為3650天籍茧,即證書有效期版述。-
kubernetes-ca
openssl genrsa -out ca.key 2048 openssl req -x509 -new -nodes -key ca.key \ -subj "/CN=kubernetes" -config openssl.conf \ -extensions v3_ca -out ca.crt -days 3560
-
etcd-ca
mkdir -p etcd openssl genrsa -out etcd/ca.key 2048 openssl req -x509 -new -nodes -key etcd/ca.key \ -subj "/CN=kubernetes" -config openssl.conf \ -extensions v3_ca -out etcd/ca.crt -days 3560
-
front-proxy-ca
openssl genrsa -out front-proxy-ca.key 2048 openssl req -x509 -new -nodes -key front-proxy-ca.key \ -subj "/CN=kubernetes" -config openssl.conf \ -extensions v3_ca -out front-proxy-ca.crt -days 3560
-
-
創(chuàng)建 Certificates
- 將要創(chuàng)建的 Certificates
Name Key Certificates Common Name Organization etcd/server etcd/server.key etcd/server.crt master etcd/peer etcd/peer.key etcd/peer.crt master etcd/healthcheck-client etcd/healthcheck-client.key etcd/healthcheck-client.crt kube-etcd-healthcheck-client system:masters apiserver-etcd-client apiserver-etcd-client.key apiserver-etcd-client.crt kube-apiserver-etcd-client system:masters apiserver apiserver.key apiserver.crt kube-apiserver apiserver-kubelet-client apiserver-kubelet-client.key apiserver-kubelet-client.crt kube-apiserver-kubelet-client system:masters front-proxy-client front-proxy-client.key front-proxy-client.crt front-proxy-client kube-scheduler kube-scheduler.key kube-scheduler.crt system:kube-scheduler sa(kube-controller-manager) sa.key(sa.pub) kube-controller-manager.crt system:kube-controller-manager admin(kubectl) admin.key admin.crt kubernetes-admin system:masters kubelet kubelet.key kubelet.crt system:node:master system:nodes
- 將要創(chuàng)建的 Certificates
mkdir etcd
cp /etc/kubernetes/pki/etcd/*.key etcd/
cp /etc/kubernetes/pki/etcd/ca.crt etcd/
-
etcd/server
~~openssl genrsa -out etcd/server.key 2048~~ openssl req -new -key etcd/server.key \ -subj "/CN=master" -out etcd/server.csr openssl x509 -in etcd/server.csr -req -CA etcd/ca.crt \ -CAkey etcd/ca.key -CAcreateserial -extensions v3_req_etcd \ -extfile openssl.conf -out etcd/server.crt -days 3560
-
etcd/peer
~~openssl genrsa -out etcd/peer.key 2048~~ openssl req -new -key etcd/peer.key \ -subj "/CN=master" -out etcd/peer.csr openssl x509 -in etcd/peer.csr -req -CA etcd/ca.crt \ -CAkey etcd/ca.key -CAcreateserial -extensions v3_req_etcd \ -extfile openssl.conf -out etcd/peer.crt -days 3560
-
etcd/healthcheck-client
~~openssl genrsa -out etcd/healthcheck-client.key 2048~~ openssl req -new -key etcd/healthcheck-client.key \ -subj "/CN=kube-etcd-healthcheck-client/O=system:masters" \ -out etcd/healthcheck-client.csr openssl x509 -in etcd/healthcheck-client.csr -req -CA etcd/ca.crt \ -CAkey etcd/ca.key -CAcreateserial -extensions v3_req_etcd \ -extfile openssl.conf -out etcd/healthcheck-client.crt -days 3560
cp /etc/kubernetes/pki/*.key .
cp /etc/kubernetes/pki/ca.crt .
cp /etc/kubernetes/pki/front-proxy-ca.crt .
-
apiserver-etcd-client
~~openssl genrsa -out apiserver-etcd-client.key 2048~~ openssl req -new -key apiserver-etcd-client.key \ -subj "/CN=kube-apiserver-etcd-client/O=system:masters" \ -out apiserver-etcd-client.csr openssl x509 -in apiserver-etcd-client.csr -req -CA etcd/ca.crt \ -CAkey etcd/ca.key -CAcreateserial -extensions v3_req_etcd \ -extfile openssl.conf -out apiserver-etcd-client.crt -days 3560
-
apiserver
~~openssl genrsa -out apiserver.key 2048~~ openssl req -new -key apiserver.key \ -subj "/CN=kube-apiserver" -config openssl.conf \ -out apiserver.csr openssl x509 -req -in apiserver.csr -CA ca.crt -CAkey ca.key \ -CAcreateserial -extensions v3_req_apiserver \ -extfile openssl.conf -out apiserver.crt -days 3560
-
apiserver-kubelet-client
~~openssl genrsa -out apiserver-kubelet-client.key 2048~~ openssl req -new -key apiserver-kubelet-client.key \ -subj "/CN=kube-apiserver-kubelet-client/O=system:masters" \ -out apiserver-kubelet-client.csr openssl x509 -req -in apiserver-kubelet-client.csr -CA ca.crt -CAkey ca.key \ -CAcreateserial -extensions v3_req_client \ -extfile openssl.conf -out apiserver-kubelet-client.crt -days 3560
-
front-proxy-client
~~openssl genrsa -out front-proxy-client.key 2048~~ openssl req -new -key front-proxy-client.key \ -subj "/CN=front-proxy-client" \ -out front-proxy-client.csr openssl x509 -req -in front-proxy-client.csr -CA front-proxy-ca.crt -CAkey front-proxy-ca.key \ -CAcreateserial -extensions v3_req_client \ -extfile openssl.conf -out front-proxy-client.crt -days 3560
-
kube-scheduler
openssl genrsa -out kube-scheduler.key 2048 openssl req -new -key kube-scheduler.key \ -subj "/CN=system:kube-scheduler" \ -out kube-scheduler.csr openssl x509 -req -in kube-scheduler.csr -CA ca.crt -CAkey ca.key \ -CAcreateserial -extensions v3_req_client \ -extfile openssl.conf -out kube-scheduler.crt -days 3560
cp /etc/kubernetes/pki/sa.pub .
-
sa(kube-controller-manager)
~~openssl genrsa -out sa.key 2048~~ ~~openssl rsa -in sa.key -pubout -out sa.pub~~ openssl req -new -key sa.key \ -subj "/CN=system:kube-controller-manager" \ -out kube-controller-manager.csr openssl x509 -req -in kube-controller-manager.csr -CA ca.crt -CAkey ca.key \ -CAcreateserial -extensions v3_req_client \ -extfile openssl.conf -out kube-controller-manager.crt -days 3560
-
admin(kubectl)
openssl genrsa -out admin.key 2048 openssl req -new -key admin.key \ -subj "/CN=kubernetes-admin/O=system:masters" \ -out admin.csr openssl x509 -req -in admin.csr -CA ca.crt -CAkey ca.key \ -CAcreateserial -extensions v3_req_client \ -extfile openssl.conf -out admin.crt -days 3560
-
kubelet
openssl genrsa -out kubelet.key 2048 # 此處為 master 節(jié)點 nodeName,每個 master 生成對應(yīng)的證書 openssl req -new -key kubelet.key \ -subj "/CN=system:node:node1/O=system:nodes" \ -out kubelet.csr openssl x509 -req -CA ca.crt -CAkey ca.key \ -CAcreateserial -extensions v3_req_client \ -extfile openssl.conf -days 3560 -in kubelet.csr -out kubelet.crt
第三步:生成kubernetes各組件配置文件并應(yīng)用
-
所要生成的配置文件列表
配置文件名稱 組件證書文件名稱 組件秘鑰文件名稱 根證書文件名稱 admin.conf(kubectl) admin.crt admin.key ca.crt kubelet.conf kubelet.crt kubelet.key ca.crt scheduler.conf kube-scheduler.crt kube-scheduler.key ca.crt controller-manager.conf kube-controller-manager.crt sa.key ca.crt - 操作前請先備份原有配置文件
- 除了
kubelet.conf
文件需注意配置為對應(yīng)節(jié)點的nodeName寞冯,其余配置文件可通用 - 以下操作請先在一臺 master 節(jié)點上操作確認(rèn)沒有問題后再進(jìn)行配置其他節(jié)點
- –certificate-authority:指定根證書
- –client-certificate渴析、–client-key:指定組件證書及秘鑰
- –embed-certs=true:將組件證書內(nèi)容嵌入到生成的配置文件中(不加時,寫入的是證書文件路徑)
-
admin.conf(kubectl)
KUBE_APISERVER="https://192.168.12.10:6443" CLUSTER_NAME="kubernetes" KUBE_USER="kubernetes-admin" KUBE_CERT="admin" KUBE_CONFIG="admin.conf" # 設(shè)置集群參數(shù) kubectl config set-cluster ${CLUSTER_NAME} \ --certificate-authority=ca.crt \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=${KUBE_CONFIG} # 設(shè)置客戶端認(rèn)證參數(shù) kubectl config set-credentials ${KUBE_USER} \ --client-certificate=${KUBE_CERT}.crt \ --client-key=${KUBE_CERT}.key \ --embed-certs=true \ --kubeconfig=${KUBE_CONFIG} # 設(shè)置上下文參數(shù) kubectl config set-context ${KUBE_USER}@${CLUSTER_NAME} \ --cluster=${CLUSTER_NAME} \ --user=${KUBE_USER} \ --kubeconfig=${KUBE_CONFIG} # 設(shè)置當(dāng)前使用的上下文 kubectl config use-context ${KUBE_USER}@${CLUSTER_NAME} --kubeconfig=${KUBE_CONFIG} # 查看生成的配置文件 kubectl config view --kubeconfig=${KUBE_CONFIG}
cp /etc/kubernetes/kubelet.conf .
-
kubelet.conf(注意配置對應(yīng)的nodeName)
KUBE_APISERVER="https://192.168.12.10:6443" CLUSTER_NAME="kubernetes" # 此處為 master 節(jié)點 nodeName吮龄,每個 master 生成對應(yīng)的 kubelet.conf KUBE_USER="system:node1:master" KUBE_CERT="kubelet" KUBE_CONFIG="kubelet.conf" # 設(shè)置集群參數(shù) kubectl config set-cluster ${CLUSTER_NAME} \ --certificate-authority=ca.crt \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=${KUBE_CONFIG} # 設(shè)置客戶端認(rèn)證參數(shù) kubectl config set-credentials ${KUBE_USER} \ --client-certificate=${KUBE_CERT}.crt \ --client-key=kubelet.key \ --embed-certs=true \ --kubeconfig=${KUBE_CONFIG} # 設(shè)置上下文參數(shù) kubectl config set-context ${KUBE_USER}@${CLUSTER_NAME} \ --cluster=${CLUSTER_NAME} \ --user=${KUBE_USER} \ --kubeconfig=${KUBE_CONFIG} # 設(shè)置當(dāng)前使用的上下文 kubectl config use-context ${KUBE_USER}@${CLUSTER_NAME} --kubeconfig=${KUBE_CONFIG} # 查看生成的配置文件 kubectl config view --kubeconfig=${KUBE_CONFIG}
-
scheduler.conf
KUBE_APISERVER="https://192.168.12.10:6443" CLUSTER_NAME="kubernetes" KUBE_USER="system:kube-scheduler" KUBE_CERT="kube-scheduler" KUBE_CONFIG="scheduler.conf" # 設(shè)置集群參數(shù) kubectl config set-cluster ${CLUSTER_NAME} \ --certificate-authority=ca.crt \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=${KUBE_CONFIG} # 設(shè)置客戶端認(rèn)證參數(shù) kubectl config set-credentials ${KUBE_USER} \ --client-certificate=${KUBE_CERT}.crt \ --client-key=${KUBE_CERT}.key \ --embed-certs=true \ --kubeconfig=${KUBE_CONFIG} # 設(shè)置上下文參數(shù) kubectl config set-context ${KUBE_USER}@${CLUSTER_NAME} \ --cluster=${CLUSTER_NAME} \ --user=${KUBE_USER} \ --kubeconfig=${KUBE_CONFIG} # 設(shè)置當(dāng)前使用的上下文 kubectl config use-context ${KUBE_USER}@${CLUSTER_NAME} --kubeconfig=${KUBE_CONFIG} # 查看生成的配置文件 kubectl config view --kubeconfig=${KUBE_CONFIG}
-
controller-manager.conf
KUBE_APISERVER="https://192.168.12.10:6443" CLUSTER_NAME="kubernetes" KUBE_USER="system:kube-controller-manager" KUBE_CERT="kube-controller-manager" KUBE_CONFIG="controller-manager.conf" # 設(shè)置集群參數(shù) kubectl config set-cluster ${CLUSTER_NAME} \ --certificate-authority=ca.crt \ --embed-certs=true \ --server=${KUBE_APISERVER} \ --kubeconfig=${KUBE_CONFIG} # 設(shè)置客戶端認(rèn)證參數(shù) kubectl config set-credentials ${KUBE_USER} \ --client-certificate=${KUBE_CERT}.crt \ --client-key=sa.key \ --embed-certs=true \ --kubeconfig=${KUBE_CONFIG} # 設(shè)置上下文參數(shù) kubectl config set-context ${KUBE_USER}@${CLUSTER_NAME} \ --cluster=${CLUSTER_NAME} \ --user=${KUBE_USER} \ --kubeconfig=${KUBE_CONFIG} # 設(shè)置當(dāng)前使用的上下文 kubectl config use-context ${KUBE_USER}@${CLUSTER_NAME} --kubeconfig=${KUBE_CONFIG} # 查看生成的配置文件 kubectl config view --kubeconfig=${KUBE_CONFIG}
rm -rf *.csr
cp -r /etc/kubernetes/ /etc/kubernetes.bak
cd /etc/kubernetes/pki
\cp -rf /root/key/* .
cd ..
\cp pki/*.conf .
rm -rf openssl.conf
-
應(yīng)用配置
- 重啟 Docker 和 Kubelet
- 查看三個kubernetes組件(kubelet俭茧,controller-manager,scheduler)的日志漓帚,確認(rèn)是否還有證書過期報錯信息母债。
-
Worker節(jié)點證書更新操作
-
停止docker和kubelet
systemctl stop docker && systemctl stop kubelet
刪除
kubelet.conf
文件,文件一般在/etc/kubernetes
目錄下尝抖。編輯bootstrap-kubelet.conf
文件(文件一般在/etc/kubernetes
目錄下)毡们,修改certificate-authority-data
內(nèi)容,與master節(jié)點中的admin.conf
文件的該區(qū)域內(nèi)容相同昧辽。備份后刪除該目錄下的文件
rm -rf /var/lib/kubelet/pki
-
重啟所有節(jié)點docker和kubelet
systemctl restart docker && systemctl restart kubelet
-
重啟服務(wù)
- 將全部kube-proxy重啟
- 將全部網(wǎng)絡(luò)插件重啟(比如:flannel)
更新權(quán)限更新token
cp /etc/kubernetes/admin.conf ~/.kube/configkubeadm token listkubeadm token create --ttl=0-
kubeadm token list
把獲取到的token更新到node節(jié)點/etc/kubernetes/bootstrap-kubelet.conf
重啟node節(jié)點kubelet