Kubelet之TLS BootStrap啟動(dòng)

<center>TLS BootStrap啟動(dòng)流程</center>

創(chuàng)建bootstrap token與ConfigMap

(1).生成token創(chuàng)建secret

kind: Secret  
metadata:  
  #Name MUST be of form   "bootstrap-token-<token id>"  
  name: bootstrap-token-07401b  
  namespace: kube-system  

#Type MUST be 'bootstrap.kubernetes.io/token'
type: bootstrap.kubernetes.io/token
stringData:
  # Human readable description. Optional.
  description: "The default bootstrap token generated by 'kubeadm init'."

  # Token ID and secret. Required.
  token-id: 07401b
  token-secret: f395accd246ae52d

  # Expiration. Optional.
  expiration: 2017-03-10T03:22:11Z

  # Allowed usages.
  usage-bootstrap-authentication: "true"
  usage-bootstrap-signing: "true"

  # Extra groups to authenticate the token as. Must start with "system:bootstrappers:"
  auth-extra-groups: system:bootstrappers:worker,system:bootstrappers:ingress

1)Extra groups:
表明anyone使用該token進(jìn)行認(rèn)證的時(shí)候?qū)碛性摻MGroup所綁定的權(quán)利.默認(rèn)情況下該token具有system:bootstrappers組下的權(quán)利.
2)usage-bootstrap-*:
表示該secret能夠用來干什么事.
3)id及token生成:
第一種方案:

echo $(openssl rand -hex 3).$(openssl rand -hex 8)

第二種方案:

echo "$(head -c 6 /dev/urandom | md5sum | head -c 6)"."$(head -c 16 /dev/urandom | md5sum | head -c 16)"

需要注意的是在使用 Bootstrap Token 進(jìn)行引導(dǎo)時(shí)果覆，Kubelet 組件使用 Token 發(fā)起的請(qǐng)求其用戶名為system:bootstrap:<tokenid>婿禽，用戶組為system:bootstrappers；so我們?cè)趧?chuàng)建ClusterRoleBinding時(shí)要綁定到這個(gè)用戶或者組上

(2)ConfigMap的生成策略
首先需要根據(jù)(1)中生成的secret及token創(chuàng)建boot-kubeconfig文件
1)設(shè)置集群

kubectl config set-cluster bootstrap \
  --kubeconfig=bootstrap-kubeconfig-public  \
  --server=https://${KUBERNETES_MASTER}:6443 \
  --certificate-authority=ca.pem \
  --embed-certs=true

2)根據(jù)生成的kubeconfig文件生成configmap

kubectl -n kube-public create configmap cluster-info \
  --from-file=kubeconfig=bootstrap-kubeconfig-public

3)獲取configmap詳情

kubectl -n kube-public get configmap cluster-info -o yaml

4)RBAC授權(quán)允許匿名用戶使用該ConfigMap

kubectl create role anonymous-for-cluster-info --resource=configmaps --resource-name=cluster-info --namespace=kube-public --verb=get,list,watch
kubectl create rolebinding anonymous-for-cluster-info-binding --role=anonymous-for-cluster-info --user=system:anonymous --namespace=kube-public

給system:bootstrappers組授權(quán)

(1)kubelet首次使用token啟動(dòng)授權(quán)

# enable bootstrapping nodes to create CSR
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: create-csrs-for-bootstrapping
subjects:
- kind: Group
  name: system:bootstrappers
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: system:node-bootstrapper
  apiGroup: rbac.authorization.k8s.io

或者直接命令行:

kubectl create clusterrolebinding create-csrs-for-bootstrapping --clusterrole=system:bootstrappers --group=system:node-bootstrapper

anyone使用token進(jìn)行認(rèn)證通過后進(jìn)入授權(quán)階段.api-server從該token中獲取namespace和name信息,并將該token特殊對(duì)待.授予anyone bootstrap權(quán)利,將該匿名用戶劃分到system:bootstraps組.至此anyone使用該token認(rèn)證的時(shí)候都具有了system:node-bootstrapper的權(quán)利.

controller-manager配置

當(dāng)api-server完成對(duì)kubelet獲取證書請(qǐng)求的認(rèn)證授權(quán)之后,需要controller-manager對(duì)kubelet生成證書并發(fā).
(1)證書配置
為了使controller-manager完成證書的簽名需要配置ca證書

--cluster-signing-cert-file="/var/lib/kubernetes/ca.pem"
--cluster-signing-key-file="/var/lib/kubernetes/ca-key.pem"

(2)Approval授權(quán)
為了完成kubelet的創(chuàng)建證書請(qǐng)求CSRs,需要告訴controller-manager通過CSR請(qǐng)求. 通過配置RBAC規(guī)則保證controller-manager只對(duì)kubelet發(fā)起的特定CSR請(qǐng)求自定批準(zhǔn). 以下配置告訴controller自動(dòng)批準(zhǔn)三種證書:

nodeClient kubelet第一次請(qǐng)求獲取證書,表明該node暫時(shí)還沒獲取過證書.

# Approve all CSRs for the group "system:bootstrappers"
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: auto-approve-csrs-for-group
subjects:
- kind: Group
  name: system:bootstrappers
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
  apiGroup: rbac.authorization.k8s.io

selfnodeclient 請(qǐng)求重新獲取證書請(qǐng)求當(dāng)證書過期之后.

# Approve renewal CSRs for the group "system:nodes"
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: auto-approve-renewals-for-nodes
subjects:
- kind: Group
  name: system:nodes
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
  apiGroup: rbac.authorization.k8s.io

3)selfnodeserver kubelet server renew自己的證書發(fā)起的CSR請(qǐng)求,即是10250端口使用的證書
白話:nodeclient 類型的 CSR 僅在第一次啟動(dòng)時(shí)會(huì)產(chǎn)生，selfnodeclient 類型的 CSR 請(qǐng)求實(shí)際上就是 kubelet renew 自己作為 client 跟 apiserver 通訊時(shí)使用的證書產(chǎn)生的氓鄙，selfnodeserver 類型的 CSR 請(qǐng)求則是 kubelet 首次申請(qǐng)或后續(xù) renew 自己的 10250 api 端口證書時(shí)產(chǎn)生的

(3)自動(dòng)續(xù)期下的引導(dǎo)過程

kubelet 讀取 bootstrap.kubeconfig璧坟，使用其 CA 與 Token 向 apiserver 發(fā)起第一次 CSR 請(qǐng)求(nodeclient)
apiserver 根據(jù) RBAC 規(guī)則自動(dòng)批準(zhǔn)首次 CSR 請(qǐng)求(approve-node-client-csr)，并下發(fā)證書(kubelet-client.crt)
kubelet 使用剛剛簽發(fā)的證書(O=system:nodes, CN=system:node:NODE_NAME)與 apiserver 通訊讶隐，并發(fā)起申請(qǐng) 10250 server 所使用證書的 CSR 請(qǐng)求
apiserver 根據(jù) RBAC 規(guī)則自動(dòng)批準(zhǔn) kubelet 為其 10250 端口申請(qǐng)的證書(kubelet-server-current.crt)
證書即將到期時(shí)起胰，kubelet 自動(dòng)向 apiserver 發(fā)起用于與 apiserver 通訊所用證書的 renew CSR 請(qǐng)求和 renew 本身 10250 端口所用證書的 CSR 請(qǐng)求
apiserver 根據(jù) RBAC 規(guī)則自動(dòng)批準(zhǔn)兩個(gè)證書
kubelet 拿到新證書后關(guān)閉所有連接，reload 新證書巫延，以后便一直如此

根據(jù)token創(chuàng)建kubeconfig文件

(1) 設(shè)置集群參數(shù)

API_SERVER = https://192.168.124.18:6443
kubectl config set-cluster bootstrap --certificate-authority=/etc/kubernetes/ssl/ca.key --embed-certs=true --server=${API_SERVER} --kubeconfig=bootstrap.kubeconfig

(2)設(shè)置客戶端認(rèn)證參數(shù)

kubectl config set-credentials kubelet-bootstrap --token=*.** --kubeconfig=bootstrap.kubeconfig

(3)設(shè)置默認(rèn)上下文context

kubectl config set-context bootstrap --user=kubelet-bootstrap --cluster=bootstrap --kube-config=bootstrap.kubeconfig

(4)設(shè)置默認(rèn)上下文
kubectl config use-context bootstrap --kubeconfig=bootstrap.kubeconfig

創(chuàng)建kubelet啟動(dòng)文件配置參數(shù)

(1)api-server配置token啟動(dòng)
--enable-bootstrap-token-auth
(2)controller-manager
1)--controllers=*,bootstrapsigner,tokenclean:自動(dòng)簽名
2)--experimental-cluster-signing-duration:指定證書超時(shí)時(shí)間
(3)kubelet設(shè)置自動(dòng)獲取證書
1)--rotate-certificates:自旋獲取證書
2)開啟自旋獲取客戶端及server端證書--feature-gates=RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true默認(rèn)是開啟的
3)--bootstrap-kubeconfig與--kubeconfig指定配置參數(shù)
(3)kubelet.service啟動(dòng)文件

[Unit]
Description=this is my kubelet
[Service]
EnvironmentFile=-/etc/kubernetes/config.conf
EnvironmentFile=-/etc/kubernetes/kubelet.conf
ExecStart=/usr/local/bin/kubelet \
       $KUBELET_CONFIG \
       $KUBELET_ADDRESS

Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-users.target

配置文件如下:

KUBELET_CONFIG="--cgroup-driver=systemd --runtime-cgroups=/systemd/system.slice
--kubelet-cgroups=/systemd/system.slice --cert-dir=/etc/kubernetes/ssl --cluster
-domain=cluster.local --cluster-dns=10.254.0.2 --bootstrap-kubeconfig=/etc/kuber
netes/bootstrap.kubeconfig --kubeconfig=/etc/kubernetes/kubeconfig --network-plu
gin=cni --pod-cidr=10.254.0.0/16 --rotate-certificates"
KUBELET_ADDRESS="--address=192.168.124.18"