Kubeadm是Kubernetes官方提供的用于快速安裝Kubernetes集群的工具，伴隨Kubernetes每個(gè)版本的發(fā)布都會(huì)同步更新，在2018年將進(jìn)入GA狀態(tài)，說明離生產(chǎn)環(huán)境中使用的距離越來越近了。

使用Kubeadm搭建Kubernetes集群本來是件很簡(jiǎn)單的事拙绊，但由于眾所周知的原因，在中國大陸是無法訪問 k8s.gcr.io的泳秀。這就使我們無法按照官方的教程來創(chuàng)建集群标沪。而國內(nèi)的教程參差不齊，大多也無法運(yùn)行成功晶默，我也是踩了很多坑谨娜，才部署成功，故在此分享出來磺陡。
另外趴梢，我把相關(guān)的命令簡(jiǎn)單整合了一下，寫成了兩個(gè)腳本币他，分別用來部署Master和Worker Node坞靶，用起來非常方便。Kubernetes自動(dòng)化部署腳本

準(zhǔn)備

安裝環(huán)境準(zhǔn)備

關(guān)閉防火墻和selinux

# 關(guān)閉防火墻
systemctl stop firewalld
systemctl disable firewalld

# 禁用SELINUX
setenforce 0

vim /etc/selinux/config
SELINUX=disabled

關(guān)閉系統(tǒng)的Swap

Kubernetes 1.8開始要求必須禁用Swap蝴悉，如果不關(guān)閉彰阴，默認(rèn)配置下kubelet將無法啟動(dòng)。

# 關(guān)閉系統(tǒng)的Swap方法如下:
# 編輯`/etc/fstab`文件拍冠，注釋掉引用`swap`的行尿这，保存并重啟后輸入:
sudo swapoff -a

驗(yàn)證Mac地址和product_uuid是否唯一（可選）

Kubernetes要求集群中所有機(jī)器具有不同的Mac地址簇抵、產(chǎn)品uuid、Hostname射众〉冢可以使用如下命令查看：

# UUID
cat /sys/class/dmi/id/product_uuid

# Mac地址
ip link

# Hostname
cat /etc/hostname

安裝Docker

Kubernetes從1.6開始使用CRI(Container Runtime Interface)容器運(yùn)行時(shí)接口。默認(rèn)的容器運(yùn)行時(shí)仍然是Docker叨橱，是使用kubelet中內(nèi)置dockershim CRI來實(shí)現(xiàn)的典蜕。

sudo yum install -y docker

安裝kubeadm, kubelet 和 kubectl

部署之前，我們需要安裝一下三個(gè)包：

• kubeadm: 引導(dǎo)啟動(dòng)k8s集群的命令行工具罗洗。
• kubelet: 在群集中所有節(jié)點(diǎn)上運(yùn)行的核心組件, 用來執(zhí)行如啟動(dòng)pods和containers等操作愉舔。
• kubectl: 操作集群的命令行工具。

修改yum源

Kubernetes國內(nèi)yum源

安裝kubeadm, kubelet 和 kubectl

查看可用版本

sudo yum list --showduplicates | grep 'kubeadm\|kubectl\|kubelet'

安裝 kubeadm1.13.1, kubelet1.13.1 和 kubectl1.13.1

yum install -y kubelet-1.13.1 kubeadm-1.13.1 kubectl-1.13.1 --disableexcludes=kubernetes

此時(shí)還不能啟動(dòng) kubelet伙菜，先設(shè)置開機(jī)啟動(dòng)：

systemctl enable kubelet

初始化 Master 節(jié)點(diǎn)

配置文件

# 生成配置文件
kubeadm config print-default > kubeadm.conf 
# 修改配置文件
# 修改鏡像倉儲(chǔ)地址
sed -i "s#imageRepository: .*#imageRepository: registry.cn-beijing.aliyuncs.com/imcto#g" kubeadm.conf
# 修改版本號(hào)
sed -i "s/kubernetesVersion: .*/kubernetesVersion: v1.13.1/g" kubeadm.conf
sed -i "s/podSubnet: .*/podSubnet: \"10.244.0.0\/16\"/g" kubeadm.conf

初始化

拉取鏡像

$ sudo kubeadm config images pull --config kubeadm.conf
[config/images] Pulled registry.cn-beijing.aliyuncs.com/imcto/kube-apiserver:v1.13.1
[config/images] Pulled registry.cn-beijing.aliyuncs.com/imcto/kube-controller-manager:v1.13.1
[config/images] Pulled registry.cn-beijing.aliyuncs.com/imcto/kube-scheduler:v1.13.1
[config/images] Pulled registry.cn-beijing.aliyuncs.com/imcto/kube-proxy:v1.13.1
[config/images] Pulled registry.cn-beijing.aliyuncs.com/imcto/pause:3.1
[config/images] Pulled registry.cn-beijing.aliyuncs.com/imcto/etcd:3.2.24
[config/images] Pulled registry.cn-beijing.aliyuncs.com/imcto/coredns:1.2.6

可以看到轩缤，已經(jīng)成功拉取了需要的鏡像

開始初始化

此處可能會(huì)遇到一個(gè)坑 安裝Kubernetes報(bào)錯(cuò)：The kubelet is not running

$ sudo kubeadm init --config kubeadm.conf
[init] Using Kubernetes version: v1.13.1
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [kube-node1 localhost] and IPs [10.136.157.23 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [kube-node1 localhost] and IPs [10.136.157.23 127.0.0.1 ::1]
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kube-node1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.136.157.23]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 18.501577 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.13" in namespace kube-system with the configuration for the kubelets in the cluster
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "kube-node1" as an annotation
[mark-control-plane] Marking the node kube-node1 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node kube-node1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 10.136.157.23:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:b9b8a2423ecb28237cf729f77a3b832bef6b5c2f300e97bd5fc76bf3ca480952

開啟8080端口

# 修改配置文件
sudo sed -i 's/insecure-port=0/insecure-port=8080/g' /etc/kubernetes/manifests/kube-apiserver.yaml
# 重啟docker鏡像
sudo docker ps |grep 'kube-apiserver_kube-apiserver'|awk '{print $1}'|head -1|xargs sudo docker restart

設(shè)置 Node 節(jié)點(diǎn)加入集群

kubeadm init 命令的輸出日志中已經(jīng)告知我們加入新節(jié)點(diǎn)的方式。如果忘記了 Master 的 Token贩绕，可以在 Master 上輸入以下命令查看：

$ sudo kubeadm token list
TOKEN                     TTL       EXPIRES                     USAGES                   DESCRIPTION   EXTRA GROUPS
abcdef.0123456789abcdef   9h        2019-03-22T19:35:14+08:00   authentication,signing   <none>        system:bootstrappers:kubeadm:default-node-token

默認(rèn)情況下 Token 過期是時(shí)間是24小時(shí)典奉，如果 Token 過期以后，可以輸入以下命令丧叽，生成新的 Token

kubeadm token create

獲取ca證書sha256編碼hash值

$ openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
b9b8a2423ecb28237cf729f77a3b832bef6b5c2f300e97bd5fc76bf3ca480952

加入新的Node節(jié)點(diǎn)

$ sudo kubeadm join 10.136.157.23:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:b9b8a2423ecb28237cf729f77a3b832bef6b5c2f300e97bd5fc76bf3ca480952
[preflight] Running pre-flight checks
[discovery] Trying to connect to API Server "10.136.157.23:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://10.136.157.23:6443"
[discovery] Requesting info from "https://10.136.157.23:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "10.136.157.23:6443"
[discovery] Successfully established connection with API Server "10.136.157.23:6443"
[join] Reading configuration from the cluster...
[join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.13" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "kube-node2" as an annotation

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.

啟用IPV6

注意：需要在新加入的Node上啟用IPV6，否則會(huì)導(dǎo)致 coredns 啟動(dòng)失敗公你，始終處于 ContainerCreating 狀態(tài)

sudo sed -i 's\ipv6.disable=1\ipv6.disable=0\g' /etc/default/grub
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo reboot

安裝網(wǎng)絡(luò)插件

為了讓Pods間可以相互通信踊淳，我們必須安裝一個(gè)網(wǎng)絡(luò)插件，并且必須在部署任何應(yīng)用之前安裝陕靠，CoreDNS也是在網(wǎng)絡(luò)插件安裝之后才會(huì)啟動(dòng)的迂尝。

網(wǎng)絡(luò)的插件完整列表，請(qǐng)參考 Networking and Network Policy剪芥。

在安裝之前垄开，我們先查看一下當(dāng)前Pods的狀態(tài)：

$ sudo kubectl get pods --all-namespaces
NAMESPACE     NAME                                 READY     STATUS    RESTARTS   AGE
kube-system   coredns-58d6b7c8d7-f5mdg             0/1       Pending   0          57m
kube-system   coredns-58d6b7c8d7-fkpmc             0/1       Pending   0          57m
kube-system   etcd-kube-node1                      1/1       Running   1          56m
kube-system   kube-apiserver-kube-node1            1/1       Running   7          10m
kube-system   kube-controller-manager-kube-node1   1/1       Running   2          56m
kube-system   kube-proxy-2xrzt                     1/1       Running   1          57m
kube-system   kube-scheduler-kube-node1            1/1       Running   2          56m

如上，可以看到CoreDND的狀態(tài)是Pending税肪，就是因?yàn)槲覀冞€沒有安裝網(wǎng)絡(luò)插件溉躲。

安裝RBAC

wget https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/canal/rbac.yaml
sudo kubectl apply -f rbac.yaml

安裝Flannel

sudo sysctl net.bridge.bridge-nf-call-iptables=1

Set /proc/sys/net/bridge/bridge-nf-call-iptables to 1 by running sysctl net.bridge.bridge-nf-call-iptables=1 to pass bridged IPv4 traffic to iptables’ chains. This is a requirement for some CNI plugins to work

$ wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
$ sudo kubectl apply -f kube-flannel.yml
podsecuritypolicy.extensions/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds-amd64 created
daemonset.extensions/kube-flannel-ds-arm64 created
daemonset.extensions/kube-flannel-ds-arm created
daemonset.extensions/kube-flannel-ds-ppc64le created
daemonset.extensions/kube-flannel-ds-s390x created

通過 ifconfig 可以看到新建了一張?zhí)摂M網(wǎng)卡：

$ ifconfig
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.244.0.0  netmask 255.255.255.255  broadcast 0.0.0.0
        ether 7a:d1:26:11:ca:ae  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Master隔離（可選）

默認(rèn)情況下，由于安全原因益兄，集群并不會(huì)將pods部署在Master節(jié)點(diǎn)上锻梳。但是在開發(fā)環(huán)境下，我們可能就只有一個(gè)Master節(jié)點(diǎn)净捅，這時(shí)可以使用下面的命令來解除這個(gè)限制：

$ sudo kubectl taint nodes --all node-role.kubernetes.io/master-
## 輸出
node/ubuntu1 untainted

等待一會(huì)疑枯，我們可以在Master節(jié)點(diǎn)上使用kubectl get nodes命令來查看節(jié)點(diǎn)的狀態(tài)：

$ sudo kubectl get nodes
NAME         STATUS     ROLES    AGE    VERSION
kube-node1   Ready   master   3h3m   v1.13.3
kube-node2   Ready   <none>   118s   v1.13.3
kube-node3   Ready   <none>   95s    v1.13.3

安裝Dashboard UI

Dashboard默認(rèn)是不安裝的，要想安裝Dashboard蛔六，可以執(zhí)行下面的命令：

$ wget https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.0/src/deploy/recommended/kubernetes-dashboard.yaml

# 修改kubernetes-dashboard.yaml
        spec:
      containers:
      - name: kubernetes-dashboard
        image: registry.cn-beijing.aliyuncs.com/imcto/kubernetes-dashboard-amd64:v1.10.1
# 創(chuàng)建Dashboard UI
$ sudo kubectl create -f kubernetes-dashboard.yaml

注意：如果沒有必要荆永，不用解除 --apiserver-host 的注釋

訪問Dashboard UI

修改kubernetes-dashboard配置

$ kubectl -n kube-system edit service kubernetes-dashboard

將type: ClusterIP 修改為 type: NodePort废亭，并保存文件。
檢查 Dashboard UI 暴露的服務(wù)端口

$ sudo kubectl -n kube-system get service kubernetes-dashboard
NAME                   TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
kubernetes-dashboard   NodePort   10.110.146.196   <none>        443:32275/TCP   42m

可以看到具钥，Dashboard UI 的服務(wù)端口是32275豆村，協(xié)議為HTTPS，我們可以https://<master-ip>:32275來進(jìn)行訪問氓拼。

遇到的問題

初始化 Kubernetes 問題（端口占用）
安裝Kubernetes報(bào)錯(cuò)：The connection to the server localhost:8080 was refused
安裝Kubernetes報(bào)錯(cuò)：STATUS NotReady
安裝Kubernetes報(bào)錯(cuò)：/proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory
安裝Kubernetes報(bào)錯(cuò)：kubernetes-dashboard狀態(tài)CrashLoopBackOff
kubelet啟動(dòng)報(bào)錯(cuò)：node "kube-master1" not found

參考文章

Centos7 使用 kubeadm 安裝Kubernetes 1.13.3
使用Kubeadm搭建Kubernetes(1.12.2)集群
在國內(nèi)如何巧妙獲取kubernetes各鏡像你画？
國內(nèi)借助阿里云快速獲取gcr.io上的鏡像
解決kubeadm部署kubernetes集群鏡像問題
kubeadm 生成的token過期后，集群增加節(jié)點(diǎn)
kubeadm 安裝 kubernetes
Kubernetes自動(dòng)化部署腳本