[TOC]

簡(jiǎn)介：

Kubernetes作為Google開(kāi)源的容器運(yùn)行平臺(tái)摊鸡，受到了大家的熱捧包竹。搭建一套完整的kubernetes平臺(tái)匕坯，也成為試用這套平臺(tái)必須邁過(guò)的坎兒氮块。kubernetes1.5版本以及之前，安裝還是相對(duì)比較方便的校哎，官方就有通過(guò)yum源在centos7安裝kubernetes两波。但是在kubernetes1.6之后，安裝就比較繁瑣了闷哆，需要證書(shū)各種認(rèn)證腰奋，對(duì)于剛接觸kubernetes的人來(lái)說(shuō)很不友好。

架構(gòu)說(shuō)明：

兩臺(tái)主機(jī)：

18.16.202.35 master
18.16.202.36 slaver

[root@localhost /]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
18.16.202.35 node1
18.16.202.36 node2              

系統(tǒng)配置:

1.1 關(guān)閉防火墻

systemctl stop firewalld
systemctl disable firewalld

1.2 禁用SELinux

setenforce 0

編輯文件/etc/selinux/config抱怔，將SELINUX修改為disabled劣坊，如下：

sed -i 's/SELINUX=permissive/SELINUX=disabled/' /etc/sysconfig/selinux

#SELINUX=disabled

1.3 關(guān)閉系統(tǒng)Swap

Kubernetes 1.8開(kāi)始要求關(guān)閉系統(tǒng)的Swap，如果不關(guān)閉屈留，默認(rèn)配置下kubelet將無(wú)法啟動(dòng)局冰。方法一,通過(guò)kubelet的啟動(dòng)參數(shù)–fail-swap-on=false更改這個(gè)限制测蘑。方法二,關(guān)閉系統(tǒng)的Swap。

swapoff -a

修改/etc/fstab文件康二，注釋掉SWAP的自動(dòng)掛載碳胳，使用free -m確認(rèn)swap已經(jīng)關(guān)閉。

#注釋掉swap分區(qū)
[root@localhost /]# sed -i 's/.*swap.*/#&/' /etc/fstab

#/dev/mapper/centos-swap swap                    swap    defaults        0 0
                                                                
[root@localhost /]# free -m
              total        used        free      shared  buff/cache   available
Mem:            962         154         446           6         361         612
Swap:             0           0           0

1.4 安裝docker

sudo yum install -y yum-utils device-mapper-persistent-data lvm2
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo yum makecache fast

sudo yum -y install docker-ce
systemctl enable docker.service
systemctl restart docker

我這里安裝的是docker-ce 18.06

docker的iptables設(shè)置：

docker 版本為1.13.1時(shí)沫勿，iptables的情況

[root@localhost /]# iptables -nvL
Chain INPUT (policy ACCEPT 423 packets, 66469 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  423 66469 KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER-ISOLATION  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 385 packets, 63638 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  385 63638 KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER-ISOLATION (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain KUBE-SERVICES (2 references)
 pkts bytes target     prot opt in     out     source               destination         

docker版本為18.06時(shí)挨约，iptables

[root@localhost /]# iptables -nvL
Chain INPUT (policy ACCEPT 12218 packets, 1299K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 608 packets, 56787 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0       

Docker從1.13版本開(kāi)始調(diào)整了默認(rèn)的防火墻規(guī)則，禁用了iptables filter表中FOWARD鏈产雹，這樣會(huì)引起Kubernetes集群中跨Node的Pod無(wú)法通信诫惭。但這里通過(guò)安裝docker 1806，發(fā)現(xiàn)默認(rèn)策略又改回了ACCEPT蔓挖，這個(gè)不知道是從哪個(gè)版本改回的夕土，因?yàn)槲覀兙€(xiàn)上版本使用的1706還是需要手動(dòng)調(diào)整這個(gè)策略的。

其他版本docker操作：

# 開(kāi)啟forward
# Docker從1.13版本開(kāi)始調(diào)整了默認(rèn)的防火墻規(guī)則
# 禁用了iptables filter表中FOWARD鏈
# 這樣會(huì)引起Kubernetes集群中跨Node的Pod無(wú)法通信
$ iptables -P FORWARD ACCEPT

使用kubeadm部署Kubernetes:

2.1 安裝kubeadm和kubelet

下面在各節(jié)點(diǎn)安裝kubeadm和kubelet：

# 配置源
$ cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

# 安裝
$ yum makecache fast
$ yum install -y kubelet kubeadm kubectl ipvsadm

配置：

# 配置轉(zhuǎn)發(fā)相關(guān)參數(shù)瘟判，否則可能會(huì)出錯(cuò)
$ cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness=0
EOF

# 使配置生效
$ sysctl --system

# 如果net.bridge.bridge-nf-call-iptables報(bào)錯(cuò)隘弊，加載br_netfilter模塊
$ modprobe br_netfilter
$ sysctl -p /etc/sysctl.d/k8s.conf

# 加載ipvs相關(guān)內(nèi)核模塊
# 如果重新開(kāi)機(jī)，需要重新加載（可以寫(xiě)在 /etc/rc.local 中開(kāi)機(jī)自動(dòng)加載）
$ modprobe ip_vs
$ modprobe ip_vs_rr
$ modprobe ip_vs_wrr
$ modprobe ip_vs_sh
$ modprobe nf_conntrack_ipv4
# 查看是否加載成功
$ lsmod | grep ip_vs

配置啟動(dòng)kubelet（所有節(jié)點(diǎn)）

# 配置kubelet使用國(guó)內(nèi)pause鏡像
# 配置kubelet的cgroups
# 獲取docker的cgroups
DOCKER_CGROUPS=$(docker info | grep 'Cgroup' | cut -d' ' -f3)
echo $DOCKER_CGROUPS
cat >/etc/sysconfig/kubelet<<EOF
KUBELET_EXTRA_ARGS="--cgroup-driver=$DOCKER_CGROUPS --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.1"
EOF

# 啟動(dòng)
$ systemctl daemon-reload
$ systemctl enable kubelet && systemctl restart kubelet

在這里使用systemctl status kubelet荒适，你會(huì)發(fā)現(xiàn)報(bào)錯(cuò)誤信息梨熙；

10月 11 00:26:43 node1 systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
10月 11 00:26:43 node1 systemd[1]: Unit kubelet.service entered failed state.
10月 11 00:26:43 node1 systemd[1]: kubelet.service failed.

運(yùn)行journalctl -xefu kubelet 命令查看systemd日志才發(fā)現(xiàn)，真正的錯(cuò)誤是：

unable to load client CA file /etc/kubernetes/pki/ca.crt: open /etc/kubernetes/pki/ca.crt: no such file or directory

這個(gè)錯(cuò)誤在運(yùn)行kubeadm init 生成CA證書(shū)后會(huì)被自動(dòng)解決刀诬，此處可先忽略咽扇。

簡(jiǎn)單地說(shuō)就是在kubeadm init 之前kubelet會(huì)不斷重啟。

2.2 配置master節(jié)點(diǎn)

直接使用命令：

kubeadm init \
   --kubernetes-version=v1.12.0 \
   --pod-network-cidr=10.244.0.0/16 \
   --apiserver-advertise-address=192.168.61.11 \
   --ignore-preflight-errors=Swap

或者使用kubeadm-master.config配置文件陕壹，在/etc/kubernetes/文件夾下面操作：

# 1.11 版本 centos 下使用 ipvs 模式會(huì)出問(wèn)題
# 參考 https://github.com/kubernetes/kubernetes/issues/65461

# 生成配置文件
cat >/etc/kubernetes/kubeadm-master.config<<EOF
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.12.0
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
api:
  advertiseAddress: 18.16.202.35 

controllerManagerExtraArgs:
  node-monitor-grace-period: 10s
  pod-eviction-timeout: 10s

networking:
  podSubnet: 10.244.0.0/16
  
kubeProxy:
  config:
    mode: ipvs
    # mode: iptables
EOF


# 提前拉取鏡像
# 如果執(zhí)行失敗 可以多次執(zhí)行
kubeadm config images pull --config /etc/kubernetes/kubeadm-master.config

# 初始化
kubeadm init --config /etc/kubernetes/kubeadm-master.config
# or
kubeadm init --config /etc/kubernetes/kubeadm-master.config --ignore-preflight-errors=all

安裝過(guò)程中遇到異常：

[preflight] Some fatal errors occurred:
  [ERROR DirAvailable--var-lib-etcd]: /var/lib/etcd is not empty

直接刪除/var/lib/etcd文件夾

如果初始化過(guò)程出現(xiàn)問(wèn)題质欲，使用如下命令重置：

kubeadm reset

rm -rf /var/lib/cni/ $HOME/.kube/config

2.3 初始化master節(jié)點(diǎn)：

[root@localhost kubernetes]#  kubeadm init --config kubeadm-master.config
[init] using Kubernetes version: v1.12.0
[preflight] running pre-flight checks
    [WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs_rr ip_vs_wrr ip_vs_sh ip_vs] or no builtin kernel ipvs support: map[ip_vs_sh:{} nf_conntrack_ipv4:{} ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{}]
you can solve this problem with following methods:
 1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support

[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [localhost.localdomain localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [localhost.localdomain localhost] and IPs [18.16.202.35 127.0.0.1 ::1]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [localhost.localdomain kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 18.16.202.35]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[certificates] Generated sa key and public key.
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests" 
[init] this might take a minute or longer if the control plane images have to be pulled
[apiclient] All control plane components are healthy after 40.510372 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.12" in namespace kube-system with the configuration for the kubelets in the cluster
[markmaster] Marking the node localhost.localdomain as master by adding the label "node-role.kubernetes.io/master=''"
[markmaster] Marking the node localhost.localdomain as master by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "localhost.localdomain" as an annotation
[bootstraptoken] using token: xc9gpo.mmv1mmsjhq6tzhdc
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

    kubeadm join 18.16.202.35:6443 --token ccxrk8.myui0xu4syp99gxu --discovery-token-ca-cert-hash sha256:e3c90ace969aa4d62143e7da6202f548662866dfe33c140095b020031bff2986

上面記錄了完成的初始化輸出的內(nèi)容，根據(jù)輸出的內(nèi)容基本上可以看出手動(dòng)初始化安裝一個(gè)Kubernetes集群所需要的關(guān)鍵步驟糠馆。

其中有以下關(guān)鍵內(nèi)容：

• [kubelet] 生成kubelet的配置文件”/var/lib/kubelet/config.yaml”

• [certificates]生成相關(guān)的各種證書(shū)

• [kubeconfig]生成相關(guān)的kubeconfig文件

• [bootstraptoken]生成token記錄下來(lái)嘶伟，后邊使用kubeadm join往集群中添加節(jié)點(diǎn)時(shí)會(huì)用到

• 下面的命令是配置常規(guī)用戶(hù)如何使用kubectl訪(fǎng)問(wèn)集群：

    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config
  

• 最后給出了將節(jié)點(diǎn)加入集群的命令:

    kubeadm join 18.16.202.35:6443 --token ccxrk8.myui0xu4syp99gxu --discovery-token-ca-cert-hash sha256:e3c90ace969aa4d62143e7da6202f548662866dfe33c140095b020031bff2986
  

如果安裝過(guò)程中爆出異常，kubelet不能啟動(dòng)又碌，但是查看kubelet實(shí)際上啟動(dòng)了九昧，查看kubelet日志發(fā)現(xiàn)

journalctl -xeu kubelet

10月 11 21:29:14 node1 kubelet[5351]: W1011 21:29:14.012763    5351 cni.go:188] Unable to update cni config: No networks found in /etc/cni/net.d
10月 11 21:29:14 node1 kubelet[5351]: E1011 21:29:14.012853    5351 kubelet.go:2167] Container runtime network not ready: NetworkReady=false reason:Ne
....
10月 11 21:29:15 node1 kubelet[5351]: E1011 21:29:15.473163    5351 event.go:212] Unable to write event: 'Post https://18.16.202.35:6443/api/v1/namesp

在國(guó)外服務(wù)器執(zhí)行：

docker pull quay.io/coreos/flannel:v0.10.0-amd64
docker tag quay.io/coreos/flannel:v0.10.0-amd64 ${username}/flannel:v0.10.0-amd64
docker push ${username}/flannel:v0.10.0-amd64
docker rmi quay.io/coreos/flannel:v0.10.0-amd64
docker rmi ${username}/flannel:v0.10.0-amd64

在國(guó)內(nèi)服務(wù)器執(zhí)行：

sudo docker pull ${username}/flannel:v0.10.0-amd64
sudo docker tag ${username}/flannel:v0.10.0-amd64 quay.io/coreos/flannel:v0.10.0-amd64
sudo docker rmi ${username}/flannel:v0.10.0-amd64

或者直接使用yqfwind/flannel

重新init, 然后apply flannel之后，可以查看/etc/cni/net.d/目錄下是有配置文件10-flannel.conflist

最后毕匀，在其他節(jié)點(diǎn)也需要下載相關(guān)鏡像

如果節(jié)點(diǎn)服務(wù)區(qū)/etc/cni/net.d/10-flannel.conflist不存在铸鹰，一是檢查flannel鏡像是否存在，也可以拷貝master服務(wù)器的文件至相關(guān)目錄下皂岔。

我使用下面的命令：

docker pull quay.io/coreos/flannel:v0.10.0-amd64 
mkdir -p /etc/cni/net.d/
cat <<EOF> /etc/cni/net.d/10-flannel.conf
{"name":"cbr0","type":"flannel","delegate": {"isDefaultGateway": true}}
EOF
mkdir /usr/share/oci-umount/oci-umount.d -p
mkdir /run/flannel/
cat <<EOF> /run/flannel/subnet.env
FLANNEL_NETWORK=172.100.0.0/16
FLANNEL_SUBNET=172.100.1.0/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
EOF
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml

2.4 配置使用kubectl

如下操作在master節(jié)點(diǎn)操作

$ rm -rf $HOME/.kube
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

# 查看node節(jié)點(diǎn)
$ kubectl get nodes
NAME    STATUS     ROLES    AGE     VERSION
node1   NotReady   master   6m19s   v1.12.0

2.5 配置使用網(wǎng)絡(luò)插件

如下操作在master節(jié)點(diǎn)操作

# 下載配置
$ cd ~ && mkdir flannel && cd flannel
$ wget https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml

修改配置文件kube-flannel.yml:

# 修改kube-flannel.yml中配置
# 此處的ip配置要與上面kubeadm的pod-network一致
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }

# 默認(rèn)的鏡像是quay.io/coreos/flannel:v0.10.0-amd64蹋笼，如果你能pull下來(lái)就不用修改鏡像地址，否則，修改yml中鏡像地址為阿里鏡像源
image: registry.cn-shanghai.aliyuncs.com/gcr-k8s/flannel:v0.10.0-amd64

# 如果Node有多個(gè)網(wǎng)卡的話(huà)剖毯，參考flannel issues 39701圾笨，
# https://github.com/kubernetes/kubernetes/issues/39701
# 目前需要在kube-flannel.yml中使用--iface參數(shù)指定集群主機(jī)內(nèi)網(wǎng)網(wǎng)卡的名稱(chēng)，
# 否則可能會(huì)出現(xiàn)dns無(wú)法解析逊谋。容器無(wú)法通信的情況擂达，需要將kube-flannel.yml下載到本地，
# flanneld啟動(dòng)參數(shù)加上--iface=<iface-name>
    containers:
      - name: kube-flannel
        image: registry.cn-shanghai.aliyuncs.com/gcr-k8s/flannel:v0.10.0-amd64
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        - --iface=ens33
        - --iface=eth0
??????--iface=ens33 的值涣狗，是你當(dāng)前的網(wǎng)卡,或者可以指定多網(wǎng)卡

# 1.12版本的kubeadm額外給node1節(jié)點(diǎn)設(shè)置了一個(gè)污點(diǎn)(Taint)：node.kubernetes.io/not-ready:NoSchedule，
# 很容易理解舒憾，即如果節(jié)點(diǎn)還沒(méi)有ready之前镀钓，是不接受調(diào)度的《朴兀可是如果Kubernetes的網(wǎng)絡(luò)插件還沒(méi)有部署的話(huà)丁溅，節(jié)點(diǎn)是不會(huì)進(jìn)入ready狀態(tài)的。
# 因此我們修改以下kube-flannel.yaml的內(nèi)容探遵，加入對(duì)node.kubernetes.io/not-ready:NoSchedule這個(gè)污點(diǎn)的容忍：
tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - key: node.kubernetes.io/not-ready
        operator: Exists
        effect: NoSchedule

啟動(dòng)：

# 啟動(dòng)
$ kubectl apply -f ~/flannel/kube-flannel.yml

# 查看
$ kubectl get pods --namespace kube-system
# kubectl get service
$ kubectl get svc --namespace kube-system

# 只有網(wǎng)絡(luò)插件也安裝配置完成之后窟赏，才能會(huì)顯示為ready狀態(tài)
# 設(shè)置master允許部署應(yīng)用pod，參與工作負(fù)載箱季，現(xiàn)在可以部署其他系統(tǒng)組件
# 如 dashboard, heapster, efk等
$ kubectl taint nodes --all node-role.kubernetes.io/master-
# 或者 kubectl taint nodes node1 node-role.kubernetes.io/master-   
  node/node1 untainted

# master不運(yùn)行pod
  kubectl taint nodes node1 node-role.kubernetes.io/master=:NoSchedule

操作記錄：

[root@localhost flannel]#  kubectl apply -f kube-flannel.yml
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds created
[root@localhost flannel]# kubectl get pods --namespace kube-system
NAME                                            READY   STATUS    RESTARTS   AGE
coredns-6c66ffc55b-ggsgx                        0/1     Pending   0          26m
coredns-6c66ffc55b-m457x                        0/1     Pending   0          26m
etcd-localhost.localdomain                      1/1     Running   0          25m
kube-apiserver-localhost.localdomain            1/1     Running   0          25m
kube-controller-manager-localhost.localdomain   1/1     Running   0          25m
kube-proxy-9jqwm                                1/1     Running   0          26m
kube-scheduler-localhost.localdomain            1/1     Running   0          25m
[root@localhost flannel]#  kubectl get svc --namespace kube-system
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP   26m

如果曝出異常：

node "master" untainted

或者

error: taint "node-role.kubernetes.io/master:" not found

一般是因?yàn)樯弦淮伟惭b沒(méi)有卸載干凈涯穷。

2.6 配置node節(jié)點(diǎn)加入集群

如下操作在所有node節(jié)點(diǎn)操作

# 此命令為初始化master成功后返回的結(jié)果
$   kubeadm join 18.16.202.35:6443 --token ccxrk8.myui0xu4syp99gxu --discovery-token-ca-cert-hash sha256:e3c90ace969aa4d62143e7da6202f548662866dfe33c140095b020031bff2986

異常信息：

[preflight] running pre-flight checks
[discovery] Trying to connect to API Server "18.16.202.35:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://18.16.202.35:6443"
[discovery] Requesting info from "https://18.16.202.35:6443" again to validate TLS against the pinned public key
[discovery] Failed to request cluster info, will try again: [Get https://18.16.202.35:6443/api/v1/namespaces/kube-public/configmaps/cluster-info: x509: certificate has expired or is not yet valid]
[discovery] Failed to request cluster info, will try again: [Get https://18.16.202.35:6443/api/v1/namespaces/kube-public/configmaps/cluster-info: x509: certificate has expired or is not yet valid]

這個(gè)問(wèn)題我這邊是因?yàn)榉?wù)器時(shí)間不對(duì)，調(diào)整以后就ok了藏雏。

查看pods:

[root@node1 flannel]# kubectl get pods -n kube-system
NAME                            READY   STATUS             RESTARTS   AGE
coredns-6c66ffc55b-l76bq        1/1     Running            0          16m
coredns-6c66ffc55b-zlsvh        1/1     Running            0          16m
etcd-node1                      1/1     Running            0          16m
kube-apiserver-node1            1/1     Running            0          16m
kube-controller-manager-node1   1/1     Running            0          15m
kube-flannel-ds-sr6tq           0/1     CrashLoopBackOff   6          7m12s
kube-flannel-ds-ttzhv           1/1     Running            0          9m24s
kube-proxy-nfbg2                1/1     Running            0          7m12s
kube-proxy-r4g7b                1/1     Running            0          16m
kube-scheduler-node1            1/1     Running            0          16m

查看異常pod信息：

[root@node1 flannel]# kubectl  describe pods kube-flannel-ds-sr6tq -n  kube-system
Name:               kube-flannel-ds-sr6tq
Namespace:          kube-system
Priority:           0
PriorityClassName:  <none>
拷况。。掘殴。赚瘦。。
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Pulling    12m                  kubelet, node2     pulling image "registry.cn-shanghai.aliyuncs.com/gcr-k8s/flannel:v0.10.0-amd64"
  Normal   Pulled     11m                  kubelet, node2     Successfully pulled image "registry.cn-shanghai.aliyuncs.com/gcr-k8s/flannel:v0.10.0-amd64"
  Normal   Created    11m                  kubelet, node2     Created container
  Normal   Started    11m                  kubelet, node2     Started container
  Normal   Created    11m (x4 over 11m)    kubelet, node2     Created container
  Normal   Started    11m (x4 over 11m)    kubelet, node2     Started container
  Normal   Pulled     10m (x5 over 11m)    kubelet, node2     Container image "registry.cn-shanghai.aliyuncs.com/gcr-k8s/flannel:v0.10.0-amd64" already present on machine
  Normal   Scheduled  7m15s                default-scheduler  Successfully assigned kube-system/kube-flannel-ds-sr6tq to node2
  Warning  BackOff    7m6s (x23 over 11m)  kubelet, node2     Back-off restarting failed container

遇到這種情況直接 刪除異常pod:

[root@node1 flannel]# kubectl delete pod kube-flannel-ds-sr6tq -n kube-system
pod "kube-flannel-ds-sr6tq" deleted
[root@node1 flannel]# kubectl get pods -n kube-system
NAME                            READY   STATUS    RESTARTS   AGE
coredns-6c66ffc55b-l76bq        1/1     Running   0          17m
coredns-6c66ffc55b-zlsvh        1/1     Running   0          17m
etcd-node1                      1/1     Running   0          16m
kube-apiserver-node1            1/1     Running   0          16m
kube-controller-manager-node1   1/1     Running   0          16m
kube-flannel-ds-7lfrh           1/1     Running   1          6s
kube-flannel-ds-ttzhv           1/1     Running   0          10m
kube-proxy-nfbg2                1/1     Running   0          7m55s
kube-proxy-r4g7b                1/1     Running   0          17m
kube-scheduler-node1            1/1     Running   0          16m

查看節(jié)點(diǎn)：

[root@node1 flannel]# kubectl get nodes -n kube-system
NAME    STATUS   ROLES    AGE     VERSION
node1   Ready    master   17m     v1.12.1
node2   Ready    <none>   8m14s   v1.12.1

參考：

https://www.cnblogs.com/liangDream/p/7358847.html

https://my.oschina.net/binges/blog/1615955?p=2&temp=1521445654544

https://blog.frognew.com/2018/10/kubeadm-install-kubernetes-1.12.html

http://www.reibang.com/p/31bee0cecaf2

https://www.zybuluo.com/ncepuwanghui/note/953929

https://www.kubernetes.org.cn/4256.html

https://note.youdao.com/share/?id=31d9d5db79cc3ae27e72c029b09ac4ab&type=note#/

https://juejin.im/post/5b45d4185188251ac062f27c

http://www.reibang.com/p/02dc13d2f651

https://blog.csdn.net/qq_34857250/article/details/82562514

https://www.cnblogs.com/ssss429170331/p/7685044.html

https://imroc.io/posts/kubernetes/install-kubernetes-1.9-on-centos7-with-kubeadm/