使用kubeadm工具箱創(chuàng)建kubernetes集群
安裝kubeadm
環(huán)境準(zhǔn)備：CentOS 7 16G Memory 16 CPUs （筆者服務(wù)器配置）
每個(gè)節(jié)點(diǎn)的唯一主機(jī)名畦粮，MAC地址和product_uuid
必須禁用swap為了使kubelet正常工作
swapoff
# 修改/ets/fatab,注釋swap的掛載配置/重要
- 驗(yàn)證每個(gè)節(jié)點(diǎn)的MAC地址和product_uuid是唯一的
# 可以使用命令下面檢查product_uuid：
sudo cat /sys/class/dmi/id/product_uuid
- 檢查網(wǎng)絡(luò)適配器
如果您有多個(gè)網(wǎng)絡(luò)適配器尝艘，并且您的Kubernetes組件在默認(rèn)路由上無法訪問孕讳，我們建議您添加IP路由，以便Kubernetes集群地址通過適當(dāng)?shù)倪m配器
- 檢查所需端口
Master node(s)
| Protocol | Direction | Port Range | Purpose |
|---|---|---|---|
| TCP | Inbound | 6443* | Kubernetes API server |
| TCP | Inbound | 2379-2380 | etcd server client API |
| TCP | Inbound | 10250 | Kubelet API |
| TCP | Inbound | 10251 | kube-scheduler |
| TCP | Inbound | 10252 | kube-controller-manager |
| TCP | Inbound | 10255 | Read-only Kubelet API |
Worker node(s)
| Protocol | Direction | Port Range | Purpose |
|---|---|---|---|
| TCP | Inbound | 10250 | Kubelet API |
| TCP | Inbound | 10255 | Read-only Kubelet API |
| TCP | Inbound | 30000-32767 | NodePort Services** |
默認(rèn)的端口范圍:https://kubernetes.io/docs/concepts/services-networking/service/
安裝Docker
參考：http://blog.csdn.net/wh211212/article/details/78662605
Docker官方安裝：https://docs.docker.com/engine/installation/
yum install -y docker
systemctl enable docker && systemctl start docker
# 不建議使用官網(wǎng)的docker-ce版本、支持性不是很好、使用epel源支持的docker即可
[root@aniu-k8s ~]# docker version
Client:
Version: 1.12.6
API version: 1.24
Package version: docker-1.12.6-68.gitec8512b.el7.centos.x86_64
Go version: go1.8.3
Git commit: ec8512b/1.12.6
Built: Mon Dec 11 16:08:42 2017
OS/Arch: linux/amd64
Server:
Version: 1.12.6
API version: 1.24
Package version: docker-1.12.6-68.gitec8512b.el7.centos.x86_64
Go version: go1.8.3
Git commit: ec8512b/1.12.6
Built: Mon Dec 11 16:08:42 2017
OS/Arch: linux/amd64
在每臺機(jī)器上安裝Docker。建議使用v1.12版本莹痢，但v1.11，v1.13和17.03版本也是可以的墓赴。版本17.06+可能有效竞膳，但尚未經(jīng)過Kubernetes節(jié)點(diǎn)團(tuán)隊(duì)的測試和驗(yàn)證。
請以root身份根據(jù)您的操作系統(tǒng)執(zhí)行以下命令竣蹦。通過SSH連接到每個(gè)主機(jī)后顶猜，您可以通過執(zhí)行sudo -i成為root用戶
- 確保kubelet使用的cgroup驅(qū)動程序與Docker使用的相同。為了確保兼容性痘括，您可以更新Docker，如下所示：
cat << EOF > /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"]
}
EOF
# 這里使用官網(wǎng)配置啟動docker報(bào)錯(cuò)/啟用使用默認(rèn)配置
安裝kubeadm, kubelet and kubectl
kubeadm：引導(dǎo)群集的命令
kubelet：運(yùn)行在集群中所有機(jī)器上的組件，并執(zhí)行諸如啟動pods和容器的組件纲菌。
kubectl: 與集群交互
配置官方kubernetes源：
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
- 配置國內(nèi)kubernetes源
# cat > /etc/yum.repo.d/kubernetes.repo <<EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
EOF
禁用SELinux并關(guān)閉防火墻
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config # 需重啟
setenforce 0
#關(guān)閉防火墻
systemctl stop firewalld && systemctl disable firewalld
RHEL / CentOS 7上的某些用戶報(bào)告了由于iptables被繞過而導(dǎo)致流量被錯(cuò)誤路由的問題挠日。應(yīng)該確保net.bridge.bridge-nf-call-iptables的sysctl配置中被設(shè)置為1，例如
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
- 安裝kubelet kubeadm kubectl
[root@aniu-k8s ~]# yum install -y kubelet kubeadm kubectl
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.163.com
* epel: mirrors.#edu.cn
*
extras: mirrors.163.com
* updates: mirrors.163.com
Package kubelet-1.9.2-0.x86_64 already installed and latest version
Package kubeadm-1.9.2-0.x86_64 already installed and latest version
Package kubectl-1.9.2-0.x86_64 already installed and latest version
Nothing to do
# systemctl enable kubelet && systemctl start kubelet
- 初始化kubeadm翰舌，否則啟動kubelet報(bào)證書錯(cuò)誤
[root@aniu-k8s ~]# kubeadm init --kubernetes-version=v1.9.2
[init] Using Kubernetes version: v1.9.2
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
[WARNING FileExisting-crictl]: crictl not found in system path
[preflight] Starting the kubelet service
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [aniu-k8s kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.10.10]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.
[apiclient] All control plane components are healthy after 75.502276 seconds
[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[markmaster] Will mark node aniu-k8s as master by adding a label and a taint
[markmaster] Master aniu-k8s tainted and labelled with key/value: node-role.kubernetes.io/master=""
[bootstraptoken] Using token: d9b013.26c2f690632cbef9
[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: kube-dns
[addons] Applied essential addon: kube-proxy
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of machines by running the following on each node
as root:
kubeadm join --token d9b013.26c2f690632cbef9 192.168.10.10:6443 --discovery-token-ca-cert-hash sha256:887a2ea3fccca1dec2caa12ad2e54f5baf806f29becf548a3b098ee3a869b518
使用kubeadm創(chuàng)建群集
參考：https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/
kubeadm是一個(gè)工具包嚣潜，可幫助您以簡單，合理安全和可擴(kuò)展的方式引導(dǎo)最佳實(shí)踐Kubernetes群集椅贱。它還支持為您管理Bootstrap令牌并升級/降級群集懂算。
它在設(shè)計(jì)上并不為您安裝網(wǎng)絡(luò)解決方案，這意味著您必須使用kubectl apply自行安裝第三方符合CNI的網(wǎng)絡(luò)解決方案
- 初始化master
[root@aniu-k8s ~]# kubeadm init --kubernetes-version=v1.9.2
要讓kubectl為非root用戶工作庇麦，您可能需要運(yùn)行以下命令（這也是kubeadm init輸出的一部分）：
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
- 查看集群狀態(tài)
[root@aniu-k8s ~]# kubectl get cs
NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health": "true"}
安裝pod network
僅在Master節(jié)點(diǎn)執(zhí)行计技，網(wǎng)絡(luò)必須在任何應(yīng)用程序之前部署。而且山橄，kube-dns是一個(gè)內(nèi)部幫助服務(wù)垮媒，在安裝網(wǎng)絡(luò)之前不會啟動。 kubeadm僅支持基于容器網(wǎng)絡(luò)接口（CNI）的網(wǎng)絡(luò)（并且不支持kubenet）
- 安裝Flannel
# 將橋接的IPv4流量傳遞給iptables的鏈
sysctl net.bridge.bridge-nf-call-iptables=1
#
[root@aniu-k8s ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml
# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
clusterrole "flannel" created
clusterrolebinding "flannel" created
serviceaccount "flannel" created
configmap "kube-flannel-cfg" created
daemonset "kube-flannel-ds" created
一旦安裝了pod網(wǎng)絡(luò)航棱，就可以通過在kubectl get pods --all-namespaces的輸出中檢查kube-dns pod是否正在運(yùn)行來確認(rèn)它正在工作睡雇。 一旦kube-dns吊艙啟動并運(yùn)行，您可以繼續(xù)加入您的節(jié)點(diǎn)
[root@aniu-k8s ~]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system etcd-aniu-k8s 1/1 Running 0 1h
kube-system kube-apiserver-aniu-k8s 1/1 Running 0 1h
kube-system kube-controller-manager-aniu-k8s 1/1 Running 0 1h
kube-system kube-dns-6f4fd4bdf-2428k 0/3 ContainerCreating 0 1h
kube-system kube-flannel-ds-2h2c6 0/1 CrashLoopBackOff 3 1m
kube-system kube-proxy-wt74z 1/1 Running 0 1h
kube-system kube-scheduler-aniu-k8s 1/1 Running 0 1h
- 注意：筆者安裝pod network采用flannel有問題饮醇，故換成Weave Net
export kubever=$(kubectl version | base64 | tr -d '\n')
kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$kubever"
- Master Isolation
使用kubeadm初始化的集群它抱，出于安全考慮Pod不會被調(diào)度到Master Node上，可使用如下命令使Master節(jié)點(diǎn)參與工作負(fù)載朴艰。：
[root@aniu-k8s ~]# kubectl taint nodes --all node-role.kubernetes.io/master-
node "aniu-k8s" untainted
- 查看節(jié)點(diǎn)狀態(tài)
[root@aniu-k8s ~]# kubectl get pod --all-namespaces -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
kube-system etcd-aniu-k8s 1/1 Running 0 49m 192.168.10.10 aniu-k8s
kube-system kube-apiserver-aniu-k8s 1/1 Running 0 49m 192.168.10.10 aniu-k8s
kube-system kube-controller-manager-aniu-k8s 1/1 Running 0 49m 192.168.10.10 aniu-k8s
kube-system kube-dns-6f4fd4bdf-n4ctn 3/3 Running 0 50m 10.32.0.2 aniu-k8s
kube-system kube-proxy-s5pnl 1/1 Running 0 47m 192.168.0.209 aniu-saas-4
kube-system kube-proxy-szs7k 1/1 Running 0 50m 192.168.10.10 aniu-k8s
kube-system kube-scheduler-aniu-k8s 1/1 Running 0 49m 192.168.10.10 aniu-k8s
kube-system weave-net-bkbs2 2/2 Running 0 49m 192.168.10.10 aniu-k8s
kube-system weave-net-cwvdk 2/2 Running 0 47m 192.168.0.209 aniu-saas-4
# 可以看到全部的pod已經(jīng)全部running
- 向K8s集群中加入節(jié)點(diǎn)
節(jié)點(diǎn)是工作負(fù)載（containers and pods）運(yùn)行的地方抗愁。要將新節(jié)點(diǎn)添加到群集，請為每臺機(jī)器執(zhí)行以下操作：
# 節(jié)點(diǎn)需要安裝 yum install -y kubelet kubeadm kubectl
[root@aniu-saas-4 ~]# kubeadm join --token dc2313.9e3daddc83109625 192.168.10.10:6443 --discovery-token-ca-cert-hash sha256:8fe62dea8e88ff957dcd712f3c5948cc43f940abb3f34e8823576434d216ed5a
[preflight] Running pre-flight checks.
[WARNING FileExisting-crictl]: crictl not found in system path
[preflight] Starting the kubelet service
[discovery] Trying to connect to API Server "192.168.10.10:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.10.10:6443"
[discovery] Requesting info from "https://192.168.10.10:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.10.10:6443"
[discovery] Successfully established connection with API Server "192.168.10.10:6443"
This node has joined the cluster:
* Certificate signing request was sent to master and a response
was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the master to see this node join the cluster.
- 查看節(jié)點(diǎn)信息
[root@aniu-k8s ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
aniu-k8s Ready master 1h v1.9.2
aniu-saas-4 NotReady <none> 17s v1.9.2
- 從其他機(jī)器或者筆記本終端操作集群
scp root@<master ip>:/etc/kubernetes/admin.conf .
kubectl --kubeconfig ./admin.conf get nodes
- 將API服務(wù)器代理到本地主機(jī)
scp root@<master ip>:/etc/kubernetes/admin.conf .
kubectl --kubeconfig ./admin.conf proxy
- 刪除Kubernetes集群節(jié)點(diǎn)
kubectl drain <node name> --delete-local-data --force --ignore-daemonsets
kubectl delete node <node name>
注意事項(xiàng)
- 如果kubeadm出錯(cuò)呵晚，修改完成之后需要 kubeadm reset在重啟初始化
- 官網(wǎng)文檔只有修改docker配置哪一步蜘腌，筆者沒有操作，其他步驟和官網(wǎng)一致
錯(cuò)誤
- 查看系統(tǒng)日志仍存在相關(guān)錯(cuò)誤：
#
Error adding network: open /run/flannel/subnet.env: no such file or directory
#
oci-systemd-hook[12470]: systemdhook <debug>: Skipping as container command is /pause, not init or systemd
# 后面繼續(xù)學(xué)習(xí)排查故障
部署Dashboard插件
- 下載Dashboard配置文件
mkdir ~/k8s
cd ~/k8s
wget https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml
- 修改Dashboard Service饵隙，編輯kubernetes-dashboard.yaml文件撮珠，在Dashboard Service中添加type: NodePort，暴露Dashboard服務(wù)
# ------------------- Dashboard Service ------------------- #
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
type: NodePort # 添加
ports:
- port: 443
targetPort: 8443
selector:
k8s-app: kubernetes-dashboard
- 安裝Dashboard插件
[root@aniu-k8s k8s]# kubectl create -f kubernetes-dashboard.yaml
secret "kubernetes-dashboard-certs" created
serviceaccount "kubernetes-dashboard" created
role "kubernetes-dashboard-minimal" created
rolebinding "kubernetes-dashboard-minimal" created
deployment "kubernetes-dashboard" created
service "kubernetes-dashboard" created
- Dashboard賬戶集群管理權(quán)限
創(chuàng)建一個(gè)kubernetes-dashboard-admin的ServiceAccount并授予集群admin的權(quán)限金矛，創(chuàng)建kubernetes-dashboard-admin.rbac.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-admin
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: kubernetes-dashboard-admin
labels:
k8s-app: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard-admin
namespace: kube-system
執(zhí)行命令：
[root@aniu-k8s ~]# kubectl create -f kubernetes-dashboard-admin.rbac.yaml
serviceaccount "kubernetes-dashboard-admin" created
clusterrolebinding "kubernetes-dashboard-admin" created
- 查看kubernete-dashboard-admin的token
[root@aniu-k8s ~]# kubectl -n kube-system get secret | grep kubernetes-dashboard-admin
kubernetes-dashboard-admin-token-c9sq2 kubernetes.io/service-account-token 3 12s
[root@aniu-k8s ~]# kubectl describe -n kube-system secret/kubernetes-dashboard-admin-token-c9sq2
Name: kubernetes-dashboard-admin-token-c9sq2
Namespace: kube-system
Labels: <none>
Annotations: kubernetes.io/service-account.name=kubernetes-dashboard-admin
kubernetes.io/service-account.uid=04821fef-061f-11e8-a2bc-d4ae528a3fba
Type: kubernetes.io/service-account-token
Data
====
ca.crt: 1025 bytes
namespace: 11 bytes
token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbi10b2tlbi1jOXNxMiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJrdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjA0ODIxZmVmLTA2MWYtMTFlOC1hMmJjLWQ0YWU1MjhhM2ZiYSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTprdWJlcm5ldGVzLWRhc2hib2FyZC1hZG1pbiJ9.csPOFxpLHxj-btcmUpOEUFO4MgL5wVL_lSuECALt9aLlb6x72lBRIQZfXnu8MbchhUlDBEX-i4dNU6_nyTKTokbiLwtCbKM12g7wG44aw1c-RjmFRvVxe9tMjjQXEN4ZExHoqtrcU5qTHrXo9qQOy5fyPBc6rbnS7YuPwp6tpofMO9WHdHCp0PejveAKSk6V6f-rPCZuh6ScfCYNF9ytLW-SGY4Kly9DXPR1AYgSdi7y1pu61iqWPgWUMqCzd5qsQ8ml4avOgK-jM-StqoG5_Rftk0sCVoVqfiN4toQhoC28_9TGBu0IKPiM-e1Fo6J4bZ8MrDULHnzs8lMWz1c0lQ
- 查看Dashboard服務(wù)端口
[root@aniu-k8s ~]# kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 14h
kubernetes-dashboard NodePort 10.96.219.54 <none> 443:30760/TCP 2m
