使用外部的 etcd 安裝 cilium 在較大的運(yùn)行環(huán)境中能夠提供更好的性能椅棺。
Requirements
- Kubernetes >= 1.9
- Linux kernel >= 4.9
- ETCD >= 3.1.0
- kubernetes 環(huán)境中安裝了 Helm 3
- Kubernetes in CNI mode
- 在所有 worker node 上掛載 BPF 文件系統(tǒng)
- 推薦：在 kube-controller-manager 上使能 PodCIDR allocation (--allocate-node-cidrs)
安裝 helm 3
- 下載解壓 helm 安裝包
[root@k8s-master-01 ~]# wget https://get.helm.sh/helm-v3.1.2-linux-amd64.tar.gz
[root@k8s-master-01 ~]# tar -zxvf helm-v3.1.2-linux-amd64.tar.gz
[root@k8s-master-01 ~]# mv linux-amd64/helm /usr/local/bin/
- verify
[root@k8s-master-01 ~]# helm help
The Kubernetes package manager
Common actions for Helm:
- helm search: search for charts
- helm pull: download a chart to your local directory to view
- helm install: upload the chart to Kubernetes
- helm list: list releases of charts
Environment variables:
+------------------+-----------------------------------------------------------------------------+
| Name | Description |
+------------------+-----------------------------------------------------------------------------+
| $XDG_CACHE_HOME | set an alternative location for storing cached files. |
| $XDG_CONFIG_HOME | set an alternative location for storing Helm configuration. |
| $XDG_DATA_HOME | set an alternative location for storing Helm data. |
| $HELM_DRIVER | set the backend storage driver. Values are: configmap, secret, memory |
| $HELM_NO_PLUGINS | disable plugins. Set HELM_NO_PLUGINS=1 to disable plugins. |
| $KUBECONFIG | set an alternative Kubernetes configuration file (default "~/.kube/config") |
+------------------+-----------------------------------------------------------------------------+
Helm stores configuration based on the XDG base directory specification, so
- cached files are stored in $XDG_CACHE_HOME/helm
- configuration is stored in $XDG_CONFIG_HOME/helm
- data is stored in $XDG_DATA_HOME/helm
Use "helm [command] --help" for more information about a command.
掛載 BPF 文件系統(tǒng)
- 在所有 k8s node 中掛載 bpf 文件系統(tǒng)
[root@k8s-master-01 ~]# mount bpffs /sys/fs/bpf -t bpf
- verify
[root@k8s-master-01 ~]# mount |grep bpf
bpffs on /sys/fs/bpf type bpf (rw,relatime)
# persistence configuration, don’t worry that ‘bpffs’ displaying as red, seems bpf was new commer, fastab desen’t update that feature.
[root@k8s-master-01 ~]# echo "bpffs /sys/fs/bpf bpf defaults 0 0" >> /etc/fstab
kubernetes 配置
在所有的 kubernetes worker node 中的 kubelet 配置使用 CNI 模式
kubelet.config 中添加--network-plugin=cni在 kube-controller-manager 中使能 PodCIDR
在 controller-manager.config 中添加--allocate-node-cidrs=true
Cilium 配置
當(dāng)使用外部 etcd 作為 cilium 的 k-v 存儲(chǔ)啥刻，etcd 的 IP 地址需要在 cilium 的 configmap 中配置沪羔。
-
使用 helm 安裝 cilium
添加 helm cilium repo
[root@k8s-master-01 ~]# helm repo add cilium https://helm.cilium.io/
- 創(chuàng)建 etcd ssl 證書
[root@k8s-master-01 ~]# kubectl create secret generic -n kube-system cilium-etcd-secrets \
--from-file=etcd-client-ca.crt=/etc/etcd/ssl/ca.crt \
--from-file=etcd-client.key=/etc/etcd/ssl/etcd.key \
--from-file=etcd-client.crt=/etc/etcd/ssl/etcd.crt
-
安裝 cilium
指定 cilium 版本為 v1.7.1, 開啟 SSL 驗(yàn)證，開啟 prometheus 監(jiān)控归斤，添加 etcd cluster 的 menber endpoints
[root@k8s-master-01 ~]# helm install cilium cilium/cilium \
--version 1.7.1\
--set global.etcd.enabled=true\
--set global.etcd.ssl=true\
--set global.prometheus.enabled=true\
--set global.etcd.endpoints[0]=https://172.19.50.7:2379\
--set global.etcd.endpoints[1]=https://172.19.60.32:2379\
--set global.etcd.endpoints[2]=https://172.19.100.16:2379\
--namespace kube-system
NAME: cilium
LAST DEPLOYED: Mon Mar 16 16:44:33 2020
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
- 驗(yàn)證 cilium pod 都安裝成功
[root@k8s-master-01 ~]# kubectl --namespace kube-system get ds cilium
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
cilium 4 4 4 4 4 <none> 13h
[root@k8s-master-01 ~]# kubectl -n kube-system get deployments cilium-operator
NAME READY UP-TO-DATE AVAILABLE AGE
cilium-operator 1/1 1 1 13h
-
安裝 cilium 連接測試用例
此用例將會(huì)部署一系列的 deployment，它們會(huì)使用多種路徑來相互訪問，連接路徑包括帶或者不帶服務(wù)負(fù)載均衡和各種網(wǎng)絡(luò)策略的組合
部署的 podName 表示連接方式颜启，readiness/liveness 探針則可指示連接是否成功
[root@k8s-master-01 ~]# kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/kubernetes/connectivity-check/connectivity-check.yaml -n app-service
[root@k8s-master-01 ~]# kubectl get pods -o wide -n app-service
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
echo-a-58dd59998d-n9g9p 1/1 Running 0 9m13s 10.244.1.50 k8s-master-02 <none> <none>
echo-b-669ccc7765-lzqn7 1/1 Running 0 9m13s 10.244.2.50 k8s-master-03 <none> <none>
host-to-b-multi-node-clusterip-6fb94d9df6-rbjwz 1/1 Running 3 9m13s 192.168.66.226 k8s-master-02 <none> <none>
host-to-b-multi-node-headless-7c4ff79cd-hm6sr 1/1 Running 3 9m13s 192.168.66.226 k8s-master-02 <none> <none>
pod-to-a-5c8dcf69f7-gldq9 1/1 Running 3 9m13s 10.244.2.30 k8s-master-03 <none> <none>
pod-to-a-allowed-cnp-75684d58cc-tf9nn 1/1 Running 1 9m13s 10.244.2.239 k8s-master-03 <none> <none>
pod-to-a-external-1111-669ccfb85f-7r4j8 1/1 Running 0 9m13s 10.244.2.251 k8s-master-03 <none> <none>
pod-to-a-l3-denied-cnp-7b8bfcb66c-wd4nj 1/1 Running 0 9m13s 10.244.2.134 k8s-master-03 <none> <none>
pod-to-b-intra-node-74997967f8-ml5ps 1/1 Running 3 9m13s 10.244.2.95 k8s-master-03 <none> <none>
pod-to-b-multi-node-clusterip-587678cbc4-4qcb2 1/1 Running 3 9m13s 10.244.1.28 k8s-master-02 <none> <none>
pod-to-b-multi-node-headless-574d9f5894-tmfwn 1/1 Running 3 9m13s 10.244.1.138 k8s-master-02 <none> <none>
pod-to-external-fqdn-allow-google-cnp-6dd57bc859-l49z2 1/1 Running 0 9m12s 10.244.2.62 k8s-master-03 <none> <none>
-
安裝 hubble https://github.com/cilium/hubble
hubble 是一個(gè)用于云原生工作負(fù)載的完全分布式網(wǎng)絡(luò)和安全可視化平臺(tái)。它建立在 Cilium 和 eBPF 的基礎(chǔ)上批什，以完全透明的方式深入了解服務(wù)以及網(wǎng)絡(luò)基礎(chǔ)結(jié)構(gòu)的通信和行為农曲。
[root@k8s-master-01 ~]# git clone https://github.com/cilium/hubble.git
[root@k8s-master-01 ~]# cd hubble/install/kubernetes
[root@k8s-master-01 ~]# helm install hubble ./hubble \
--namespace kube-system \
--set metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,http}" \
--set ui.enabled=true
hubble 對(duì)前面安裝的測試用例監(jiān)控信息
image.png
