目錄
Calico簡介
Calico 網(wǎng)絡(luò)Node之間通信網(wǎng)絡(luò)
Calico網(wǎng)絡(luò)模型主要工作組件
示例1:安裝calico
calicoctl命令安裝與使用
示例2:修改BGP網(wǎng)絡(luò)

Calico簡介

Calico 是一種容器之間互通的網(wǎng)絡(luò)方案。在虛擬化平臺中舷嗡，比如 OpenStack轴猎、Docker 等都需要實現(xiàn) workloads 之間互連，但同時也需要對容器做隔離控制进萄，就像在 Internet 中的服務(wù)僅開放80端口捻脖、公有云的多租戶一樣，提供隔離和管控機制中鼠。而在多數(shù)的虛擬化平臺實現(xiàn)中可婶，通常都使用二層隔離技術(shù)來實現(xiàn)容器的網(wǎng)絡(luò)，這些二層的技術(shù)有一些弊端援雇，比如需要依賴 VLAN、bridge 和隧道等技術(shù)，其中 bridge 帶來了復(fù)雜性尺迂，vlan 隔離和 tunnel 隧道則消耗更多的資源并對物理環(huán)境有要求掠拳，隨著網(wǎng)絡(luò)規(guī)模的增大，整體會變得越加復(fù)雜筐赔。我們嘗試把 Host 當(dāng)作 Internet 中的路由器铣猩，同樣使用 BGP 同步路由，并使用 iptables 來做安全訪問策略茴丰，最終設(shè)計出了 Calico 方案达皿。

設(shè)計思想：Calico 不使用隧道或 NAT 來實現(xiàn)轉(zhuǎn)發(fā)天吓，而是巧妙的把所有二三層流量轉(zhuǎn)換成三層流量，并通過 host 上路由配置完成跨 Host 轉(zhuǎn)發(fā)

設(shè)計優(yōu)勢：

1.更優(yōu)的資源利用
二層網(wǎng)絡(luò)通訊需要依賴廣播消息機制峦椰，廣播消息的開銷與 host 的數(shù)量呈指數(shù)級增長失仁，Calico 使用的三層路由方法，則完全抑制了二層廣播们何，減少了資源開銷萄焦。

2.可擴展性
Calico 使用與 Internet 類似的方案，Internet 的網(wǎng)絡(luò)比任何數(shù)據(jù)中心都大冤竹，Calico 同樣天然具有可擴展性拂封。

3.簡單而更容易 debug
因為沒有隧道，意味著 workloads 之間路徑更短更簡單鹦蠕，配置更少冒签，在 host 上更容易進(jìn)行 debug 調(diào)試。

4.更少的依賴
Calico 僅依賴三層路由可達(dá)钟病。

5.可適配性
Calico 較少的依賴性使它能適配所有 VM萧恕、Container、白盒或者混合環(huán)境場景肠阱。

Calico 網(wǎng)絡(luò)Node之間通信網(wǎng)絡(luò)

IPIP(可跨網(wǎng)段通信)
從字面來理解票唆，就是把一個IP數(shù)據(jù)包又套在一個IP包里，即把 IP 層封裝到 IP 層的一個 tunnel屹徘。它的作用其實基本上就相當(dāng)于一個基于IP層的網(wǎng)橋走趋！一般來說，普通的網(wǎng)橋是基于mac層的噪伊，根本不需 IP簿煌，而這個 ipip 則是通過兩端的路由做一個 tunnel，把兩個本來不通的網(wǎng)絡(luò)通過點對點連接起來鉴吹。
類似vxlan 但封裝開銷比vxlan小 效率相對更高一些姨伟，但安全性也更差

Vxlan(可跨網(wǎng)段通信)
與Flannel Vxlan原理相同

BGP(二層網(wǎng)絡(luò)通信)
邊界網(wǎng)關(guān)協(xié)議（Border Gateway Protocol, BGP）是互聯(lián)網(wǎng)上一個核心的去中心化自治路由協(xié)議。它通過維護(hù)IP路由表或‘前綴’表來實現(xiàn)自治系統(tǒng)（AS）之間的可達(dá)性豆励，屬于矢量路由協(xié)議夺荒。BGP不使用傳統(tǒng)的內(nèi)部網(wǎng)關(guān)協(xié)議（IGP）的指標(biāo)，而使用基于路徑肆糕、網(wǎng)絡(luò)策略或規(guī)則集來決定路由般堆。因此诚啃，它更適合被稱為矢量性協(xié)議，而不是路由協(xié)議和橙。BGP仔燕，通俗的講就是講接入到機房的多條線路（如電信魔招、聯(lián)通、移動等）融合為一體办斑，實現(xiàn)多線單IP外恕，BGP 機房的優(yōu)點：服務(wù)器只需要設(shè)置一個IP地址乡翅，最佳訪問路由是由網(wǎng)絡(luò)上的骨干路由器根據(jù)路由跳數(shù)與其它技術(shù)指標(biāo)來確定的，不會占用服務(wù)器的任何系統(tǒng)
實際上尚洽，Calico 項目提供的 BGP 網(wǎng)絡(luò)解決方案靶累，與 Flannel 的 host-gw 模式幾乎一樣挣柬。也就是說，Calico也是基于路由表實現(xiàn)容器數(shù)據(jù)包轉(zhuǎn)發(fā)澈灼，但不同于Flannel使用flanneld進(jìn)程來維護(hù)路由信息的做法，而Calico項目使用BGP協(xié)議來自動維護(hù)整個集群的路由信息委乌。

部署推薦方案:
BGP+Vxlan

其中BGP在官方的推薦方案中 以50個節(jié)點為界區(qū)別了不同規(guī)模使用不同的部署方案

• 小規(guī)模網(wǎng)絡(luò):BGP peer 一對一網(wǎng)絡(luò)：每個節(jié)點都是有N-1條路由遭贸，小型網(wǎng)絡(luò)適用壕吹，當(dāng)節(jié)點數(shù)N變多時，路由表更新及AIP-SERVER都需要承受很大的壓力 類似網(wǎng)絡(luò)拓?fù)浣Y(jié)構(gòu)中的 網(wǎng)狀拓?fù)浣Y(jié)構(gòu)

• 大規(guī)模網(wǎng)絡(luò):BGP Reflector 路由反射器：選擇一到多個節(jié)點做為Reflector踏堡，所有節(jié)點路由都匯總給Reflector顷蟆，所有節(jié)點都路由都指向Reflector ，適合大型網(wǎng)絡(luò)逐纬，類似網(wǎng)絡(luò)拓?fù)浣Y(jié)構(gòu)中的星型網(wǎng)絡(luò)

Calico網(wǎng)絡(luò)模型主要工作組件：###

1. Felix：運行在每一臺 Host 的 agent 進(jìn)程，主要負(fù)責(zé)網(wǎng)絡(luò)接口管理和監(jiān)聽豁生、路由甸箱、ARP 管理绕辖、ACL 管理和同步仪际、狀態(tài)上報等。
2. etcd：分布式鍵值存儲肯适，主要負(fù)責(zé)網(wǎng)絡(luò)元數(shù)據(jù)一致性框舔，確保Calico網(wǎng)絡(luò)狀態(tài)的準(zhǔn)確性刘绣，可以與kubernetes共用挣输；
3. BGP Client（BIRD）：Calico 為每一臺 Host 部署一個 BGP Client撩嚼，使用 BIRD 實現(xiàn)完丽，BIRD 是一個單獨的持續(xù)發(fā)展的項目，實現(xiàn)了眾多動態(tài)路由協(xié)議比如 BGP蜻底、OSPF朱躺、RIP 等长搀。在 Calico 的角色是監(jiān)聽 Host 上由 Felix 注入的路由信息，然后通過 BGP 協(xié)議廣播告訴剩余 Host 節(jié)點枪芒，從而實現(xiàn)網(wǎng)絡(luò)互通舅踪。
4. BGP Route Reflector：在大型網(wǎng)絡(luò)規(guī)模中抽碌，如果僅僅使用 BGP client 形成 mesh 全網(wǎng)互聯(lián)的方案就會導(dǎo)致規(guī)模限制决瞳，因為所有節(jié)點之間倆倆互聯(lián)皮胡，需要 N^2 個連接屡贺，為了解決這個規(guī)模問題甩栈，可以采用 BGP 的 Router Reflector 的方法量没，使所有 BGP Client 僅與特定 RR 節(jié)點互聯(lián)并做路由同步，從而大大減少連接數(shù)。

Calico有兩種運行方式饶套，

1. 是讓calico/node獨立運行于Kubernetes集群之外妓蛮，但calico/kube-controllers依然需要以Pod資源運行中集群之上;
2. 是以CNI插件方式配置Calico完全托管運行于Kubernetes集群之上蛤克，類似于我們前面曾經(jīng)部署托管Flannel網(wǎng)絡(luò)插件的方式。
   對于后一種方式,Calico提供了在線的部署清單髓介，它分別為50節(jié)點及以下規(guī)模和50節(jié)點以上規(guī)模的Kubernetes集群使用Kubernetes API作為Dabastore提供了不同的配置清單唐础，也為使用獨立的etcd集群提供了專用配置清單一膨。但這3種類型的配置清單中,Calico默認(rèn)啟用的是基于IPIP隧道的疊加網(wǎng)絡(luò)豹绪，因而它會在所有流量上使用IPIP隧道而不是BGP路由瞒津。以下配置定義在部署清單中DaemonSet/calico-node資源的Pod模板中的calico-node容器之上豺型。

配置選項
在IPv4類型的地址池上啟用的IPIP及其類型姻氨，支持3種可用值
Always(全局流量)肴焊、Cross-SubNet(跨子網(wǎng)流量)和Never3種可用值

• name: CALICO_IPV4POOL_IPIP
  value: "Always"

• 是否在IPV4地址池上啟用VXLAN隧道協(xié)議娶眷，取值及意義與Flannel的VXLAN后端相同;但在全局流量啟用VXLAN時將完全不再需要BGP網(wǎng)絡(luò)届宠，建議將相關(guān)的組件禁用

• name: CALICO_ IPV4POOL_VXLAN
  value: "Never"

• 需要注意的是，Calico分配的地址池需要同Ktbernetes集群的Pod網(wǎng)絡(luò)的定義保持一致伤塌。Pod網(wǎng)絡(luò)通常由kubeadm init初始化集群時使用--pod-network-cidr選項指定的網(wǎng)絡(luò)每聪，而Calico在其默認(rèn)的配置清單中默認(rèn)使用192.168.0.0/16作為Pod網(wǎng)絡(luò)药薯，因而部署Kubernetes集群時應(yīng)該規(guī)劃好要使用的網(wǎng)絡(luò)地址童本，并設(shè)定此二者相匹配巾陕。對于曾經(jīng)使用了flannel的默認(rèn)的10.244.0.0/16網(wǎng)絡(luò)的環(huán)境而言,我們也可以選擇修改資源清單中的定義，從而將其修改為其他網(wǎng)絡(luò)地址,它定義在DaemonSet/calico-node資源的Pod模板中的calico-node容器之上晾匠。

官網(wǎng)鏈接：

https://docs.projectcalico.org/getting-started/kubernetes/self-managed-onprem/onpremises

示例1:安裝calico

wget https://docs.projectcalico.org/manifests/calico.yaml

[root@k8s-master ~]# cd /etc/kubernetes/manifests/
[root@k8s-master manifests]# cat kube-controller-manager.yaml 
...
System Info:
  Machine ID:                 32599e2a74704b2e95443e24ea15d4f6
  System UUID:                34979a62-16de-4287-b149-2d4c2d8a70fb
  Boot ID:                    f31de60e-4f89-4553-ba7a-99a46d049936
  Kernel Version:             5.4.109-1.el7.elrepo.x86_64
  OS Image:                   CentOS Linux 7 (Core)
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://20.10.7
  Kubelet Version:            v1.19.9
  Kube-Proxy Version:         v1.19.9
PodCIDR:                      10.244.1.0/24  #第個節(jié)點的地址塊都是由K8S分配
PodCIDRs:                     10.244.1.0/24   
Non-terminated Pods:          (17 in total)

[root@k8s-master ~]# kubectl describe node k8s-node1
System Info:
  Machine ID:                 32599e2a74704b2e95443e24ea15d4f6
  System UUID:                34979a62-16de-4287-b149-2d4c2d8a70fb
  Boot ID:                    f31de60e-4f89-4553-ba7a-99a46d049936
  Kernel Version:             5.4.109-1.el7.elrepo.x86_64
  OS Image:                   CentOS Linux 7 (Core)
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://20.10.7
  Kubelet Version:            v1.19.9
  Kube-Proxy Version:         v1.19.9
PodCIDR:                      10.244.1.0/24   #每個Node Pod都是由K8S分配IP
PodCIDRs:                     10.244.1.0/24

[root@k8s-master Network]# vim calico.yaml
...
 "ipam": {
              "type": "host-local",
              "subnet": "usePodCidr"   #使用k8s ipam插件分配地址 
          },
          "policy": {
              "type": "k8s"
          },

...
- name: FELIX_WIREGUARDMTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            # The default IPv4 pool to create on startup if none exists. Pod IPs will be
            # chosen from this range. Changing this value after installation will have
            # no effect. This should fall within `--cluster-cidr`.
            - name: CALICO_IPV4POOL_CIDR
              value: "10.244.0.0/16"  #為了和之前的flannel 10.244.0.0/16適配
            - name: CALICO_IPV4POOL_BLOCK_SIZE  #添加這一行修改默認(rèn)塊大小
              value: "24"
            - name: USE_POD_CIDR  #使用K8S的分配的IP地址亡资，不然calico和K8S分配的地址會不一樣
              value: "true"

• 安裝calico

[root@k8s-master plugin]# kubectl delete -f kube-flannel.yml
podsecuritypolicy.policy "psp.flannel.unprivileged" deleted
clusterrole.rbac.authorization.k8s.io "flannel" deleted
clusterrolebinding.rbac.authorization.k8s.io "flannel" deleted
serviceaccount "flannel" deleted
configmap "kube-flannel-cfg" deleted
daemonset.apps "kube-flannel-ds" deleted

[root@k8s-master plugin]# kubectl apply -f calico.yaml 
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
poddisruptionbudget.policy/calico-kube-controllers created

• calico幾個組件

[root@k8s-master ~]# ps aux|grep calico
root     10867  0.0  0.1 112816  2156 pts/1    S+   13:51   0:00 grep --color=auto calico
root     20680  0.0  2.3 1215184 35216 ?       Sl   10:27   0:06 calico-node -allocate-tunnel-addrs
root     20681  0.0  2.1 1215184 32672 ?       Sl   10:27   0:06 calico-node -monitor-addresses
root     20682  2.4  3.3 1510624 51636 ?       Sl   10:27   4:54 calico-node -felix
root     20683  0.0  2.3 1657832 35496 ?       Sl   10:27   0:09 calico-node -confd
root     20686  0.0  2.0 1214928 31628 ?       Sl   10:27   0:05 calico-node -monitor-token

• 因為calico并沒有使用k8s的ipam分配IP,所以節(jié)點會有2個IP,一個是K8S分配的IP 一個是calico分配的IP

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.54.2    0.0.0.0         UG    101    0        0 eth4
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
192.168.4.0     0.0.0.0         255.255.255.0   U     100    0        0 eth0
192.168.12.0    192.168.4.172   255.255.255.0   UG    0      0        0 tunl0  #可以看到tunl0的路由信息
192.168.51.0    192.168.4.173   255.255.255.0   UG    0      0        0 tunl0  #同時可以看到節(jié)點的IP不像之前一定是連續(xù)的
192.168.54.0    0.0.0.0         255.255.255.0   U     101    0        0 eth4
192.168.113.0   192.168.4.171   255.255.255.0   UG    0      0        0 tunl0  #隧道接口
192.168.237.0   0.0.0.0         255.255.255.0   U     0      0        0 *
192.168.237.1   0.0.0.0         255.255.255.255 UH    0      0        0 cali7c0fb624285
192.168.237.2   0.0.0.0         255.255.255.255 UH    0      0        0 caliedaf285d4ef
192.168.237.3   0.0.0.0         255.255.255.255 UH    0      0        0 cali854da94d42a


[root@k8s-master calico]# ip route list
default via 192.168.54.2 dev eth4 proto static metric 101 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
192.168.4.0/24 dev eth0 proto kernel scope link src 192.168.4.170 metric 100 
192.168.12.0/24 via 192.168.4.172 dev tunl0 proto bird onlink   #可以看到tunl0的路由信息
192.168.51.0/24 via 192.168.4.173 dev tunl0 proto bird onlink    
192.168.54.0/24 dev eth4 proto kernel scope link src 192.168.54.170 metric 101 
192.168.113.0/24 via 192.168.4.171 dev tunl0 proto bird onlink 
blackhole 192.168.237.0/24 proto bird 
192.168.237.1 dev cali7c0fb624285 scope link 
192.168.237.2 dev caliedaf285d4ef scope link 
192.168.237.3 dev cali854da94d42a scope link 

• 192.168.51.0/24 via 192.168.4.173 dev tunl0 proto bird onlink 下面的路由表可以看到 calico會為每個節(jié)點分配網(wǎng)絡(luò)地址段 并不是使用節(jié)點的網(wǎng)絡(luò)地址

[root@k8s-node1 ~]# ip route list
default via 192.168.54.2 dev eth4 proto static metric 101 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
192.168.4.0/24 dev eth0 proto kernel scope link src 192.168.4.171 metric 100 
192.168.12.0/24 via 192.168.4.172 dev tunl0 proto bird onlink 
192.168.51.0/24 via 192.168.4.173 dev tunl0 proto bird onlink 
192.168.54.0/24 dev eth4 proto kernel scope link src 192.168.54.171 metric 101 
blackhole 192.168.113.0/24 proto bird   #黑洞 代表自己網(wǎng)段 
192.168.237.0/24 via 192.168.4.170 dev tunl0 proto bird onlink

• 查看目前工作模式

[root@k8s-master calico]# kubectl api-resources
NAME                              SHORTNAMES   APIGROUP                       NAMESPACED   KIND
bgpconfigurations                              crd.projectcalico.org          false        BGPConfiguration
bgppeers                                       crd.projectcalico.org          false        BGPPeer
blockaffinities                                crd.projectcalico.org          false        BlockAffinity
clusterinformations                            crd.projectcalico.org          false        ClusterInformation
felixconfigurations                            crd.projectcalico.org          false        FelixConfiguration
globalnetworkpolicies                          crd.projectcalico.org          false        GlobalNetworkPolicy
globalnetworksets                              crd.projectcalico.org          false        GlobalNetworkSet
hostendpoints                                  crd.projectcalico.org          false        HostEndpoint
ipamblocks                                     crd.projectcalico.org          false        IPAMBlock
ipamconfigs                                    crd.projectcalico.org          false        IPAMConfig
ipamhandles                                    crd.projectcalico.org          false        IPAMHandle
ippools                                        crd.projectcalico.org          false        IPPool  #calico地址池
kubecontrollersconfigurations                  crd.projectcalico.org          false        KubeControllersConfiguration
networkpolicies                                crd.projectcalico.org          true         NetworkPolicy
networksets                                    crd.projectcalico.org          true         NetworkSet


[root@k8s-master calico]# kubectl get  ippools -o yaml
....
  spec:
    blockSize: 24   #掩碼長度
    cidr: 192.168.0.0/16  #地址池
    ipipMode: Always      #可以看到目前為ipip模式
    natOutgoing: true
    nodeSelector: all()

• 訪問抓包

[root@k8s-master PodControl]# kubectl get pod -o wide
NAME                              READY   STATUS    RESTARTS   AGE    IP             NODE        NOMINATED NODE   READINESS GATES
deployment-demo-fb544c5d8-r7pc8   1/1     Running   0          8m3s   192.168.51.1   k8s-node3   <none>           <none>
deployment-demo-fb544c5d8-splfr   1/1     Running   0          8m3s   192.168.12.1   k8s-node2   <none>           <none>
[root@k8s-master PodControl]# kubectl exec deployment-demo-fb544c5d8-r7pc8 -it -- /bin/sh
[root@deployment-demo-fb544c5d8-r7pc8 /]# ifconfig
eth0      Link encap:Ethernet  HWaddr 16:96:97:3F:F3:C5  
          inet addr:192.168.51.1  Bcast:192.168.51.1  Mask:255.255.255.255
          UP BROADCAST RUNNING MULTICAST  MTU:1480  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

[root@deployment-demo-fb544c5d8-r7pc8 /]# curl 192.168.12.1
iKubernetes demoapp v1.0 !! ClientIP: 192.168.51.1, ServerName: deployment-demo-fb544c5d8-splfr, ServerIP: 192.168.12.1!
[root@deployment-demo-fb544c5d8-r7pc8 /]# curl 192.168.12.1
iKubernetes demoapp v1.0 !! ClientIP: 192.168.51.1, ServerName: deployment-demo-fb544c5d8-splfr, ServerIP: 192.168.12.1!
[root@deployment-demo-fb544c5d8-r7pc8 /]# curl 192.168.12.1



[root@k8s-node2 ~]# tcpdump -i eth0 -nn ip host 192.168.4.172  and host 192.168.4.173
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:48:24.421003 IP 192.168.4.173 > 192.168.4.172: IP 192.168.51.1.33436 > 192.168.12.1.80: Flags [S], seq 3259804851, win 64800, options [mss 1440,sackOK,TS val 2008488248 ecr 0,nop,wscale 7], length 0 (ipip-proto-4)
11:48:24.421093 IP 192.168.4.172 > 192.168.4.173: IP 192.168.12.1.80 > 192.168.51.1.33436: Flags [S.], seq 3234480084, ack 3259804852, win 64260, options [mss 1440,sackOK,TS val 1053230437 ecr 2008488248,nop,wscale 7], length 0 (ipip-proto-4)  #可以看到(ipip-proto-4)為IPIP模式
11:48:24.422305 IP 192.168.4.173 > 192.168.4.172: IP 192.168.51.1.33436 > 192.168.12.1.80: Flags [.], ack 1, win 507, options [nop,nop,TS val 2008488250 ecr 1053230437], length 0 (ipip-proto-4)
11:48:24.422308 IP 192.168.4.173 > 192.168.4.172: IP 192.168.51.1.33436 > 192.168.12.1.80: Flags [P.], seq 1:77, ack 1, win 507, options [nop,nop,TS val 2008488250 ecr 1053230437], length 76: HTTP: GET / HTTP/1.1 (ipip-proto-4)
11:48:24.422554 IP 192.168.4.172 > 192.168.4.173: IP 192.168.12.1.80 > 192.168.51.1.33436: Flags [.], ack 77, win 502, options [nop,nop,TS val 1053230439 ecr 2008488250], length 0 (ipip-proto-4)
11:48:24.431688 IP 192.168.4.172 > 192.168.4.173: IP 192.168.12.1.80 > 192.168.51.1.33436: Flags [P.], seq 1:18, ack 77, win 502, options [nop,nop,TS val 1053230447 ecr 2008488250], length 17: HTTP: HTTP/1.0 200 OK (ipip-proto-4)
11:48:24.432638 IP 192.168.4.172 > 192.168.4.173: IP 192.168.12.1.80 > 192.168.51.1.33436: Flags [FP.], seq 18:276, ack 77, win 502, options [nop,nop,TS val 1053230449 ecr 2008488250], length 258: HTTP (ipip-proto-4)
11:48:24.433660 IP 192.168.4.173 > 192.168.4.172: IP 192.168.51.1.33436 > 192.168.12.1.80: Flags [.], ack 18, win 507, options [nop,nop,TS val 2008488261 ecr 1053230447], length 0 (ipip-proto-4)
11:48:24.437531 IP 192.168.4.173 > 192.168.4.172: IP 192.168.51.1.33436 > 192.168.12.1.80: Flags [F.], seq 77, ack 277, win 505, options [nop,nop,TS val 2008488261 ecr 1053230449], length 0 (ipip-proto-4)
11:48:24.437775 IP 192.168.4.172 > 192.168.4.173: IP 192.168.12.1.80 > 192.168.51.1.33436: Flags [.], ack 78, win 502, options [nop,nop,TS val 1053230454 ecr 2008488261], length 0 (ipip-proto-4)
IP 192.168.4.172 > 192.168.4.173: IP 192.168.12.1.80 > 192.168.51.1.33436 

• 可以看到默認(rèn)為ipip模式 也是經(jīng)過封裝在轉(zhuǎn)發(fā)的 和Flannel很類似,但相對Flannel經(jīng)過虛擬網(wǎng)橋CNI calico直接內(nèi)核(內(nèi)核的路由由 kube-proxy或IPVS生成)到在由tunl0傳輸想對Flannel少了一層交換機交換的過程,性能相比Flannel會快一些 但這并不是calico最佳的模式

calicoctl命令安裝與使用

calicoctl安裝的2種方式
第1種方式 calicoctl

https://docs.projectcalico.org/getting-started/clis/calicoctl/install

• 幾種方式運行calicoctl 常用方式1:直接下載2進(jìn)制calicoctl 直接運行

[root@k8s-master ~]# curl -o calicoctl -O -L  "https://github.com/projectcalico/calicoctl/releases/download/v3.20.0/calicoctl"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   615  100   615    0     0    498      0  0:00:01  0:00:01 --:--:--   504
100 43.2M  100 43.2M    0     0   518k      0  0:01:25  0:01:25 --:--:--  920k

[root@k8s-master ~]# mv calicoctl /usr/bin/

[root@k8s-master ~]# chmod +x /usr/bin/calicoctl 
[root@k8s-master ~]# calicoctl  --help
Usage:
  calicoctl [options] <command> [<args>...]

    create       Create a resource by file, directory or stdin.
    replace      Replace a resource by file, directory or stdin.
    apply        Apply a resource by file, directory or stdin.  This creates a resource
                 if it does not exist, and replaces a resource if it does exists.
    patch        Patch a pre-exisiting resource in place.
    delete       Delete a resource identified by file, directory, stdin or resource type and
                 name.
    get          Get a resource identified by file, directory, stdin or resource type and
                 name.
    label        Add or update labels of resources.
    convert      Convert config files between different API versions.
    ipam         IP address management.
    node         Calico node management.
    version      Display the version of this binary.
    export       Export the Calico datastore objects for migration
    import       Import the Calico datastore objects for migration
    datastore    Calico datastore management.

• calicoctl 命令使用

- calicoctl 默認(rèn)會讀取 ~/.kube/下文件加載認(rèn)證信息,也可以通過配置文件指定認(rèn)證信息位置

[root@k8s-master calico]# mkdir /etc/calico/^C
[root@k8s-master calico]# cd /etc/calico/
[root@k8s-master calico]# cat calicoctl.cfg 
apiVersion: projectcalico.org/v3
kind: CalicoAPIConfig
metadata:
spec:
  datastoreType: "kubernetes"
  kubeconfig: "/etc/kubernetes/admin.conf" #指定conf路徑
[root@k8s-master calico]# 

[root@k8s-master calico]# kubectl get ippools
NAME                  AGE
default-ipv4-ippool   23h
[root@k8s-master calico]# calicoctl get ippool  #可以用calicoctl直接訪問 calico資源
NAME                  CIDR             SELECTOR   
default-ipv4-ippool   192.168.0.0/16   all() 


[root@k8s-master calico]# calicoctl get ippool default-ipv4-ippool -o yaml
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  creationTimestamp: "2021-08-29T14:33:53Z"
  name: default-ipv4-ippool
  resourceVersion: "1305"
  uid: c01d73f3-c0c9-4674-b27e-725a1eaa5717
spec:
  blockSize: 24
  cidr: 192.168.0.0/16
  ipipMode: Always
  natOutgoing: true
  nodeSelector: all()
  vxlanMode: Never


[root@k8s-master calico]# calicoctl ipam --help  
Usage:
  calicoctl [options] [<args>...]

Options:
  -h --help                 Show this screen.
  -c --config=<config>      Path to the file containing connection
                            configuration in YAML or JSON format.
                            [default: /etc/calico/calicoctl.cfg]
  --context=<context>       The name of the kubeconfig context to use.
  -a
  -A --all-namespaces
     --as=<AS_NUM>
     --backend=(bird|gobgp|none)
     --dryrun
     --export
     --felix-config=<CONFIG>
  -f --filename=<FILENAME>
     --force
     --from-report=<REPORT>
     --ignore-validation
     --init-system
     --ip6-autodetection-method=<IP6_AUTODETECTION_METHOD>
     --ip6=<IP6>
     --ip-autodetection-method=<IP_AUTODETECTION_METHOD>
...

[root@k8s-master calico]# calicoctl ipam show
+----------+----------------+-----------+------------+--------------+
| GROUPING |      CIDR      | IPS TOTAL | IPS IN USE |   IPS FREE   |
+----------+----------------+-----------+------------+--------------+
| IP Pool  | 192.168.0.0/16 |     65536 | 9 (0%)     | 65527 (100%) |
+----------+----------------+-----------+------------+--------------+


[root@k8s-master calico]# calicoctl ipam show  --show-blocks  #每個地址段使用了多少個
+----------+------------------+-----------+------------+--------------+
| GROUPING |       CIDR       | IPS TOTAL | IPS IN USE |   IPS FREE   |
+----------+------------------+-----------+------------+--------------+
| IP Pool  | 192.168.0.0/16   |     65536 | 9 (0%)     | 65527 (100%) |
| Block    | 192.168.113.0/24 |       256 | 1 (0%)     | 255 (100%)   |
| Block    | 192.168.12.0/24  |       256 | 2 (1%)     | 254 (99%)    |
| Block    | 192.168.237.0/24 |       256 | 4 (2%)     | 252 (98%)    |
| Block    | 192.168.51.0/24  |       256 | 2 (1%)     | 254 (99%)    |
+----------+------------------+-----------+------------+--------------+

[root@k8s-master calico]# calicoctl ipam show  --show-config  #查看配置信息
+--------------------+-------+
|      PROPERTY      | VALUE |
+--------------------+-------+
| StrictAffinity     | false |
| AutoAllocateBlocks | true  |
| MaxBlocksPerHost   |     0 |
+--------------------+-------+

第2種方式 以kubectl插件方式運行

[root@k8s-master calico]# cp -p /usr/bin/calicoctl  /usr/bin/kubectl-calico  #把之前的文件改個名字就可以了

[root@k8s-master calico]# kubectl calico
Usage:
  kubectl-calico [options] <command> [<args>...]
Invalid option: ''. Use flag '--help' to read about a specific subcommand

[root@k8s-master calico]# kubectl calico get nodes  #和第1種方式相比加kubectl
NAME         
k8s-master   
k8s-node1    
k8s-node2    
k8s-node3    

[root@k8s-master calico]# kubectl calico ipam show
+----------+----------------+-----------+------------+--------------+
| GROUPING |      CIDR      | IPS TOTAL | IPS IN USE |   IPS FREE   |
+----------+----------------+-----------+------------+--------------+
| IP Pool  | 192.168.0.0/16 |     65536 | 9 (0%)     | 65527 (100%) |
+----------+----------------+-----------+------------+--------------+

示例2:修改BGP網(wǎng)絡(luò)

#獲取現(xiàn)有配置在此基礎(chǔ)上修改

[root@k8s-master calico]# kubectl calico get ippool -o yaml > default-ipv4-ippool.yaml 

[root@k8s-master calico]# cat default-ipv4-ippool.yaml 
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: default-ipv4-ippool
spec:
  blockSize: 24
  cidr: 192.168.0.0/16
  ipipMode: CrossSubnet #跨節(jié)點子網(wǎng)時使用IPIP 沒有跨子網(wǎng)使用BGP
  natOutgoing: true
  nodeSelector: all()
  vxlanMode: Never  #vxlanMode與ipipMode不能同時打開 必須有一個為Never
  
#通過ipipMode廊勃、vxlanMode不同選項可以使calico運行在純GBP经窖、ipip画侣、vxlanMode或混合模式下
#如:ipipMode: Never vxlanMode: Never 為純BGP模式  ipipMode: Never vxlanMode: CrossSubnet 為BGP+vxlan模式

[root@k8s-master calico]# calicoctl apply -f default-ipv4-ippool.yaml 
Successfully applied 1 'IPPool' resource(s)

#在來看路由信息 已經(jīng)沒有之前的tunl0 直接從節(jié)點網(wǎng)絡(luò)出去
[root@k8s-master calico]# ip route list
default via 192.168.54.2 dev eth4 proto static metric 101 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
192.168.4.0/24 dev eth0 proto kernel scope link src 192.168.4.170 metric 100 
192.168.12.0/24 via 192.168.4.172 dev eth0 proto bird 
192.168.51.0/24 via 192.168.4.173 dev eth0 proto bird     #已經(jīng)沒有之前的tunl0隧道
192.168.54.0/24 dev eth4 proto kernel scope link src 192.168.54.170 metric 101 
192.168.113.0/24 via 192.168.4.171 dev eth0 proto bird 
blackhole 192.168.237.0/24 proto bird 
192.168.237.1 dev cali7c0fb624285 scope link 
192.168.237.2 dev caliedaf285d4ef scope link 
192.168.237.3 dev cali854da94d42a scope link 
[root@k8s-master calico]# 

抓包測試

[root@k8s-master ~]# kubectl get pod -o wide
NAME                              READY   STATUS    RESTARTS   AGE   IP             NODE        NOMINATED NODE   READINESS GATES
deployment-demo-fb544c5d8-r7pc8   1/1     Running   0          10h   192.168.51.1   k8s-node3   <none>           <none>
deployment-demo-fb544c5d8-splfr   1/1     Running   0          10h   192.168.12.1   k8s-node2   <none>           <none>

#從節(jié)點3訪問節(jié)點3 
[root@k8s-master calico]# kubectl exec deployment-demo-fb544c5d8-r7pc8 -it -- /bin/sh
[root@deployment-demo-fb544c5d8-r7pc8 /]# curl 192.168.12.1
iKubernetes demoapp v1.0 !! ClientIP: 192.168.51.1, ServerName: deployment-demo-fb544c5d8-splfr, ServerIP: 192.168.12.1!
[root@deployment-demo-fb544c5d8-r7pc8 /]# curl 192.168.12.1
iKubernetes demoapp v1.0 !! ClientIP: 192.168.51.1, ServerName: deployment-demo-fb544c5d8-splfr, ServerIP: 192.168.12.1!

#直接抓Pod IP的包 因為沒有封裝 所以是Pod IP直接通信 沒有外層IP

[root@k8s-node2 ~]# tcpdump -i eth0 -nn ip host 192.168.51.1  and host 192.168.12.1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
22:11:54.704770 IP 192.168.51.1.33464 > 192.168.12.1.80: Flags [S], seq 4075444778, win 64800, options [mss 1440,sackOK,TS val 2045898534 ecr 0,nop,wscale 7], length 0
22:11:54.705866 IP 192.168.12.1.80 > 192.168.51.1.33464: Flags [S.], seq 402120893, ack 4075444779, win 64260, options [mss 1440,sackOK,TS val 1090640722 ecr 2045898534,nop,wscale 7], length 0
22:11:54.706670 IP 192.168.51.1.33464 > 192.168.12.1.80: Flags [.], ack 1, win 507, options [nop,nop,TS val 2045898537 ecr 1090640722], length 0
22:11:54.707077 IP 192.168.51.1.33464 > 192.168.12.1.80: Flags [P.], seq 1:77, ack 1, win 507, options [nop,nop,TS val 2045898537 ecr 1090640722], length 76: HTTP: GET / HTTP/1.1
22:11:54.707132 IP 192.168.12.1.80 > 192.168.51.1.33464: Flags [.], ack 77, win 502, options [nop,nop,TS val 1090640723 ecr 2045898537], length 0
22:11:54.737231 IP 192.168.12.1.80 > 192.168.51.1.33464: Flags [P.], seq 1:18, ack 77, win 502, options [nop,nop,TS val 1090640754 ecr 2045898537], length 17: HTTP: HTTP/1.0 200 OK
22:11:54.738439 IP 192.168.51.1.33464 > 192.168.12.1.80: Flags [.], ack 18, win 507, options [nop,nop,TS val 2045898568 ecr 1090640754], length 0
22:11:54.739117 IP 192.168.12.1.80 > 192.168.51.1.33464: Flags [P.], seq 18:155, ack 77, win 502, options [nop,nop,TS val 1090640755 ecr 2045898568], length 137: HTTP
22:11:54.739630 IP 192.168.12.1.80 > 192.168.51.1.33464: Flags [FP.], seq 155:276, ack 77, win 502, options [nop,nop,TS val 1090640756 ecr 2045898568], length 121: HTTP
22:11:54.739810 IP 192.168.51.1.33464 > 192.168.12.1.80: Flags [.], ack 155, win 506, options [nop,nop,TS val 2045898570 ecr 1090640755], length 0


[root@k8s-master calico]# calicoctl  node status 
Calico process is running.

IPv4 BGP status  #可以看到已經(jīng)BGP模式了 這里看到是除去自己其它的3個節(jié)點
+---------------+-------------------+-------+----------+-------------+
| PEER ADDRESS  |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+---------------+-------------------+-------+----------+-------------+
| 192.168.4.171 | node-to-node mesh | up    | 02:27:59 | Established |
| 192.168.4.172 | node-to-node mesh | up    | 02:27:58 | Established |
| 192.168.4.173 | node-to-node mesh | up    | 02:27:58 | Established |
+---------------+-------------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.

• 到目前為止 如果是小規(guī)模的集群 比如50臺以下 就可以直接使用了
• 如果是大規(guī)模集群 部署reflector路由反射器，避免過多的路由表更新 減輕AIP-SERVER壓力

#把maseter配置成reflector節(jié)點

[root@k8s-master calico]# cat reflector-node.yaml 
apiVersion: projectcalico.org/v3
kind: Node
metadata:
  labels:
    route-reflector: true
  name: k8s-master   #節(jié)點名
spec:
  bgp:
    ipv4Address: 192.168.4.170/24  #Master IP
    ipv4IPIPTunnelAddr: 192.168.237.0  #tunl0網(wǎng)絡(luò)地址
    routeReflectorClusterID: 1.1.1.1  #ID信息 如果有多個node 不能和其它重復(fù)就行


[root@k8s-master calico]# calicoctl apply -f reflector-node.yaml 
Successfully applied 1 'Node' resource(s)

• 配置所有節(jié)點與reflector節(jié)點通信

[root@k8s-master calico]# cat bgppeer-demo.yaml
kind: BGPPeer
apiVersion: projectcalico.org/v3
metadata:
  name: bgppeer-demo
spec:
  nodeSelector: all()   #所有節(jié)點
  peerSelector: route-reflector=="true" #與有這個標(biāo)簽的節(jié)點通信
  
[root@k8s-master calico]# calicoctl apply -f bgppeer-demo.yaml 
Successfully applied 1 'BGPPeer' resource(s)
[root@k8s-master calico]# calicoctl node status
Calico process is running.

IPv4 BGP status
+---------------+-------------------+-------+----------+-------------+
| PEER ADDRESS  |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+---------------+-------------------+-------+----------+-------------+
| 192.168.4.171 | node-to-node mesh | up    | 02:27:59 | Established |#之前的mesh工作模式還在
| 192.168.4.172 | node-to-node mesh | up    | 02:27:58 | Established |
| 192.168.4.173 | node-to-node mesh | up    | 02:27:58 | Established |
| 192.168.4.171 | node specific     | start | 14:36:40 | Idle        |#基于reflector工作模式
| 192.168.4.172 | node specific     | start | 14:36:40 | Idle        |
| 192.168.4.173 | node specific     | start | 14:36:40 | Idle        |
+---------------+-------------------+-------+----------+-------------+

IPv6 BGP status

#關(guān)掉mesh 點對點的工作模式

[root@k8s-master calico]# cat  default-bgpconfiguration.yaml 
apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
  name: default
spec:
  logSeverityScreen: Info
  nodeToNodeMeshEnabled: false  #是赤允許點對點通信
  asNumber : 63400

[root@k8s-master calico]# calicoctl apply -f default-bgpconfiguration.yaml 
Successfully applied 1 'BGPConfiguration' resource(s)
[root@k8s-master calico]# calicoctl node status
Calico process is running.

IPv4 BGP status
+---------------+---------------+-------+----------+-------------+
| PEER ADDRESS  |   PEER TYPE   | STATE |  SINCE   |    INFO     |
+---------------+---------------+-------+----------+-------------+
| 192.168.4.171 | node specific | up    | 14:45:26 | Established |
| 192.168.4.172 | node specific | up    | 14:45:26 | Established |
| 192.168.4.173 | node specific | up    | 14:45:26 | Established |
+---------------+---------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.