Flannel
Flannel是 CoreOS 團(tuán)隊(duì)針對(duì) Kubernetes 設(shè)計(jì)的一個(gè)覆蓋網(wǎng)絡(luò)（Overlay Network）工具，其目的在于幫助每一個(gè)使用 Kuberentes 的 CoreOS 主機(jī)擁有一個(gè)完整的子網(wǎng)天揖。這次的分享內(nèi)容將從Flannel的介紹夺欲、工作原理及安裝和配置三方面來介紹這個(gè)工具的使用方法。
Flannel通過給每臺(tái)宿主機(jī)分配一個(gè)子網(wǎng)的方式為容器提供虛擬網(wǎng)絡(luò)今膊，它基于Linux TUN/TAP些阅，使用UDP封裝IP包來創(chuàng)建overlay網(wǎng)絡(luò)，并借助etcd維護(hù)網(wǎng)絡(luò)的分配情況斑唬。
Flannel is a simple and easy way to configure a layer 3 network fabric designed for Kubernetes.
Flannel原理
Flannel是CoreOS團(tuán)隊(duì)針對(duì)Kubernetes設(shè)計(jì)的一個(gè)網(wǎng)絡(luò)規(guī)劃服務(wù)市埋，簡(jiǎn)單來說黎泣，它的功能是讓集群中的不同節(jié)點(diǎn)主機(jī)創(chuàng)建的Docker容器都具有全集群唯一的虛擬IP地址。但在默認(rèn)的Docker配置中腰素，每個(gè)Node的Docker服務(wù)會(huì)分別負(fù)責(zé)所在節(jié)點(diǎn)容器的IP分配聘裁。Node內(nèi)部得容器之間可以相互訪問,但是跨主機(jī)(Node)網(wǎng)絡(luò)相互間是不能通信。Flannel設(shè)計(jì)目的就是為集群中所有節(jié)點(diǎn)重新規(guī)劃IP地址的使用規(guī)則弓千，從而使得不同節(jié)點(diǎn)上的容器能夠獲得"同屬一個(gè)內(nèi)網(wǎng)"且"不重復(fù)的"IP地址衡便，并讓屬于不同節(jié)點(diǎn)上的容器能夠直接通過內(nèi)網(wǎng)IP通信。
Flannel 使用etcd存儲(chǔ)配置數(shù)據(jù)和子網(wǎng)分配信息洋访。flannel 啟動(dòng)之后镣陕，后臺(tái)進(jìn)程首先檢索配置和正在使用的子網(wǎng)列表，然后選擇一個(gè)可用的子網(wǎng)姻政，然后嘗試去注冊(cè)它呆抑。etcd也存儲(chǔ)這個(gè)每個(gè)主機(jī)對(duì)應(yīng)的ip。flannel 使用etcd的watch機(jī)制監(jiān)視/coreos.com/network/subnets下面所有元素的變化信息汁展，并且根據(jù)它來維護(hù)一個(gè)路由表鹊碍。為了提高性能，flannel優(yōu)化了Universal TAP/TUN設(shè)備食绿，對(duì)TUN和UDP之間的ip分片做了代理侈咕。
在Flannel的GitHub頁面有如下的一張?jiān)韴D：
packet-01.png
對(duì)上圖的簡(jiǎn)單說明 (Flannel的工作原理可以解釋如下)：
- 數(shù)據(jù)從源容器中發(fā)出后，經(jīng)由所在主機(jī)的docker0虛擬網(wǎng)卡轉(zhuǎn)發(fā)到flannel0虛擬網(wǎng)卡器紧，這是個(gè)P2P的虛擬網(wǎng)卡耀销，flanneld服務(wù)監(jiān)聽在網(wǎng)卡的另外一端。
- Flannel通過Etcd服務(wù)維護(hù)了一張節(jié)點(diǎn)間的路由表铲汪，該張表里保存了各個(gè)節(jié)點(diǎn)主機(jī)的子網(wǎng)網(wǎng)段信息熊尉。
- 源主機(jī)的flanneld服務(wù)將原本的數(shù)據(jù)內(nèi)容UDP封裝后根據(jù)自己的路由表投遞給目的節(jié)點(diǎn)的flanneld服務(wù)，數(shù)據(jù)到達(dá)以后被解包掌腰，然后直接進(jìn)入目的節(jié)點(diǎn)的flannel0虛擬網(wǎng)卡狰住，然后被轉(zhuǎn)發(fā)到目的主機(jī)的docker0虛擬網(wǎng)卡，最后就像本機(jī)容器通信一樣的由docker0路由到達(dá)目標(biāo)容器齿梁。
除了UDP转晰，F(xiàn)lannel還支持很多其他的Backend：
- udp：使用用戶態(tài)udp封裝，默認(rèn)使用8285端口士飒。由于是在用戶態(tài)封裝和解包查邢，性能上有較大的損失
- vxlan：vxlan封裝，需要配置VNI酵幕，Port（默認(rèn)8472）和GBP
- host-gw：直接路由的方式扰藕，將容器網(wǎng)絡(luò)的路由信息直接更新到主機(jī)的路由表中，僅適用于二層直接可達(dá)的網(wǎng)絡(luò)
- aws-vpc：使用 Amazon VPC route table 創(chuàng)建路由芳撒，適用于AWS上運(yùn)行的容器
- gce：使用Google Compute Engine Network創(chuàng)建路由邓深，所有instance需要開啟IP forwarding未桥，適用于GCE上運(yùn)行的容器
- ali-vpc：使用阿里云VPC route table 創(chuàng)建路由，適用于阿里云上運(yùn)行的容器
官方文檔
下載地址
https://github.com/coreos/flannel/releases/download/v0.11.0/flannel-v0.11.0-linux-amd64.tar.gz
環(huán)境準(zhǔn)備
flannel只需要在Node節(jié)點(diǎn)安裝,Master節(jié)點(diǎn)無需安裝
| 操作系統(tǒng) | IP 地址 | HostName |
|---|---|---|
| CentOS7.x-86_x64 | 10.0.52.14 | k8s.node1 |
| CentOS7.x-86_x64 | 10.0.52.6 | k8s.node2 |
Flannel 安裝
etcd注冊(cè)網(wǎng)段,供flanneld使用
[root@k8s ~]# /opt/etcd/bin/etcdctl \
> --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem \
> --endpoints="https://10.0.52.13:2379,https://10.0.52.14:2379,https://10.0.52.6:2379" \
> set /coreos.com/network/config '{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}'
{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}
[root@k8s ~]#
解壓縮
- mk-docker-opts.sh 腳本將分配給 flanneld 的 Pod 子網(wǎng)網(wǎng)段信息寫入 /run/flannel/docker 文件芥备，后續(xù) docker 啟動(dòng)時(shí) 使用這個(gè)文件中的環(huán)境變量配置 docker0 網(wǎng)橋冬耿；
- flanneld 使用系統(tǒng)缺省路由所在的接口與其它節(jié)點(diǎn)通信，對(duì)于有多個(gè)網(wǎng)絡(luò)接口（如內(nèi)網(wǎng)和公網(wǎng)）的節(jié)點(diǎn)萌壳，可以用 -iface 參數(shù)指定通信接口亦镶，如上面的 eth0 接口;
- flanneld 運(yùn)行時(shí)需要 root 權(quán)限；
[root@k8s ~]# mkdir flannel
[root@k8s ~]# tar zxf flannel-v0.11.0-linux-amd64.tar.gz -C ./flannel
[root@k8s ~]# ls
anaconda-ks.cfg flannel flannel-v0.11.0-linux-amd64.tar.gz
[root@k8s ~]# cd flannel
[root@k8s flannel]# ls
flanneld mk-docker-opts.sh README.md
[root@k8s flannel]#
安裝
[root@k8s flannel]# mkdir /opt/kubernetes/{bin,cfg,ssl} -p
[root@k8s flannel]# cp flanneld mk-docker-opts.sh /opt/kubernetes/bin/
[root@k8s flannel]# ls /opt/kubernetes/bin/
flanneld mk-docker-opts.sh
[root@k8s flannel]#
配置Flannel
cat << EOF | tee /opt/kubernetes/cfg/flanneld
FLANNEL_OPTIONS="--etcd-endpoints=https://10.0.52.13:2379,https://10.0.52.14:2379,https://10.0.52.6:2379 \
-etcd-cafile=/opt/etcd/ssl/ca.pem \
-etcd-certfile=/opt/etcd/ssl/server.pem \
-etcd-keyfile=/opt/etcd/ssl/server-key.pem \
-etcd-prefix=/coreos.com/network"
EOF
創(chuàng)建flanneld systemd啟動(dòng)管理文件
cat << EOF | tee /usr/lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld overlay address etcd agent
After=network-online.target network.target
Before=docker.service
[Service]
Type=notify
EnvironmentFile=/opt/kubernetes/cfg/flanneld
ExecStart=/opt/kubernetes/bin/flanneld --ip-masq \$FLANNEL_OPTIONS
ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
啟動(dòng)服務(wù)
[root@k8s flannel]# systemctl daemon-reload
[root@k8s flannel]# systemctl start flanneld
[root@k8s flannel]# systemctl enable flanneld
Created symlink from /etc/systemd/system/multi-user.target.wants/flanneld.service to /usr/lib/systemd/system/flanneld.service.
[root@k8s flannel]# ps -ef |grep flanneld
root 13688 1 0 13:21 ? 00:00:00 /opt/kubernetes/bin/flanneld --ip-masq --etcd-endpoints=https://10.0.52.13:2379,https://10.0.52.14:2379,https://10.0.52.6:2379 -etcd-cafile=/opt/etcd/ssl/ca.pem -etcd-certfile=/opt/etcd/ssl/server.pem -etcd-keyfile=/opt/etcd/ssl/server-key.pem -etcd-prefix=/coreos.com/network
root 13794 12107 0 13:21 pts/0 00:00:00 grep --color=auto flanneld
[root@k8s flannel]#
配置Docker啟動(dòng)指定子網(wǎng)
cat << EOF | tee /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/run/flannel/subnet.env
ExecStart=/usr/bin/dockerd \$DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP \$MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl restart docker
將flannel得相關(guān) 文件到所有節(jié)點(diǎn)
10.0.52.6 節(jié)點(diǎn)執(zhí)行:
scp -r /opt/kubernetes root@10.0.52.14:/opt/
scp /usr/lib/systemd/system/docker.service root@10.0.52.14:/usr/lib/systemd/system/docker.service
scp /usr/lib/systemd/system/flanneld.service root@10.0.52.14:/usr/lib/systemd/system/flanneld.service
10.0.52.14節(jié)點(diǎn)執(zhí)行:
systemctl daemon-reload
systemctl start flanneld
systemctl enable flanneld
systemctl restart docker
驗(yàn)證服務(wù)
查看 /run/flannel/subnet.env 得到flannel為docker分配得網(wǎng)段為--bip=172.17.100.1/24
[root@k8s flannel]# cat /run/flannel/subnet.env
DOCKER_OPT_BIP="--bip=172.17.100.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=false"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_NETWORK_OPTIONS=" --bip=172.17.100.1/24 --ip-masq=false --mtu=1450"
[root@k8s flannel]#
ifconfig 查看docker0得IP為172.17.100.1 ,flannel.1得IP為172.17.100.0,意味著其他node發(fā)給本node上得容器包都會(huì)被flannel.1捕獲,并解分包轉(zhuǎn)發(fā)給docker0,docker0再和本node上得容器進(jìn)行通信
[root@k8s docker]# ifconfig
docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.100.1 netmask 255.255.255.0 broadcast 172.17.100.255
ether 02:42:52:03:df:a7 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.52.6 netmask 255.255.255.0 broadcast 10.0.52.255
inet6 fe80::5c0e:d0d1:f594:8dcf prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:8c:22:cd txqueuelen 1000 (Ethernet)
RX packets 8299777 bytes 1626481176 (1.5 GiB)
RX errors 0 dropped 25 overruns 0 frame 0
TX packets 8437318 bytes 1462496730 (1.3 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 172.17.100.0 netmask 255.255.255.255 broadcast 0.0.0.0
inet6 fe80::9cb7:83ff:fe66:9823 prefixlen 64 scopeid 0x20<link>
ether 9e:b7:83:66:98:23 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 8 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1 (Local Loopback)
RX packets 30618 bytes 36997134 (35.2 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 30618 bytes 36997134 (35.2 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
測(cè)試兩個(gè)node中容器跨網(wǎng)絡(luò)通信 可以看到不同node節(jié)點(diǎn)的容器之間可以正常通信
node1 中執(zhí)行:
[root@k8s cfg]# docker run -it busybox
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
53071b97a884: Pull complete
Digest: sha256:4b6ad3a68d34da29bf7c8ccb5d355ba8b4babcad1f99798204e7abb43e54ee3d
Status: Downloaded newer image for busybox:latest
/ # ifconfig
eth0 Link encap:Ethernet HWaddr 02:42:AC:11:05:02
inet addr:172.17.5.2 Bcast:172.17.5.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:16 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1296 (1.2 KiB) TX bytes:0 (0.0 B)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
/ # ping 172.17.100.2
PING 172.17.100.2 (172.17.100.2): 56 data bytes
64 bytes from 172.17.100.2: seq=0 ttl=62 time=0.398 ms
64 bytes from 172.17.100.2: seq=1 ttl=62 time=0.233 ms
64 bytes from 172.17.100.2: seq=2 ttl=62 time=0.232 ms
64 bytes from 172.17.100.2: seq=3 ttl=62 time=0.237 ms
64 bytes from 172.17.100.2: seq=4 ttl=62 time=0.246 ms
64 bytes from 172.17.100.2: seq=5 ttl=62 time=0.229 ms
64 bytes from 172.17.100.2: seq=6 ttl=62 time=0.246 ms
64 bytes from 172.17.100.2: seq=7 ttl=62 time=0.236 ms
^C
--- 172.17.100.2 ping statistics ---
8 packets transmitted, 8 packets received, 0% packet loss
round-trip min/avg/max = 0.229/0.257/0.398 ms
/ #
node2 中執(zhí)行:
[root@k8s docker]# docker run -it busybox
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
53071b97a884: Pull complete
Digest: sha256:4b6ad3a68d34da29bf7c8ccb5d355ba8b4babcad1f99798204e7abb43e54ee3d
Status: Downloaded newer image for busybox:latest
/ # ifconfig
eth0 Link encap:Ethernet HWaddr 02:42:AC:11:64:02
inet addr:172.17.100.2 Bcast:172.17.100.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:16 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1296 (1.2 KiB) TX bytes:0 (0.0 B)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
/ # ping 172.17.5.2
PING 172.17.5.2 (172.17.5.2): 56 data bytes
64 bytes from 172.17.5.2: seq=0 ttl=62 time=0.296 ms
64 bytes from 172.17.5.2: seq=1 ttl=62 time=0.218 ms
64 bytes from 172.17.5.2: seq=2 ttl=62 time=0.204 ms
64 bytes from 172.17.5.2: seq=3 ttl=62 time=0.215 ms
64 bytes from 172.17.5.2: seq=4 ttl=62 time=0.231 ms
64 bytes from 172.17.5.2: seq=5 ttl=62 time=0.216 ms
^C
--- 172.17.5.2 ping statistics ---
6 packets transmitted, 6 packets received, 0% packet loss
round-trip min/avg/max = 0.204/0.230/0.296 ms
/ #
查看etcd注冊(cè)的ip地址
/opt/etcd/bin/etcdctl
--ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem
--endpoints="https://10.0.52.13:2379,https://10.0.52.14:2379,https://10.0.52.6:2379"
ls /coreos.com/network/subnets
[root@k8s ~]# /opt/etcd/bin/etcdctl \
> --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem \
> --endpoints="https://10.0.52.13:2379,https://10.0.52.14:2379,https://10.0.52.6:2379" \
> ls /coreos.com/network/subnets
/coreos.com/network/subnets/172.17.100.0-24
/coreos.com/network/subnets/172.17.5.0-24
[root@k8s ~]#
/opt/etcd/bin/etcdctl
--ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem
--endpoints="https://10.0.52.13:2379,https://10.0.52.14:2379,https://10.0.52.6:2379"
get /coreos.com/network/subnets/172.17.100.0-24
[root@k8s ~]# /opt/etcd/bin/etcdctl \
> --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem \
> --endpoints="https://10.0.52.13:2379,https://10.0.52.14:2379,https://10.0.52.6:2379" \
> get /coreos.com/network/subnets/172.17.100.0-24
{"PublicIP":"10.0.52.6","BackendType":"vxlan","BackendData":{"VtepMAC":"9e:b7:83:66:98:23"}}
[root@k8s ~]#
- PublicIP： 節(jié)點(diǎn)ip地址
- BackendType： 類型
- VtepMAC： 虛擬的mac
查看路由表
node1的路由表:
[root@k8s cfg]# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default gateway 0.0.0.0 UG 100 0 0 ens192
10.0.52.0 0.0.0.0 255.255.255.0 U 100 0 0 ens192
172.17.5.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
172.17.100.0 172.17.100.0 255.255.255.0 UG 0 0 0 flannel.1
node2的路由表:
[root@k8s docker]# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default gateway 0.0.0.0 UG 100 0 0 ens192
10.0.52.0 0.0.0.0 255.255.255.0 U 100 0 0 ens192
172.17.5.0 172.17.5.0 255.255.255.0 UG 0 0 0 flannel.1
172.17.100.0 0.0.0.0 255.255.255.0 U 0 0 0 docker0
至此,flannel安裝完成.
