實(shí)驗(yàn)網(wǎng)絡(luò)示意圖
-
docker-node1和docer-node2是 Etcd 的成員炫彩，并且都安裝了 Docker -
redis-client和redis分別是宿主機(jī)docker-node1和docer-node2的容器實(shí)例 - 每個(gè) Docker 宿主機(jī)上凳谦，都創(chuàng)建了名為
overlay_net的overlay network
宿主機(jī)操作系統(tǒng)： CentOS Linux release 7.6.1810 (Core)
Docker CE:
Version: 18.09.0
API version: 1.39 (minimum version 1.12)
下面就來一步一步的實(shí)現(xiàn)骂远。
1. 搭建Etcd Cluster
這里僅僅是為了演示Doker 夸主機(jī)通信目的而搭建及其簡單的 Etcd 集群刻恭，不可用于生產(chǎn)。
簡單介紹
這將調(diào)出etcd偵聽端口2379以進(jìn)行客戶端通信硼一，并在端口2380上進(jìn)行服務(wù)器到服務(wù)器通信累澡。
etcd是一個(gè)分布式可靠的鍵值存儲(chǔ)，用于分布式系統(tǒng)的最關(guān)鍵數(shù)據(jù)般贼，重點(diǎn)是：
- 簡單：面向用戶定義明確的API（gRPC）
- 安全：具有可選客戶端證書身份驗(yàn)證的自動(dòng)TLS
- 快速：基準(zhǔn)測試10,000次/秒
- 可靠：使用Raft正確分布
etcd是用Go編寫的愧哟，使用Raft一致性算法來管理高度可用的復(fù)制日志。
Etcd 的監(jiān)聽端口
-
2380用于集群成員之間的通信 -
2379用于監(jiān)聽客戶端的請求
下載
官方提供了二進(jìn)制包 https://github.com/etcd-io/etcd/releases/
可以根據(jù)自己的系統(tǒng)進(jìn)行有選擇的下載哼蛆。
本測試環(huán)境下載的是適合 Linux 64位系統(tǒng)環(huán)境的包
image.png
Etcd 機(jī)器信息
| 主機(jī)名 | IP |
|---|---|
| docker-node1 | 192.168.60.10 |
| docker-node2 | 192.168.60.20 |
a. docker-node1 的操作
下載解壓
[vagrant@docker-node1 ~]$ wget https://github.com/etcd-io/etcd/releases/download/v3.3.10/etcd-v3.3.10-linux-amd64.tar.gz
[vagrant@docker-node1 ~]$ tar -xf etcd-v3.3.10-linux-amd64.tar.gz
[vagrant@docker-node1 ~]$ cd etcd-v3.3.10-linux-amd64/
[vagrant@docker-node1 etcd-v3.3.10-linux-amd64]$
后臺(tái)啟動(dòng)運(yùn)行
[vagrant@docker-node2 etcd-v3.3.10-linux-amd64]$ export NODE1_IP=192.168.60.10
[vagrant@docker-node2 etcd-v3.3.10-linux-amd64]$ export NODE2_IP=192.168.60.20
[vagrant@docker-node2 etcd-v3.3.10-linux-amd64]$ export NODE1_NAME=docker-node1
[vagrant@docker-node2 etcd-v3.3.10-linux-amd64]$ export NODE2_NAME=docker-node2
[vagrant@docker-node1 etcd-v3.3.10-linux-amd64]$ nohup ./etcd \
--name ${NODE1_NAME} \
--initial-advertise-peer-urls http://${NODE1_IP}:2380 \
--listen-peer-urls http://${NODE1_IP}:2380 \
--listen-client-urls http://${NODE1_IP}:2379,http://127.0.0.1:2379 \
--advertise-client-urls http://${NODE1_IP}:2379 \
--initial-cluster-token etcd-cluster-1 \
--initial-cluster ${NODE1_NAME}=http://${NODE1_IP}:2380,${NODE2_NAME}=http://${NODE2_IP}:2380 \
--initial-cluster-state new &
- –name 節(jié)點(diǎn)名稱
- –initial-advertise-peer-urls 告知集群其他節(jié)點(diǎn)自己的URL蕊梧，tcp2380端口用于集群通信
- –listen-peer-urls 監(jiān)聽URL，用于與其他節(jié)點(diǎn)通訊
- –advertise-client-urls 告知客戶端的URL,tcp2379端口用于監(jiān)聽客戶端請求
- –initial-cluster-token 集群的ID
- --initial-cluster-token # 集群 token腮介，用于成員之間認(rèn)證
- –initial-cluster 集群中所有節(jié)點(diǎn)
- –initial-cluster-state 集群狀態(tài)肥矢，new為新創(chuàng)建集群，existing為已存在的集群
b. docker-node2 的操作
下載解壓
[vagrant@docker-node2 ~]$ wget https://github.com/etcd-io/etcd/releases/download/v3.3.10/etcd-v3.3.10-linux-amd64.tar.gz
[vagrant@docker-node2 ~]$ tar -xf etcd-v3.3.10-linux-amd64.tar.gz
[vagrant@docker-node2 ~]$ cd etcd-v3.3.10-linux-amd64/
后臺(tái)啟動(dòng)運(yùn)行
[vagrant@docker-node2 etcd-v3.3.10-linux-amd64]$ export NODE1_IP=192.168.60.10
[vagrant@docker-node2 etcd-v3.3.10-linux-amd64]$ export NODE2_IP=192.168.60.20
[vagrant@docker-node2 etcd-v3.3.10-linux-amd64]$ export NODE1_NAME=docker-node1
[vagrant@docker-node2 etcd-v3.3.10-linux-amd64]$ export NODE2_NAME=docker-node2
[vagrant@docker-node2 etcd-v3.3.10-linux-amd64]$ nohup ./etcd \
--name ${NODE2_NAME} \
--initial-advertise-peer-urls http://${NODE2_IP}:2380 \
--listen-peer-urls http://${NODE2_IP}:2380 \
--listen-client-urls http://${NODE2_IP}:2379,http://127.0.0.1:2379 \
--advertise-client-urls http://${NODE2_IP}:2379 \
--initial-cluster-token etcd-cluster-1 \
--initial-cluster ${NODE1_NAME}=http://${NODE1_IP}:2380,${NODE2_NAME}=http://${NODE2_IP}:2380 \
--initial-cluster-state new &
集群健康檢查
分別在每個(gè)成員中執(zhí)行如下命令
[vagrant@docker-node2 etcd-v3.3.10-linux-amd64]$ ./etcdctl cluster-health
member d860400699ce61e is healthy: got healthy result from http://192.168.60.10:2379
member f96c6ec64b2090e3 is healthy: got healthy result from http://192.168.60.20:2379
cluster is healthy
[vagrant@docker-node2 etcd-v3.3.10-linux-amd64]$
返回 cluster is healthy 說明正常叠洗。
查看集群成員
在任意節(jié)點(diǎn)上運(yùn)行
[vagrant@docker-node2 etcd-v3.3.10-linux-amd64]$ ./etcdctl member list
d860400699ce61e: name=docker-node1 peerURLs=http://192.168.60.10:2380 clientURLs=http://192.168.60.10:2379 isLeader=true
f96c6ec64b2090e3: name=docker-node2 peerURLs=http://192.168.60.20:2380 clientURLs=http://192.168.60.20:2379 isLeader=false
集群正常后甘改，接下來就可以對 Docker 進(jìn)行配置了
關(guān)于 Etcd 的更多配置參考官方文檔
2. 配置 Docker 使用 Etcd
分別在兩臺(tái)宿主機(jī)上停止 dockerd 服務(wù)。
這里一 systemd 為例
sudo systemctl stop docker
docker-node1 上執(zhí)行
sudo dockerd -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --cluster-store=etcd://${NODE1_IP}:2379 --cluster-advertise=${NODE1_IP}:2375 &
docker-node2 上執(zhí)行
sudo dockerd -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --cluster-store=etcd://${NODE2_IP}:2379 --cluster-advertise=${NODE2_IP}:2375 &
在其中的一個(gè)節(jié)點(diǎn)上創(chuàng)建 overlay network
- 這里我在
docker-node1上創(chuàng)建名字為:overlay_net的覆蓋網(wǎng)絡(luò)
docker@docker-node1:~$ docker network create -d overlay overlay_net
833aa03d003739bf1f5802f8c1ebe0ae617e02e46b09000c03b348b66e1e27c5
docker@docker-node1:~$ docker network inspect overlay_net
[
{
"Name": "overlay_net",
"Id": "833aa03d003739bf1f5802f8c1ebe0ae617e02e46b09000c03b348b66e1e27c5",
"Created": "2018-12-16T01:52:28.046320149Z",
"Scope": "global",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "10.0.0.0/24",
"Gateway": "10.0.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {},
"Options": {},
"Labels": {}
}
]
- 之后在
docker-node2上查看網(wǎng)絡(luò)信息
docker@docker-node2:~$ docker network ls
NETWORK ID NAME DRIVER SCOPE
31e870259bb0 bridge bridge local
ee8cec8939af host host local
8d6c3e3ceb94 none null local
833aa03d0037 overlay_net overlay global
docker@docker-node2:~$
可以看到在一個(gè)節(jié)點(diǎn)上創(chuàng)建的網(wǎng)絡(luò)信息灭抑，可以同步到集群中的所有節(jié)點(diǎn)上十艾。
運(yùn)行容器并將其連接到 overlay_net 網(wǎng)絡(luò)中
-
docker-node1運(yùn)行容器redis-client
docker@docker-node1:~$ sudo docker run -itd --rm --name=redis-client --network=overlay_net python:3.6-alpine
65bc05e3801242b2e917775d079a15272a926dab0e38b8039564f50547bef770
ERRO[0000] enabling default vlan on bridge br0 failed open /sys/class/net/br0/bridge/default_pvid: permission denied
ERRO[2018-12-16T01:59:00.903303768Z] reexec to set bridge default vlan failed exit status 1
INFO[0839] shim containerd-shim started address="/containerd-shim/moby/65bc05e3801242b2e917775d079a15272a926dab0e38b8039564f50547bef770/shim.sock" debug=false pid=4668
這里的錯(cuò)誤信息暫時(shí)沒解決
-
docker-node2運(yùn)行容器redis
docker@docker-node2:~$ sudo docker run -itd --rm --name=redis --network=overlay_net redis:alpine
58b6cf076675f005d0a0d944fd85762bfe0a6e0bf8a63f94fac3e65d9f74e955
ERRO[0000] enabling default vlan on bridge br0 failed open /sys/class/net/br0/bridge/default_pvid: permission denied
ERRO[2018-12-16T01:59:56.673683047Z] reexec to set bridge default vlan failed exit status 1
INFO[0781] shim containerd-shim started address="/containerd-shim/moby/58b6cf076675f005d0a0d944fd85762bfe0a6e0bf8a63f94fac3e65d9f74e955/shim.sock" debug=false pid=4729
- 在
docker-node2進(jìn)入容器redis測試連通性
docker@docker-node2:~$ docker exec -it redis sh
/data # ping redis-client
PING redis-client (10.0.0.2): 56 data bytes
64 bytes from 10.0.0.2: seq=0 ttl=64 time=0.989 ms
64 bytes from 10.0.0.2: seq=1 ttl=64 time=0.702 ms
^C
--- redis-client ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.702/0.845/0.989 ms
- 在
docker-node1進(jìn)入容器redis-client測試連通性
docker@docker-node1:~$ docker exec -it redis-client sh
/ # ping redis
PING redis (10.0.0.3): 56 data bytes
64 bytes from 10.0.0.3: seq=0 ttl=64 time=0.998 ms
64 bytes from 10.0.0.3: seq=1 ttl=64 time=0.657 ms
^C
--- redis ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.657/0.827/0.998 ms
- 最后在容器
redis-client中安裝python3用的redis模塊，并且進(jìn)行端口連通性測試
/ # pip3 install redis
Collecting redis
Downloading https://files.pythonhosted.org/packages/f5/00/5253aff5e747faf10d8ceb35fb5569b848cde2fdc13685d42fcf63118bbc/redis-3.0.1-py2.py3-none-any.whl (61kB)
100% |████████████████████████████████| 71kB 115kB/s
Installing collected packages: redis
Successfully installed redis-3.0.1
/ # python3
Python 3.6.7 (default, Nov 16 2018, 06:52:39)
[GCC 6.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import redis
>>> r = redis.Redis(host="redis", port=6379)
>>> r.set("name", "shark")
True
>>> r.get("name")
b'shark'
>>> exit()
/ # exit
docker@docker-node1:~$
