通過docker-machine +consul +overlay實現(xiàn)跨主機通信

一烁涌、基礎環(huán)境配置

三臺服務器均執(zhí)行以下操作

1. 配置固定IP

[root@localhost ~]#? vim? /etc/sysconfig/network-scripts/ifcfg-ens32

TYPE=Ethernet

PROXY_METHOD=none

BROWSER_ONLY=no

BOOTPROTO=static

DEFROUTE=yes

IPV4_FAILURE_FATAL=no

IPV6INIT=yes

IPV6_AUTOCONF=yes

IPV6_DEFROUTE=yes

IPV6_FAILURE_FATAL=no

IPV6_ADDR_GEN_MODE=stable-privacy

NAME=ens32

UUID=75963e3f-b289-4bbd-8489-44f6f2b8c7f0

DEVICE=ens32

ONBOOT=yes

IPADDR=192.168.0.10

PREFIX=24

GATEWAY=192.168.0.1

DNS1=114.114.114.114

[root@localhost ~]# systemctl restart network

2. 更改主機名

[root@localhost ~]#? hostnamectl set-hostname docker-ce

[root@localhost ~]#? exit? ? //重新登陸即可

[root@docker-ce ~]#

在20，30兩臺服務器上重復上面的操作，docker1是IPADDR=192.168.0.20切平，docker2是IPADDR=192.168.0.30

3.關(guān)閉防火墻

[root@docker-ce ~]# systemctl stop firewalld

[root@docker-ce ~]# systemctl disable firewalld

4. 同步系統(tǒng)時間

[root@docker-ce ~]# yum -y install ntp

[root@docker-ce ~]#? systemctl enable ntpd.service

[root@docker-ce ~]# ntpdate cn.pool.ntp.org

[root@docker-ce ~]# hwclock -w

[root@docker-ce ~]#? crontab -e

0 2 * * * ntpdate ntpdate cn.pool.ntp.org? && hwclock -w

5.關(guān)閉selinux

[root@dockerce ~]# vim /etc/sysconfig/selinux

SELINUX=disabled

[root@docker-ce ~]#? reboot

二蜓氨、docker-ce配置

1. 安裝docker-machine

方式一：

[root@docker-ce ~]# curl -L https://github.com/docker/machine/releases/download/v0.14.0/docker-machine-`uname -s`-`uname -m` >/tmp/docker-machine && chmod +x /tmp/docker-machine && cp /tmp/docker-machine /usr/local/bin/docker-machine

%Total? ? % Received % Xferd? Average Speed? Time? Time? ? Time? Current

? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Dload? Upload? Total? Spent? ? Left Speed

100? 617? ? 0? 617? 0? ? 0? ? 582? ? 0 --:--:--? 0:00:01 --:--:--? 583

100 25.1M 100 25.1M? ? 0? ? 0? 644k? ? ? 0? 0:00:40 0:00:40 --:--:-- 2801k

此連接安裝時，有時會鏈接超時,多鏈接幾次或是過會再鏈接應該會成功。

方式二：

直接下載安裝包

[root@docker-ce ~]# yum -y install wget

[root@docker-ce ~]#? wgethttps://github.com/docker/machine/releases/download/v0.14.0/docker-machine-Linux-x86_64

[root@docker-ce ~]# mv docker-machine-Linux-x86_64 /usr/local/bin/docker-machine

[root@docker-ce ~]# chmod +x? /usr/local/bin/docker-machine

[root@docker-ce ~]# docker-machine version

docker-machine version 0.14.0, build 89b8332

//docker-machine軟件包需要翻墻下載

2. docker1,docker2 配置免密鑰登錄

[root@docker-ce ~]# ssh-keygen? ? //（生成主機密鑰對横漏，一路回車即可）

Generating public/private rsa key pair.

Enter file in which to save the key (/root/.ssh/id_rsa):

Created directory '/root/.ssh'.

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /root/.ssh/id_rsa.

Your public key has been saved in /root/.ssh/id_rsa.pub.

The key fingerprint is:

SHA256:27whAu+mDPZzqj25FcbF47Z8nbIav14KbxGeCl6aZac root@docker-ce

The key's randomart image is:

+---[RSA 2048]----+

|? ? ? ? ? ? ? ? |

|? ? ? .? ? ? ? |

|? ? ? ? +? ? ? ? |

|? ? . o ..? ? ? |

|? ? . + S. o? ? |

|? ? +.++==. .? |

|? o? o+BB+*.+? ? |

| . =+o*.EB.B? ? |

|? ..BO. .+O.? ? |

+----[SHA256]-----+

[root@docker-ce ~]# ssh-copy-id 192.168.0.20? ? //（復制公鑰到20服務器）

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"

The authenticity of host '192.168.0.20 (192.168.0.20)' can't be established.

ECDSA key fingerprint is SHA256:eOpJf2pvOii5sgnKZS+Wb3G3hc/7deEqGzuPcb2Ymhs.

ECDSA key fingerprint is MD5:1f:20:29:32:84:1e:59:cd:47:a5:c9:c3:82:77:50:f6.

Are you sure you want to continue connecting (yes/no)? yes

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

root@192.168.0.20's password:? ? //輸入遠程主機20的root用戶的密碼

Number of key(s) added: 1

Now try logging into the machine, with:? "ssh '192.168.0.20'"

and check to make sure that only the key(s) you wanted were added.

[root@docker-ce ~]# ssh-copy-id 192.168.0.30? //（復制公鑰到30服務器）

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"

The authenticity of host '192.168.0.30 (192.168.0.30)' can't be established.

ECDSA key fingerprint is SHA256:eOpJf2pvOii5sgnKZS+Wb3G3hc/7deEqGzuPcb2Ymhs.

ECDSA key fingerprint is MD5:1f:20:29:32:84:1e:59:cd:47:a5:c9:c3:82:77:50:f6.

Are you sure you want to continue connecting (yes/no)? yes

/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys

root@192.168.0.30's password:? ? //輸入遠程主機30的root用戶的密碼

Number of key(s) added: 1

Now try logging into the machine, with:? "ssh '192.168.0.30'"

and check to make sure that only the key(s) you wanted were added.

3. 登陸驗證

[root@docker-ce ~]# ssh 192.168.0.20

Last login: Fri May 18 10:42:53 2018 from 192.168.0.110

[root@docker02 ~]# exit

[root@docker-ce ~]# ssh 192.168.0.30

Last login: Fri May 18 10:43:11 2018 from 192.168.0.110

[root@docker03 ~]# exit

4. 批量安裝docker環(huán)境并配置啟動

[root@docker-ce ~]# docker-machine ls （查看安裝docker的機器）

NAME? ACTIVE? DRIVER? STATE? URL? SWARM? DOCKER? ERRORS

在進行批量安裝的時候，主要做以下幾件事：

1> 使用ssh連接到遠程主機上（配置無密碼登錄等原因）

2> 安裝docker（從dockerhub上下載最新的安裝包進行安裝）

3> 將本機生成證書并將證書拷貝到遠程主機上

4> 設置遠程主機的啟動程序并啟動遠程主機的docker進程

批量安裝時花費的時間比較長

[root@docker-ce ~]# docker-machine create -d generic --generic-ip-address=192.168.0.20 docker1（使用generic驅(qū)動安裝ip為192.168.0.20的主機熟掂，并且將其主機名設置為docker1）

Creating CA: /root/.docker/machine/certs/ca.pem

Creating client certificate: /root/.docker/machine/certs/cert.pem

Running pre-create checks...

Creating machine...

(docker1) No SSH key specified. Assuming an existing key at the default location.

Waiting for machine to be running, this may take a few minutes...

Detecting operating system of created instance...

Waiting for SSH to be available...

Detecting the provisioner...

Provisioning with centos...

Copying certs to the local machine directory...

Copying certs to the remote machine...

Setting Docker configuration on the remote daemon...

Checking connection to Docker...

Docker is up and running!

To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env docker1

[root@docker-ce ~]# docker-machine create -d generic --generic-ip-address=192.168.0.30 docker2

Running pre-create checks...

Creating machine...

(docker2) No SSH key specified. Assuming an existing key at the default location.

Waiting for machine to be running, this may take a few minutes...

Detecting operating system of created instance...

Waiting for SSH to be available...

Detecting the provisioner...

Provisioning with centos...

Copying certs to the local machine directory...

Copying certs to the remote machine...

Setting Docker configuration on the remote daemon...

Checking connection to Docker...

Docker is up and running!

To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env docker2

[root@docker-ce ~]# docker-machine ls （檢查是否安裝成功缎浇，查看遠程主機上的版本）

NAME? ? ? ACTIVE? DRIVER? ? STATE? ? URL? ? ? ? ? ? ? ? ? ? ? SWARM? DOCKER? ? ? ? ERRORS

docker1? -? ? ? ? generic? Running? tcp://192.168.0.20:2376? ? ? ? ? v18.05.0-ce?

docker2? -? ? ? ? generic? Running? tcp://192.168.0.30:2376? ? ? ? ? v18.05.0-ce

5. 其他情況說明

在生成證書的時候，在本機上和遠程主機上都會保存相關(guān)的證書信息赴肚，而且docker-mahine的配置信息也是保存在此位置素跺，相當于控制的主機，如下所示：

[root@docker-ce ~]# ls -l .docker/（在家目錄的隱藏目錄.docker目錄下）

total 4

-rw-------. 1 root root 173 Jan 17 14:52 config.json（保存的是控制機的配置信息誉券，例如docker客戶端的版本）

drwx------. 4 root root? 35 Jan 13 14:34 machine（保存本機和遠程主機上證書信息及相關(guān)遠程主機的配置信息）

遠程主機運行的進程時候指厌，已經(jīng)做了相關(guān)的配置，進程也已經(jīng)啟動横朋，查看啟動的進程信息：

[root@docker2 ~]# ls -l /etc/docker/（證書保存的配置仑乌，使用https進行通信，保證安全性）

total 20

-rw-r--r--. 1 root root 1029 Jan 20 05:18 ca.pem

-rw-r--r--. 1 root root? 56 Jan 19 03:12 daemon.json

-rw-------. 1 root root? 245 Dec 12 21:40 key.json

-rw-r--r--. 1 root root 1679 Jan 20 05:18 server-key.pem

-rw-r--r--. 1 root root 1103 Jan 20 05:18 server.pem

[root@docker2 ~]# cat /etc/systemd/system/docker.service.d/10-machine.conf

（啟動的配置文件琴锭，在單機host環(huán)境中不存在此目錄晰甚，此文件會覆蓋默認的啟動的配置文件）

[Service]

ExecStart=

ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --storage-driver devicemapper--tlsverify --tlscacert/etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=generic

Environment=

[root@docker2 ~]# ps -ef|grep docker（查看啟動的docker進程）

root? ? ? 21741? ? ? 1? 0 05:18 ?? ? ? ? 00:00:11 /usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --storage-driver devicemapper --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=generic

root? ? ? 21746? 21741? 0 05:19 ?? ? ? ? 00:00:11 docker-containerd --config /var/run/docker/containerd/containerd.toml

root? ? ? 21950? 17184? 0 05:39 pts/0? ? 00:00:00 grep --color=auto docker

[root@docker2 ~]# netstat -tunlpx |grep dockerd（查看docker監(jiān)聽的端口）

tcp6? ? ? 0? ? ? 0 :::2376:::*? ? ? ? ? ? ? ? ? ? LISTEN? ? ? 21741/dockerd

unix? 2? ? ? [ ACC ]? ? STREAM? ? LISTENING? ? 324400? 21741/dockerd? ? ? ? /var/run/docker.sock

unix? 2? ? ? [ ACC ]? ? STREAM? ? LISTENING? ? 323509? 21741/dockerd? ? ? ? /var/run/docker/metrics.sock

unix? 2? ? ? [ ACC ]? ? STREAM? ? LISTENING? ? 324567? 21741/dockerd? ? ? ? /run/docker/libnetwork/fc7eb55d830a15800059dcf61156a80314b2ba7354834170cf198702e586f22e.sock

在使用多機環(huán)境的時候需要注意的是，啟動的配置文件已經(jīng)被覆蓋了决帖，如果去修改單機環(huán)境中的啟動的配置文件是不能生效的厕九，必須要修改新創(chuàng)建的目錄下的文件內(nèi)容。

三地回、跨主機通信

使用overlay驅(qū)動進行跨主機通信扁远，在使用此驅(qū)動的時候，是通過udp的4789端口進行通信刻像。

1. 創(chuàng)建數(shù)據(jù)庫來保存網(wǎng)絡信息

docker-ce配置

直接使用容器consul來進行運行畅买，如下所示：

[root@docker-ce]# docker run -d -p 8500:8500 -h consul --name consul progrium/consul --server -bootstrap

Unable to find image 'progrium/consul:latest' locally

latest: Pulling from progrium/consul

c862d82a67a2: Pull complete

0e7f3c08384e: Pull complete

0e221e32327a: Pull complete

09a952464e47: Pull complete

60a1b927414d: Pull complete

4c9f46b5ccce: Pull complete

417d86672aa4: Pull complete

b0d47ad24447: Pull complete

fd5300bd53f0: Pull complete

a3ed95caeb02: Pull complete

d023b445076e: Pull complete

ba8851f89e33: Pull complete

5d1cefca2a28: Pull complete

Digest: sha256:8cc8023462905929df9a79ff67ee435a36848ce7a10f18d6d0faba9306b97274

Status: Downloaded newer image for progrium/consul:latest

6ac42624732d189e0e8913c69632b266e0fe67103c4f1403ffd8b155cffc1e61

[root@docker-ce]# docker ps

CONTAINER ID? ? ? ? IMAGE? ? ? ? ? ? ? COMMAND? ? ? ? ? ? ? ? ? CREATED? ? ? ? ? ? STATUS? ? ? ? ? ? ? PORTS? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? NAMES

6ac42624732d? ? ? ? progrium/consul? ? "/bin/start --server…"? 18 seconds ago? ? ? Up 14 seconds? ? ? 53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp? consul

使用ip加端口直接進行訪問，此處是http://192.168.0.10:8500

2. 遠程主機配置

在遠程主機上要在這個數(shù)據(jù)庫上注冊细睡，從而需要修改配置文件谷羞，如下所示：

[root@docker1 ~]# vim? /etc/systemd/system/docker.service.d/10-machine.conf （加粗內(nèi)容是需要添加的）

[Service]

ExecStart=

ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --storage-driver devicemapper --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=generic --cluster-store=consul://192.168.0.10:8500 --cluster-advertise=ens32:2376

Environment=

參數(shù)說明：

--cluster-store主要表示為需要連接的kv存儲的地址，--cluster-advertise表示使用哪個網(wǎng)卡和端口來進行通信溜徙。ens32表示為網(wǎng)卡的名稱或者使用主機的ip地址湃缎。

[root@docker1 ~]# systemctl daemon-reload（重新加載配置文件）

[root@docker1 ~]# systemctl restart docker（重新啟動服務）

docker2上同樣進行上述操作

再次查看kv存儲，發(fā)現(xiàn)已經(jīng)注冊蠢壹，注意在點擊的時候嗓违，需要先點擊key/value,然后再點擊docker，最后點擊nodes才能看到注冊的主機

3. 創(chuàng)建overlay網(wǎng)絡

[root@docker1 ~]# docker network ls

NETWORK ID? ? ? ? ? NAME? ? ? ? ? ? ? ? DRIVER? ? ? ? ? ? ? SCOPE

ef17c0c56159? ? ? ? bridge? ? ? ? ? ? ? bridge? ? ? ? ? ? ? local

34963f83928c? ? ? ? host? ? ? ? ? ? ? ? host? ? ? ? ? ? ? ? local

a79f72191b90? ? ? ? none? ? ? ? ? ? ? ? null? ? ? ? ? ? ? ? local

[root@docker1 ~]#? docker network create -d overlay kel? ? //（創(chuàng)建overlay網(wǎng)絡）

8675d048411335e1738996b62fb9e505e270379a3efd1a4af2a9f7b96c03dd1f

[root@docker1 ~]# docker network ls

NETWORK ID? ? ? ? ? NAME? ? ? ? ? ? ? ? DRIVER? ? ? ? ? ? ? SCOPE

ef17c0c56159? ? ? ? bridge? ? ? ? ? ? ? bridge? ? ? ? ? ? ? local

34963f83928c? ? ? ? host? ? ? ? ? ? ? ? host? ? ? ? ? ? ? ? local

8675d0484113? ? ? ? kel? ? ? ? ? ? ? ? overlay? ? ? ? ? ? global

a79f72191b90? ? ? ? none? ? ? ? ? ? ? ? null? ? ? ? ? ? ? ? local

[root@docker2 ~]# docker network ls? ? ? ?//（在主機docker1上也能看到此網(wǎng)絡）

NETWORK ID? ? ? ? ? NAME? ? ? ? ? ? ? ? DRIVER? ? ? ? ? ? ? SCOPE

d78fa5f31c2a? ? ? ? bridge? ? ? ? ? ? ? bridge? ? ? ? ? ? ? local

34963f83928c? ? ? ? host? ? ? ? ? ? ? ? host? ? ? ? ? ? ? ? local

8675d0484113? ? ? ? kel? ? ? ? ? ? ? ? overlay? ? ? ? ? ? global

a79f72191b90? ? ? ? none? ? ? ? ? ? ? ? null? ? ? ? ? ? ? ? local

4. 創(chuàng)建容器測試網(wǎng)絡連通性

[root@docker1 ~]# docker pull cirros

Using default tag: latest

latest: Pulling from library/cirros

480d57c7bf4d: Pull complete

0f45bfe9a805: Pull complete

5aa6c26e64dc: Pull complete

Digest: sha256:e67f6f4a0521e326ba2dd697950046aba5ce836edda79cb818d45a56841c7ca2

Status: Downloaded newer image for cirros:latest

[root@docker1 ~]# docker run -itd --name b1 --network=kel cirros? //（運行一個網(wǎng)絡在kel網(wǎng)絡中的容器）

/ # ifconfig

eth0? ? ? Link encap:Ethernet? HWaddr 02:42:0A:00:00:04?

? ? ? ? ? inet addr:10.0.0.2? Bcast:10.0.0.255? Mask:255.255.255.0

? ? ? ? ? UP BROADCAST RUNNING MULTICAST? MTU:1450? Metric:1

? ? ? ? ? RX packets:0 errors:0 dropped:0 overruns:0 frame:0

? ? ? ? ? TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

? ? ? ? ? collisions:0 txqueuelen:0

? ? ? ? ? RX bytes:0 (0.0 B)? TX bytes:0 (0.0 B)

[root@docker2 ~]# docker run -itd --name b2 --network=kel cirros? ? //（在另一臺主機上運行一個kel網(wǎng)段中的容器）

/ # ifconfig

eth0? ? ? Link encap:Ethernet? HWaddr 02:42:0A:00:00:03?

? ? ? ? ? inet addr:10.0.0.3? Bcast:10.0.0.255? Mask:255.255.255.0

? ? ? ? ? UP BROADCAST RUNNING MULTICAST? MTU:1450? Metric:1

? ? ? ? ? RX packets:0 errors:0 dropped:0 overruns:0 frame:0

? ? ? ? ? TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

? ? ? ? ? collisions:0 txqueuelen:0

? ? ? ? ? RX bytes:0 (0.0 B)? TX bytes:0 (0.0 B)

/# ip r? ?//(查看路由）

default via 172.18.0.1 dev eth1

10.0.0.0/24 dev eth0 scope link? src 10.0.0.2

172.18.0.0/16 dev eth1 scope link? src 172.18.0.2

/ # cat /etc/resolv.conf? ? //(自帶的DNS解析）

nameserver 127.0.0.11

options ndots:0

/ # ip addr show? ? //（有兩個網(wǎng)絡接口）

1: lo: mtu 65536 qdisc noqueue qlen 1

? ? link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

? ? inet 127.0.0.1/8 scope host lo

? ? ? valid_lft forever preferred_lft forever

7: eth0@if8: mtu 1450 qdisc noqueue

? ? link/ether 02:42:0a:00:00:02 brd ff:ff:ff:ff:ff:ff

? ? inet 10.0.0.2/24 brd 10.0.0.255 scope global eth0

? ? ? valid_lft forever preferred_lft forever

10: eth1@if11: mtu 1500 qdisc noqueue

? ? link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff

? ? inet 172.18.0.2/16 brd 172.18.255.255 scope global eth1

? ? ? valid_lft forever preferred_lft forever

5. 不同宿主機內(nèi)容器互通測試

docker1內(nèi)容器

/ # ping b2 （跨主機ping)

PING b2 (10.0.0.3): 56 data bytes

64 bytes from 10.0.0.3: seq=0 ttl=64 time=19.570 ms

64 bytes from 10.0.0.3: seq=1 ttl=64 time=3.206 ms

--- b2 ping statistics ---

2 packets transmitted, 2 packets received, 0% packet loss

round-trip min/avg/max = 3.206/11.388/19.570 ms

docker2內(nèi)容器

/# ping b1

PING 10.0.0.2 (10.0.0.2): 56 data bytes

64 bytes from 10.0.0.2: seq=0 ttl=64 time=37.227 ms

64 bytes from 10.0.0.2: seq=1 ttl=64 time=0.448 ms

--- 10.0.0.2 ping statistics ---

2 packets transmitted, 2 packets received, 0% packet loss

round-trip min/avg/max = 0.448/18.837/37.227 ms

原創(chuàng)作品图贸，允許轉(zhuǎn)載蹂季，轉(zhuǎn)載時請務必以超鏈接形式標明文章 原始出處 冕广、作者信息和本聲明。