二進(jìn)制安裝-k8s高可用集群04-部署Etcd集群

本文檔介紹部署一個(gè)，有三個(gè)節(jié)點(diǎn)高可用 etcd 集群的步驟：

下載和分發(fā) etcd 二進(jìn)制文件；
創(chuàng)建 etcd 集群各節(jié)點(diǎn)的 x509 證書(shū)咆耿，用于加密客戶(hù)端 (如 etcdctl) 與 etcd 集群蜕窿、etcd 集群之間的數(shù)據(jù)流；
創(chuàng)建 etcd 的 systemd unit 文件塔插，配置服務(wù)參數(shù)；
檢查集群工作狀態(tài)

1）下載和分發(fā) etcd 二進(jìn)制文件

到 https://github.com/coreos/etcd/releases 頁(yè)面下載最新版本的發(fā)布包：

wget https://github.com/coreos/etcd/releases/download/v3.3.7/etcd-v3.3.7-linux-amd64.tar.gz
tar -zxvf etcd-v3.3.7-linux-amd64.tar.gz

分發(fā)二進(jìn)制文件,推送到所有節(jié)點(diǎn)里：

cat > magic12_distribute_etcd.sh << "EOF"
#!/bin/bash
# 分發(fā)二進(jìn)制文件
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}"
    scp /data/etcd-v3.3.7-linux-amd64/etcd* root@${node_ip}:/opt/k8s/bin
    ssh root@${node_ip} "chmod +x /opt/k8s/bin/*"
done
EOF

2）創(chuàng)建 etcd 證書(shū)和私鑰

創(chuàng)建證書(shū)簽名請(qǐng)求：

cat > /data/cert/etcd-csr.json <<EOF
{
  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "172.68.96.101",
    "172.68.96.102",
    "172.68.96.103"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "4Paradigm"
    }
  ]
}
EOF

hosts 字段指定授權(quán)使用該證書(shū)的 etcd 節(jié)點(diǎn) IP 或域名列表拓哟，這里將 etcd 集群的三個(gè)節(jié)點(diǎn) IP 都列在其中

生成證書(shū)和私鑰：

[root@master cert]# 
cfssl gencert -ca=/etc/kubernetes/cert/ca.pem \
    -ca-key=/etc/kubernetes/cert/ca-key.pem \
    -config=/etc/kubernetes/cert/ca-config.json \
    -profile=kubernetes etcd-csr.json | cfssljson -bare etcd

[root@master cert]# ls etcd*

分發(fā)生成的證書(shū)和私鑰到各 etcd 節(jié)點(diǎn)：

cat > magic13_distribute_certndoe.sh << "EOF"
#!/bin/bash
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p /etc/etcd/cert && chown -R k8s /etc/etcd/cert"
    scp /data/cert/etcd*.pem root@${node_ip}:/etc/etcd/cert/
done
EOF

3）創(chuàng)建 etcd 的 systemd unit 模板文件

source /opt/k8s/bin/environment.sh
mkdir -pv /data/service/ /data/template/

[root@master template]# vim /data/template/etcd.service.template
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
User=root
Type=notify
WorkingDirectory=/var/lib/etcd/
ExecStart=/opt/k8s/bin/etcd \
  --data-dir=/var/lib/etcd \
  --name=##NODE_NAME## \
  --cert-file=/etc/etcd/cert/etcd.pem \
  --key-file=/etc/etcd/cert/etcd-key.pem \
  --trusted-ca-file=/etc/kubernetes/cert/ca.pem \
  --peer-cert-file=/etc/etcd/cert/etcd.pem \
  --peer-key-file=/etc/etcd/cert/etcd-key.pem \
  --peer-trusted-ca-file=/etc/kubernetes/cert/ca.pem \
  --listen-peer-urls=https://##NODE_IP##:2380 \
  --initial-advertise-peer-urls=https://##NODE_IP##:2380 \
  --listen-client-urls=https://##NODE_IP##:2379,http://127.0.0.1:2379 \
  --advertise-client-urls=https://##NODE_IP##:2379,http://127.0.0.1:2379 \
  --initial-cluster-token=etcd-cluster-0 \
  --initial-cluster=master=https://172.68.96.101:2380,node01=https://172.68.96.102:2380,node02=https://172.68.96.103:2380 \
  --initial-cluster-state=new
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

User：指定以 k8或root （可選）
WorkingDirectory想许、--data-dir：指定工作目錄和數(shù)據(jù)目錄為 /var/lib/etcd，需在啟動(dòng)服務(wù)前創(chuàng)建這個(gè)目錄断序；
--name：指定節(jié)點(diǎn)名稱(chēng)流纹，當(dāng) –initial-cluster-state 值為 new 時(shí)，–name 的參數(shù)值必須位于 –initial-cluster 列表中违诗；
--cert-file漱凝、--key-file：etcd server 與 client 通信時(shí)使用的證書(shū)和私鑰；
--trusted-ca-file：簽名 client 證書(shū)的 CA 證書(shū)诸迟，用于驗(yàn)證 client 證書(shū)茸炒；
--peer-cert-file愕乎、--peer-key-file：etcd 與 peer 通信使用的證書(shū)和私鑰；
--peer-trusted-ca-file：簽名 peer 證書(shū)的 CA 證書(shū)壁公，用于驗(yàn)證 peer 證書(shū)感论；

4）將剛剛的文件初始化成節(jié)點(diǎn)可用

替換模板文件中的變量，為各節(jié)點(diǎn)創(chuàng)建 systemd unit 文件：

cat > magic14_init_NodeCluster.sh << "EOF"
#!/bin/bash
source /opt/k8s/bin/environment.sh
for (( i=0; i < 110; i++ ))
do
    sed -e "s/##NODE_NAME##/${NODE_NAMES[i]}/" -e "s/##NODE_IP##/${NODE_IPS[i]}/" /data/template/etcd.service.template > /data/service/etcd-${NODE_IPS[i]}.service 
done
EOF

[root@master service]# bash -x magic14_init_NodeCluster.sh
[root@master service]# ls *.service
etcd-172.68.96.101.service  etcd-172.68.96.102.service  etcd-172.68.96.103.service

NODE_NAMES 和 NODE_IPS 為相同長(zhǎng)度的 bash 數(shù)組紊册，分別為節(jié)點(diǎn)名稱(chēng)和對(duì)應(yīng)的 IP比肄；

分發(fā)生成的 systemd unit 文件：

cat > magic15_distribute_SystemdUnit.sh << "EOF"
#!/bin/bash
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}"
    ssh root@${node_ip} "mkdir -p /var/lib/etcd && chown -R k8s /var/lib/etcd" 
    scp /data/service/etcd-${node_ip}.service root@${node_ip}:/etc/systemd/system/etcd.service
done
EOF

5）啟動(dòng) etcd 服務(wù)

cat > magic16_start_etcd_Service.sh << "EOF" 
#!/bin/bash
# 啟動(dòng) etcd 服務(wù)
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}" 
    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable etcd && systemctl start etcd &"
done
EOF

etcd 進(jìn)程首次啟動(dòng)時(shí)會(huì)等待其它節(jié)點(diǎn)的 etcd 加入集群，命令 systemctl start etcd 會(huì)卡住一段時(shí)間湿硝，為正承角埃現(xiàn)象

6）檢查啟動(dòng)結(jié)果

cat > magic17_check_etcd_Service.sh << "EOF" 
#!/bin/bash
# 檢查啟動(dòng)結(jié)果
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}" 
    ssh root@${node_ip} "systemctl status etcd|grep Active"
done
EOF

結(jié)果如下，則說(shuō)明正常：

[root@master scriptable]# bash magic17_check_etcd_Service.sh 
>>> 172.68.96.101
   Active: active (running) since Wed 20XX-XX-XX XX:XX:XX CST; XXh ago
>>> 172.68.96.102
   Active: active (running) since Wed 20XX-XX-XX XX:XX:XX CST; XXh ago
>>> 172.68.96.103
   Active: active (running) since Wed 20XX-XX-XX XX:XX:XX CST; XXh ago

否則查看日志关斜，確認(rèn)原因：

journalctl -xu etcd

7）驗(yàn)證服務(wù)狀態(tài)

部署完 etcd 集群后示括，在任一個(gè)etc節(jié)點(diǎn)上執(zhí)行如下命令：

cat > magic18_verification_service.sh << "EOF"
#!/bin/bash
# 驗(yàn)證服務(wù)狀態(tài)
source /opt/k8s/bin/environment.sh
for node_ip in ${NODE_IPS[@]}
do
    echo ">>> ${node_ip}" 
    ETCDCTL_API=3 /opt/k8s/bin/etcdctl \
    --endpoints=https://${node_ip}:2379 \
    --cacert=/etc/kubernetes/cert/ca.pem \
    --cert=/etc/etcd/cert/etcd.pem \
    --key=/etc/etcd/cert/etcd-key.pem endpoint health
done
EOF