二進(jìn)制安裝K8s 之 部署ETCD集群
一泞辐、下載安裝cfssl，用于k8s證書簽名
二進(jìn)制包地址:https://pkg.cfssl.org/
所需軟件包：
- cfssl 1.6.0
- cfssljson 1.6.0
- cfssl-certinfo 1.6.0
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.0/cfssl_1.6.0_linux_amd64 -O cfssl
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.0/cfssljson_1.6.0_linux_amd64 -O cfssljson
wget https://github.com/cloudflare/cfssl/releases/download/v1.6.0/cfssl-certinfo_1.6.0_linux_amd64 -O cfssl-certinfo
chmod +x cfssl*
mv cfssl* /usr/local/bin/
2约计、生成etcd證書
- 自簽CA:
#生成默認(rèn)的證書配置文件【可以省略此步驟】,如果沒有證書配置文件模板可以使用
cfssl print-defaults config >ca-config.json
cfssl print-defaults csr >ca-csr.json
- 修改證書
cat > ca-config.json <<EOF
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"www": {
"expiry": "87600h",
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
]
}
}
}
}
EOF
cat > ca-csr.json <<EOF
{
"CN": "etcd CA",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "Beijing",
"ST": "Beijing"
}
]
}
EOF
- 生成證書
生成ca.pem ca-key.pem 根證書文件
cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
- 使用自簽CA簽發(fā)Etcd HTTPS證書
創(chuàng)建證書申請(qǐng)文件：
#注：上述文件hosts字段中IP為所有etcd節(jié)點(diǎn)的集群內(nèi)部通信IP，一個(gè)都不能少迁筛！為了方便后期擴(kuò)容可以多寫幾個(gè)預(yù)留的IP煤蚌。
cat > server-csr.json <<EOF
{
"CN": "etcd",
"hosts": [
"192.168.100.170",
"192.168.100.171",
"192.168.100.172",
"192.168.100.173"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing"
}
]
}
EOF
# 生成域名證書 server-key.pem server.pem
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
3、下載安裝etcd
- 下載二進(jìn)制包
#下載
wget https://github.com/etcd-io/etcd/releases/download/v3.5.0/etcd-v3.5.0-linux-amd64.tar.gz -O /data/download/
#解壓
tar -zxvf etcd-v3.5.0-linux-amd64.tar.gz
#建議復(fù)制到/usr/local/bin/ 目錄下
mv /data/download/etcd-v3.5.0-linux-amd64/{etcd,etcdctl} /usr/local/bin/
mv /data/download/etcd-v3.5.0-linux-amd64/{etcd,etcdctl} /data/k8s/etcd/bin/
- 創(chuàng)建etcd配置文件
cat > /data/etcd/config/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.100.170:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.100.170:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.100.170:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.100.170:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.100.170:2380,etcd-2=https://192.168.100.171:2380,etcd-3=https://192.168.100.172:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
- 3细卧、systemd管理etcd
注意證書路徑
cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/data/etcd/config/etcd.conf
ExecStart=/usr/local/bin/etcd \
--cert-file=/data/etcd/ssl/server.pem \
--key-file=/data/etcd/ssl/server-key.pem \
--peer-cert-file=/data/etcd/ssl/server.pem \
--peer-key-file=/data/etcd/ssl/server-key.pem \
--trusted-ca-file=/data/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/data/etcd/ssl/ca.pem \
--logger=zap
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
- 4尉桩、拷貝剛才生成的證書
把剛才生成的證書拷貝到配置文件中的路徑：
cp /data/docker/TSL/etcd/*.pem /data/etcd/ssl/
- 5、啟動(dòng)并設(shè)置開機(jī)啟動(dòng)
systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
6 贪庙、將上面節(jié)點(diǎn)1所有生成的文件拷貝到節(jié)點(diǎn)2和節(jié)點(diǎn)3
#復(fù)制整個(gè)目錄
scp -r /data/etcd/* root@192.168.100.171:/data/etcd/
scp -r /data/etcd/* root@192.168.100.172:/data/etcd/
#復(fù)制systemd文件
scp /usr/lib/systemd/system/etcd.service root@192.168.100.171:/usr/lib/systemd/system/
scp /usr/lib/systemd/system/etcd.service root@192.168.100.172:/usr/lib/systemd/system/
#cp etcd 二進(jìn)制文件 集群其他機(jī)器上操作
cp /data/etcd/bin/etc* /usr/local/bin/
#然后在節(jié)點(diǎn)2和節(jié)點(diǎn)3分別修改etcd.conf配置文件中的節(jié)點(diǎn)名稱和當(dāng)前服務(wù)器IP：
vi /opt/etcd/cfg/etcd.conf
#[Member]
ETCD_NAME="etcd-1" # 修改此處蜘犁，節(jié)點(diǎn)2改為etcd-2，節(jié)點(diǎn)3改為etcd-3
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.100.71:2380" # 修改此處為當(dāng)前服務(wù)器IP
ETCD_LISTEN_CLIENT_URLS="https://192.168.100.71:2379" # 修改此處為當(dāng)前服務(wù)器IP
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.100.71:2380" # 修改此處為當(dāng)前服務(wù)器IP
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.100.71:2379" # 修改此處為當(dāng)前服務(wù)器IP
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.100.71:2380,etcd-2=https://192.168.31.72:2380,etcd-3=https://192.168.31.73:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
#啟動(dòng)止邮，并設(shè)置開始啟動(dòng)
systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
7这橙、查看集群狀態(tài)
systemctl status etcd
● etcd.service - Etcd Server
Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
Active: active (running) since 日 2021-07-18 19:04:49 CST; 18s ago
Main PID: 1875 (etcd)
Tasks: 8
Memory: 33.5M
CGroup: /system.slice/etcd.service
└─1875 /usr/local/bin/etcd --cert-file=/data/k8s/etcd/ssl/server.pem --key-file=/data/k8s/etcd/ssl/server-key.pem --peer-cert-file=/data/k8s/etcd/ssl/server.pem --peer-key-file=/data/k8s/etcd/ssl/serve...
7月 18 19:04:51 master01 etcd[1875]: {"level":"info","ts":"2021-07-18T19:04:51.863+0800","caller":"rafthttp/peer_status.go:53","msg":"peer became active","peer-id":"1bd67ef396fd86"}
7月 18 19:04:51 master01 etcd[1875]: {"level":"info","ts":"2021-07-18T19:04:51.864+0800","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection with remote peer","stream-rea...bd67ef396fd86"}
7月 18 19:04:51 master01 etcd[1875]: {"level":"info","ts":"2021-07-18T19:04:51.865+0800","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection with remote peer","stream-rea...bd67ef396fd86"}
7月 18 19:04:51 master01 etcd[1875]: {"level":"info","ts":"2021-07-18T19:04:51.866+0800","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"7f0b6bf57639838f","to":"1bd67ef396fd8...eam MsgApp v2"}
7月 18 19:04:51 master01 etcd[1875]: {"level":"info","ts":"2021-07-18T19:04:51.866+0800","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-wri...bd67ef396fd86"}
7月 18 19:04:51 master01 etcd[1875]: {"level":"info","ts":"2021-07-18T19:04:51.899+0800","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"7f0b6bf57639838f","to":"1bd67ef396fd8...tream Message"}
7月 18 19:04:51 master01 etcd[1875]: {"level":"info","ts":"2021-07-18T19:04:51.899+0800","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-wri...bd67ef396fd86"}
7月 18 19:04:53 master01 etcd[1875]: {"level":"info","ts":"2021-07-18T19:04:53.988+0800","caller":"etcdserver/server.go:2481","msg":"updating cluster version using v2 API","from":"3.0","to":"3.5"}
7月 18 19:04:53 master01 etcd[1875]: {"level":"info","ts":"2021-07-18T19:04:53.991+0800","caller":"membership/cluster.go:523","msg":"updated cluster version","cluster-id":"a89a4473c024c0a2","local....0","to":"3.5"}
7月 18 19:04:53 master01 etcd[1875]: {"level":"info","ts":"2021-07-18T19:04:53.991+0800","caller":"etcdserver/server.go:2500","msg":"cluster version is updated","cluster-version":"3.5"}
Hint: Some lines were ellipsized, use -l to show in full.
etcdctl --cacert=/data/etcd/ssl/ca.pem \
--cert=/data/etcd/ssl/server.pem \
--key=/data/etcd/ssl/server-key.pem \
--endpoints="https://192.168.0.170:2379,https://192.168.100.171:2379,https://192.168.100.172:2379" endpoint status -w table
etcdctl --cacert=/data/etcd/ssl/ca.pem \
--cert=/data/etcd/ssl/server.pem \
--key=/data/etcd/ssl/server-key.pem \
--endpoints="https://192.168.100.170:2379,https://192.168.100.171:2379,https://192.168.100.172:2379" endpoint health
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.100.170:2379 | 32e07c4d987eefc | 3.5.0 | 29 kB | true | false | 2 | 9 | 9 | |
| https://192.168.100.171:2379 | 7ec2542a2723e9e3 | 3.5.0 | 20 kB | false | false | 2 | 9 | 9 | |
| https://192.168.100.172:2379 | 2186647c238c4402 | 3.5.0 | 20 kB | false | false | 2 | 9 | 9 | |
+----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
https://192.168.100.170:2379 is healthy: successfully committed proposal: took = 32.498535ms
https://192.168.100.171:2379 is healthy: successfully committed proposal: took = 37.070854ms
https://192.168.100.172:2379 is healthy: successfully committed proposal: took = 37.475938ms
#如果輸出上面信息奏窑，就說明集群部署成功。如果有問題第一步先看日志：/var/log/message 或 journalctl -u etcd
