版本信息
- OpenShift v3.11
故障場景
- 3/5 master 節(jié)點(diǎn)的集群
- 因機(jī)器故障只剩一個(gè) master 可用，沒有備份
- 集群 etcd 不可用
2020-01-07 14:20:58.154738 E | etcdserver: publish error: etcdserver: request timed out
2020-01-07 14:20:58.163767 W | rafthttp: health check for peer d9e07999afebd092 could not connect: dial tcp 192.168.0.128:2380: getsockopt: connection refused
2020-01-07 14:20:58.164548 W | rafthttp: health check for peer ebcee2e9c4c528b2 could not connect: dial tcp 192.168.0.110:2380: getsockopt: connection refused
2020-01-07 14:21:00.624133 I | raft: e00e9e227c7f69e4 is starting a new election at term 25
2020-01-07 14:21:00.624175 I | raft: e00e9e227c7f69e4 became candidate at term 26
2020-01-07 14:21:00.624189 I | raft: e00e9e227c7f69e4 received MsgVoteResp from e00e9e227c7f69e4 at term 26
2020-01-07 14:21:00.624199 I | raft: e00e9e227c7f69e4 [logterm: 4, index: 40800] sent MsgVote request to d9e07999afebd092 at term 26
2020-01-07 14:21:00.624219 I | raft: e00e9e227c7f69e4 [logterm: 4, index: 40800] sent MsgVote request to ebcee2e9c4c528b2 at term 26
- oc 命令不可用
# oc get nodes
Unable to connect to the server: EOF
故障處理過程
- 先恢復(fù)etcd辩尊，使剩余的1個(gè)etcd可用
- 修改配置赎婚，使剩下的一個(gè)etcd形成一個(gè)可用的etcd集群
- 在剩余的節(jié)點(diǎn)操作，比如我實(shí)驗(yàn)環(huán)境的master3.b8d8.internal
[root@master3 bak]# mv /etc/origin/node/pods/etcd.yaml .
[root@master3 bak]# cp /etc/etcd/etcd.conf etcd.conf.bak
[root@master3 bak]# echo "ETCD_FORCE_NEW_CLUSTER=true" >> /etc/etcd/etcd.conf
[root@master3 bak]# mv etcd.yaml /etc/origin/node/pods/.
- 去掉ETCD_FORCE_NEW_CLUSTER=true變量玖像，重新啟動(dòng)etcd
[root@master3 bak]# mv /etc/origin/node/pods/etcd.yaml .
[root@master3 bak]# rm -rf /etc/etcd/etcd.conf
[root@master3 bak]# mv etcd.conf.bak /etc/etcd/etcd.conf
[root@master3 bak]# mv etcd.yaml /etc/origin/node/pods/.
- 恢復(fù)一個(gè)etcd后集群已經(jīng)可用正常使用紫谷，只是沒有高可用了
[root@master3 bak]# oc get nodes
NAME STATUS ROLES AGE VERSION
infranode1.b8d8.internal Ready infra 2h v1.11.0+d4cacc0
infranode2.b8d8.internal Ready infra 2h v1.11.0+d4cacc0
master1.b8d8.internal NotReady master 2h v1.11.0+d4cacc0
master2.b8d8.internal NotReady master 2h v1.11.0+d4cacc0
master3.b8d8.internal Ready master 2h v1.11.0+d4cacc0
node1.b8d8.internal Ready compute 2h v1.11.0+d4cacc0
node2.b8d8.internal Ready compute 2h v1.11.0+d4cacc0
node3.b8d8.internal Ready compute 2h v1.11.0+d4cacc0
- 刪除故障節(jié)點(diǎn)
[root@master3 bak]# oc delete node master1.b8d8.internal master2.b8d8.internal
node "master1.b8d8.internal" deleted
node "master2.b8d8.internal" deleted
[root@master3 bak]# oc get nodes
NAME STATUS ROLES AGE VERSION
infranode1.b8d8.internal Ready infra 2h v1.11.0+d4cacc0
infranode2.b8d8.internal Ready infra 2h v1.11.0+d4cacc0
master3.b8d8.internal Ready master 2h v1.11.0+d4cacc0
node1.b8d8.internal Ready compute 2h v1.11.0+d4cacc0
node2.b8d8.internal Ready compute 2h v1.11.0+d4cacc0
node3.b8d8.internal Ready compute 2h v1.11.0+d4cacc0
- 重建etcd ca證書，因?yàn)槲覀兪チ薸nventory file文件里“第一”個(gè)master節(jié)點(diǎn)捐寥◇宰颍“第一”個(gè)master節(jié)點(diǎn)非常重要，這個(gè)節(jié)點(diǎn)有ca證書的詳細(xì)信息上真。
- 修改inventory file,去掉故障節(jié)點(diǎn)
[OSEv3:children]
lb
masters
etcd
nodes
nfs
[lb]
loadbalancer.b8d8.internal
[masters]
## 刪掉故障節(jié)點(diǎn)
#master1.b8d8.internal
#master2.b8d8.internal
master3.b8d8.internal
[etcd]
## 刪掉故障節(jié)點(diǎn)
#master1.b8d8.internal
#master2.b8d8.internal
master3.b8d8.internal
[nodes]
## These are the masters
## 刪掉故障節(jié)點(diǎn)
#master1.b8d8.internal openshift_node_group_name='node-config-master' openshift_node_problem_detector_install=true
#master2.b8d8.internal openshift_node_group_name='node-config-master' openshift_node_problem_detector_install=true
master3.b8d8.internal openshift_node_group_name='node-config-master' openshift_node_problem_detector_install=true
## These are infranodes
infranode1.b8d8.internal openshift_node_group_name='node-config-infra' openshift_node_problem_detector_install=true
infranode2.b8d8.internal openshift_node_group_name='node-config-infra' openshift_node_problem_detector_install=true
## These are regular nodes
node1.b8d8.internal openshift_node_group_name='node-config-compute' openshift_node_problem_detector_install=true
node2.b8d8.internal openshift_node_group_name='node-config-compute' openshift_node_problem_detector_install=true
node3.b8d8.internal openshift_node_group_name='node-config-compute' openshift_node_problem_detector_install=true
···
- 跑重建etcd ca證書腳本
- 在ansible節(jié)點(diǎn)運(yùn)行
[root@bastion ~]# cd /usr/share/ansible/openshift-ansible
[root@bastion openshift-ansible]# ansible-playbook -i /etc/ansible/hosts_redeploy-certificates playbooks/openshift-etcd/redeploy-ca.yml -vv
- 擴(kuò)展master節(jié)點(diǎn)咬腋，恢復(fù)3/5個(gè)
- 修改inventory file,將需要添加回來的節(jié)點(diǎn)寫上去
OSEv3:children]
lb
masters
etcd
## 需要添加這兩個(gè)flag new_nodes/new_masters
new_nodes
new_masters
nodes
nfs
[lb]
loadbalancer.b8d8.internal
[masters]
#master1.b8d8.internal
#master2.b8d8.internal
master3.b8d8.internal
## 重新添加回來的master
[new_masters]
master1.b8d8.internal
master2.b8d8.internal
[etcd]
#master1.b8d8.internal
#master2.b8d8.internal
master3.b8d8.internal
[nodes]
## These are the masters
#master1.b8d8.internal openshift_node_group_name='node-config-master' openshift_node_problem_detector_install=true
#master2.b8d8.internal openshift_node_group_name='node-config-master' openshift_node_problem_detector_install=true
master3.b8d8.internal openshift_node_group_name='node-config-master' openshift_node_problem_detector_install=true
## These are infranodes
infranode1.b8d8.internal openshift_node_group_name='node-config-infra' openshift_node_problem_detector_install=true
infranode2.b8d8.internal openshift_node_group_name='node-config-infra' openshift_node_problem_detector_install=true
## These are regular nodes
node1.b8d8.internal openshift_node_group_name='node-config-compute' openshift_node_problem_detector_install=true
node2.b8d8.internal openshift_node_group_name='node-config-compute' openshift_node_problem_detector_install=true
node3.b8d8.internal openshift_node_group_name='node-config-compute' openshift_node_problem_detector_install=true
## 重新添加回來的master
[new_nodes]
master1.b8d8.internal openshift_node_group_name='node-config-master' openshift_node_problem_detector_install=true
master2.b8d8.internal openshift_node_group_name='node-config-master' openshift_node_problem_detector_install=true
···
- 在剩余節(jié)點(diǎn)去掉yum的unexclude, grep的結(jié)果不能含有atomic-openshift*
[root@master3 bak]# atomic-openshift-excluder unexclude
[root@master3 bak]# grep exclude /etc/yum.conf
exclude= docker*1.20* docker*1.19* docker*1.18* docker*1.17* docker*1.16* docker*1.15* docker*1.14*
- 把ca.serial.txt文件放在剩余master節(jié)點(diǎn)的/etc/origin/master/，這個(gè)文件是在部署集群的時(shí)候生成的睡互，在集群的“第一”個(gè)master節(jié)點(diǎn)上根竿，如果沒有這個(gè)文件，這個(gè)集群就擴(kuò)不回3/5個(gè)master節(jié)點(diǎn)的高可用集群了.
- 如果真的沒有怎么辦就珠？可以嘗試手寫一個(gè)寇壳，值為0F
[root@master3 bak]# cp ca.serial.txt /etc/origin/master/
- 運(yùn)行 add master腳本
[root@bastion ~]# cd /usr/share/ansible/openshift-ansible
[root@bastion openshift-ansible]# ansible-playbook -i /etc/ansible/hosts_add_masters playbooks/openshift-master/scaleup.yml -vv
- approve csr
- 在剩余的節(jié)點(diǎn)操作
[root@master3 bak]# oc get csr -o name | xargs oc adm certificate approve
- 擴(kuò)展etcd節(jié)點(diǎn)，恢復(fù)3/5個(gè)節(jié)點(diǎn)的集群
- 修改inventory file,將需要添加的etcd節(jié)點(diǎn)寫上去妻怎，注意目前的master節(jié)點(diǎn)順序
[OSEv3:children]
lb
masters
etcd
## 添加etcd節(jié)點(diǎn)
new_etcd
nodes
nfs
[lb]
loadbalancer.b8d8.internal
[masters]
## 注意這里的順序壳炎，現(xiàn)在master3.b8d8.internal才是“第一”個(gè)master了
master3.b8d8.internal
master1.b8d8.internal
master2.b8d8.internal
[etcd]
#master1.b8d8.internal
#master2.b8d8.internal
master3.b8d8.internal
[new_etcd]
## 添加etcd節(jié)點(diǎn)
master1.b8d8.internal
master2.b8d8.internal
[nodes]
## These are the masters
master1.b8d8.internal openshift_node_group_name='node-config-master' openshift_node_problem_detector_install=true
master2.b8d8.internal openshift_node_group_name='node-config-master' openshift_node_problem_detector_install=true
master3.b8d8.internal openshift_node_group_name='node-config-master' openshift_node_problem_detector_install=true
## These are infranodes
infranode1.b8d8.internal openshift_node_group_name='node-config-infra' openshift_node_problem_detector_install=true
infranode2.b8d8.internal openshift_node_group_name='node-config-infra' openshift_node_problem_detector_install=true
## These are regular nodes
node1.b8d8.internal openshift_node_group_name='node-config-compute' openshift_node_problem_detector_install=true
node2.b8d8.internal openshift_node_group_name='node-config-compute' openshift_node_problem_detector_install=true
node3.b8d8.internal openshift_node_group_name='node-config-compute' openshift_node_problem_detector_install=true
···
- 運(yùn)行add etcd腳本
[root@bastion ~]# cd /usr/share/ansible/openshift-ansible
[root@bastion ~]# ansible-playbook -i /etc/ansible/hosts_add_etcd playbooks/openshift-etcd/scaleup.yml -vv
驗(yàn)證
- 查看etcd/api pod
[root@master3 bak]# oc get pod --all-namespaces |grep etcd
kube-system master-etcd-master1.b8d8.internal 1/1 Running 0 18s
kube-system master-etcd-master2.b8d8.internal 1/1 Running 0 1m
kube-system master-etcd-master3.b8d8.internal 1/1 Running 3 3h
[root@master3 bak]# oc get pod --all-namespaces |grep master-api
kube-system master-api-master1.b8d8.internal 1/1 Running 2 9m
kube-system master-api-master2.b8d8.internal 1/1 Running 2 9m
kube-system master-api-master3.b8d8.internal 1/1 Running 12 3h
- 檢查etcd集群狀態(tài)
[root@master3 bak]# oc get pod --all-namespaces -owide |grep etcd
kube-system master-etcd-master1.b8d8.internal 1/1 Running 0 3m 192.168.0.128 master1.b8d8.internal <none>
kube-system master-etcd-master2.b8d8.internal 1/1 Running 0 4m 192.168.0.110 master2.b8d8.internal <none>
kube-system master-etcd-master3.b8d8.internal 1/1 Running 3 3h 192.168.0.152 master3.b8d8.internal <none>
[root@master3 bak]# oc get pod --all-namespaces -owide |grep master-api
kube-system master-api-master1.b8d8.internal 1/1 Running 2 12m 192.168.0.128 master1.b8d8.internal <none>
kube-system master-api-master2.b8d8.internal 1/1 Running 2 12m 192.168.0.110 master2.b8d8.internal <none>
kube-system master-api-master3.b8d8.internal 1/1 Running 12 3h 192.168.0.152 master3.b8d8.internal <none>
- 停掉原來的節(jié)點(diǎn)檢查集群是否還正常工作
完
