引子
目前屿附，kubeadm 已經(jīng)支持了搭建高可用的 Kubernetes 集群矾削，大大降低了搭建的難度壤玫，官方的文檔也非常地簡潔明了，但是還是有些坑點(diǎn)哼凯，下面我就將我的操作記錄下欲间。
架構(gòu)
Master節(jié)點(diǎn)上只要運(yùn)行一下三個(gè)服務(wù):
- kube-apiserver: 無狀態(tài)的，需要通過LB來實(shí)現(xiàn)高可用断部。
- kube-controller-manager: 自帶leader-elected功能猎贴，kubeadm初始化時(shí)，默認(rèn)
--leader-elect=true; - kube-scheduler: 自帶leader-elected功能，kubeadm初始化時(shí)嘱能，默認(rèn)
--leader-elect=true;
環(huán)境說明
- OS: Ubuntu 16.04
- Kubernetes: 1.9
- CNI Network: Calico
- 節(jié)點(diǎn)信息:
| Hostname | IP | 備注 |
|---|---|---|
| k8s-master-01 | 192.168.4.24 | |
| k8s-master-02 | 192.168.4.25 | |
| k8s-master-03 | 192.168.4.26 | |
| k8s-node-01 | 192.168.4.27 | kubernetes 工作節(jié)點(diǎn) |
| lb-haproxy | 192.168.4.40 | 通過 haproxy 來實(shí)現(xiàn) kube-api 的 LB |
前置準(zhǔn)備
開始之前，請確保你有個(gè)能翻墻的環(huán)境虱疏，推薦使用 shadowsocket惹骂，并開啟 http proxy。
1. LB 準(zhǔn)備
開源的LB實(shí)現(xiàn)有很多做瞪，比如 ipvs, nginx, haproxy等等对粪。在選型時(shí)，請注意装蓬，這是我踩的第一個(gè)坑著拭，我剛開始選用的方案是 ipvs 的 DR 模式來實(shí)現(xiàn)一個(gè)四層的負(fù)載均衡，請看下面的流程：
master kube-proxy -> LB -> master
# 等同于牍帚，這在ipvs DR下是有問題的儡遮，因?yàn)閙aster是ipvs后端實(shí)際的負(fù)載，它上面有條路由是：route add -host $vip lo:0, $vip的路由被直接發(fā)送到了Loopback接口了暗赶；
master kube-proxy -> master
后來該用 haproxy 來實(shí)現(xiàn)鄙币，就沒有問題了，安裝步驟就略過蹂随，給個(gè)我自己的最簡單配置：
haproxy.cfg
########## Kube-API LB #####################
listen kube-api-lb
bind 0.0.0.0:6443
mode tcp
balance roundrobin
server k8s-master-01 192.168.4.24:6443 weight 1 maxconn 10000 check inter 10s
server k8s-master-02 192.168.4.25:6443 weight 1 maxconn 10000 check inter 10s
server k8s-master-03 192.168.4.26:6443 weight 1 maxconn 10000 check inter 10s
######## stast ############################
listen admin_stats
bind 0.0.0.0:8099
mode http
option httplog
maxconn 10
stats refresh 30s
stats uri /stats
2. 關(guān)閉 swap 分區(qū)
swapoff -a
#要永久禁掉swap分區(qū)十嘿，打開如下文件注釋掉swap那一行
vi /etc/fstab
3. 開啟 http_proxy
后續(xù)安裝 kubeadm 等資源都需要翻墻，這里需要開命令行下開啟代理:
export http_proxy="http://192.168.4.18:1080"
export https_proxy="http://192.168.4.18:1080"
export no_proxy="192.168.4.24,192.168.4.25,192.168.4.26,127.0.0.1"
4. ETCD 集群
我這里 etcd 集群沒有開啟TLS的支持岳锁，需要的可以參考官方的文檔绩衷。這里還需要注意下，如何你想查看下 kubernetes 或者 calico 存在 etcd 中的內(nèi)容激率，請使用api3的版本：export ETCDCTL_API=3, 不然你使用etcdctl ls /來查看是沒有任何內(nèi)容的咳燕。
下載 etcd 二進(jìn)制包:
export ETCD_VERSION=v3.1.10
curl -sSL https://github.com/coreos/etcd/releases/download/${ETCD_VERSION}/etcd-${ETCD_VERSION}-linux-amd64.tar.gz | tar -xzv --strip-components=1 -C /usr/local/bin/
rm -rf etcd-$ETCD_VERSION-linux-amd64*
配置啟動(dòng)腳本:
# node1
touch /etc/etcd.env
echo "PEER_NAME=k8s-master-01" >> /etc/etcd.env
echo "PRIVATE_IP=192.168.4.24" >> /etc/etcd.env
# node2
touch /etc/etcd.env
echo "PEER_NAME=k8s-master-02" >> /etc/etcd.env
echo "PRIVATE_IP=192.168.4.25" >> /etc/etcd.env
# node3
touch /etc/etcd.env
echo "PEER_NAME=k8s-master-03" >> /etc/etcd.env
echo "PRIVATE_IP=192.168.4.26" >> /etc/etcd.env
cat >/etc/systemd/system/etcd.service <<EOL
[Unit]
Description=etcd
Documentation=https://github.com/coreos/etcd
Conflicts=etcd.service
Conflicts=etcd2.service
[Service]
EnvironmentFile=/etc/etcd.env
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0
ExecStart=/usr/local/bin/etcd --name ${PEER_NAME} \
--data-dir /var/lib/etcd \
--listen-client-urls http://${PRIVATE_IP}:2379,http://127.0.0.1:2379 \
--advertise-client-urls http://${PRIVATE_IP}:2379 \
--listen-peer-urls http://${PRIVATE_IP}:2380 \
--initial-advertise-peer-urls http://${PRIVATE_IP}:2380 \
--initial-cluster k8s-master-01=http://192.168.4.24:2380,k8s-master-02=http://192.168.4.25:2380,k8s-master-03=http://192.168.4.26:2380 \
--initial-cluster-token my-etcd-token \
--initial-cluster-state new
[Install]
WantedBy=multi-user.target
EOL
啟動(dòng):
systemctl daemon-reload
systemctl start etcd
檢查集群運(yùn)行狀態(tài):
etcdctl cluster-health
5. Docker Daemon
docker的版本貌似也有要求:
On each of your machines, install Docker. Version v1.12 is recommended, but v1.11, v1.13 and 17.03 are known to work as well. Versions 17.06+ might work, but have not yet been tested and verified by the Kubernetes node team.
apt-get update
apt-get install -y docker.io=1.13.1-0ubuntu1~16.04.2
docker daemon使用http代理
mkdir -p /etc/systemd/system/docker.service.d
vim /etc/systemd/system/docker.service.d/http-proxy.conf
[Service]
Environment="HTTP_PROXY=http://192.168.4.18:1080"
Environment="HTTPS_PROXY=http://192.168.4.18:1080"
Environment="NO_PROXY=192.168.4.0/24"
systemctl daemon-reload
systemctl restart docker
6. 安裝 kubeadm kubelet kubectl
apt-get update && apt-get install -y apt-transport-https
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
EOF
apt-get update
apt-get install -y kubelet kubeadm kubectl
建立 Master 節(jié)點(diǎn)
1. kubeadm init
通過配置文件進(jìn)行初始化，這里需要注意的是: podSubnet 需要跟后面CNI網(wǎng)絡(luò)保持一致柱搜；
cat >config.yaml <<EOL
apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
api:
advertiseAddress: 192.168.4.24
etcd:
endpoints:
- http://192.168.4.24:2379
- http://192.168.4.25:2379
- http://192.168.4.26:2379
networking:
podSubnet: 10.1.0.0/16
apiServerCertSANs:
- 192.168.4.24
- 192.168.4.25
- 192.168.4.26
- 192.168.4.27
- 192.168.4.40
apiServerExtraArgs:
endpoint-reconciler-type: lease
EOL
kubeadm init --config=config.yaml
2. 拷貝證書
待第一個(gè)master節(jié)點(diǎn)完成初始化后迟郎，將/etc/kubernetes/pki目錄下的所有文件拷貝至其他兩個(gè)節(jié)點(diǎn)。
同上步驟聪蘸，其他兩個(gè)節(jié)點(diǎn)使用kubeadm init --config=config.yaml來完成初始化宪肖。
3. 配置kubectl的配置文件
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
如果是 root 用戶:
export KUBECONFIG=/etc/kubernetes/admin.conf
4. 準(zhǔn)備CNI網(wǎng)絡(luò)
kubernetes 前提是需要Pod能跨主機(jī)通訊，這里我選擇的方案是Calico, 在CNI網(wǎng)絡(luò)沒有準(zhǔn)備好之前健爬，Master節(jié)點(diǎn)的狀態(tài)為NotReady控乾。我這是使用上面的etcd集群作為calico的存儲(chǔ)，更多請參考calico Standard Hosted Install娜遵。
注意: 這里又一個(gè)坑點(diǎn)來了蜕衡，kubeadm初始化的Master默認(rèn)是不允許調(diào)度Pod的， Calico的node卻使用DaemonSet來部署的设拟，所以我們先要把這個(gè)限制給去掉先, Master節(jié)點(diǎn)通過taints來限制的慨仿，更多請參考: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/:
去除限制：
kubectl taint nodes k8s-master-01 node-role.kubernetes.io/master:NoSchedule-
kubectl taint nodes k8s-master-02 node-role.kubernetes.io/master:NoSchedule-
kubectl taint nodes k8s-master-03 node-role.kubernetes.io/master:NoSchedule-
建立 RBAC 的角色
https://docs.projectcalico.org/v3.0/getting-started/kubernetes/installation/hosted/hosted
通過calico.yaml部署CNI
wget https://docs.projectcalico.org/v3.0/getting-started/kubernetes/installation/hosted/calico.yaml
# calico.yaml 需要修改兩個(gè)地方：
1. etcd_endpoints: "http://192.168.4.24:2379,http://192.168.4.25:2379,http://192.168.4.26:2379"
2. - name: CALICO_IPV4POOL_CIDR
value: "10.1.0.0/16" # 這個(gè)應(yīng)該與kubeadm初始化文件config.yaml一致
kubectl apply -f calico.yaml
Master 節(jié)點(diǎn)加上Pod調(diào)度限制:
kubectl taint nodes k8s-master-01 node-role.kubernetes.io/master=:NoSchedule
kubectl taint nodes k8s-master-02 node-role.kubernetes.io/master=:NoSchedule
kubectl taint nodes k8s-master-03 node-role.kubernetes.io/master=:NoSchedule
Worker Node 節(jié)點(diǎn)
1. 準(zhǔn)備
worker node上也需要安裝docker damon, kubelet, kubeadm, kubectl, 具體請參考Master節(jié)點(diǎn)久脯。
2. worker node join
kubeadm join --token a65269.b45f13a6e90114e3 192.168.4.24:6443 --discovery-token-ca-cert-hash sha256:f71fb0208c16a54d782a2a05f33d5b22a062ede8f14c901384e9020534dca169
3. 將kubep-proxy, kubelet的kube-api地址指向LB
kubectl get configmap -n kube-system kube-proxy -o yaml > kube-proxy.yaml
sudo sed -i 's#server:.*#server: https://<masterLoadBalancerFQDN>:6443#g' kube-proxy.yaml
kubectl apply -f kube-proxy.yaml --force
# restart all kube-proxy pods to ensure that they load the new configmap
kubectl delete pod -n kube-system -l k8s-app=kube-proxy
sudo sed -i 's#server:.*#server: https://<masterLoadBalancerFQDN>:6443#g' /etc/kubernetes/kubelet.conf
sudo systemctl restart kubelet
參考
https://kubernetes.io/docs/setup/independent/high-availability/
