集群方案：
- 發(fā)行版：CentOS 7
- 容器運行時
- 內(nèi)核： 4.18.12-1.el7.elrepo.x86_64
- 版本：Kubernetes: 1.14.0
- 網(wǎng)絡(luò)方案: Calico
- kube-proxy mode: IPVS
- master高可用方案：HAProxy keepalived LVS
- DNS插件: CoreDNS
- metrics插件：metrics-server
- 界面：kubernetes-dashboard
Kubernetes集群搭建
| Host Name | Role | IP |
|---|---|---|
| master1 | master1 | 192.168.56.103 |
| master2 | master2 | 192.168.56.104 |
| master3 | master3 | 192.168.56.105 |
| node1 | node1 | 192.168.56.106 |
| node2 | node2 | 192.168.56.107 |
| node3 | node3 | 192.168.56.108 |
1牙丽、離線安裝包準備（基于能夠訪問外網(wǎng)的服務(wù)器下載相應(yīng)安裝包）
# 設(shè)置yum緩存路徑，cachedir 緩存路徑 keepcache=1保持安裝包在軟件安裝之后不刪除
cat /etc/yum.conf
[main]
cachedir=/home/yum
keepcache=1
...
# 安裝ifconfig
yum install net-tools -y
# 時間同步
yum install -y ntpdate
# 安裝docker（建議19.8.06）
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager \
--add-repo \
https://download.docker.com/linux/centos/docker-ce.repo
yum makecache fast
## 列出Docker版本
yum list docker-ce --showduplicates | sort -r
## 安裝指定版本
sudo yum install docker-ce-<VERSION_STRING>
# 安裝文件管理器，XShell可通過rz sz命令上傳或者下載服務(wù)器文件
yum intall lrzsz -y
# 安裝keepalived续誉、haproxy
yum install -y socat keepalived ipvsadm haproxy
# 安裝kubernetes相關(guān)組件
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
yum install -y kubelet kubeadm kubectl ebtables
# 其他軟件安裝
yum install wget
...
2、節(jié)點系統(tǒng)配置
-
關(guān)閉SELinux初肉、防火墻
systemctl stop firewalld systemctl disable firewalld setenforce 0 sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config -
關(guān)閉系統(tǒng)的Swap（Kubernetes 1.8開始要求）
swapoff -a yes | cp /etc/fstab /etc/fstab_bak cat /etc/fstab_bak |grep -v swap > /etc/fstab -
配置L2網(wǎng)橋在轉(zhuǎn)發(fā)包時會被iptables的FORWARD規(guī)則所過濾酷鸦，該配置被CNI插件需要，更多信息請參考Network Plugin Requirements
echo """ vm.swappiness = 0 net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 """ > /etc/sysctl.conf sysctl -pcentos7添加bridge-nf-call-ip6tables出現(xiàn)No such file or directory,簡單來說就是執(zhí)行一下 modprobe br_netfilter
-
同步時間
ntpdate -u ntp.api.bz -
升級內(nèi)核到最新（已準備內(nèi)核離線安裝包牙咏，可選）
grub2-set-default 0 && grub2-mkconfig -o /etc/grub2.cfg grubby --default-kernel grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)" -
重啟系統(tǒng)臼隔，確認內(nèi)核版本后，開啟IPVS（如果未升級內(nèi)核妄壶，去掉ip_vs_fo）
uname -a cat > /etc/sysconfig/modules/ipvs.modules <<EOF #!/bin/bash ipvs_modules="ip_vs ip_vs_lc ip_vs_wlc ip_vs_rr ip_vs_wrr ip_vs_lblc ip_vs_lblcr ip_vs_dh ip_vs_sh ip_vs_fo ip_vs_nq ip_vs_sed ip_vs_ftp nf_conntrack" for kernel_module in \${ipvs_modules}; do /sbin/modinfo -F filename \${kernel_module} > /dev/null 2>&1 if [ $? -eq 0 ]; then /sbin/modprobe \${kernel_module} fi done EOF chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep ip_vs執(zhí)行sysctl -p報錯可執(zhí)行modprobe br_netfilter摔握，請參考centos7添加bridge-nf-call-ip6tables出現(xiàn)No such file or directory
-
所有機器需要設(shè)定/etc/sysctl.d/k8s.conf的系統(tǒng)參數(shù)
# https://github.com/moby/moby/issues/31208 # ipvsadm -l --timout # 修復(fù)ipvs模式下長連接timeout問題 小于900即可 cat <<EOF > /etc/sysctl.d/k8s.conf net.ipv4.tcp_keepalive_time = 600 net.ipv4.tcp_keepalive_intvl = 30 net.ipv4.tcp_keepalive_probes = 10 net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 net.ipv6.conf.lo.disable_ipv6 = 1 net.ipv4.neigh.default.gc_stale_time = 120 net.ipv4.conf.all.rp_filter = 0 net.ipv4.conf.default.rp_filter = 0 net.ipv4.conf.default.arp_announce = 2 net.ipv4.conf.lo.arp_announce = 2 net.ipv4.conf.all.arp_announce = 2 net.ipv4.ip_forward = 1 net.ipv4.tcp_max_tw_buckets = 5000 net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_max_syn_backlog = 1024 net.ipv4.tcp_synack_retries = 2 net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.netfilter.nf_conntrack_max = 2310720 fs.inotify.max_user_watches=89100 fs.may_detach_mounts = 1 fs.file-max = 52706963 fs.nr_open = 52706963 net.bridge.bridge-nf-call-arptables = 1 vm.swappiness = 0 vm.overcommit_memory=1 vm.panic_on_oom=0 EOF sysctl --system -
設(shè)置開機啟動
# 啟動docker sed -i "13i ExecStartPost=/usr/sbin/iptables -P FORWARD ACCEPT" /usr/lib/systemd/system/docker.service systemctl daemon-reload systemctl enable docker systemctl start docker # 設(shè)置kubelet開機啟動 systemctl enable kubelet systemctl enable keepalived systemctl enable haproxy -
設(shè)置免密登錄
# 1、三次回車后丁寄，密鑰生成完成 ssh-keygen # 2氨淌、拷貝密鑰到其他節(jié)點 ssh-copy-id -i ~/.ssh/id_rsa.pub 用戶名字@192.168.x.xxx
**、 Kubernetes要求集群中所有機器具有不同的Mac地址伊磺、產(chǎn)品uuid盛正、Hostname。
3奢浑、keepalived+haproxy配置
cd ~/
# 創(chuàng)建集群信息文件
echo """
CP0_IP=192.168.56.103
CP1_IP=192.168.56.104
CP2_IP=192.168.56.105
VIP=192.168.56.102
NET_IF=eth0
CIDR=10.244.0.0/16
""" > ./cluster-info
bash -c "$(curl -fsSL https://raw.githubusercontent.com/hnbcao/kubeadm-ha-master/v1.14.0/keepalived-haproxy.sh)"
4蛮艰、部署HA Master
HA Master的部署過程已經(jīng)自動化，請在master-1上執(zhí)行如下命令，并注意修改IP;
腳本主要執(zhí)行三步：
1)壤蚜、重置kubelet設(shè)置
kubeadm reset -f
rm -rf /etc/kubernetes/pki/
2)即寡、編寫節(jié)點配置文件并初始化master1的kubelet
echo """
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: v1.13.0
controlPlaneEndpoint: "${VIP}:8443"
maxPods: 100
networkPlugin: cni
imageRepository: registry.aliyuncs.com/google_containers
apiServer:
certSANs:
- ${CP0_IP}
- ${CP1_IP}
- ${CP2_IP}
- ${VIP}
networking:
# This CIDR is a Calico default. Substitute or remove for your CNI provider.
podSubnet: ${CIDR}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
""" > /etc/kubernetes/kubeadm-config.yaml
kubeadm init --config /etc/kubernetes/kubeadm-config.yaml
mkdir -p $HOME/.kube
cp -f /etc/kubernetes/admin.conf ${HOME}/.kube/config
- 關(guān)于默認網(wǎng)關(guān)問題，如果有多張網(wǎng)卡袜刷，需要先將默認網(wǎng)關(guān)切換到集群使用的那張網(wǎng)卡上聪富，否則可能會出現(xiàn)etcd無法連接等問題。（應(yīng)用我用的虛擬機著蟹，有一張網(wǎng)卡無法做到各個節(jié)點胡同墩蔓；route查看當前網(wǎng)關(guān)信息，route del default刪除默認網(wǎng)關(guān)萧豆，route add default enth0設(shè)置默認網(wǎng)關(guān)enth0為網(wǎng)卡名）
3)奸披、拷貝相關(guān)證書到master2、master3
for index in 1 2; do
ip=${IPS[${index}]}
ssh $ip "mkdir -p /etc/kubernetes/pki/etcd; mkdir -p ~/.kube/"
scp /etc/kubernetes/pki/ca.crt $ip:/etc/kubernetes/pki/ca.crt
scp /etc/kubernetes/pki/ca.key $ip:/etc/kubernetes/pki/ca.key
scp /etc/kubernetes/pki/sa.key $ip:/etc/kubernetes/pki/sa.key
scp /etc/kubernetes/pki/sa.pub $ip:/etc/kubernetes/pki/sa.pub
scp /etc/kubernetes/pki/front-proxy-ca.crt $ip:/etc/kubernetes/pki/front-proxy-ca.crt
scp /etc/kubernetes/pki/front-proxy-ca.key $ip:/etc/kubernetes/pki/front-proxy-ca.key
scp /etc/kubernetes/pki/etcd/ca.crt $ip:/etc/kubernetes/pki/etcd/ca.crt
scp /etc/kubernetes/pki/etcd/ca.key $ip:/etc/kubernetes/pki/etcd/ca.key
scp /etc/kubernetes/admin.conf $ip:/etc/kubernetes/admin.conf
scp /etc/kubernetes/admin.conf $ip:~/.kube/config
ssh ${ip} "${JOIN_CMD} --experimental-control-plane"
done
4)涮雷、master2阵面、master3加入節(jié)點
JOIN_CMD=`kubeadm token create --print-join-command`
ssh ${ip} "${JOIN_CMD} --experimental-control-plane"
完整腳本：
# 部署HA master
bash -c "$(curl -fsSL https://raw.githubusercontent.com/hnbcao/kubeadm-ha-master/v1.14.0/kube-ha.sh)"
5、加入節(jié)點
-
各個節(jié)點需要配置keepalived 和 haproxy
#/etc/haproxy/haproxy.cfg global log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon stats socket /var/lib/haproxy/stats defaults mode tcp log global option tcplog option dontlognull option redispatch retries 3 timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout check 10s maxconn 3000 listen stats mode http bind :10086 stats enable stats uri /admin?stats stats auth admin:admin stats admin if TRUE frontend k8s_https *:8443 mode tcp maxconn 2000 default_backend https_sri backend https_sri balance roundrobin server master1-api ${MASTER1_IP}:6443 check inter 10000 fall 2 rise 2 weight 1 server master2-api ${MASTER2_IP}:6443 check inter 10000 fall 2 rise 2 weight 1 server master3-api ${MASTER3_IP}:6443 check inter 10000 fall 2 rise 2 weight 1#/etc/keepalived/keepalived.conf global_defs { router_id LVS_DEVEL } vrrp_script check_haproxy { script /etc/keepalived/check_haproxy.sh interval 3 } vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 80 priority 100 advert_int 1 authentication { auth_type PASS auth_pass just0kk } virtual_ipaddress { ${VIP}/24 } track_script { check_haproxy } } }
注意兩個配置中的{MASTER2 _ IP},
{VIP}需要替換為自己集群相應(yīng)的IP地址
-
重啟keepalived和haproxy
systemctl stop keepalived systemctl enable keepalived systemctl start keepalived systemctl stop haproxy systemctl enable haproxy systemctl start haproxy -
節(jié)點加入命令獲取
#master節(jié)點執(zhí)行該命令样刷，再在節(jié)點執(zhí)行獲取到的命令 kubeadm token create --print-join-command
6、結(jié)束安裝
當前集群安裝完畢览爵，還需要安裝cni插件置鼻，推薦使用calico，性能優(yōu)于其他蜓竹。
