公網(wǎng)部署K8S
場景：
- 跨云廠商的機器
- 非同一網(wǎng)段的機器
機器準(zhǔn)備：
Linux 主機3臺 (CETOS7.6 2CPU 4G)
節(jié)點 MAC 地址 和 product_uuid 的唯一性
端口檢查:
nc 127.0.0.1 6443-
申請域名，解析指向master機器IP（可選）
master node1 node2 IP 114.132.94.160 43.138.235.139 43.139.23.242 系統(tǒng) Centos7.6 Centos7.6 Centos7.6
版本說明：
- K8S @ v1.25
- docker-engine @ 20.10.18
過程中問題整理：
目前已整理的過程中問題， 更多問題歡迎大家以issue方式提交給我爹殊， 謝謝。
部署步驟
- 清除舊K8S部署痕跡
- 安裝docker(每個主機)
- 配置cri-docker使kubernetes以docker作為運行時(每個主機)
- 配置基礎(chǔ)環(huán)境(每個主機)
- 建立虛擬機網(wǎng)卡(每個主機)
- 安裝kubernetes(每個主機)
- 修改kubelet啟動參數(shù)文件(每個主機)
- 開啟云服務(wù)器端口(每個主機)
- 初始化集群(Master)
- 工作節(jié)點加入集群(Node)
- 安裝flannel網(wǎng)絡(luò)插件(Master)
- 部署nginx驗證安裝
清除舊K8S部署痕跡
- 安裝失敗需要reset集群
sudo kubeadm reset --cri-socket /var/run/cri-dockerd.sock - 清除文件
rm -rf /root/.kube/ sudo rm -rf /etc/kubernetes/ sudo rm -rf /var/lib/kubelet/ sudo rm -rf /var/lib/dockershim sudo rm -rf /var/run/kubernetes sudo rm -rf /var/lib/cni sudo rm -rf /var/lib/etcd sudo rm -rf /etc/cni/net.d - 刪除掉k8s對本機網(wǎng)卡iptables轉(zhuǎn)發(fā)的配置
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X ipvsadm -C ipvsadm --clear - 執(zhí)行 ip addr 會發(fā)現(xiàn)還有一些虛擬veth奸绷、cni梗夸、flannel等設(shè)備
ip link delete xxx
安裝dcoker
- 已安裝建議刪除，重新安裝
# 殺死所有運行容器 docker kill $(docker ps -a -q) # 刪除所有容器 docker rm $(docker ps -a -q) # 刪除所有鏡像 docker rmi $(docker images -q) # 停止 docker 服務(wù) systemctl stop docker # 刪除存儲目錄 rm -rf /etc/docker rm -rf /run/docker rm -rf /var/lib/dockershim rm -rf /var/lib/docker # 卸載 docker yum remove docker docker-engine docker-common docker-selinux - yum安裝docker
# 安裝部分依賴 yum install -y yum-utils device-mapper-persistent-data lvm2 # 添加docker yum源 sudo yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo # 安裝docker yum install -y docker-ce docker-ce-selinux # 啟動docker并設(shè)置開機自啟動 systemctl enable docker systemctl start docker
配置cri-docker使kubernetes以docker作為運行時
- 下載最新版 cri-docker
- 解壓出cri-docker
tar -zxf cri-dockerd-0.2.5.amd64.tgz cp cri-dockerd/cri-dockerd /usr/bin/ - 創(chuàng)建cri-docker啟動文件
cat > /usr/lib/systemd/system/cri-docker.service << EOF [Unit] Description=CRI Interface for Docker Application Container Engine Documentation=https://docs.mirantis.com After=network-online.target firewalld.service docker.service Wants=network-online.target Requires=cri-docker.socket [Service] Type=notify ExecStart=/usr/bin/cri-dockerd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.8 ExecReload=/bin/kill -s HUP $MAINPID TimeoutSec=0 RestartSec=2 Restart=always StartLimitBurst=3 StartLimitInterval=60s LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity TasksMax=infinity Delegate=yes KillMode=process [Install] WantedBy=multi-user.target EOFcat > /usr/lib/systemd/system/cri-docker.socket << EOF [Unit] Description=CRI Docker Socket for the API PartOf=cri-docker.service [Socket] ListenStream=%t/cri-dockerd.sock SocketMode=0660 SocketUser=root SocketGroup=docker [Install] WantedBy=sockets.target EOF - 啟動cri-docker并設(shè)置開機自動啟動
systemctl daemon-reload systemctl enable cri-docker --now systemctl status cri-docker
配置基礎(chǔ)環(huán)境
- 準(zhǔn)備工作
# 禁用防火墻和iptables systemctl stop firewalld systemctl disable firewalld systemctl stop iptables systemctl disable iptables # 禁用selinux sudo setenforce 0 sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config # 禁用swap分區(qū) swapoff -a sed -i '/swap/s/^/#/' /etc/fstab #允許 iptables 檢查橋接流量 cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf br_netfilter EOF cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 EOF sudo sysctl --system
建立虛擬機網(wǎng)卡
- 創(chuàng)建一個虛擬網(wǎng)卡
cat > /etc/sysconfig/network-scripts/ifcfg-eth0:1 <<EOF BOOTPROTO=static DEVICE=eth0:1 IPADDR=機器公網(wǎng)IP PREFIX=32 TYPE=Ethernet USERCTL=no ONBOOT=yes EOF - 重啟網(wǎng)卡及檢查
systemctl restart network ip addr
安裝kubernetes
- 添加阿里云k8s鏡像源（國內(nèi)網(wǎng)絡(luò)）
cat > /etc/yum.repos.d/kubernetes.repo << EOF [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF - 執(zhí)行yum命令
sudo yum install -y kubelet kubeadm kubectl--disableexcludes=kubernetes - 設(shè)置kubelet開機自啟
systemctl enable kubelet.service
修改kubelet啟動參數(shù)文件
添加 kubelet的啟動參數(shù)--node-ip=公網(wǎng)IP号醉， 每個主機都要添加并指定對應(yīng)的公網(wǎng)ip
- 修改
/usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf, 此文件安裝kubeadm后就存在了vim /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf - 在末尾添加參數(shù) --node-ip=公網(wǎng)IP
..... ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS --node-ip=機器公網(wǎng)IP
開啟云服務(wù)器端口
- 先默認(rèn)開啟所有的出入站規(guī)則
- 需要配置最小化Kubernetes出入站規(guī)則反症， 參考Kubernetes服務(wù)間需要用到的端口范圍
初始化集群
- kubeadm init
sudo kubeadm init \ --kubernetes-version v1.25.0 \ --control-plane-endpoint=114.132.94.160 \ --apiserver-advertise-address=114.132.94.160 \ --image-repository registry.aliyuncs.com/google_containers \ --service-cidr=10.96.0.0/12 \ --pod-network-cidr=10.244.0.0/16 \ --v=5 \ --cri-socket /run/cri-dockerd.sock - 初始化kubectl
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config - 修改kube-apiserver參數(shù)
在 master 節(jié)點，kube-apiserver 添加--bind-address=0.0.0.0和修改--advertise-addres=<公網(wǎng)IP>
sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml - 檢查Nodes 及 Pods
kubectl get nodes -o wide kubectl get pods -o wide --all-namespaces
工作節(jié)點加入集群
- 使用
kubeadm join命令加入集群kubeadm join 114.132.94.160:6443 --token 0bd2ih.7afjzcq0lpcy17lt \ --discovery-token-ca-cert-hash sha256:fc83b436652b4c1501862ae971bab0fa1762de541e9115b6ecfcf1032033703b \ --cri-socket /var/run/cri-dockerd.sock - 如果token過期畔派，可生成新token：
kubeadm token create --ttl 0 --print-join-command
安裝flannel網(wǎng)絡(luò)插件
- 下載flannel的yml文件
curl -OL https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml - 修改kube-flannel.yml文件
...略 apiVersion: apps/v1 kind: DaemonSet metadata: name: kube-flannel-ds-amd64 namespace: kube-system labels: tier: node app: flannel spec: selector: matchLabels: app: flannel template: metadata: labels: tier: node app: flannel spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: beta.kubernetes.io/os operator: In values: - linux - key: beta.kubernetes.io/arch operator: In values: - amd64 hostNetwork: true tolerations: - operator: Exists effect: NoSchedule serviceAccountName: flannel initContainers: - name: install-cni image: quay.io/coreos/flannel:v0.11.0-amd64 command: - cp args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist volumeMounts: - name: cni mountPath: /etc/cni/net.d - name: flannel-cfg mountPath: /etc/kube-flannel/ containers: - name: kube-flannel image: quay.io/coreos/flannel:v0.11.0-amd64 command: - /opt/bin/flanneld args: - --ip-masq - --public-ip=$(PUBLIC_IP) # 添加此參數(shù)铅碍，申明公網(wǎng)IP - --iface=eth0 # 添加此參數(shù)，綁定網(wǎng)卡 - --kube-subnet-mgr resources: requests: cpu: "100m" memory: "50Mi" limits: cpu: "100m" memory: "50Mi" securityContext: privileged: false capabilities: add: ["NET_ADMIN"] env: - name: PUBLIC_IP # 添加環(huán)境變量 valueFrom: # * fieldRef: # * fieldPath: status.podIP # * - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name ... 略 - 創(chuàng)建flannel
kubectl create -f kube-flannel.yml - 檢查flannel Pod
kubectl logs -n kube-flannel pods/kube-flannel-ds-dj2cb kubectl logs -n kube-flannel pods/kube-flannel-ds-gxlx6 kubectl logs -n kube-flannel pods/kube-flannel-ds-rvxc4
部署nginx驗證安裝
- 新建
nginx-deployment.yamlapiVersion: v1 kind: Service metadata: name: nginx-service labels: app: nginx spec: ports: - protocol: TCP port: 80 targetPort: 80 selector: app: nginx --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80 -
kubectl createkubectl create -f nginx-deployment.yaml - 驗證
kubectl get svc -o wide --all-namespaces curl nginx-service對應(yīng)的CLUSTER-IP
