記錄下ubuntu 22.04安裝最新Kubernets 1.29 蓄愁，全部按照官方教程去執(zhí)行

安裝容器

因?yàn)镵ubernetes 1.20版本中双炕，Docker支持已經(jīng)被棄用，并在1.24版本后續(xù)的版本不再包含docker依賴了撮抓。后續(xù)使用containerd作為容器實(shí)現(xiàn)妇斤，它是OCI的規(guī)范，理論上所以符合OCI規(guī)范的容器都可以做Kubernetes的容器實(shí)現(xiàn)。雖然Docker已經(jīng)被棄用而改用contrainerd了站超，但是containerd也是Docker公司開發(fā)的荸恕，在docker官網(wǎng)找到安裝方法，我這里使用官網(wǎng)推薦安裝死相。
這種方式安裝的containerd是最新的版本融求，必須要對(duì)應(yīng)是最新Kubernetes，如果要安裝指定版本Kuberenets算撮，最好去github下載指定版本.

sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
apt install containerd.io cri-tools -y

安裝Kubernetes組件

安裝Kubernetes本質(zhì)就是安裝下面3個(gè)工具

• kubeadm: 安裝Kubernetes集群執(zhí)行工具生宛，安裝集群需要各類組件、初始化集群肮柜。
• kubelet: 可以理解成集群節(jié)點(diǎn)運(yùn)行agent陷舅，通過API方式向服務(wù)器注冊(cè)，處理Master下發(fā)到本節(jié)點(diǎn)的任務(wù)审洞，管理Pod及Pod中的容器蔑赘。
• kubectl: Kubernetes命令行客戶端，可以命令方式向集群調(diào)用预明，查看資源。

由于官網(wǎng)Kubenretes官網(wǎng)源下載很慢耙箍，我這里使用的國內(nèi)阿里云源

apt-get update && apt-get install -y apt-transport-https
curl -fsSL https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.29/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://mirrors.aliyun.com/kubernetes-new/core/stable/v1.29/deb/ /" |
tee /etc/apt/sources.list.d/kubernetes.list
apt-get update
apt-get install -y kubelet kubeadm kubectl

驗(yàn)證安裝結(jié)果

kubeadm version

kubeadm version: &version.Info{Major:"1", Minor:"28", GitVersion:"v1.28.7", GitCommit:"c8dcb00be9961ec36d141d2e4103f85f92bcf291", GitTreeState:"clean", BuildDate:"2024-02-14T10:39:01Z", GoVersion:"go1.21.7", Compiler:"gc", Platform:"linux/amd64"}

設(shè)置Linux內(nèi)核

禁用swap 交換分區(qū)

sudo swapoff -a

寫入配置中撰糠，保證重啟后依然生效

sudo sed -i '/ swap / s/^(.*)$/#\1/g' /etc/fstab

設(shè)置容器模板

vim /etc/modules-load.d/containerd.conf

寫入以下內(nèi)容

overlay
br_netfilter

開啟模塊

sudo modprobe overlay
sudo modprobe br_netfilter

配置kubernetes網(wǎng)絡(luò)

vim /etc/sysctl.d/kubernetes.conf

寫入下面內(nèi)容

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1

重啟，讓配置生效

sudo sysctl --system

安裝Kubernetes集群

獲取kubeadm 默認(rèn)配置

kubeadm config print init-defaults > kubeadm.conf

修改4處地方

• localAPIEndpoint.advertiseAddress為master的ip辩昆；
• nodeRegistration.name 為當(dāng)前節(jié)點(diǎn)名稱阅酪；
• imageRepository為國內(nèi)源：registry.cn-hangzhou.aliyuncs.com/google_containers
• podSubnet 設(shè)置pod網(wǎng)段，這里必須 192.168.0.0/16汁针，因?yàn)橄旅姘惭bCNI 插件Calico已經(jīng)固定網(wǎng)段,如果要安裝其他插件自行修改
• kubernetesVersion： 1.29.2 术辐，查看了kubeadm 版本為1.29.2 ，默認(rèn)是1.29.0

修改配置如下

apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 172.28.14.94 #當(dāng)前主機(jī)IP
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
  name: ser
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers #改成國內(nèi)鏡像地址
kind: ClusterConfiguration
kubernetesVersion: v1.29.2 # 改成跟kubelet 一致
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
  podSubnet: 192.168.0.0/16 #固定網(wǎng)段施无，這是cni插件Calico要求的
scheduler: {}

我這里先執(zhí)行鏡像下載辉词，一步步來

kubeadm config images pull --config=kubeadm.conf

出現(xiàn)錯(cuò)誤

root@syf-CN15S:~# kubeadm config images pull --config=kubeadm.conf
W0323 23:12:47.468488 54593 initconfiguration.go:312] error unmarshaling configuration schema.GroupVersionKind{Group:"kubeadm.k8s.io", Version:"v1beta3", Kind:"ClusterConfiguration"}: strict decoding error: unknown field "networking.pod-network-cidr"
failed to pull image "registry.lank8s.cn/kube-apiserver:v1.29.0": output: time="2024-03-23T23:12:47+08:00" level=fatal msg="validate service connection: validate CRI v1 image API for endpoint "unix:///var/run/containerd/containerd.sock": rpc error: code = Unimplemented desc = unknown service runtime.v1.ImageService"
, error: exit status 1
To see the stack trace of this error execute with --v=5 or higher

這是cri沒有使用containerd 作為容器，在github 找到解決方法

vim /etc/crictl.yaml

新增下面內(nèi)容

runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 2
debug: true
pull-image-on-create: false

修改containerd默認(rèn)配置

因?yàn)閏ontainerd沒有配置猾骡，使用下面命令生成默認(rèn)配置

containerd config default > /etc/containerd/config.toml

修改鏡像地址瑞躺，改用國內(nèi)倉庫

[plugins."io.containerd.grpc.v1.cri"]
    device_ownership_from_security_context = false
    disable_apparmor = false
    disable_cgroup = false
    disable_hugetlb_controller = true
    disable_proc_mount = false
    disable_tcp_service = true
    enable_selinux = false
    enable_tls_streaming = false
    enable_unprivileged_icmp = false
    enable_unprivileged_ports = false
    ignore_image_defined_volumes = false
    max_concurrent_downloads = 3
    max_container_log_line_size = 16384
    netns_mounts_under_state_dir = false
    restrict_oom_score_adj = false
    sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9" # 修改這一行就可以

同樣修改Cgroup設(shè)置，會(huì)導(dǎo)致etcd無限重啟

          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            BinaryName = ""
            CriuImagePath = ""
            CriuPath = ""
            CriuWorkPath = ""
            IoGid = 0
            IoUid = 0
            NoNewKeyring = false
            NoPivotRoot = false
            Root = ""
            ShimCgroup = ""
            SystemdCgroup = false //改成  true 就可以

配置containerd 容器倉庫加速兴想，不然使用默認(rèn)docker.io 非常慢的
新創(chuàng)建 /etc/containerd/certs.d/docker.io/ 目錄, 向下面結(jié)構(gòu) docker.io 表示默認(rèn)Docker 官方倉庫地址

/etc/containerd/
├── certs.d
│   ├── docker.io
│   │   └── host.toml
└── config.toml

編輯host.toml

vim /etc/containerd/certs.d/docker.io/host.toml

添加下面內(nèi)容

 [host."https://xxxxx.mirror.aliyuncs.com"] # docker 倉庫加速地址
  capabilities = ["pull", "resolve","push"]
  skip_verify = true

其他配置可以參看Github
重啟幢哨，讓配置生效

systemctl restart containred

重新執(zhí)行命令，拉取鏡像嫂便，出現(xiàn)下面記錄捞镰，表示拉取鏡像成功

root@syf-CN15S:~# kubeadm config images pull --config=kubeadm.conf
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.29.2
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.29.2
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.29.2
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.29.2
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.11.1
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9
[config/images] Pulled registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.10-0

執(zhí)行初始化命令

kubeadm init --config=kubeadm.con

看見下面信息，說明集群搭建成功

.... 省略

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

根據(jù)安裝提示，執(zhí)行下面命令

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

執(zhí)行下下面命令岸售，驗(yàn)證下kubernets是否安裝成功

kubectl get node

如果你和我一樣践樱，狀態(tài)不是Ready,那就是有問題

NAME STATUS ROLES AGE VERSION
syf-cn15s NotReady control-plane 5m14s v1.29.3

首先成kubelete 排查kubelet，查看是否正常運(yùn)行

systemctl status kubelet

看見下面

3月 24 00:09:59 syf-CN15S kubelet[59351]: E0324 00:09:59.243528   59351 kubelet.go:2892] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Networ>
3月 24 00:10:04 syf-CN15S kubelet[59351]: E0324 00:10:04.244612   59351 kubelet.go:2892] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Networ>
3月 24 00:10:09 syf-CN15S kubelet[59351]: E0324 00:10:09.246425   59351 kubelet.go:2892] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Networ>
3月 24 00:10:14 syf-CN15S kubelet[59351]: E0324 00:10:14.248467   59351 kubelet.go:2892] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Networ>
3月 24 00:10:19 syf-CN15S kubelet[59351]: E0324 00:10:19.250277   59351 kubelet.go:2892] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Networ>
3月 24 00:10:24 syf-CN15S kubelet[59351]: E0324 00:10:24.251682   59351 kubelet.go:2892] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Networ>
3月 24 00:10:29 syf-CN15S kubelet[59351]: E0324 00:10:29.253199   59351 kubelet.go:2892] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Networ>
3月 24 00:10:34 syf-CN15S kubelet[59351]: E0324 00:10:34.255030   59351 kubelet.go:2892] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Networ>
3月 24 00:10:39 syf-CN15S kubelet[59351]: E0324 00:10:39.256634   59351 kubelet.go:2892] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Networ>
3月 24 00:10:44 syf-CN15S kubelet[59351]: E0324 00:10:44.258525   59351 kubelet.go:2892] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Networ

這是沒有安裝CNI 插件導(dǎo)致

安裝CNI插件

我這里安裝calico 作為Kubernetes網(wǎng)絡(luò)CNI插件冰评，安裝步驟來自官網(wǎng)教程
先現(xiàn)在yaml文件映胁，這個(gè)步驟很關(guān)鍵，接下來如果安裝失敗了甲雅，可以使用kubectl delete 直接刪除

wget https://raw.githubusercontent.com/projectcalico/calico/v3.27.2/manifests/tigera-operator.yaml
wget https://raw.githubusercontent.com/projectcalico/calico/v3.27.2/manifests/custom-resources.yaml
kubectl create -f tigera-operator.yaml
kubectl create -f custom-resources.yaml

等待一會(huì)calico 安裝完成解孙，使用下面命令查看鏡像運(yùn)行情況

watch kubectl get pods -n calico-system

看到下面信息，則說明安裝完成了

NAME READY STATUS RESTARTS AGE
calico-kube-controllers-df447b98-44n5j 1/1 Running 0 10m
calico-node-8jrzf 1/1 Running 0 10m
calico-typha-78f4459ccc-xgh4h 1/1 Running 0 10m
csi-node-driver-ggjql 2/2 Running 0 10m

再次查看k8s運(yùn)行情況

root@syf-CN15S:~# kubectl get node
NAME STATUS ROLES AGE VERSION
syf-cn15s Ready control-plane 37m v1.29.3

正常運(yùn)行了

如果是單節(jié)點(diǎn)運(yùn)行抛人，還有執(zhí)行以下命令弛姜，才能讓master 充當(dāng)工作節(jié)點(diǎn)

kubectl taint nodes --all node-role.kubernetes.io/control-plane-

問題排查思路

我遇到問題先看kubelet 狀態(tài)，日志

sudo systemctl status kubelet

查看詳細(xì)日志

sudo journalctl -u kubelet.service

定位到具體容器后妖枚，集群沒有安裝好廷臼，kubectl還能使用的，可以使用containerd的客戶端工具crictl 查看绝页，用法和docker基本一樣
查看日志

sudo crictl logs <containerd-id>

進(jìn)入容器

sudo crictl exec -it <containerd-id> /bin/bash

以下文章參考的資料
https://phoenixnap.com/kb/install-kubernetes-on-ubuntu
https://docs.docker.com/engine/install/ubuntu/
https://developer.aliyun.com/mirror/kubernetes?spm=a2c6h.12873639.article-detail.8.73cd4579RZH8Z4
https://github.com/containerd/containerd/blob/main/docs/hosts.md