1.1 服務(wù)版本信息

pod錯(cuò)誤排查可以看下https://zhuanlan.zhihu.com/p/34332367這個(gè)博主的內(nèi)容

1.2 為什么棄用docker倾芝？

Docker 是 Kubernetes 使用的第一個(gè)容器運(yùn)行時(shí)四苇。 這也是許多 Kubernetes 用戶和愛(ài)好者如此熟悉Docker的原因之一鹃愤。 對(duì)Docker 的支持被硬編碼到 Kubernetes 中——一個(gè)被項(xiàng)目稱為 dockershim的組件。

隨著容器化成為行業(yè)標(biāo)準(zhǔn)撑毛，Kubernetes 項(xiàng)目增加了對(duì)其他運(yùn)行時(shí)的支持。 最終實(shí)現(xiàn)了容器運(yùn)行時(shí)接口（CRI）佩迟，讓系統(tǒng)組件（如 kubelet）以標(biāo)準(zhǔn)化的方式與容器運(yùn)行時(shí)通信郑口。 因此，dockershim 成為了 Kubernetes 項(xiàng)目中的一個(gè)異掣鞯恚現(xiàn)象懒鉴。

對(duì) Docker 和 dockershim 的依賴已經(jīng)滲透到 CNCF 生態(tài)系統(tǒng)中的各種工具和項(xiàng)目中，這導(dǎo)致了代碼脆弱碎浇。

通過(guò)刪除 dockershim CRI临谱，我們擁抱了 CNCF 的第一個(gè)價(jià)值： “快比慢好”。

2.添加k8s軟件源-阿里云(all節(jié)點(diǎn))

2.1 ubuntu

curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-get add -
cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF

命令說(shuō)明：
1.通過(guò)下載工具下載位于https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg的deb軟件包密鑰奴璃，然后通過(guò)"apt-key"命令添加密鑰
2.通過(guò)cat把源deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main寫(xiě)入到"/etc/apt/sources.list.d/kubernetes.list"

2.2 centos

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
        http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

3. 安裝 kubeadm吴裤、kubelet、kubectl(all節(jié)點(diǎn))

查看可選擇版本

$ apt-get update
$ apt-cache madison kubeadm kubelet kubectl
#centos
$ yum list kubelet --showduplicates

安裝指定版本

#ubuntu
$ apt-get install -y kubelet=1.24.2-00  kubeadm=1.24.2-00 kubectl=1.24.2-00
#centos
$ yum install -y kubelet-1.24.2 kubeadm-1.24.2  kubectl-1.24.2 

$ kubelet --version
Kubernetes v1.24.2

4. 啟動(dòng)kubelet服務(wù)(all節(jié)點(diǎn))

# 設(shè)置開(kāi)機(jī)啟動(dòng)
$ systemctl enable kubelet.service --now
# 查看是否啟動(dòng)
$ systemctl status kubelet.service

如果啟動(dòng)失敗溺健，可以執(zhí)行命令journalctl -u kubelet -n 100看下是否有錯(cuò)誤，也可以看下最近幾條的日志tail -n 10 /var/log/messages|grep kube

啟動(dòng)錯(cuò)誤：Error: failed to load kubelet config file, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubele>
解決：配置文件路徑不存在钮蛛，后面kubeadm init會(huì)自動(dòng)生成

5. 初始化集群（master節(jié)點(diǎn)）

5.1 配置文件

a. 下載默認(rèn)配置文件

# 創(chuàng)建目錄 /usr/local/kubernetes
$ mkdir -p  /usr/local/kubernetes
# 下載默認(rèn)配置
$ kubeadm config print init-defaults --component-configs KubeletConfiguration > /usr/local/kubernetes/kubeadm.yaml

b. 修改配置文件

# 修改1: advertiseAddress
localAPIEndpoint:
  advertiseAddress: 192.168.77.133  # 指定master節(jié)點(diǎn)內(nèi)網(wǎng)IP
...
# 修改2：修改master節(jié)點(diǎn)名稱
nodeRegistration:
  ...
  name: master # 修改master節(jié)點(diǎn)名稱


# 修改3：此處新增一個(gè) kubeproxy.config節(jié)點(diǎn)安聘，并把kube-proxy模式為ipvs辛润，默認(rèn)為iptables
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
...
# 修改4: 設(shè)置imageRepository為阿里云的registry，避免因gcr被墻，無(wú)法直接拉取鏡像
imageRepository: registry.aliyuncs.com/google_containers 
...

# 修改5: 指定k8s版本號(hào)蕉世，默認(rèn)這里忽略了小版本號(hào)
kubernetesVersion: 1.24.2 
...

# 修改6: 確認(rèn)設(shè)置kubelet的cgroupDriver為systemd
---
apiVersion: kubelet.config.k8s.io/v1beta1
...
cgroupDriver: systemd

# 修改7: 指定 pod 子網(wǎng)
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
  podSubnet: 10.244.0.0/16  # 指定 pod 子網(wǎng)

5.2 拉取容器鏡像

# 根據(jù)配置文件拉取鏡像
$ kubeadm config images pull --config /usr/local/kubernetes/kubeadm.yaml
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.24.2
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.24.2
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.24.2
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.24.2
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.7
[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.5.3-0
[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.8.6

5.3 啟動(dòng)集群

# 根據(jù)配置文件啟動(dòng)鏡像
$ kubeadm init --config /usr/local/kubernetes/kubeadm.yaml
[init] Using Kubernetes version: v1.24.2
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master] and IPs [10.96.0.1 192.168.77.128]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost master] and IPs [192.168.77.128 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost master] and IPs [192.168.77.128 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 12.012612 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.77.128:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:ab661eca8399a920e0811ff88ba4238919a588c9bd5b6d4998c35d69c5efbb12 

問(wèn)題1：如果一直報(bào)：Error getting node" err="node \"master\" not found，可能是因?yàn)橹皥?zhí)行了init失敗導(dǎo)致了某些信息被污染了耕皮，解決：嘗試更新了證書(shū)撑碴，kubeadm certs renew all仑濒，然后查看systemctl status kubelet服務(wù)狀態(tài)正常，重新執(zhí)行init偷遗，如果報(bào)部分文件存在墩瞳，先執(zhí)行kubeadm reset再執(zhí)行init

問(wèn)題2：報(bào)錯(cuò)[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist，
解決：執(zhí)行modprobe br_netfilter

問(wèn)題3：報(bào)錯(cuò)：Failed to create sandbox for pod,此種情況如果在kubelet日志里面發(fā)現(xiàn)不了其他問(wèn)題氏豌，可以看下containerd日志喉酌。journalctl -u containerd -f|grep error監(jiān)聽(tīng)日志發(fā)現(xiàn)runc did not terminate successfully: exit status 127: runc: symbol lookup error: runc: undefined symbol: seccomp_notify_respond\\n\" runtime=io.containerd.runc.v2\ntime=\"2024-09-12T08:23:47+08:00\" level=warning msg=\"failed to read init pid file\" error=\"open /run/containerd/io.containerd.runtime.v2.task/k8s.io/9aeffbdf60c32558fb60e1f56224f5db2e6464e970ae11da3cccd06d6c494cc7/init.pid: no such file or directory，此種情況大概率是libseccomp版本引起的泵喘，更新一下yum update libseccomp泪电，重置后重新初始化

5.4 查看狀態(tài)

$ kubectl get node
NAME     STATUS   ROLES           AGE   VERSION
master   Ready    control-plane   11m   v1.24.2

問(wèn)題：執(zhí)行命令報(bào)錯(cuò)：The connection to the server localhost:8080 was refused - did you specify the right host or port?
原因是kubectl命令需要使用kubernetes-admin來(lái)運(yùn)行;
解決方法如下:將主節(jié)點(diǎn)中的/etc/kubernetes/admin.conf文件拷貝到從節(jié)點(diǎn)相同目錄下，然后配置環(huán)境變量

$ echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bashrc 
$ source ~/.bashrc

6. 添加節(jié)點(diǎn)

6.1 加入集群(在節(jié)點(diǎn)執(zhí)行)

# 在 `node1`和`node2`執(zhí)行以下命令纪铺，加入集群
$ kubeadm join 192.168.77.133:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:b3edc1a3f9d8aba888a2b33a50b6b1e0293d21d5fb1bc76e586c3fc600f2b74112 
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

此命令是在啟動(dòng)master的時(shí)候給出的相速。具體請(qǐng)看5.3 啟動(dòng)集群，如果找不到了或者卡住了（過(guò)期了）可以執(zhí)行kubeadm token create --print-join-command

如果報(bào)錯(cuò)[WARNING FileExisting-tc]: tc not found in system path
執(zhí)行yum install iproute-tc -y安裝一下

查看是否成功

root@k8s-master# kubectl get nodes
NAME     STATUS   ROLES           AGE     VERSION
master   Ready    control-plane   2d1h    v1.24.2
node1    Ready    <none>          2m54s   v1.24.2
node2    Ready    <none>          83s     v1.24.2

NotReady是因?yàn)闆](méi)有安裝網(wǎng)絡(luò)組件鲜锚。

7.包管理器helm

7.1 什么是helm

每個(gè)成功的軟件平臺(tái)都有一個(gè)優(yōu)秀的打包系統(tǒng)突诬，比如Debian、Ubuntu的apt烹棉，Red Hat攒霹、CentOS的yum。Helm則是Kubernetes上的包管理器浆洗。

Helm到底解決了什么問(wèn)題催束？為什么Kubernetes需要Helm？

答案是：Kubernetes能夠很好地組織和編排容器伏社，但它缺少一個(gè)更高層次的應(yīng)用打包工具抠刺，而Helm就是來(lái)干這件事的。

7.2 安裝

# 下載包
[root@master tmp]$ wget https://get.helm.sh/helm-v3.9.0-linux-amd64.tar.gz
# 解壓
[root@master tmp]$ tar -zxvf helm-v3.9.0-linux-amd64.tar.gz
# 移動(dòng)二進(jìn)制未加
[root@master tmp]$ mv linux-amd64/helm  /usr/local/bin/
# 驗(yàn)證
[root@master tmp]# helm version
version.BuildInfo{Version:"v3.9.0", GitCommit:"7ceeda6c585217a19a1131663d8cd1f7d641b2a7", GitTreeState:"clean", GoVersion:"go1.17.5"}

8. 使用helm部署Calico

選擇calico作為k8s的Pod網(wǎng)絡(luò)組件摘昌，下面使用helm在k8s集群中安裝calico速妖。

8.1 下載helm chart

[root@master helm] $ wget https://github.com/projectcalico/calico/releases/download/v3.23.1/tigera-operator-v3.23.1.tgz

8.2 安裝

# 指定安裝在命名空間為kube-system下
[root@master helm] $ helm install calico tigera-operator-v3.23.1.tgz -n kube-system
# 等待runing
$ kubectl get pod -n kube-system -w | grep tigera-operator
tigera-operator-5fb55776df-hnkbw   1/1     Running   1 (18h ago)     19h
# 所有都變成runing
[root@master helm]$ kubectl get pods -n calico-system -w
NAME                                       READY   STATUS    RESTARTS        AGE
calico-kube-controllers-68884f975d-b6vqq   1/1     Running   5 (4m37s ago)   18h
calico-node-4gknz                          1/1     Running   3 (4m54s ago)   19h
calico-node-8fx7n                          1/1     Running   1 (42m ago)     19h
calico-node-qzbh6                          1/1     Running   1 (18h ago)     19h
calico-typha-5d75c97db9-pwgnm              1/1     Running   3 (4m54s ago)   19h
calico-typha-5d75c97db9-vfpfq              1/1     Running   1 (18h ago)     19h

如果一直提示No resources found in calico-system namespace.，可能是節(jié)點(diǎn)有問(wèn)題聪黎，此時(shí)執(zhí)行下kubectl get pod -n calico-system -o wide可以看到一直在創(chuàng)建中

這個(gè)時(shí)候可以在主節(jié)點(diǎn)執(zhí)行kubectl describe pod <podname> -n kube-system（推薦）或者進(jìn)入到node1和node2查看具體是什么問(wèn)題罕容，執(zhí)行journalctl -u kubelet -f

問(wèn)題：Back-off pulling image "docker.io/calico/cni:v3.23.1"一直報(bào)某些鏡像拉去不下來(lái)
解決：可以手動(dòng)拉取，ctr images pull docker.io/calico/cni:v3.23.1

8.3 查看當(dāng)時(shí)節(jié)點(diǎn)狀態(tài)(全部變成Ready)

[root@master helm]$ kubectl get nodes
NAME     STATUS   ROLES           AGE   VERSION
master   Ready    control-plane   20m   v1.24.2
node1    Ready    <none>          16m   v1.24.2
node2    Ready    <none>          15m   v1.24.2

9.驗(yàn)證DNS是否可用

# 運(yùn)行并進(jìn)入驗(yàn)證容器
$ kubectl run curl --image=radial/busyboxplus:curl -it

# 驗(yàn)證
[ root@curl:/ ]$ nslookup kubernetes.default
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local

如果報(bào)錯(cuò)：nslookup: can't resolve 'kubernetes.default',
請(qǐng)參考：https://kubernetes.io/zh-cn/docs/tasks/administer-cluster/dns-debugging-resolution/稿饰，我是停用了systemd-resolved锦秒，重啟就好了

10.使用helm部署ingress-nginx

為了便于將集群中的服務(wù)暴露到集群外部，需要使用Ingress喉镰。接下來(lái)使用Helm將ingress-nginx部署到Kubernetes上旅择。Nginx Ingress Controller被部署在Kubernetes的邊緣節(jié)點(diǎn)上。

使用kubernetes-dashboard作為面板侣姆，默認(rèn)安裝完成的dashboard的訪問(wèn)方式是ClusterIP,進(jìn)而想訪問(wèn)dashboard需更改為nodeport或者loadbalancer或者配置為ingress的方式才能訪問(wèn)dashbaord生真。本文將以ingress-nginx發(fā)布dashboard在外部訪問(wèn)

10.1 設(shè)置邊緣節(jié)點(diǎn)

這里將node1(192.168.77.129)作為邊緣節(jié)點(diǎn)沉噩，打上Label：

[root@master kubernetes]$ kubectl label node node1 node-role.kubernetes.io/edge=
node/node1 labeled
[root@master kubernetes]$ kubectl get nodes
NAME     STATUS   ROLES           AGE     VERSION
master   Ready    control-plane   2d21h   v1.24.2
node1    Ready    <none>          20h     v1.24.2
node2    Ready    edge            20h     v1.24.2

10.2 下載ingress-nginx的helm chart

$ wget https://github.com/kubernetes/ingress-nginx/releases/download/helm-chart-4.1.2/ingress-nginx-4.1.2.tgz

10.3 編寫(xiě)install-ingress.yaml

vim install-ingress.yaml

controller:
  ingressClassResource:
    name: nginx
    enabled: true
    default: true
    controllerValue: "k8s.io/ingress-nginx"
  admissionWebhooks:
    enabled: false
  replicaCount: 1
  image:
    registry: docker.io
    image: unreachableg/k8s.gcr.io_ingress-nginx_controller
    tag: "v1.2.0"
    digest: sha256:314435f9465a7b2973e3aa4f2edad7465cc7bcdc8304be5d146d70e4da136e51
  hostNetwork: true
  nodeSelector:
    node-role.kubernetes.io/edge: ''
  affinity:
    podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - nginx-ingress
            - key: component
              operator: In
              values:
              - controller
          topologyKey: kubernetes.io/hostname
  tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: PreferNoSchedule

nginx ingress controller的副本數(shù)replicaCount為1，將被調(diào)度到node1這個(gè)邊緣節(jié)點(diǎn)上柱蟀。這里并沒(méi)有指定nginx ingress controller service的externalIPs川蒙，而是通過(guò)hostNetwork: true設(shè)置nginx ingress controller使用宿主機(jī)網(wǎng)絡(luò)。

10.4 安裝

[root@master kubernetes]$ helm install ingress-nginx ingress-nginx-4.1.2.tgz --create-namespace -n ingress-nginx -f install-ingress.yaml
NAME: ingress-nginx
LAST DEPLOYED: Thu May  9 10:49:08 2024
NAMESPACE: ingress-nginx
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status by running 'kubectl --namespace ingress-nginx get services -o wide -w ingress-nginx-controller'
...
If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:

  apiVersion: v1
  kind: Secret
  metadata:
    name: example-tls
    namespace: foo
  data:
    tls.crt: <base64 encoded cert>
    tls.key: <base64 encoded key>
  type: kubernetes.io/tls

測(cè)試訪問(wèn)http://192.168.77.133返回默認(rèn)的nginx 404頁(yè)产弹，則部署完成派歌。

11. 使用helm部署DashBoard

為了提供更豐富的用戶體驗(yàn)，Kubernetes還開(kāi)發(fā)了一個(gè)基于Web的Dashboard痰哨，用戶可以用Kubernetes Dashboard部署容器化的應(yīng)用胶果、監(jiān)控應(yīng)用的狀態(tài)、執(zhí)行故障排查任務(wù)以及管理Kubernetes的各種資源斤斧。

在Kubernetes Dashboard中可以查看集群中應(yīng)用的運(yùn)行狀態(tài)早抠，也能夠創(chuàng)建和修改各種Kubernetes資源，比如Deployment撬讽、Job蕊连、DaemonSet等。用戶可以Scale Up/Down Deployment游昼、執(zhí)行Rolling Update甘苍、重啟某個(gè)Pod或者通過(guò)向?qū)Р渴鹦碌膽?yīng)用。Dashboard能顯示集群中各種資源的狀態(tài)以及日志信息烘豌≡赝ィ可以說(shuō)，Kubernetes Dashboard提供了kubectl的絕大部分功能廊佩。

本文是通過(guò)ingress-nginx訪問(wèn)dashboard

11.1 部署metrics-server

Metrics-Server是集群核心監(jiān)控?cái)?shù)據(jù)的聚合器囚聚。通俗地說(shuō)，它存儲(chǔ)了集群中各節(jié)點(diǎn)的監(jiān)控?cái)?shù)據(jù)标锄，并且提供了API以供分析和使用顽铸。

a. 下載配置文件

# 下載配置文件
[root@master ~]$ wget https://github.com/kubernetes-sigs/metrics-server/releases/download/metrics-server-helm-chart-3.8.2/components.yaml

b. 修改配置

vim components.yaml

....
---
apiVersion: apps/v1
kind: Deployment
metadata:
  ...
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  template:
    metadata:
      labels:
        k8s-app: metrics-server
    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --metric-resolution=15s
        #修改1：添加容器啟動(dòng)參數(shù),方便跳過(guò)安全tls,生產(chǎn)不推薦使用
        - --kubelet-insecure-tls 
        #修改2：更換鏡像地址
        image: docker.io/unreachableg/k8s.gcr.io_metrics-server_metrics-server:v0.6.1
        imagePullPolicy: IfNotPresent
...

c. 部署

# 部署資源
[root@master kubernetes]$ kubectl apply -f components.yaml
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
# 查看狀態(tài):ContainerCreating等待變成Running
[root@master kubernetes]$ kubectl get pod -n kube-system | grep metrics
metrics-server-77cffb4988-25wsk    1/1     Running   0              2m4s

11.2 添加對(duì)應(yīng)的chart repo

# 添加chart
[root@master kubernetes]$ helm repo add kubernetes-dashboard https://kubernetes.github.io/dashboard/
"kubernetes-dashboard" has been added to your repositories

# 更新包
[root@master kubernetes]$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "kubernetes-dashboard" chart repository
Update Complete. ?Happy Helming!?

11.3 制作ssl證書(shū)

本文因?yàn)槭潜镜匕惭b，如果是生產(chǎn)環(huán)境請(qǐng)使用真實(shí)的證書(shū)

# 生成證書(shū)請(qǐng)求的key
openssl genrsa -out dashboard.key 2048
# 生成證書(shū)請(qǐng)求
openssl req -new -key dashboard.key -out dashboard.csr -subj /C=CN/ST=JiangSu/L=NanJing/O=Shanhy/OU=Shanhy/CN=*.yzj-k8s.com
# 生成自簽證書(shū)（證書(shū)文件 dashboard.crt 和私鑰 dashboad.key）
openssl x509 -req -in dashboard.csr -signkey dashboard.key -out dashboard.crt -days 3650
# 查看證書(shū)信息
openssl x509 -in dashboard.crt -text -noout

11.4 配置https證書(shū)為secret

# 創(chuàng)建secret到kube-system命名空間下
# 之后我們的dashboard也會(huì)創(chuàng)建在這個(gè)命名空間下料皇，需要依賴這個(gè)谓松，所以提前創(chuàng)建
kubectl create secret tls dashboard-tls --key dashboard.key --cert dashboard.crt -n kube-system
kubectl create secret tls dashboard-tls  -n kube-system --from-file=tls.crt=dashboard.crt --from-file=tls.key=dashboard.key

# 查看secret
kubectl get secret -n kube-system
NAME                              TYPE                DATA   AGE
dashboard-tls                     kubernetes.io/tls   2      30s

說(shuō)明：
如果你使用--key --cert方式則創(chuàng)建的secret中data的默認(rèn)2個(gè)文件名就是tls.key和tls.crt，你可以使用命令kubectl describe secret -n kubernetes-dashboard kubernetes-dashboard-certs查看践剂。
如果你使用第二條命令的--from-file的方式毒返，則你需要手工指定文件名稱tls.crt和tls.key（看示例的寫(xiě)法），如果你把上面的命令直接寫(xiě)成--from-file=dashboard.crt舷手，那么掛載后的文件就是dashboard.crt，這樣你需要把第二步第2點(diǎn)中參數(shù)的tls.crt修改為dashboard.crt劲绪。
你還可以直接使用--from-file=mycert/這樣直接指定一個(gè)目錄男窟，那么會(huì)把改目錄下的所有文件都掛載到容器的/certs中盆赤，文件名保持不變。

11.4 定制char配置

image:
  repository: kubernetesui/dashboard
  tag: v2.5.1
ingress:
  enabled: true
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
  hosts:
  - yzj-k8s.com # 上面定義的證書(shū)域名
  tls:
    - secretName: dashboard-tls # 上面添加的證書(shū)名稱
      hosts:
      - yzj-k8s.com # 上面定義的證書(shū)域名
metricsScraper:
  enabled: true

11.5 安裝部署

$ helm install kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard -n kube-system -f dashboard.yaml

這里遇到了問(wèn)題歉眷，Error: INSTALLATION FAILED: template: kubernetes-dashboard/templates/secrets/csrf.yaml:22:6: executing "kubernetes-dashboard/templates/secrets/csrf.yaml" at <include "kubernetes-dashboard.app.csrf.secret.value" .>: error calling include: template: kubernetes-dashboard/templates/_helpers.tpl:83:63: executing "kubernetes-dashboard.app.csrf.secret.value" at <$secret.data>: wrong type for value; expected map[string]interface {}; got interface {}
此問(wèn)題一直沒(méi)有找到解決辦法Ｎ！：辜瘛淑际！

如果上面安裝成功，忽略下面的11.6到11.8

11.6 單獨(dú)安裝kubernetes-dashboard

在github上選擇合適的版本扇住，本文檔是k8s是1.24版本春缕，所以安裝2.6.1

• 下載配置文件

wget -O dashboard.yaml https://raw.githubusercontent.com/kubernetes/dashboard/v2.6.1/aio/deploy/recommended.yaml
# 如果上面的下載不下來(lái)，選擇直接打開(kāi)網(wǎng)址艘蹋，復(fù)制
vim dashboard.yaml 

• 修改配置文件

  
  

其中auto-generate-certificates不能注釋锄贼，因?yàn)槲铱吹竭^(guò)有帖子說(shuō)要注釋掉（這個(gè)參數(shù)不僅僅是自動(dòng)證書(shū)的開(kāi)關(guān)，還是總的HTTPS的開(kāi)關(guān)女阀，當(dāng)我們手工配置了證書(shū)后宅荤，容器不會(huì)自動(dòng)生成）。
另外兩個(gè)tls參數(shù)指定的是被掛載到容器中的證書(shū)的名字浸策，下面我們使用 tls secret 處理的證書(shū)冯键，通過(guò)配置mountPath: /certs可以得知被掛載到容器的/certs目錄中，其名字為tls.crt和tls.key（為什么叫這2個(gè)名字或者是否可以配置其他名字庸汗，請(qǐng)繼續(xù)往下看）惫确。

11.7 安裝dashboard

$ kubectl apply -f dashboard.yaml
namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created

#查看安裝情況
$ kubectl get service,pod,secret -n kubernetes-dashboard
NAME                                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/dashboard-metrics-scraper   ClusterIP   10.109.30.117   <none>        8000/TCP   142m
service/kubernetes-dashboard        ClusterIP   10.103.156.58   <none>        443/TCP    142m

NAME                                            READY   STATUS    RESTARTS   AGE
pod/dashboard-metrics-scraper-8c47d4b5d-g9gz9   1/1     Running   0          139m
pod/kubernetes-dashboard-f47c7f849-hbvtr        1/1     Running   0          139m

NAME                                     TYPE                DATA   AGE
secret/dashboard-tls                     kubernetes.io/tls   2      134m
secret/kubernetes-dashboard-certs        Opaque              0      142m
secret/kubernetes-dashboard-csrf         Opaque              1      142m
secret/kubernetes-dashboard-key-holder   Opaque              2      142m

問(wèn)題：pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/kubernetesui/metrics-scraper:v1.0.8": failed to resolve reference "docker.io/kubernetesui/metrics-scraper:v1.0.8": failed to do request: Head "https://registry-1.docker.io/v2/kubernetesui/metrics-scraper/manifests/v1.0.8": dial tcp 202.160.128.203:443: connect: connection refused，
解決：鏡像拉不下來(lái)夫晌，解決辦法如下

#通過(guò)其他源拉取
$ ctr image pull docker.m.daocloud.io/kubernetesui/metrics-scraper:v1.0.8
$ ctr image list
docker.m.daocloud.io/kubernetesui/metrics-scraper:v1.0.8                                                                                  application/vnd.docker.distribution.manifest.list.v2+json sha256:76049887f07a0476dc93efc2d3569b9529bf982b22d29f356092ce206e98765c 18.8 MiB  linux/amd64,linux/arm,linux/arm64,linux/ppc64le,linux/s390x -
#導(dǎo)出鏡像
$ tr image export metrics.tar docker.io/kubernetesui/metrics-scraper:v1.0.8
#導(dǎo)入到`crictl`,導(dǎo)入命令雕薪， 所有鏡像都位于 http://k8s.io 命名空間下， 并且需要添加平臺(tái) --platform
$ ctr -n k8s.io images import  metrics.tar --platform linux/amd64
#查看鏡像列表
$ crictl image
docker.io/kubernetesui/metrics-scraper                       v1.0.8              115053965e86b       19.7MB

11.8 配置ingress方式訪問(wèn)

Nginx Ingress Controller默認(rèn)使用HTTP協(xié)議轉(zhuǎn)發(fā)請(qǐng)求到后端業(yè)務(wù)容器晓淀。當(dāng)您的業(yè)務(wù)容器為HTTPS協(xié)議時(shí)所袁，可以通過(guò)使用注解nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"來(lái)使得Nginx Ingress Controller使用HTTP協(xié)議轉(zhuǎn)發(fā)請(qǐng)求到后端業(yè)務(wù)容器。

vim ingress-dashboard.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dashboard-ingress
  namespace: kubernetes-dashboard
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  ingressClassName: "nginx" # 控制器的類型為nginx
  tls:
  - hosts:
    - yzj-k8s.com   #主機(jī)名
    secretName: dashboard-tls  #這里引用創(chuàng)建的secrets
  rules:
  - host: yzj-k8s.com
    http:
      paths:
      - path: /
        pathType: Prefix   #起始與根都進(jìn)行代理凶掰。
        backend:
          service:
            name: kubernetes-dashboard   #service名稱
            port:     #后端端口
              number: 443

#加載配置文件
$ kubectl apply -f ingress-dashboard.yaml
ingress.networking.k8s.io/dashboard-ingress created
# 檢查Ingress配置文件
$ kubectl describe ingress -n kubernetes-dashboard
Name:             dashboard-ingress
Labels:           <none>
Namespace:        kubernetes-dashboard
Address:          
Ingress Class:    nginx
Default backend:  <default>
TLS:
  dashboard-tls terminates yzj-k8s.com
Rules:
  Host         Path  Backends
  ----         ----  --------
  yzj-k8s.com  
               /   kubernetes-dashboard:443 (10.244.166.153:8443)
Annotations:   nginx.ingress.kubernetes.io/backend-protocol: HTTPS
Events:
  Type    Reason  Age   From                      Message
  ----    ------  ----  ----                      -------
  Normal  Sync    135m  nginx-ingress-controller  Scheduled for sync

11.9 編輯本地hosts解析并訪問(wèn)UI

a.hosts文件添加

# 192.168.77.134是ingress-nginx的節(jié)點(diǎn)地址
192.168.77.134  yzj-k8s.com

b. 訪問(wèn)

c. 創(chuàng)建管理token

# 創(chuàng)建管理員  dashboard-admin 
$ kubectl create serviceaccount dashboard-admin -n kube-system

# 在整個(gè)集群中為管理員 dashboard-admin 授權(quán)
$ kubectl create clusterrolebinding dashboard-admin \
--clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin

# 創(chuàng)建集群管理員登錄dashboard所需token:
$ kubectl create token dashboard-admin -n kube-system --duration=87600h
eyJhbGciOiJSUzI1NiIsImtpZCI6ImxwOXhhbnJQTHZYWWlrQUx3WENFZzZZUlBhRW1kOVcxUDJsSzludERpQUEifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoyMDMwOTM3MTMxLCJpYXQiOjE3MTU1NzcxMzEsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJkYXNoYm9hcmQtYWRtaW4iLCJ1aWQiOiIwMTQyYTZmOC1hMGVlLTQwMzctODJmYS0wOGRjM2RiN2QxMmQifX0sIm5iZiI6MTcxNTU3NzEzMSwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.0ZMAG1E_PnPcd-GcpSm6HLOCkaZBMHM3_Q_6ewLbXBztcAxMEHbCenUAERSXgsWPd6AZKcuT_HtIc1f-DG9jyR6JkQ8jWiH5cAcdzKjtlNrDkqVeIOz4K-2xZewJfGTOvMLVtxtHyvOOZe8JXXGKO2bb7anYN8Fe4r0nCSEp7Sc1AZRL-S1993-Qe6HPPaNXvejiSi7-g1UeCyHZHXjGx8Ociluu9TV1MxsYso4Bmxflug0HVtpv0t3lYliT0EaJWkHjU2zkooXI8l7ltdeogTXdQBWd2KrNArtId4EfQ21Qt-B-SjwbHlEyPX8qAhnOUyGb4hD1hoGiT8Msk1q3Og

通過(guò)token登錄燥爷，輸入上面的token，再次訪問(wèn)