經(jīng)過一周的折騰,終于把kubernetes給搭建起來了,簡單做個記錄。
如果沒有GTW的話,通過官網(wǎng)給的教程那簡直就是so easy膝捞。可是渴频,你知道的拔第,國人在軟件方面總不是那么一帆風順的。
先來說說kubernetes核心組件:
kubelet:每個節(jié)點都需要安裝的組件,負責操作docker并與主節(jié)點通信得封。
kube-apiservice:主節(jié)點提供給用戶操作的接口,同時操作各個節(jié)點。
kube-controller-manager:控制中心,負責各個節(jié)點的狀態(tài)管理等悯恍。
kube-scheduler:調(diào)度中心瞬欧。
kubernetes幾個主要的概念:
- namespace:命名空間唉侄。我們將按照項目的性質(zhì)進行劃分候生,例如:showdoc:可以劃分到support命名空間须蜗。開發(fā)的服務(wù)劃分到dev下面缭付。
- Deployment:部署陷猫。每個項目都需要一個部署的節(jié)點舅巷,可以理解成守護進程忘蟹。如果具體的容器掛了媚值,部署的節(jié)點會自動重新啟動一個新的容器嚼松。
- Pod:容器寝受。每個項目具體的容器節(jié)點。可以通過這個容器節(jié)點查看日志,運行一些簡單的容器命令肄扎。
- Service:服務(wù)反浓。這個服務(wù)的功能是將kubernetes集群IP和容器進行綁定肪笋。外部可直接訪問集群IP進行訪問容器。(我們可以通過nginx將端口代理到kubernetes集群IP上面而姐,達到外部局域網(wǎng)直接訪問容器的目的。)
安裝docker
本機裝的docker版本是18.06.0-ce.
安裝依賴
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
設(shè)置阿里云鏡像
sudo yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
安裝docker
sudo yum install docker-ce
啟動 Docker
sudo systemctl enable docker
sudo systemctl start docker
配置阿里云鏡像加速
sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
"registry-mirrors": ["你的加速器地址"]
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker
準備kubernetes安裝條件
配置阿里云的kubernetes源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
EOF
重建 Yum 緩存
yum -y install epel-release
yum clean all
yum makecache
安裝kubernetes核心組件
yum -y install kubelet kubeadm kubectl kubernetes-cni
啟動服務(wù)
systemctl enable kubelet && systemctl start kubelet
配置 Kubeadm 所用到的鏡像,新建一個kubeadm-config.sh文件
#!/bin/bash
images=(kube-proxy-amd64:v1.11.0 kube-scheduler-amd64:v1.11.0 kube-controller-manager-amd64:v1.11.0 kube-apiserver-amd64:v1.11.0
etcd-amd64:3.2.18 coredns:1.1.3 pause-amd64:3.1 kubernetes-dashboard-amd64:v1.8.3 k8s-dns-sidecar-amd64:1.14.9 k8s-dns-kube-dns-amd64:1.14.9
k8s-dns-dnsmasq-nanny-amd64:1.14.9 )
for imageName in ${images[@]} ; do
docker pull keveon/$imageName
docker tag keveon/$imageName k8s.gcr.io/$imageName
docker rmi keveon/$imageName
done
docker tag k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1
執(zhí)行這個文件
chmod -R 777 ./kubeadm-config.sh
./kubeadm-config.sh
關(guān)閉 Swap
sudo swapoff -a
#要永久禁掉swap分區(qū),打開如下文件注釋掉swap那一行
sudo vi /etc/fstab
關(guān)閉 SELinux
# 永久關(guān)閉 修改/etc/sysconfig/selinux文件設(shè)置
sed -i 's/SELINUX=permissive/SELINUX=disabled/' /etc/sysconfig/selinux
# 臨時禁用selinux
setenforce 0
關(guān)閉 防火墻
systemctl stop firewalld&&systemctl disable firewalld
配置轉(zhuǎn)發(fā)參數(shù)
# 配置轉(zhuǎn)發(fā)相關(guān)參數(shù)瞬雹,否則可能會出錯
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness=0
EOF
# 這里按回車状婶,下面是第二條命令
sysctl --system
正式安裝 Kuberentes(主機)###
執(zhí)行命令
kubeadm init --kubernetes-version=v1.11.0 --pod-network-cidr=10.244.0.0/16
看到如下信息即安裝成功
[init] using Kubernetes version: v1.11.0
[preflight] running pre-flight checks
[WARNING Firewalld]: firewalld is active, please ensure ports [6443 10250] are open or your cluster may not function correctly
I0822 17:20:10.828319 14125 kernel_validator.go:81] Validating kernel version
I0822 17:20:10.828402 14125 kernel_validator.go:96] Validating kernel config
[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.06.0-ce. Max validated version: 17.03
[WARNING Hostname]: hostname "test121" could not be reached
[WARNING Hostname]: hostname "test121" lookup test121 on 192.168.192.1:53: no such host
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [test121 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.192.121]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [test121 localhost] and IPs [127.0.0.1 ::1]
[certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [test121 localhost] and IPs [192.168.192.121 127.0.0.1 ::1]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] this might take a minute or longer if the control plane images have to be pulled
[apiclient] All control plane components are healthy after 36.501252 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.11" in namespace kube-system with the configuration for the kubelets in the cluster
[markmaster] Marking the node test121 as master by adding the label "node-role.kubernetes.io/master=''"
[markmaster] Marking the node test121 as master by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "test121" as an annotation
[bootstraptoken] using token: zw54ho.7rcy6bzcxjlxo0j6
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of machines by running the following on each node
as root:
kubeadm join 192.168.192.121:6443 --token zw54ho.7rcy6bzcxjlxo0j6 --discovery-token-ca-cert-hash sha256:b9162792dd5ad6719c87bbc3938240f6dc23de4be6f0fefff5069f11b3710f79
配置 kubectl 認證信息
export KUBECONFIG=/etc/kubernetes/admin.conf
# 如果你想持久化的話,直接執(zhí)行以下命令【推薦】
echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
安裝 Flannel 網(wǎng)絡(luò)
mkdir -p /etc/cni/net.d/
cat <<EOF> /etc/cni/net.d/10-flannel.conf
{
“name”: “cbr0”,
“type”: “flannel”,
“delegate”: {
“isDefaultGateway”: true
}
}
EOF
mkdir /usr/share/oci-umount/oci-umount.d -p
mkdir /run/flannel/
cat <<EOF> /run/flannel/subnet.env
FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.1.0/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true
EOF
最后,我們需要新建一個 flannel.yml 文件,內(nèi)容如下:
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: flannel
rules:
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- ""
resources:
- nodes
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes/status
verbs:
- patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: flannel
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: flannel
subjects:
- kind: ServiceAccount
name: flannel
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: flannel
namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-flannel-cfg
namespace: kube-system
labels:
tier: node
app: flannel
data:
cni-conf.json: |
{
"name": "cbr0",
"type": "flannel",
"delegate": {
"isDefaultGateway": true
}
}
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: kube-flannel-ds
namespace: kube-system
labels:
tier: node
app: flannel
spec:
template:
metadata:
labels:
tier: node
app: flannel
spec:
hostNetwork: true
nodeSelector:
beta.kubernetes.io/arch: amd64
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay.io/coreos/flannel:v0.9.1-amd64
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conf
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.9.1-amd64
command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr" ]
securityContext:
privileged: true
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
執(zhí)行命令
kubectl create -f ./flannel.yml
執(zhí)行完成之后,我們可以運行一下命令闻蛀,查看現(xiàn)在的節(jié)點信息:
kubectl get nodes
# 會打印如下信息
NAME STATUS ROLES AGE VERSION
test121 Ready master 6m v1.11.0
如果想讓master運行pod,執(zhí)行如下命令
kubectl taint nodes master.k8s node-role.kubernetes.io/master-
#或者
kubectl uncordon test121
好了手蝎,以上我們就完成了主機的配置棵介。
正式安裝 Kuberentes(節(jié)點安裝)
Node 節(jié)點所需要做的都在 準備kubernetes安裝條件里面贸营,做完之后直接執(zhí)行剛剛主機輸出的:
kubeadm join 192.168.192.121:6443 --token zw54ho.7rcy6bzcxjlxo0j6 --discovery-token-ca-cert-hash sha256:b9162792dd5ad6719c87bbc3938240f6dc23de4be6f0fefff5069f11b3710f79
執(zhí)行完就 OK 了揣云⊙忠悖回到主節(jié)點查看運行狀態(tài)
kubectl get nodes
#會輸出
NAME STATUS ROLES AGE VERSION
test119 Ready <none> 4d v1.11.2
test121 Ready master 4d v1.11.2
Dashboard 配置(測試環(huán)境)
Kuberentes 配置 DashBoard 也不簡單汪榔,當然你可以使用官方的 dashboard 的 yaml 文件進行部署雌团,也可以使用 Mr.Devin 這位博主所提供的修改版猛蔽,避免踩坑略板。
地址在:https://github.com/gh-Devin/kubernetes-dashboard藐鹤,將這些 Yaml 文件下載下來赂韵,在其目錄下(注意在 Yaml 文件所在目錄),執(zhí)行以下命令:
kubectl -n kube-system create -f .
繼續(xù)創(chuàng)建文件dashboard-admin.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: kubernetes-dashboard
labels:
k8s-app: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kube-system
kubectl -f ./dashboard-admin.yaml create
訪問你 MASTER 主機的 IP:30090即可祭示。
Dashboard 配置(生產(chǎn)環(huán)境)
- 配置Dashboard
Dashboard需要用到k8s.gcr.io/kubernetes-dashboard的鏡像,由于網(wǎng)絡(luò)原因稠歉,可以采用預(yù)先拉取并打Tag或者修改yaml文件中的鏡像地址蹂窖,本文使用后者:
kubectl apply -f http://mirror.faasx.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml
上面使用的yaml只是將 https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml 中的 k8s.gcr.io 替換為了 reg.qiniu.com/k8s。
查看Dashboard是否啟動
kubectl get pods --all-namespaces
# 輸出
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system kubernetes-dashboard-7d5dcdb6d9-mf6l2 1/1 Running 0 9m
創(chuàng)建服務(wù)用戶,創(chuàng)建admin-user.yaml文件
# admin-user.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kube-system
執(zhí)行kubectl create命令:
kubectl create -f admin-user.yaml
綁定角色,創(chuàng)建admin-user-role-binding.yaml
# admin-user-role-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kube-system
執(zhí)行kubectl create命令:
kubectl create -f admin-user-role-binding.yaml
獲取Token,現(xiàn)在我們需要找到新創(chuàng)建的用戶的Token横媚,以便用來登錄dashboard:
kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')
#輸出:
Name: admin-user-token-qrj82
Namespace: kube-system
Labels: <none>
Annotations: kubernetes.io/service-account.name=admin-user
kubernetes.io/service-account.uid=6cd60673-4d13-11e8-a548-00155d000529
Type: kubernetes.io/service-account-token
Data
====
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLXFyajgyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI2Y2Q2MDY3My00ZDEzLTExZTgtYTU0OC0wMDE1NWQwMDA1MjkiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06YWRtaW4tdXNlciJ9.C5mjsa2uqJwjscWQ9x4mEsWALUTJu3OSfLYecqpS1niYXxp328mgx0t-QY8A7GQvAr5fWoIhhC_NOHkSkn2ubn0U22VGh2msU6zAbz9sZZ7BMXG4DLMq3AaXTXY8LzS3PQyEOCaLieyEDe-tuTZz4pbqoZQJ6V6zaKJtE9u6-zMBC2_iFujBwhBViaAP9KBbE5WfREEc0SQR9siN8W8gLSc8ZL4snndv527Pe9SxojpDGw6qP_8R-i51bP2nZGlpPadEPXj-lQqz4g5pgGziQqnsInSMpctJmHbfAh7s9lIMoBFW7GVE8AQNSoLHuuevbLArJ7sHriQtDB76_j4fmA
ca.crt: 1025 bytes
namespace: 11 bytes
保存好token信息灯蝴。
- 集成Heapster
Heapster是容器集群監(jiān)控和性能分析工具孝宗,天然的支持Kubernetes和CoreOS狡忙。
Heapster支持多種儲存方式北专,本示例中使用influxdb描孟,直接執(zhí)行下列命令即可
kubectl create -f http://mirror.faasx.com/kubernetes/heapster/deploy/kube-config/influxdb/influxdb.yaml
kubectl create -f http://mirror.faasx.com/kubernetes/heapster/deploy/kube-config/influxdb/grafana.yaml
kubectl create -f http://mirror.faasx.com/kubernetes/heapster/deploy/kube-config/influxdb/heapster.yaml
kubectl create -f http://mirror.faasx.com/kubernetes/heapster/deploy/kube-config/rbac/heapster-rbac.yaml
上面命令中用到的yaml是從 https://github.com/kubernetes/heapster/tree/master/deploy/kube-config/influxdb 復(fù)制的酬核,并將
k8s.gcr.io
修改為國內(nèi)鏡像。
然后忍些,查看一下Pod的狀態(tài):
kubectl get pods --namespace=kube-system
#輸出
NAME READY STATUS RESTARTS AGE
...
heapster-5869b599bd-kxltn 1/1 Running 0 5m
monitoring-grafana-679f6b46cb-xxsr4 1/1 Running 0 5m
monitoring-influxdb-6f875dc468-7s4xz 1/1 Running 0 6m
...
- 訪問
通過API Server
Dashboard的訪問地址為:https://<master-ip>:<apiserver-port>/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/
但是返回的結(jié)果可能如下:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "services \"https:kubernetes-dashboard:\" is forbidden: User \"system:anonymous\" cannot get services/proxy in the namespace \"kube-system\"",
"reason": "Forbidden",
"details": {
"name": "https:kubernetes-dashboard:",
"kind": "services"
},
"code": 403
}
這是因為最新版的k8s默認啟用了RBAC,并為未認證用戶賦予了一個默認的身份:anonymous。
對于API Server來說众弓,它是使用證書進行認證的滨达,我們需要先創(chuàng)建一個證書:
1.首先找到kubectl
命令的配置文件辆飘,默認情況下為/etc/kubernetes/admin.conf
紧卒,復(fù)制到了$HOME/.kube/config
中聋亡。
mkdir $HOME/.kube/config
cp /etc/kubernetes/admin.conf $HOME/.kube/config/admin.conf
2.然后我們使用client-certificate-data
和client-key-data
生成一個p12文件,可使用下列命令:
# 生成client-certificate-data
grep 'client-certificate-data' ~/.kube/config | head -n 1 | awk '{print $2}' | base64 -d >> kubecfg.crt
# 生成client-key-data
grep 'client-key-data' ~/.kube/config | head -n 1 | awk '{print $2}' | base64 -d >> kubecfg.key
# 生成p12
openssl pkcs12 -export -clcerts -inkey kubecfg.key -in kubecfg.crt -out kubecfg.p12 -name "kubernetes-client"
3.最后導(dǎo)入上面生成的p12文件佃蚜,重新打開瀏覽器,顯示如下:
然后輸入Token,進入控制臺陕贮。
我們可以使用一開始創(chuàng)建的admin-user用戶的token進行登錄峦甩,一切OK灸促。
對于生產(chǎn)系統(tǒng),我們應(yīng)該為每個用戶應(yīng)該生成自己的證書,因為不同的用戶會有不同的命名空間訪問權(quán)限汁尺。
以上就是kubernetes配置的全部過程了帮碰。