helm
背景
上一篇文章簡單介紹了Kubernetes 單機版的開發(fā)測試環(huán)境minikube的搭建和環(huán)境配置,在很長的一段時間里面我也沒有搭建了,今天突然下了一個最新版,發(fā)現(xiàn)啟動啟動不了,于是又乖乖minikube 退回到了0.25.2版本,由于最近都在使用Helm與Swift進行項目的部署發(fā)布,本文簡單記錄步驟.
概念
Helm
Helm是 k8s 集群部署和管理應用的一個包管理工具,使用Chart文件配置項目,隱藏了 k8s 本身的一些基本概念,具有版本回滾,方便升級的一些特性.
Helm分為客戶端和服務端,服務端tiller運行在 k8s 集群中,用戶管理和部署,更新 k8s集群中的應用,客戶端有 Kubernetes-helm的客戶端(命令行)工具,用于和服務端Tiller進行連接和通訊.
Swift
因為為了方便在代碼中間調(diào)用Helm的一些接口不是特別方便,例如tiller都是走grpc協(xié)議等,所以社區(qū)的小伙伴們開發(fā)了這么一個代理(Swift),封裝成為restful 的 http的接口形式,方便各種語言和tiller進行通信和操作.
搭建
本文 環(huán)境:
kubernetes 1.9.4
minikube 0.25.2
helm 2.12.1
swift 0.10.0
swift的版本需要和 helm 版本對應,參考https://github.com/appscode/swift
啟動 minikube 環(huán)境
minikube start --kubernetes-version v1.9.4 --extra-config=apiserver.Authorization.Mode=RBAC
- 必須開啟 RBAC 的權(quán)限的模式,否則后續(xù)安裝會有很多問題,比如默認的模式?jīng)]有
cluster-admin的集群權(quán)限,無法安裝swift等. - 如果用 helm 安裝 swift 遇到這樣的問題,就是這個原因.
Error: release my-searchlight failed: clusterroles.rbac.authorization.k8s.io "my-searchlight" is forbidden: attempt to grant extra privileges: [PolicyRule{APIGroups:["apiextensions.k8s.io"], Resources:["customresourcedefinitions"], Verbs:["*"]} PolicyRule{APIGroups:["extensions"], Resources:["thirdpartyresources"], Verbs:["*"]} PolicyRule{APIGroups:["monitoring.appscode.com"], Resources:["*"], Verbs:["*"]} PolicyRule{APIGroups:["storage.k8s.io"], Resources:["*"], Verbs:["get"]} PolicyRule{APIGroups:[""], Resources:["secrets"], Verbs:["get"]} PolicyRule{APIGroups:[""], Resources:["secrets"], Verbs:["list"]} PolicyRule{APIGroups:[""], Resources:["componentstatuses"], Verbs:["get"]} PolicyRule{APIGroups:[""], Resources:["componentstatuses"], Verbs:["list"]} PolicyRule{APIGroups:[""], Resources:["persistentvolumes"], Verbs:["get"]} PolicyRule{APIGroups:[""], Resources:["persistentvolumes"], Verbs:["list"]} PolicyRule{APIGroups:[""], Resources:["persistentvolumeclaims"], Verbs:["get"]} PolicyRule{APIGroups:[""], Resources:["persistentvolumeclaims"], Verbs:["list"]} PolicyRule{APIGroups:[""], Resources:["pods"], Verbs:["get"]} PolicyRule{APIGroups:[""], Resources:["pods"], Verbs:["list"]} PolicyRule{APIGroups:[""], Resources:["pods"], Verbs:["patch"]} PolicyRule{APIGroups:[""], Resources:["pods"], Verbs:["watch"]} PolicyRule{APIGroups:[""], Resources:["nodes"], Verbs:["get"]} PolicyRule{APIGroups:[""], Resources:["nodes"], Verbs:["list"]} PolicyRule{APIGroups:[""], Resources:["nodes"], Verbs:["patch"]} PolicyRule{APIGroups:[""], Resources:["nodes"], Verbs:["watch"]} PolicyRule{APIGroups:[""], Resources:["namespaces"], Verbs:["get"]} PolicyRule{APIGroups:[""], Resources:["namespaces"], Verbs:["list"]} PolicyRule{APIGroups:[""], Resources:["namespaces"], Verbs:["patch"]} PolicyRule{APIGroups:[""], Resources:["namespaces"], Verbs:["watch"]} PolicyRule{APIGroups:[""], Resources:["pods/exec"], Verbs:["create"]} PolicyRule{APIGroups:[""], Resources:["events"], Verbs:["create"]} PolicyRule{APIGroups:[""], Resources:["events"], Verbs:["list"]}] user=&{system:serviceaccount:kube-system:tilleruser a61ce1ed-0a6d-11e9-babc-0800274b952b [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]} ownerrules=[] ruleResolutionErrors=[clusterroles.rbac.authorization.k8s.io "cluster-admin" not found]
- 用 k8s 1.9.4的原因在于,1.10.0沒有默認的
kube-dns模塊了,有些不方便,而 swift 和 tiller 通信依賴 dns. - 和上文一樣配置一下代理,便于獲取一些墻外的資源.
Helm 客戶端安裝
brew install kubernetes-helm
helm version
Kube-dns安裝
- 等待
kube-dns安裝完成
kubectl get deployment -n kube-system --watch
- 創(chuàng)建 serviceaccount 賬戶
kubectl create serviceaccount kube-dns -n kube-system
- 為
kube-dns的 deployment設置關(guān)聯(lián)關(guān)系
kubectl patch deployment kube-dns -n kube-system -p '{"spec":{"template":{"spec":{"serviceAccountName":"kube-dns"}}}}'
- 等待 dns 相關(guān) pods 啟動完成
kubectl get pods -n kube-system --watch
dns
部署 Tiller
- Tiller 作為 Helm 的服務端,版本和客戶端保持一致.
- 部署 Tiller 之前需要為 tiller 賦予 k8s 的操作權(quán)限,創(chuàng)建流程如下
$ kubectl create serviceaccount tiller --namespace kube-system
$ kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
$ helm init --service-account tiller
$ helm version --short
Client: v2.12.1+g02a47c7
Server: v2.12.1+g02a47c7
-
等待 tiller 部署完成
tiller
部署 Swift
- 先為 helm 添加個 appscode 的倉庫
$ helm repo add appscode https://charts.appscode.com/stable/
$ helm repo update
$ helm repo list
NAME URL
stable https://kubernetes-charts.storage.googleapis.com
local http://127.0.0.1:8879/charts
appscode https://charts.appscode.com/stable/
- helm中搜索 swift
$ helm search swift
NAME CHART VERSION APP VERSION DESCRIPTION
appscode/swift 0.10.0 0.10.0 Swift by AppsCode - Ajax friendly Helm Tiller Proxy
stable/swift 0.6.3 0.7.3 DEPRECATED swift by AppsCode - Ajax friendly Helm Tiller ...
- 安裝 0.10.0
helm install appscode/swift --name my-swift
- 檢查是否安裝完成
kubectl get pods --all-namespaces -l app=swift --watch
swift
測試
- 獲取service 查看一下 ip 和端口
$ kubectl get svc swift-my-swift
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
swift-my-swift NodePort 10.107.55.194 <none> 9855:31743/TCP,50055:32523/TCP,56790:30092/TCP 3h58m
-
minikube ssh后進入 minikube 訪問,成功返回,說明已經(jīng) ok
$ curl http://10.107.55.194:9855/tiller/v2/version/json
{"Version":{"sem_ver":"v2.12.1","git_commit":"02a47c7249b1fc6d8fd3b94e6b4babf9d818144e","git_tree_state":"clean"}}
- 上面 service 中其實默認的 Type 應該是
Cluster IP,為了宿主機能夠訪問,改成了NodePort,方法:
kubectl patch svc swift -n kube-system -p '{"spec":{"type":"NodePort"}}'
- 宿主機試一下
$ minikube ip
192.168.99.100
$ curl http://192.168.99.100:31743/tiller/v2/version/json
{"Version":{"sem_ver":"v2.12.1","git_commit":"02a47c7249b1fc6d8fd3b94e6b4babf9d818144e","git_tree_state":"clean"}}%
-
dashborad查看
dashboard
大功告成!
ps:剛開始啟動minikube 只用了minikube start折騰了半天.....
如何測試
kube-dns是否在工作,參考官方案例
https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/
相關(guān)的錯誤
- 無法解析內(nèi)部 dns
I1228 09:25:08.241821 1 pickfirst.go:71] pickfirstBalancer: HandleSubConnStateChange: 0xc00043da40, TRANSIENT_FAILURE
I1228 09:25:08.243483 1 pickfirst.go:71] pickfirstBalancer: HandleSubConnStateChange: 0xc00043da40, CONNECTING
W1228 09:25:28.242336 1 clientconn.go:1158] grpc: addrConn.createTransport failed to connect to {tiller-deploy.kube-system.svc:44134 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp: i/o timeout". Reconnecting...
I1228 09:25:28.242368 1 pickfirst.go:71] pickfirstBalancer: HandleSubConnStateChange: 0xc00043da40, TRANSIENT_FAILURE
I1228 09:25:28.242629 1 pickfirst.go:71] pickfirstBalancer: HandleSubConnStateChange: 0xc00043da40, CONNECTING
W1228 09:25:53.349620 1 clientconn.go:1158] grpc: addrConn.createTransport failed to connect to {tiller-deploy.kube-system.svc:44134 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp: i/o timeout". Reconnecting...
I1228 09:25:53.349990 1 pickfirst.go:71] pickfirstBalancer: HandleSubConnStateChange: 0xc00043da40, TRANSIENT_FAILURE
I1228 09:25:53.350133 1 pickfirst.go:71] pickfirstBalancer: HandleSubConnStateChange: 0xc00043da40, CONNECTING
W1228 09:26:32.635786 1 clientconn.go:1158] grpc: addrConn.createTransport failed to connect to {tiller-deploy.kube-system.svc:44134 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp: i/o timeout". Reconnecting...
I1228 09:26:32.635949 1 pickfirst.go:71] pickfirstBalancer: HandleSubConnStateChange: 0xc00043da40, TRANSIENT_FAILURE
I1228 09:26:32.636553 1 pickfirst.go:71] pickfirstBalancer: HandleSubConnStateChange: 0xc00043da40, CONNECTING
W1228 09:27:12.647474 1 clientconn.go:1158] grpc: addrConn.createTransport failed to connect to {tiller-deploy.kube-system.svc:44134 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp: lookup tiller-deploy.kube-system.svc on 10.96.0.10:53: read udp 172.17.0.4:58290->10.96.0.10:53: i/o timeout". Reconnecting...
I1228 09:27:12.647527 1 pickfirst.go:71] pickfirstBalancer: HandleSubConnStateChange: 0xc00043da40, TRANSIENT_FAILURE
I1228 09:27:44.000042 1 pickfirst.go:71] pickfirstBalancer: HandleSubConnStateChange: 0xc00043da40, CONNECTING
W1228 09:28:08.235280 1 clientconn.go:830] Failed to dial tiller-deploy.kube-system.svc:44134: grpc: the connection is closing; please retry.
- 發(fā)現(xiàn) minikube 起來后,根本沒有
kube-dns和kube-dashborad,但是minikube addons list中這兩個卻是enabled的狀態(tài),參考解決方案https://github.com/kubernetes/minikube/issues/2619
原理就是手動裝了kube-dns組件讓其正常工作.
參考
[1] https://github.com/appscode/swift/blob/master/docs/setup/rbac.md
[2] https://appscode.com/products/swift/0.10.0/guides/api/
