環(huán)境
- K8S集群
- 集群一: k8s-admin 集群訪問(wèn)憑證保存為 ~/kubeconfig-k8s-admin
- 集群二: k8s-cube 集群訪問(wèn)憑證保存為 ~/kubeconfig-k8s-cube
- OPS主機(jī)需要軟件列表：helm痒玩，kubectl晕拆，ansible
- MacOS系統(tǒng)：
brew install helm kubectl ansible
cat >> ~/.zshrc << EOF
export PATH="/opt/homebrew/opt/ansible/:$PATH"
export PATH="/opt/homebrew/opt/helm/bin/:$PATH"
export PATH="/opt/homebrew/opt/kubectl/bin/:$PATH"
EOF
source ~/.zshrc
方式一：使用多個(gè) kubeconfig 文件來(lái)管理多集群
可以通過(guò)--kubeconfig 命令行參數(shù)來(lái)選擇操作不同集群
kubectl --kubeconfig ~/kubeconfig-k8s-admin get ns
kubectl --kubeconfig ~/kubeconfig-k8s-cube get ns
helm --kubeconfig ~/kubeconfig-k8s-admin list -A
helm --kubeconfig ~/kubeconfig-k8s-cube list -A
方式二：使用單一 kubeconfig 文件來(lái)管理多集群
使用默認(rèn)的 kubeconfig 文件， 需要將多個(gè) kubeconfig 合并，保存在kubeconfig 的默認(rèn)位置$HOME/.kube/config, 合并后的參考格式如下：
apiVersion: v1
kind: Config
preferences: {}
clusters:
- name: cluster-a
cluster:
certificate-authority-data: <cluster-a-token>
server: https://cluster-a-api-lb:6443
- name: cluster-b
cluster:
certificate-authority-data: <cluster-b-token>
server: https://cluster-b-api-lb:6443
users:
- name: cluster-a-user
user:
token: < cluster-a-user-token >
- name: cluster-b-user
user:
token: < cluster-b-user-token >
contexts:
- name: cluster-a-context
context:
cluster: cluster-a
user: cluster-a-user
- name: cluster-b-context
context:
cluster: cluster-b
user: cluster-b-user
current-context: cluster-a-context
kubeconfig 中定義了 (clusters)集群坤溃、(users)用戶和 以及相關(guān)聯(lián)的(contexts)上下文甲雅，如果使用Uk8s 可以在控制臺(tái): 概覽-> 內(nèi)網(wǎng)憑證/外網(wǎng)憑證 查看需要的憑證昨稼，然后按照上述格式補(bǔ)全即可
在執(zhí)行kubectl 命令時(shí)需要執(zhí)行
--context命令行參數(shù), 來(lái)選擇操作不同集群在執(zhí)行helm 命令時(shí)需要執(zhí)行 --kube-context 命令行參數(shù), 來(lái)選擇操作不同集群
在執(zhí)行kubectl helm 命令不指定參數(shù)瞒瘸，則選則文件中的
current-context作為默認(rèn)集群
kubectl get pods -A #操作的是 current-context 定義的集群
kubectl get pods -A --context cluster-a-context #操作的是 cluster-a 集群
kubectl get pods -A --context cluster-b-context #操作的是 cluster-b 集群
helm list -A --kube-context cluster-a-context #操作的是 cluster-a 集群
helm list -A --kube-context cluster-b-context #操作的是 cluster-b 集群
方式三：使用ansible Playbook 來(lái)管理集群內(nèi)的容器應(yīng)用
sudo ansible-galaxy collection install kubernetes.core
sudo pip3 install kubernetes
場(chǎng)景描述: 目前我們需要使用helm安裝一個(gè)external-dns 用來(lái)將 ingess對(duì)接的域名解析規(guī)則，自動(dòng)同步到DNS服務(wù)器裹纳，使用shell命令操作參考如下：
helm repo add stable https://harbor.onwalk.net/chartrepo/knative
helm repo update
cat > cat > admin-values.yaml << EOF
clusterDomain: admin.local
sources:
- service
- ingress
domainFilters:
- onwalk.net
policy: upsert-only
provider: alibabacloud
alibabacloud:
accessKeyId: xxxxxxxxxx
accessKeySecret: xxxxxxxxx
regionId: rg-xxxxxx
zoneType: public
EOF
helm upgrade -i external-dns stable/external-dns --version '5.4.11' -f admin-values.yaml -n external-dns --create-namespace --kube-context k8s-admin
將以上操作轉(zhuǎn)化為 ansible-playbook 的tasks 可以拆分為四個(gè) tasks 择葡，
- task1: Add stable chart repo 調(diào)用 kubernetes.core.helm_repository模塊
- task2: Update repo 調(diào)用 shell 模塊
- task3: Create NameSpace 調(diào)用 kubernetes.core.k8s 模塊
- Task4: Deploy External Dns 調(diào)用 kubernetes.core.helm 模塊
- hosts: localhost
connection: local
gather_facts: false
tasks:
- name: Add stable chart repo
kubernetes.core.helm_repository:
name: stable
repo_url: "https://harbor.onwalk.net/chartrepo/knative"
- name: Update repo
shell: "helm repo update"
- name: Create NameSpace
kubernetes.core.k8s:
api_version: v1
kind: Namespace
context: cube-admin
name: external-dns
state: present
- name: Deploy External Dns
kubernetes.core.helm:
name: external-dns
chart_ref: stable/external-dns
chart_version: 5.4.11
context: cube-admin
release_namespace: external-dns
values:
clusterDomain: cube.local
sources:
- service
- ingress
domainFilters:
- onwalk.net
policy: upsert-only
provider: alibabacloud
alibabacloud:
accessKeyId: xxxxxxxxxx
accessKeySecret: xxxxxxxxx
regionId: rg-xxxxxx
zoneType: public
將上訴文件保存為 deploy_external_dns.yaml, 執(zhí)行命令 ansible-playbook deploy_external_dns.yaml 命令執(zhí)行成功回看到返回如下類似結(jié)果：
PLAY [localhost] *****************************************************************************************************************************************************************
TASK [Add stable chart repo] *****************************************************************************************************************************************************
ok: [localhost]
TASK [Update repo] ***************************************************************************************************************************************************************
changed: [localhost]
TASK [Create NameSpace] **********************************************************************************************************************************************************
ok: [localhost]
TASK [Deploy External Dns] *******************************************************************************************************************************************************
ok: [localhost]
PLAY RECAP ***********************************************************************************************************************************************************************
localhost : ok=4 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
方式四：使用ansible-playbook roles來(lái)組織管理集群變更tasks
在方式三中，如果是管理一個(gè)集群剃氧，編寫一個(gè)deploy_external_dns.yaml就能完成工作敏储，如果管理多個(gè)集群，就要編寫多個(gè)yaml文件朋鞍，并且重復(fù)編寫很多差異不多的tasks 實(shí)際操作可能類似這個(gè)樣子：
ansible-playbook k8s_dev1_deploy_external_dns.yaml
ansible-playbook k8s_pre1_deploy_external_dns.yaml
ansible-playbook k8s_prd1_deploy_external_dns.yaml
ansible-playbook ...
經(jīng)過(guò)對(duì)比分析已添，這些tasks 主要差異變量：
context: cube-admin
clusterDomain: xxx.local
只要將以上兩個(gè)變量參考可配置化，最原始的四個(gè) tasks就可以復(fù)用滥酥，然后使用 Ansible role 重新組織tasks 文件更舞，拆分為兩個(gè)role：
helm-repository
external-dns
其中 external_dns 依賴 helm-repository 然后目錄結(jié)構(gòu)如下
roles/helm-repository
└── tasks
└── main.yml
roles/external-dns
├── meta
│ └── main.yml
└── tasks
└── main.yml
roles/helm-repository/main.yml
- name: Add stable chart repo
kubernetes.core.helm_repository:
name: stable
repo_url: "https://harbor.onwalk.net/chartrepo/knative"
- name: Update repo
shell: "helm repo update"
roles/external-dns/tasks/main.yml
- name: "cluster {{ clusterContext }} : Create NameSpace"
kubernetes.core.k8s:
api_version: v1
kind: Namespace
context: "{{ clusterContext }}"
name: external-dns
state: present
- name: "cluster {{ clusterContext }} : Deploy External Dns"
kubernetes.core.helm:
name: external-dns
chart_ref: stable/external-dns
chart_version: 5.4.11
context: "{{ clusterContext }}"
release_namespace: external-dns
values:
clusterDomain: "{{ clusterDomain }}"
sources:
- service
- ingress
domainFilters:
- onwalk.net
policy: upsert-only
provider: alibabacloud
alibabacloud:
accessKeyId: xxxxxxxxx
accessKeySecret: xxxxxxxxx
regionId: rg-xxxxxxxxx
zoneType: public
roles/external-dns/meta/main.yml
dependencies:
- role: helm-repository
新建一個(gè)文件 deploy-chart-external-dns 來(lái)引用 role：helm-repository
- hosts: localhost
connection: local
gather_facts: false
tasks:
- include_role:
name: external_dns
vars:
clusterContext: "{{ item.clusterContext }}"
clusterDomain: "{{ item.clusterDomain }}"
with_items:
- { clusterContext: 'k8s-admin', clusterDomain: 'admin.local' }
- { clusterContext: 'k8s-cube', clusterDomain: 'cube.local' }
- { clusterContext: 'k8s-dev', clusterDomain: 'dev.local' }
- { clusterContext: 'k8s-pre', clusterDomain: 'pre.local' }
- ...
最后，僅僅需要維護(hù)一個(gè)可復(fù)用的 role：helm-repository 坎吻，以及在deploy-chart-external-dns 定義要集群屬性等變量缆蝉，就可以輕松的維護(hù)多集群內(nèi)的各類容器應(yīng)用了
執(zhí)行命令：ansible-playbook deploy-chart-external-dns 返回結(jié)果如下：
PLAY [localhost] *****************************************************************************************************************************************************************
TASK [include_role : external_dns] ***********************************************************************************************************************************************
TASK [helm-repository : Add stable chart repo] ***********************************************************************************************************************************
ok: [localhost]
TASK [helm-repository : Update repo] *********************************************************************************************************************************************
changed: [localhost]
TASK [external_dns : cluster k8s-admin : Create NameSpace] ***********************************************************************************************************************
ok: [localhost]
TASK [external_dns : cluster k8s-admin : Deploy External Dns] ********************************************************************************************************************
ok: [localhost]
TASK [external_dns : cluster cube-admin : Create NameSpace] **********************************************************************************************************************
ok: [localhost]
TASK [external_dns : cluster cube-admin : Deploy External Dns] *******************************************************************************************************************
ok: [localhost]
PLAY RECAP ***********************************************************************************************************************************************************************
localhost : ok=6 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
參考
- Ansible kubernetes.core.helm 模塊
- 使用 kubeconfig 文件組織多集群訪問(wèn):
- https://kubernetes.io/zh/docs/concepts/configuration/organize-cluster-access-kubeconfig/
- [https://kubernetes.io/zh/docs/tasks/access-application-cluster/configure-access-multiple-clusters/#set-the-kubeconfig-environment-variable](http://2. https://kubernetes.io/zh/docs/tasks/access-application-cluster/configure-access-multiple-clusters/#set-the-kubeconfig-environment-variable)
