前言
本節(jié)中 K8S 使用 NFS 遠(yuǎn)程存儲(chǔ)，為托管的 pod 提供了動(dòng)態(tài)存儲(chǔ)服務(wù)臊诊，pod 創(chuàng)建者無需關(guān)心數(shù)據(jù)以何種方式存在哪里坡慌，只需要提出需要多大空間的申請(qǐng)即可。
總體流程是：
- 創(chuàng)建 NFS 服務(wù)器侨嘀。
- 創(chuàng)建 Service Account臭挽。用來管控 NFS provisioner 在k8s集群中運(yùn)行的權(quán)限。
- 創(chuàng)建 StorageClass咬腕。負(fù)責(zé)創(chuàng)建 PVC 并調(diào)用 NFS provisioner 進(jìn)行預(yù)定的工作欢峰，并關(guān)聯(lián) PV 和 PVC。
- 創(chuàng)建 NFS provisioner涨共。有兩個(gè)功能,一個(gè)是在NFS共享目錄下創(chuàng)建掛載點(diǎn)(volume),二是建立 PV 并將 PV 與 NFS 掛載點(diǎn)建立關(guān)聯(lián)纽帖。
更新歷史
- 20200610 - 初稿 - 左程立
- 原文地址 - https://blog.zuolinux.com/2020/06/10/nfs-client-provisioner.html
配置NFS服務(wù)器
server ip: 192.168.10.17
[root@work03 ~]# yum install nfs-utils rpcbind -y
[root@work03 ~]# systemctl start nfs
[root@work03 ~]# systemctl start rpcbind
[root@work03 ~]# systemctl enable nfs
[root@work03 ~]# systemctl enable rpcbind
[root@work03 ~]# mkdir -p /data/nfs/
[root@work03 ~]# chmod 777 /data/nfs/
[root@work03 ~]# cat /etc/exports
/data/nfs/ 192.168.10.0/24(rw,sync,no_root_squash,no_all_squash)
[root@work03 ~]# exportfs -arv
exporting 192.168.10.0/24:/data/nfs
[root@work03 ~]# showmount -e localhost
Export list for localhost:
/data/nfs 192.168.10.0/24
參數(shù):
sync：將數(shù)據(jù)同步寫入內(nèi)存緩沖區(qū)與磁盤中，效率低举反，但可以保證數(shù)據(jù)的一致性
async：將數(shù)據(jù)先保存在內(nèi)存緩沖區(qū)中懊直，必要時(shí)才寫入磁盤
所有work節(jié)點(diǎn)安裝 nfs-utils rpcbind
yum install nfs-utils rpcbind -y
systemctl start nfs
systemctl start rpcbind
systemctl enable nfs
systemctl enable rpcbind
創(chuàng)建動(dòng)態(tài)卷提供者
創(chuàng)建RBAC授權(quán)
# wget https://raw.githubusercontent.com/kubernetes-incubator/external-storage/master/nfs-client/deploy/rbac.yaml
# kubectl apply -f rbac.yaml
創(chuàng)建 Storageclass
# cat class.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: managed-nfs-storage
provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
archiveOnDelete: "false"
創(chuàng)建nfs-client-provisioner自動(dòng)配置程序，以便自動(dòng)創(chuàng)建持久卷(PV)
自動(dòng)創(chuàng)建的 PV 以
{pvcName}-${pvName} 的命名格式創(chuàng)建在 NFS 上
當(dāng)這個(gè) PV 被回收后會(huì)以 archieved-
# cat deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nfs-client-provisioner
labels:
app: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: nfs-client-provisioner
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccountName: nfs-client-provisioner
containers:
- name: nfs-client-provisioner
image: quay.io/external_storage/nfs-client-provisioner:latest
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: fuseim.pri/ifs
- name: NFS_SERVER
value: 192.168.10.17
- name: NFS_PATH
value: /data/nfs
volumes:
- name: nfs-client-root
nfs:
server: 192.168.10.17
path: /data/nfs
創(chuàng)建一個(gè)有狀態(tài)應(yīng)用
# cat statefulset-nfs.yaml
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
app: nginx
spec:
ports:
- port: 80
name: web
clusterIP: None
selector:
app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: nfs-web
spec:
serviceName: "nginx"
replicas: 3
selector:
matchLabels:
app: nfs-web # has to match .spec.template.metadata.labels
template:
metadata:
labels:
app: nfs-web
spec:
terminationGracePeriodSeconds: 10
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: www
annotations:
volume.beta.kubernetes.io/storage-class: managed-nfs-storage
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1Gi
[root@master01 ~]# kubectl apply -f statefulset-nfs.yaml
查看 Pod/PV/PVC
[root@master01 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
nfs-client-provisioner-5f5fff65ff-2pmxh 1/1 Running 0 26m
nfs-web-0 1/1 Running 0 2m33s
nfs-web-1 1/1 Running 0 2m27s
nfs-web-2 1/1 Running 0 2m21s
[root@master01 ~]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
www-nfs-web-0 Bound pvc-62f4868f-c6f7-459e-a280-26010c3a5849 1Gi RWO managed-nfs-storage 2m35s
www-nfs-web-1 Bound pvc-47b68872-35f2-4d3b-bc70-fc59d3bcdbf9 1Gi RWO managed-nfs-storage 2m29s
www-nfs-web-2 Bound pvc-0af3ac53-56d9-4526-8c60-eb0ce3f281e0 1Gi RWO managed-nfs-storage 2m23s
[root@master01 ~]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-0af3ac53-56d9-4526-8c60-eb0ce3f281e0 1Gi RWO Delete Bound default/www-nfs-web-2 managed-nfs-storage 2m25s
pvc-47b68872-35f2-4d3b-bc70-fc59d3bcdbf9 1Gi RWO Delete Bound default/www-nfs-web-1 managed-nfs-storage 2m31s
pvc-62f4868f-c6f7-459e-a280-26010c3a5849 1Gi RWO Delete Bound default/www-nfs-web-0 managed-nfs-storage 2m36s
查看 nfs server 目錄中信息火鼻，同時(shí)各子目錄中內(nèi)容為空
[root@work03 ~]# ls -l /data/nfs/
total 12
default-www-nfs-web-0-pvc-62f4868f-c6f7-459e-a280-26010c3a5849
default-www-nfs-web-1-pvc-47b68872-35f2-4d3b-bc70-fc59d3bcdbf9
default-www-nfs-web-2-pvc-0af3ac53-56d9-4526-8c60-eb0ce3f281e0
破壞性測(cè)試
將每個(gè) pod 中寫入內(nèi)容
[root@master01 ~]# for i in 0 1 2; do kubectl exec nfs-web-$i -- sh -c 'echo $(hostname) > /usr/share/nginx/html/index.html'; done
遠(yuǎn)程nfs各子目錄中不再為空室囊，出現(xiàn)了內(nèi)容
[root@work03 ~]# ls /data/nfs/default-www-nfs-web-0-pvc-62f4868f-c6f7-459e-a280-26010c3a5849/
index.html
[root@work03 ~]#
查看每個(gè)容器中內(nèi)容，均為各自主機(jī)名
[root@master01 ~]# for i in 0 1 2; do kubectl exec -it nfs-web-$i -- cat /usr/share/nginx/html/index.html; done
nfs-web-0
nfs-web-1
nfs-web-2
刪除對(duì)應(yīng) pod
[root@master01 ~]# kubectl get pod -l app=nfs-web
NAME READY STATUS RESTARTS AGE
nfs-web-0 1/1 Running 0 7m7s
nfs-web-1 1/1 Running 0 7m3s
nfs-web-2 1/1 Running 0 7m
[root@master01 ~]# kubectl delete pod -l app=nfs-web
pod "nfs-web-0" deleted
pod "nfs-web-1" deleted
pod "nfs-web-2" deleted
可以看到又被自動(dòng)創(chuàng)建了
[root@master01 ~]# kubectl get pod -l app=nfs-web
NAME READY STATUS RESTARTS AGE
nfs-web-0 1/1 Running 0 15s
nfs-web-1 1/1 Running 0 11s
nfs-web-2 1/1 Running 0 8s
再次查看每個(gè)pod中內(nèi)容魁索，可以看到文件內(nèi)容沒有變化
[root@master01 ~]# for i in 0 1 2; do kubectl exec -it nfs-web-$i -- cat /usr/share/nginx/html/index.html; done
nfs-web-0
nfs-web-1
nfs-web-2
結(jié)束語(yǔ)
可以看到融撞， statefulset 控制器通過固定的 pod 創(chuàng)建順序可以確保 pod 之間的拓?fù)潢P(guān)系一直處于穩(wěn)定不變的狀態(tài)，通過 nfs-client-provisioner 自動(dòng)創(chuàng)建和每個(gè) pod 有固定對(duì)應(yīng)關(guān)系的遠(yuǎn)程存儲(chǔ)卷粗蔚，確保 pod 重建后數(shù)據(jù)不會(huì)丟失尝偎。
