K8S搭建rook-ceph

一 Rook概述

1.1 Ceph簡介

Ceph是一種高度可擴(kuò)展的分布式存儲解決方案太示，提供對象控轿、文件和塊存儲。在每個(gè)存儲節(jié)點(diǎn)上财饥，將找到Ceph存儲對象的文件系統(tǒng)和Ceph OSD（對象存儲守護(hù)程序）進(jìn)程墩衙。在Ceph集群上务嫡，還存在Ceph MON（監(jiān)控）守護(hù)程序，它們確保Ceph集群保持高可用性漆改。

更多Ceph介紹參考：https://www.cnblogs.com/itzgr/category/1382602.html

1.2 Rook簡介

Rook 是一個(gè)開源的cloud-native storage編排, 提供平臺和框架心铃；為各種存儲解決方案提供平臺、框架和支持籽懦，以便與云原生環(huán)境本地集成于个。目前主要專用于Cloud-Native環(huán)境的文件、塊暮顺、對象存儲服務(wù)厅篓。它實(shí)現(xiàn)了一個(gè)自我管理的、自我擴(kuò)容的捶码、自我修復(fù)的分布式存儲服務(wù)羽氮。

Rook支持自動(dòng)部署、啟動(dòng)惫恼、配置档押、分配（provisioning）、擴(kuò)容/縮容祈纯、升級令宿、遷移、災(zāi)難恢復(fù)腕窥、監(jiān)控粒没，以及資源管理。為了實(shí)現(xiàn)所有這些功能簇爆，Rook依賴底層的容器編排平臺癞松，例如 kubernetes、CoreOS 等入蛆。响蓉。

Rook 目前支持Ceph、NFS哨毁、Minio Object Store枫甲、Edegefs、Cassandra、CockroachDB 存儲的搭建言秸。

Rook機(jī)制：

• Rook 提供了卷插件软能，來擴(kuò)展了 K8S 的存儲系統(tǒng)，使用 Kubelet 代理程序 Pod 可以掛載 Rook 管理的塊設(shè)備和文件系統(tǒng)举畸。
• Rook Operator 負(fù)責(zé)啟動(dòng)并監(jiān)控整個(gè)底層存儲系統(tǒng)，例如 Ceph Pod凳枝、Ceph OSD 等抄沮，同時(shí)它還管理 CRD、對象存儲岖瑰、文件系統(tǒng)叛买。
• Rook Agent 代理部署在 K8S 每個(gè)節(jié)點(diǎn)上以 Pod 容器運(yùn)行，每個(gè)代理 Pod 都配置一個(gè) Flexvolume 驅(qū)動(dòng)蹋订，該驅(qū)動(dòng)主要用來跟 K8S 的卷控制框架集成起來率挣，每個(gè)節(jié)點(diǎn)上的相關(guān)的操作，例如添加存儲設(shè)備露戒、掛載椒功、格式化、刪除存儲等操作智什，都有該代理來完成动漾。

更多參考如下官網(wǎng)：https://rook.io;https://ceph.com/

1.3 Rook架構(gòu)

Rook架構(gòu)如下：
Kubernetes集成Rook架構(gòu)如下：

二 Rook部署

2.1 前期規(guī)劃

裸磁盤規(guī)劃

2.2 獲取YAML 拉取項(xiàng)目

#外網(wǎng)速度較慢 建議提前下載好
git clone --single-branch --branch v1.5.1 https://github.com/rook/rook.git

2.3 部署Rook Operator

cd rook/cluster/examples/kubernetes/ceph
kubectl create -f crds.yaml -f common.yaml -f operator.yaml

2.4 獲取鏡像

由于鏡像默認(rèn)采用國外的鏡像  直接運(yùn)行yaml 會因下載速度過慢導(dǎo)致無法成功創(chuàng)建
建議先從阿里云下載 然后再打上官方的tag
# 拉取鏡像
docker pull ceph/ceph:v15.2.5
docker pull rook/ceph:v1.5.1
docker pull registry.aliyuncs.com/it00021hot/cephcsi:v3.1.2
docker pull registry.aliyuncs.com/it00021hot/csi-node-driver-registrar:v2.0.1
docker pull registry.aliyuncs.com/it00021hot/csi-attacher:v3.0.0
docker pull registry.aliyuncs.com/it00021hot/csi-provisioner:v2.0.0
docker pull registry.aliyuncs.com/it00021hot/csi-snapshotter:v3.0.0
docker pull registry.aliyuncs.com/it00021hot/csi-resizer:v1.0.0

# 設(shè)置tag
docker tag registry.aliyuncs.com/it00021hot/csi-snapshotter:v3.0.0 k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.0
docker tag registry.aliyuncs.com/it00021hot/csi-resizer:v1.0.0 k8s.gcr.io/sig-storage/csi-resizer:v1.0.0
docker tag registry.aliyuncs.com/it00021hot/cephcsi:v3.1.2 quay.io/cephcsi/cephcsi:v3.1.2
docker tag registry.aliyuncs.com/it00021hot/csi-node-driver-registrar:v2.0.1 k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.0.1
docker tag registry.aliyuncs.com/it00021hot/csi-attacher:v3.0.0 k8s.gcr.io/sig-storage/csi-attacher:v3.0.0
docker tag registry.aliyuncs.com/it00021hot/csi-provisioner:v2.0.0 k8s.gcr.io/sig-storage/csi-provisioner:v2.0.0
# 保存鏡像
docker save \
ceph/ceph:v15.2.5 \
rook/ceph:v1.5.1  \
k8s.gcr.io/sig-storage/csi-snapshotter:v3.0.0 \
k8s.gcr.io/sig-storage/csi-resizer:v1.0.0 \
quay.io/cephcsi/cephcsi:v3.1.2 \
k8s.gcr.io/sig-storage/csi-attacher:v3.0.0 \
k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.0.1 \
k8s.gcr.io/sig-storage/csi-provisioner:v2.0.0 | gzip -1 > rook.tar
#分發(fā)鏡像到個(gè)節(jié)點(diǎn)
yum install -y sshpass
echo 'StrictHostKeyChecking no'>>/etc/ssh/ssh_config
export SSHPASS='password' # ssh認(rèn)證密碼
export ALL_IPS=(192.168.12.88 192.168.12.89 192.168.12.90 192.168.12.91 192.168.12.92 192.168.12.93)
export TAR_NAME=rook.tar
for NODE in ${ALL_IPS[*]} ; do
    echo ">>>>>${NODE}"
    sshpass -e scp ${TAR_NAME} root@"${NODE}":/root
    sshpass -e ssh root@"${NODE}" "docker load -i ${TAR_NAME} && rm -rf ${TAR_NAME} "
done

2.5 配置cluster

vi cluster.yaml
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion:
    image: ceph/ceph:v15.2.5
    allowUnsupported: false
  dataDirHostPath: /var/lib/rook
  skipUpgradeChecks: false
  mon:
    count: 3
    allowMultiplePerNode: false
  dashboard:
    enabled: true
    ssl: true #ssl開關(guān)
  monitoring:
    enabled: false
    rulesNamespace: rook-ceph
  network:
    hostNetwork: false
#  rbdMirroring:  #會報(bào)錯(cuò) 故而注釋
#    workers: 0
  placement:                           #配置特定節(jié)點(diǎn)親和力保證Node作為存儲節(jié)點(diǎn)
#    all:
#      nodeAffinity:
#        requiredDuringSchedulingIgnoredDuringExecution:
#          nodeSelectorTerms:
#          - matchExpressions:
#            - key: role
#              operator: In
#              values:
#              - storage-node
#      tolerations:
#      - key: storage-node
#        operator: Exists
    mon:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: ceph-mon
              operator: In
              values:
              - enabled
      tolerations:
      - key: ceph-mon
        operator: Exists
    ods:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: ceph-osd
              operator: In
              values:
              - enabled
      tolerations:
      - key: ceph-osd
        operator: Exists
    mgr:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: ceph-mgr
              operator: In
              values:
              - enabled
      tolerations:
      - key: ceph-mgr
        operator: Exists
  annotations:
  resources:
  removeOSDsIfOutAndSafeToRemove: false
  storage:
    useAllNodes: false                  #關(guān)閉使用所有Node
    useAllDevices: false                #關(guān)閉使用所有設(shè)備
    deviceFilter: sdb
    config:
        metadataDevice:
        databaseSizeMB: "1024"
        journalSizeMB: "1024"
    nodes:
    - name: "k8snode01"                 #指定存儲節(jié)點(diǎn)主機(jī)
      config:
        storeType: bluestore    #指定類型為裸磁盤
      devices:
      - name: "sdb"                         #指定磁盤為sdb
    - name: "k8snode02"
      config:
        storeType: bluestore
      devices:
      - name: "sdb"
    - name: "k8snode03"
      config:
        storeType: bluestore
      devices:
      - name: "sdb"
  disruptionManagement:
    managePodBudgets: false
    osdMaintenanceTimeout: 30
    manageMachineDisruptionBudgets: false
    machineDisruptionBudgetNamespace: openshift-machine-api

2.6 部署cluster&ToolBox

kubectl create -f cluster.yaml
kubectl create -f toolbox.yaml

三 測試ROOK

3.1 查看ceph集群的狀態(tài)

滿足以下的條件 被視為健康：

所有mons應(yīng)該達(dá)到法定數(shù)量

mgr應(yīng)該是激活狀態(tài)

至少有一個(gè)OSD處于激活狀態(tài)

如果不是HEALTH_OK狀態(tài) 則應(yīng)該查看告警或者錯(cuò)誤信息

進(jìn)入tool容器

kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash
ceph status
  cluster:
    id:     be0ad378-ad31-4745-9e08-e72200021f37
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum a,b,c (age 41h)
    mgr: a(active, since 94s)
    mds: myfs:1 {0=myfs-a=up:active} 1 up:standby-replay
    osd: 3 osds: 3 up (since 41h), 3 in (since 41h)
 
  task status:
    scrub status:
        mds.myfs-a: idle
        mds.myfs-b: idle
 
  data:
    pools:   4 pools, 97 pgs
    objects: 31 objects, 4.9 KiB
    usage:   3.0 GiB used, 147 GiB / 150 GiB avail
    pgs:     97 active+clean
 
  io:
    client:   1.3 KiB/s rd, 170 B/s wr, 2 op/s rd, 0 op/s wr
#ceph osd status
#ceph osd df
#ceph osd utilization
#ceph osd pool stats
#ceph osd tree
#ceph pg sta
#ceph df
#rados df

3.2 ceph 集群dashboard

#vi dashboard-external-https.yaml
apiVersion: v1
kind: Service
metadata:
  name: rook-ceph-mgr-dashboard-external-https
  namespace: rook-ceph
  labels:
    app: rook-ceph-mgr
    rook_cluster: rook-ceph
spec:
  ports:
  - name: dashboard
    port: 8443
    protocol: TCP
    targetPort: 8443
  selector:
    app: rook-ceph-mgr
    rook_cluster: rook-ceph
  sessionAffinity: None
  type: NodePort

kubectl create -f dashboard-external-https.yaml

登錄 dashboard 需要安全訪問。Rook 在運(yùn)行 Rook Ceph 集群的名稱空間中創(chuàng)建一個(gè)默認(rèn)用戶荠锭，admin 并生成一個(gè)稱為的秘密rook-ceph-dashboard-admin-password

要檢索生成的密碼旱眯，可以運(yùn)行以下命令：

kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo

[root@k8smaster01 ~]# kubectl get svc -n rook-ceph | grep dashboard-external-https
rook-ceph-mgr-dashboard-external-https   NodePort    10.111.29.80    <none>        8443:32477/TCP      41h

https://192.168.12.91:32477/

[圖片上傳失敗...(image-c30571-1610011437134)]

3.3 實(shí)現(xiàn)在k8s宿主機(jī)對rook-ceph集群的簡單查看

3.3.1 復(fù)制key和config

kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') cat /etc/ceph/ceph.conf > /etc/ceph/ceph.conf
kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') cat /etc/ceph/keyring > /etc/ceph/keyring

3.3.2 配置ceph的repo源

[root@k8smaster01 ceph]# tee /etc/yum.repos.d/ceph.repo <<-'EOF'
[Ceph]
name=Ceph packages for $basearch
baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/$basearch
enabled=1
gpgcheck=0
type=rpm-md
gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc
priority=1

[Ceph-noarch]
name=Ceph noarch packages
baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/noarch
enabled=1
gpgcheck=0
type=rpm-md
gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc
priority=1

[ceph-source]
name=Ceph source packages
baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/SRPMS
enabled=1
gpgcheck=0
type=rpm-md
gpgkey=https://mirrors.aliyun.com/ceph/keys/release.asc
priority=1
EOF

3.3.3 安裝客戶端

yum -y install ceph-common ceph-fuse
#之后便可以 直接在k8smaster集群 運(yùn)行查詢的命令
[root@k8smaster01 ~]# ceph status
  cluster:
    id:     be0ad378-ad31-4745-9e08-e72200021f37
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum a,b,c (age 42h)
    mgr: a(active, since 100m)
    mds: myfs:1 {0=myfs-a=up:active} 1 up:standby-replay
    osd: 3 osds: 3 up (since 43h), 3 in (since 43h)
 
  task status:
    scrub status:
        mds.myfs-a: idle
        mds.myfs-b: idle
 
  data:
    pools:   4 pools, 97 pgs
    objects: 31 objects, 4.9 KiB
    usage:   3.0 GiB used, 147 GiB / 150 GiB avail
    pgs:     97 active+clean
 
  io:
    client:   1.2 KiB/s rd, 2 op/s rd, 0 op/s wr

3.4 塊設(shè)備創(chuàng)建及測試

3.4.1 創(chuàng)建wordpress 進(jìn)行測試

cd /tmp/rook/cluster/examples/kubernetes/ceph/csi/rbd
sed -i 's/failureDomain: host/failureDomain: osd/g' storageclass.yaml
kubectl apply -f storageclass.yaml
kubectl get sc -n rook-ceph

[圖片上傳失敗...(image-3e60c8-1610011437134)]

# 創(chuàng)建 Wordpress 進(jìn)行測試
cd /tmp/rook/cluster/examples/kubernetes
sed -i 's|mysql:5.6|registry.cn-hangzhou.aliyuncs.com/vinc-auto/mysql:5.6|g' mysql.yaml
sed -i 's|wordpress:4.6.1-apache|registry.cn-hangzhou.aliyuncs.com/vinc-auto/wordpress:4.6.1-apache|g' wordpress.yaml
sed -i 's/LoadBalancer/NodePort/g' wordpress.yaml
kubectl create -f mysql.yaml
kubectl create -f wordpress.yaml

kubectl get pvc -o wide
kubectl get deploy -o wide
kubectl get pod -o wide
kubectl get service -o wide
kubectl get svc wordpress -o wide
# 瀏覽器訪問 wordpress 進(jìn)行部署

# 查看Ceph集群中的相關(guān)數(shù)據(jù)
kubectl -n rook-ceph get pod -l "app=rook-ceph-tools"
NAME=$(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}')
kubectl -n rook-ceph exec -it ${NAME} sh
ceph osd pool stats
rbd ls -p replicapool
rbd info replicapool/'csi-vol-a15dc75d-69a0-11ea-a3b7-2ef116ca54b6'
rbd info replicapool/'csi-vol-a18385ed-69a0-11ea-a3b7-2ef116ca54b6'
exit

# 刪除測試環(huán)境
cd /tmp/rook/cluster/examples/kubernetes
kubectl delete -f wordpress.yaml
kubectl delete -f mysql.yaml
kubectl delete -n rook-ceph cephblockpools.ceph.rook.io replicapool
kubectl delete storageclass rook-ceph-block

[圖片上傳失敗...(image-8f2c3-1610011437134)]

[圖片上傳失敗...(image-17f3b6-1610011437134)]

3.5 CephFS創(chuàng)建和測試

• CephFS 允許用戶掛載一個(gè)兼容posix的共享目錄到多個(gè)主機(jī)，該存儲和NFS共享存儲以及CIFS共享目錄相似

# filesystem.yaml: 3份副本的生產(chǎn)環(huán)境配置证九，需要至少3個(gè)節(jié)點(diǎn)
# filesystem-ec.yaml: 糾錯(cuò)碼的生產(chǎn)環(huán)境配置删豺，需要至少3個(gè)節(jié)點(diǎn)
# filesystem-test.yaml: 1份副本的測試環(huán)境，只需要一個(gè)節(jié)點(diǎn)
cd /tmp/rook/cluster/examples/kubernetes/ceph
sed -i 's/failureDomain: host/failureDomain: osd/g' filesystem.yaml
kubectl apply -f filesystem.yaml
kubectl -n rook-ceph get pod -l app=rook-ceph-mds

# 簡單查看
kubectl -n rook-ceph get pod -l "app=rook-ceph-tools"
NAME=$(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}')
kubectl -n rook-ceph exec -it ${NAME} sh
ceph status
ceph osd lspools
ceph mds stat
ceph fs ls
exit

• 如果要使用CephFS愧怜，則必須先創(chuàng)建對應(yīng)的storageclass

cd /tmp/rook/cluster/examples/kubernetes/ceph/csi/cephfs/
kubectl apply -f storageclass.yaml

[圖片上傳失敗...(image-dfd0a5-1610011437134)]

測試

# 部署多個(gè)私有倉庫共享同一個(gè)數(shù)據(jù)目錄進(jìn)行測試
docker pull registry:2
kubectl create -f kube-registry.yaml
# 在kube-system下創(chuàng)建了一個(gè)deployment作為私有倉庫
# 將目錄/var/lib/registry掛接到CephFS呀页，并且是3個(gè)副本共享掛載的
kubectl get pod -n kube-system -l k8s-app=kube-registry -o wide
kubectl -n kube-system exec -it kube-registry-65df7d789d-9bwzn sh
df -hP|grep '/var/lib/registry'
cd /var/lib/registry
touch abc
exit
kubectl -n kube-system exec -it kube-registry-65df7d789d-sf55j ls /var/lib/registry

# 刪除環(huán)境
cd /tmp/rook/cluster/examples/kubernetes/ceph/csi/cephfs/
kubectl delete -f kube-registry.yaml
kubectl delete -f storageclass.yaml
cd /tmp/rook/cluster/examples/kubernetes/ceph
kubectl delete -f filesystem.yaml

3.6 對象存儲創(chuàng)建和測試

3.6.1 創(chuàng)建CephObjectStore

在提供（object）對象存儲之前，需要先創(chuàng)建相應(yīng)的支持叫搁，使用如下官方提供的默認(rèn)yaml可部署對象存儲的CephObjectStore赔桌。

#kubectl create -f object.yaml
apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
  name: my-store
  namespace: rook-ceph
spec:
  metadataPool:
    failureDomain: host
    replicated:
      size: 3
  dataPool:
    failureDomain: host
    replicated:
      size: 3
  preservePoolsOnDelete: false
  gateway:
    type: s3
    sslCertificateRef:
    port: 80
    securePort:
    instances: 1
    placement:
    annotations:
    resources:

kubectl -n rook-ceph get pod -l app=rook-ceph-rgw #查看部署完成會創(chuàng)建rgw的Pod

3.6.2 創(chuàng)建StorageClass

使用如下官方提供的默認(rèn)yaml可部署對象存儲的StorageClass。

#kubectl create -f storageclass-bucket-delete.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: rook-ceph-delete-bucket
provisioner: ceph.rook.io/bucket
reclaimPolicy: Delete
parameters:
  objectStoreName: my-store
  objectStoreNamespace: rook-ceph
  region: us-east-1

kubectl get sc #查看StorageClass 是否成功創(chuàng)建

3.6.3 創(chuàng)建bucket

# kubectl create -f object-bucket-claim-delete.yaml
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: ceph-delete-bucket
spec:
  generateBucketName: ceph-bkt
  storageClassName: rook-ceph-delete-bucket

3.6.4 設(shè)置對象存儲訪問信息

kubectl -n default get cm ceph-delete-bucket -o yaml | grep BUCKET_HOST | awk '{print $2}'
rook-ceph-rgw-my-store.rook-ceph
kubectl -n rook-ceph get svc rook-ceph-rgw-my-store
export AWS_HOST=$(kubectl -n default get cm ceph-delete-bucket -o yaml | grep BUCKET_HOST | awk '{print $2}')
export AWS_ACCESS_KEY_ID=$(kubectl -n default get secret ceph-delete-bucket -o yaml | grep AWS_ACCESS_KEY_ID | awk '{print $2}' | base64 --decode)
export AWS_SECRET_ACCESS_KEY=$(kubectl -n default get secret ceph-delete-bucket -o yaml | grep AWS_SECRET_ACCESS_KEY | awk '{print $2}' | base64 --decode)
export AWS_ENDPOINT='10.102.165.187'
echo '10.102.165.187 rook-ceph-rgw-my-store.rook-ceph' >> /etc/hosts

3.6.5 測試訪問

radosgw-admin bucket list           #查看bucket
yum --assumeyes install s3cmd           #安裝S3客戶端
echo "Hello Rook" > /tmp/rookObj        #創(chuàng)建測試文件
s3cmd put /tmp/rookObj --no-ssl --host=${AWS_HOST} --host-bucket= s3://ceph-bkt-377bf96f-aea8-4838-82bc-2cb2c16cccfb/test.txt                   #測試上傳至bucket