openshift netapp
什么是NetApp SolidFire?
NetApp公司是一家存儲(chǔ)和數(shù)據(jù)管理公司,主要提供軟件，系統(tǒng)和服務(wù)來(lái)管理和存儲(chǔ)數(shù)據(jù)筐咧，包括其專有的Data ONTAP操作系統(tǒng)。NetApp公司主要向全球數(shù)據(jù)密集型企業(yè)提供統(tǒng)一存儲(chǔ)解決方案其 Data ONTAP是全球首屈一指的存儲(chǔ)操作系統(tǒng)，公司存儲(chǔ)解決方案涵蓋了專業(yè)化的硬件、軟件和服務(wù)分预，為開(kāi)放網(wǎng)絡(luò)環(huán)境提供了無(wú)縫的存儲(chǔ)管理⌒胶矗——(來(lái)自搜狗百科)
SolidFire成立于2010年笼痹，是一家全閃存陣列的存儲(chǔ)廠商，其存儲(chǔ)控制器基于標(biāo)準(zhǔn)的x86服務(wù)器酪穿，最大可擴(kuò)展到100個(gè)節(jié)點(diǎn)凳干，2015年12月，SolidFire被NetApp收購(gòu)被济；2017年6月救赐，NetApp基于SolidFire推出超融合一體機(jī)。
SolidFire提供分布式塊存儲(chǔ)只磷，類似于ceph rbd经磅，非常靈活，支持動(dòng)態(tài)擴(kuò)縮容喳瓣，具有良好的性能馋贤。同時(shí)具有很多企業(yè)特性：如快照，組快照畏陕，豐富的API，非常靈活的QOS配置等仿滔。
Solidfire的Reporting界面
什么是Trident?
NetApp是CNCF的金牌會(huì)員惠毁，它開(kāi)發(fā)的Trident是一款開(kāi)源存儲(chǔ)配置程序和流程編排程序。
在沒(méi)有Trident的環(huán)境下崎页，K8s/Openshift環(huán)境要使用NetApp存儲(chǔ)鞠绰，就需要，手動(dòng)在NetApp控制臺(tái)上創(chuàng)建volume飒焦，并設(shè)置創(chuàng)建PV蜈膨，再創(chuàng)建PVC屿笼。這些過(guò)程需要在兩個(gè)平臺(tái)切換操作，而且很麻煩翁巍。
部署了Trident后驴一，配置好相應(yīng)的storageclass，K8s/Openshift平臺(tái)就可以直接通過(guò)storageclass動(dòng)態(tài)自動(dòng)創(chuàng)建PVC灶壶。K8s/Openshift平臺(tái)通過(guò)Trident控制器調(diào)用NetApp設(shè)備的API從而達(dá)到控制NetApp設(shè)備目的肝断，如創(chuàng)建volume，并自動(dòng)創(chuàng)建PV驰凛，及PVC胸懈，進(jìn)而讓Pod能夠使用，此過(guò)程是自動(dòng)的恰响，對(duì)平臺(tái)使用者是無(wú)感知的趣钱。
Trident本身也是以Pod的形式在K8s/Openshift平臺(tái)上運(yùn)行的。
trident架構(gòu)圖
Openshift上部署與使用Trident
準(zhǔn)備工作
- 用system:admin登錄集群
$ oc login -u system:admin
- 集群能夠訪問(wèn)SolidFire機(jī)器的MVIP(管理VIP)及SVIP(存儲(chǔ)SVIP)
$ telnet $MVIP 443
$ telnet $SVIP 3260
- 安裝基本包
$ ansible all -m package -a 'name=lsscsi,iscsi-initiator-utils,sg3_utils,device-mapper-multipath state=present'
$ ansible all -m shell -a 'mpathconf --enable --with_multipathd y'
$ ansible all -m service -a 'name=iscsid enabled=true state=started'
$ ansible all -m service -a 'name=multipathd enabled=true state=started'
$ ansible all -m service -a 'name=iscsi enabled=true state=started'
部署
- 下載安裝文件胚宦，并解壓
$ wget https://github.com/NetApp/trident/releases/download/v18.10.0/trident-installer-18.10.0.tar.gz
$ tar -xf trident-installer-18.10.0.tar.gz
$ cd trident-installer
- 配置安裝backend.json文件
$ cp sample-input/backend-solidfire.json setup/backend.json
# 修改里面的配置
$ cat setup/backend.json
{
"version": 1,
"storageDriverName": "solidfire-san",
"Endpoint": "https://{{用戶名}}:{{密碼}}@{{管理VIP}}/json-rpc/11.0",
"SVIP": "{{存儲(chǔ)VIP}}:3260",
"TenantName": "trident",
"UseCHAP": true,
"InitiatorIFace": "default",
"Types": [{"Type": "Bronze", "Qos": {"minIOPS": 1000, "maxIOPS": 2000, "burstIOPS": 4000}},
{"Type": "Silver", "Qos": {"minIOPS": 4000, "maxIOPS": 6000, "burstIOPS": 8000}},
{"Type": "Gold", "Qos": {"minIOPS": 6000, "maxIOPS": 8000, "burstIOPS": 10000}}]
}
- 創(chuàng)建trident project
$ oc new-project trident
- 安裝檢查
$ ./tridentctl install --dry-run -n trident
這個(gè)步驟會(huì)模擬安裝過(guò)程進(jìn)行執(zhí)行一遍羔挡，并會(huì)刪除所有資源。通過(guò)模擬對(duì)整個(gè)環(huán)境進(jìn)行全面的檢測(cè)间唉。以下是執(zhí)行的日志
[root@master02 trident-installer]# ./tridentctl install --dry-run -n trident -d
DEBU Initialized logging. logLevel=debug
DEBU Running outside a pod, creating CLI-based client.
DEBU Initialized Kubernetes CLI client. cli=oc flavor=openshift namespace=trident version=1.11.0+d4cacc0
DEBU Validated installation environment. installationNamespace=trident kubernetesVersion=
DEBU Deleted Kubernetes configmap. label="app=trident-installer.netapp.io" namespace=trident
DEBU Namespace exists. namespace=trident
DEBU Deleted Kubernetes object by YAML.
DEBU Deleted installer cluster role binding.
DEBU Deleted Kubernetes object by YAML.
DEBU Deleted installer cluster role.
DEBU Deleted Kubernetes object by YAML.
DEBU Deleted installer service account.
DEBU Removed security context constraint user. scc=privileged user=trident-installer
DEBU Created Kubernetes object by YAML.
INFO Created installer service account. serviceaccount=trident-installer
DEBU Created Kubernetes object by YAML.
INFO Created installer cluster role. clusterrole=trident-installer
DEBU Created Kubernetes object by YAML.
INFO Created installer cluster role binding. clusterrolebinding=trident-installer
INFO Added security context constraint user. scc=privileged user=trident-installer
DEBU Created Kubernetes configmap from directory. label="app=trident-installer.netapp.io" name=trident-installer namespace=trident path=/root/trident-installer/setup
INFO Created installer configmap. configmap=trident-installer
DEBU Created Kubernetes object by YAML.
INFO Created installer pod. pod=trident-installer
INFO Waiting for Trident installer pod to start.
DEBU Trident installer pod not yet started, waiting. increment=280.357322ms message="pod not yet started (Pending)"
DEBU Trident installer pod not yet started, waiting. increment=523.702816ms message="pod not yet started (Pending)"
DEBU Trident installer pod not yet started, waiting. increment=914.246751ms message="pod not yet started (Pending)"
DEBU Trident installer pod not yet started, waiting. increment=1.111778662s message="pod not yet started (Pending)"
DEBU Pod started. phase=Succeeded
INFO Trident installer pod started. namespace=trident pod=trident-installer
DEBU Getting logs. cmd="oc --namespace=trident logs trident-installer -f"
DEBU Initialized logging. logLevel=debug
DEBU Running in a pod, creating API-based client. namespace=trident
DEBU Initialized Kubernetes API client. cli=oc flavor=openshift namespace=trident version=v1.11.0+d4cacc0
DEBU Validated installation environment. installationNamespace=trident kubernetesVersion=v1.11.0+d4cacc0
DEBU Parsed requested volume size. quantity=2Gi
DEBU Dumping RBAC fields. ucpBearerToken= ucpHost= useKubernetesRBAC=true
DEBU Namespace exists. namespace=trident
DEBU PVC does not exist. pvc=trident
DEBU PV does not exist. pv=trident
INFO Starting storage driver. backend=/setup/backend.json
DEBU config: {"Endpoint":"https://admin:root1234@99.248.106.82/json-rpc/11.0","InitiatorIFace":"default","SVIP":"99.248.82.55:3260","TenantName":"trident","Types":[{"Qos":{"burstIOPS":4000,"maxIOPS":2000,"minIOPS":1000},"Type":"Bronze"},{"Qos":{"burstIOPS":8000,"maxIOPS":6000,"minIOPS":4000},"Type":"Silver"},{"Qos":{"burstIOPS":10000,"maxIOPS":8000,"minIOPS":6000},"Type":"Gold"}],"UseCHAP":true,"storageDriverName":"solidfire-san","version":1}
DEBU Storage prefix is absent, will use default prefix.
DEBU Parsed commonConfig: {Version:1 StorageDriverName:solidfire-san BackendName: Debug:false DebugTraceFlags:map[] DisableDelete:false StoragePrefixRaw:[] StoragePrefix:<nil> SerialNumbers:[] DriverContext: LimitVolumeSize:}
DEBU Initializing storage driver. driver=solidfire-san
DEBU Configuration defaults Size=1G StoragePrefix= UseCHAP=true
DEBU Parsed into solidfireConfig DisableDelete=false StorageDriverName=solidfire-san Version=1
DEBU Decoded to &{CommonStorageDriverConfig:0xc42064e0a0 TenantName:trident EndPoint:https://admin:root1234@99.248.106.82/json-rpc/11.0 SVIP:99.248.82.55:3260 InitiatorIFace:default Types:0xc4206d26e0 LegacyNamePrefix: AccessGroups:[] UseCHAP:true DefaultBlockSize:0 SolidfireStorageDriverConfigDefaults:{CommonStorageDriverConfigDefaults:{Size:1G}}}
DEBU Set default block size. defaultBlockSize=512
DEBU Using SF API version from config file. version=11.0
DEBU Initializing SolidFire API client. cfg="{trident https://admin:root1234@99.248.106.82/json-rpc/11.0 99.248.82.55:3260 default 0xc4206d26e0 [] 512 map[]}" endpoint="https://admin:root1234@99.248.106.82/json-rpc/11.0" svip="99.248.82.55:3260"
ERRO Error detected in API response. ID=637 code=500 message=xUnknownAccount name=xUnknownAccount
DEBU Account not found, creating. error="device API error: xUnknownAccount" tenantName=trident
DEBU Created account. accountID=0 tenantName=trident
DEBU SolidFire driver initialized. AccountID=2 InitiatorIFace=default
DEBU Using CHAP, skipped Volume Access Group logic. AccessGroups="[]" SVIP="99.248.82.55:3260" UseCHAP=true driver=solidfire-san
DEBU Added pool for SolidFire backend. attributes="map[media:{Offers: ssd} IOPS:{Min: 1000, Max: 2000} snapshots:{Offer: true} clones:{Offer: true} encryption:{Offer: false} provisioningType:{Offers: thin} backendType:{Offers: solidfire-san}]" backend=solidfire_99.248.82.55 pool=Bronze
DEBU Added pool for SolidFire backend. attributes="map[clones:{Offer: true} encryption:{Offer: false} provisioningType:{Offers: thin} backendType:{Offers: solidfire-san} media:{Offers: ssd} IOPS:{Min: 4000, Max: 6000} snapshots:{Offer: true}]" backend=solidfire_99.248.82.55 pool=Silver
DEBU Added pool for SolidFire backend. attributes="map[snapshots:{Offer: true} clones:{Offer: true} encryption:{Offer: false} provisioningType:{Offers: thin} backendType:{Offers: solidfire-san} media:{Offers: ssd} IOPS:{Min: 6000, Max: 8000}]" backend=solidfire_99.248.82.55 pool=Gold
DEBU Storage driver initialized. driver=solidfire-san
INFO Storage driver loaded. driver=solidfire-san
INFO Dry run completed, no problems found.
DEBU Received EOF from pod logs. container= pod=trident-installer
INFO Waiting for Trident installer pod to finish.
DEBU Pod finished. phase=Succeeded
INFO Trident installer pod finished. namespace=trident pod=trident-installer
DEBU Deleted Kubernetes pod. label="app=trident-installer.netapp.io" namespace=trident
INFO Deleted installer pod. pod=trident-installer
DEBU Deleted Kubernetes configmap. label="app=trident-installer.netapp.io" namespace=trident
INFO Deleted installer configmap. configmap=trident-installer
INFO In-cluster installation completed.
DEBU Deleted Kubernetes object by YAML.
INFO Deleted installer cluster role binding.
DEBU Deleted Kubernetes object by YAML.
INFO Deleted installer cluster role.
DEBU Deleted Kubernetes object by YAML.
INFO Deleted installer service account.
INFO Removed security context constraint user. scc=privileged user=trident-installer
- 正式安裝
$ ./tridentctl install -n trident
該步驟是真正的執(zhí)行绞灼。會(huì)創(chuàng)建serviceaccount, clusterrolebinding,configmap配置，trident-install pod（該pod在部署完trident deployment后會(huì)刪除）等呈野， 并會(huì)創(chuàng)建一個(gè)pv與trident pvc進(jìn)行初始化操作低矮，最終會(huì)創(chuàng)建trident deployment，完成trident的安裝被冒。
- trident的安裝支持自定義一些配置军掂。
- --etcd-image可指定etcd的鏡像（默認(rèn)是quay.io/coreos/etcd，下載會(huì)比較慢）
- --trident-image指定trident的鏡像
- --volume-size指定trident持久存儲(chǔ)的大凶虻俊（默認(rèn)為2GiB）
- --volume-name指定volume名字（默認(rèn)是etcd-vol）
- --pv指定pv名字（默認(rèn)是trident）
- --pvc指定pvc名字（默認(rèn)是trident）
- --generate-custom-yaml將使用的所有配置進(jìn)行導(dǎo)出到一個(gè)setup文件夾蝗锥，不會(huì)對(duì)集群做任何操作
- --use-custom-yaml安裝setup下的所有yaml文件進(jìn)行部署trident
以下是執(zhí)行的日志
[root@master02 trident-installer]# ./tridentctl install -n trident
INFO Created installer service account. serviceaccount=trident-installer
INFO Created installer cluster role. clusterrole=trident-installer
INFO Created installer cluster role binding. clusterrolebinding=trident-installer
INFO Added security context constraint user. scc=privileged user=trident-installer
INFO Created installer configmap. configmap=trident-installer
INFO Created installer pod. pod=trident-installer
INFO Waiting for Trident installer pod to start.
INFO Trident installer pod started. namespace=trident pod=trident-installer
INFO Starting storage driver. backend=/setup/backend.json
INFO Storage driver loaded. driver=solidfire-san
INFO Starting Trident installation. namespace=trident
INFO Created service account.
INFO Created cluster role.
INFO Created cluster role binding.
INFO Added security context constraint user. scc=anyuid user=trident
INFO Created PVC.
INFO Controller serial numbers. serialNumbers="4BZXJB2,85Q8JB2,4BXXJB2,4BXTJB2"
INFO Created iSCSI CHAP secret. secret=trident-chap-solidfire-99-248-82-55-trident
INFO Created PV. pv=trident
INFO Waiting for PVC to be bound. pvc=trident
INFO Created Trident deployment.
INFO Waiting for Trident pod to start.
INFO Trident pod started. namespace=trident pod=trident-57ccdff48f-gtflx
INFO Waiting for Trident REST interface.
INFO Trident REST interface is up. version=18.10.0
INFO Trident installation succeeded.
INFO Waiting for Trident installer pod to finish.
INFO Trident installer pod finished. namespace=trident pod=trident-installer
INFO Deleted installer pod. pod=trident-installer
INFO Deleted installer configmap. configmap=trident-installer
INFO In-cluster installation completed.
INFO Deleted installer cluster role binding.
INFO Deleted installer cluster role.
INFO Deleted installer service account.
INFO Removed security context constraint user. scc=privileged user=trident-installer
- 添加第一個(gè)backend
執(zhí)行完install后，trident并不會(huì)安裝之前配置的backend率触，需要另外再單獨(dú)添加终议。（個(gè)人覺(jué)得netapp這點(diǎn)考慮得有點(diǎn)多余，因?yàn)閐ry-run的時(shí)候已經(jīng)對(duì)backend.json作了檢查了葱蝗，直接install將它添加上豈不是更方便）
$ ./tridentctl -n trident create backend -f setup/backend.json
$ ./tridentctl -n trident get backend
+------------------------+----------------+--------+---------+
| NAME | STORAGE DRIVER | ONLINE | VOLUMES |
+------------------------+----------------+--------+---------+
| solidfire_99.248.82.55 | solidfire-san | true | 0 |
+------------------------+----------------+--------+---------+
- 添加基本的storageclass
將sample-input/storage-class-basic.yaml.templ中的BACKEND_TYPE用指定的backend中的STORAGE DRIVER值替換（此例中為solidfire-san）
$ cat sample-input/storage-class-basic.yaml.templ
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: basic
provisioner: netapp.io/trident
parameters:
backendType: "__BACKEND_TYPE__"
$ sed "s/__BACKEND_TYPE__/solidfire-san/" sample-input/storage-class-basic.yaml.templ | oc create -f -
- 根據(jù)backend中的Type創(chuàng)建對(duì)應(yīng)的storageclass
$ cat storage-class-gold.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: gold
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: netapp.io/trident
parameters:
storagePools: "solidfire_99.248.82.55:Gold" # solidfire_99.248.82.55為backend name;Gold為指定的Type
$ oc create -f storage-class-gold.yaml
查看當(dāng)前的storageclass
$ oc get sc
NAME PROVISIONER AGE
basic netapp.io/trident 2h
gold (default) netapp.io/trident 1h
使用：創(chuàng)建PVC
- 創(chuàng)建第一個(gè)PVC
$ cat test-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
volume.beta.kubernetes.io/storage-class: gold
volume.beta.kubernetes.io/storage-provisioner: netapp.io/trident
trident.netapp.io/reclaimPolicy: "Retain"
name: testpvc
namespace: test
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
$ oc create -f test-pvc.yaml
PVC創(chuàng)建的說(shuō)明:
- volume.beta.kubernetes.io/storage-class為10穴张，11步創(chuàng)建的storageclass
- volume.beta.kubernetes.io/storage-provisioner指定為netapp的trident
- trident.netapp.io/reclaimPolicy指定創(chuàng)建PV的reclaimPolicy,默認(rèn)為"Delete",支持"Delete"和"Retain"，不支持"Recycle"
- accessModes因SolidFire是塊存儲(chǔ)两曼，只支持ReadWriteOnce
SolidFire功能測(cè)試
快照恢復(fù)數(shù)據(jù)
創(chuàng)建快照
創(chuàng)建快照
基于已有快照恢復(fù)pvc數(shù)據(jù)
基于快照創(chuàng)建新的PVC
指定快照皂甘，創(chuàng)建新的存儲(chǔ)
基于已有快照恢復(fù)volume
查看新建的volume的IQN
查看新建的volume的IQN
基于新的volume創(chuàng)建PV
$ cat test-clone-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
pv.kubernetes.io/provisioned-by: netapp.io/trident
volume.beta.kubernetes.io/storage-class: gold
name: test-dd-testxx-volume
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 100Gi
iscsi:
chapAuthDiscovery: true
chapAuthSession: true
fsType: ext4
iqn: iqn.2010-01.com.solidfire:fs69.test-dd-testxx-volume.169
iscsiInterface: default
lun: 0
secretRef:
name: trident-chap-solidfire-99-248-82-55-trident
namespace: trident
targetPortal: 99.248.82.55:3260
persistentVolumeReclaimPolicy: Delete
storageClassName: gold
$ oc create -f test-clone-pv.yaml
創(chuàng)建pvc使用手動(dòng)創(chuàng)建的pv
$ cat test-clone-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test111x
namespace: test-dd
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
組快照
組快照與快照類似，不同之處悼凑，它把多個(gè)存儲(chǔ)卷在同一時(shí)間的數(shù)據(jù)做快照偿枕，從而避免數(shù)據(jù)不一致的情況璧瞬。同時(shí)在恢復(fù)的時(shí)候，也同時(shí)將備份時(shí)刻的數(shù)據(jù)進(jìn)行恢復(fù)渐夸。
組快照創(chuàng)建.png
克隆已有的pvc數(shù)據(jù)
添加annotations配置trident.netapp.io/cloneFromPVC: test-pvc,創(chuàng)建新的pvc基于已有的PVC test-pvc
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
trident.netapp.io/cloneFromPVC: test-pvc
name: test-clone-pvc
namespace: test-dd
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
storageClassName: gold
SolidFire性能測(cè)試
測(cè)試環(huán)境說(shuō)明
- openshift 3.11物理機(jī)部署：
3 Masters4 Nodes - SolidFire 4臺(tái)Node：型號(hào)SF9605
每臺(tái)Node上為10塊SSD盤(pán)嗤锉，每個(gè)Node的IOPS為5w，集群最高IOPS 20w - 每塊PV存儲(chǔ)設(shè)置為gold類型storageclass：
{"Type": "Gold", "Qos": {"minIOPS": 6000, "maxIOPS": 8000, "burstIOPS": 10000}}
dd測(cè)試
# 測(cè)試命令
$ dd if=/dev/zero of=/data/dd.test bs=4k count=200000 oflag=direct
- 單個(gè)pod捺萌，單個(gè)pv作dd命令測(cè)試
創(chuàng)建deployment進(jìn)行測(cè)試
$ cat test0-pvc.yaml
kind: PersistentVolumeClaim
metadata:
annotations:
volume.beta.kubernetes.io/storage-class: gold
volume.beta.kubernetes.io/storage-provisioner: netapp.io/trident
name: test0
namespace: test-dd
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
$ oc create -f test0-pvc.yaml ## 創(chuàng)建測(cè)試的存儲(chǔ)
$ cat dd.yaml
apiVersion: apps.openshift.io/v1
kind: DeploymentConfig
metadata:
labels:
run: ddtest
name: ddtest
spec:
replicas: 1
selector:
run: ddtest
strategy:
type: Recreate
template:
metadata:
labels:
run: ddtest
spec:
containers:
- command:
- /bin/bash
- '-c'
- |
#/bin/bash
dd if=/dev/zero of=/data/out.test1 bs=4k count=200000 oflag=direct
image: tools/iqperf:latest
imagePullPolicy: Always
name: ddtest
volumeMounts:
- mountPath: /data
name: volume-spq10
volumes:
- name: volume-spq10
persistentVolumeClaim:
claimName: test0
triggers:
- type: ConfigChange
$ oc create -f dd.yaml
在webconsole上查看日志如下
200000+0 records in
200000+0 records out
819200000 bytes (819 MB) copied, 68.8519 s, 11.9 MB/s
NetApp的管理平臺(tái)上查看集群IO狀態(tài)档冬，如圖（只需要看11:32時(shí)間以后部分）
單個(gè)pod，單個(gè)pv作dd命令測(cè)試.png
IOPS為2908
- 1個(gè)pod桃纯，1個(gè)pv酷誓，8個(gè)dd進(jìn)程
將1中的deploymentconfig中的command內(nèi)容更新為：
...
- command:
- '/bin/bash'
- '-c'
- |
#/bin/bash
for i in {1..8}
do
dd if=/dev/zero of=/data/dd.test$i bs=4k count=200000 oflag=direct &
done
sleep 1000000
...
1個(gè)pod，1個(gè)pv态坦，8個(gè)dd進(jìn)程.png
IOPS為10000
額外補(bǔ)充
- 1個(gè)pod盐数，1個(gè)pv，50->150個(gè)dd進(jìn)程伞梯，bs=512玫氢，在netapp控制臺(tái)將volume的max iops設(shè)置為200000，最終測(cè)試結(jié)果谜诫，該volume的寫(xiě)iops最大為40000漾峡。
也就是該版本solidfire下，單個(gè)volume最大寫(xiě)iops為4w喻旷。 - 同樣方法測(cè)試讀生逸，單個(gè)volume最大達(dá)到寫(xiě)iops為7.5w。
$ dd if=/data/out.test1 of=/dev/zero bs=512 count=20000000 iflag=direct
- 8個(gè)pod且预，8個(gè)pv同時(shí)使用dd命令測(cè)試
創(chuàng)建statefulset槽袄，設(shè)置volumeClaimTemplates批量創(chuàng)建存儲(chǔ)
$ cat dd-statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: testdd
namespace: test-dd
spec:
serviceName: testdd
replicas: 8
selector:
matchLabels:
app: testdd
template:
metadata:
labels:
app: testdd
spec:
terminationGracePeriodSeconds: 10
containers:
- name: testdd
containers:
- command:
- /bin/bash
- '-c'
- |
#!/bin/bash
dd if=/dev/zero of=/data/out.test1 bs=4k count=2000000 oflag=direct
image: 'harbor.apps.it.mbcloud.com/tools/iqperf:latest'
imagePullPolicy: Always
name: testdd
image: 'tools/dd:latest'
volumeMounts:
- name: data
mountPath: /data
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes:
- ReadWriteOnce
storageClassName: gold
resources:
requests:
storage: 100Gi
8個(gè)pod，8個(gè)pv同時(shí)使用dd命令測(cè)試.png
IOPS為33883
- 8個(gè)pod锋谐，8個(gè)pv同時(shí)每個(gè)pod啟用8個(gè)dd進(jìn)程遍尺，共64個(gè)dd進(jìn)程測(cè)試
更改3中statefulset的command命令如下：
...
- command:
- '/bin/bash'
- '-c'
- |
#/bin/bash
for i in {1..8}
do
dd if=/dev/zero of=/data/dd.test$i bs=4k count=200000 oflag=direct &
done
sleep 1000000
...
8個(gè)pod，8個(gè)pv同時(shí)每個(gè)pod啟用8個(gè)dd進(jìn)程涮拗，共64個(gè)dd進(jìn)程測(cè)試.png
IOPS為76832達(dá)到了gold Type下設(shè)置的IOPS上限
- 50個(gè)pod乾戏，50個(gè)pv同時(shí)每個(gè)pod啟用3個(gè)dd進(jìn)程，共150個(gè)dd進(jìn)程測(cè)試
圖片.png
此時(shí)單個(gè)PV存儲(chǔ)的詳情
圖片.png
IOPS為205545達(dá)到了gold Type下設(shè)置的IOPS上限
綜合結(jié)果如下：
| pod數(shù) | pv數(shù) | dd進(jìn)程數(shù) | IOPS |
|---|---|---|---|
| 1 | 1 | 1 | 2908 |
| 1 | 1 | 8 | 10000 |
| 8 | 8 | 8 | 33883 |
| 8 | 8 | 64 | 76832 |
| 50 | 50 | 150 | 205545 |
數(shù)據(jù)庫(kù)測(cè)試
測(cè)試工具mydbtest
測(cè)試配置
$ mysql -uapp -h172.30.213.17 -papp app -e "create table t_mytest(col1 int);"
$ cat test.conf
option
name app
loop 20000
user app/app@172.30.213.17:3306:app
declare
a int 10 30000
begin
#select * from t_mytest where col1 = :a; # 查詢
insert into t_mytest set col1 = :a; # 插入
end
執(zhí)行測(cè)試過(guò)程
./mydbtest_64.bin query=test.conf degree=40
執(zhí)行結(jié)果
# 插入數(shù)據(jù)
2019-01-23 17:59:35 Total tran=20000=312/s, qtps=40000=624/s, ela=64046 ms, avg=3202 us
Summary: SQL01 exec=800000, rows=0=0/e, avg=65 us
Summary: SQL02 exec=800000, rows=800000=100/e, avg=3135 us
Summary: exec=12307/s, qtps=24615/s
# 創(chuàng)建完索引后多搀，讀數(shù)據(jù)（參考意義不大）
2019-01-23 17:56:31 Total tran=20000=3835/s, qtps=40000=7670/s, ela=5203 ms, avg=260 us
Summary: SQL01 exec=800000, rows=22668078=2833/e, avg=174 us
Summary: SQL02 exec=800000, rows=0=0/e, avg=69 us
Summary: exec=133333/s, qtps=266666/s
插入的qtps為24615/s,性能不錯(cuò)歧蕉。
