Jewel版本Ceph集群功能性能測試

參考文檔

http://docs.ceph.com/docs/master/start/quick-start-preflight/#rhel-centos
https://www.linuxidc.com/Linux/2017-09/146760.htm
http://s3browser.com/
http://docs.ceph.org.cn/man/8/rbd/
https://hub.packtpub.com/working-ceph-block-device/#
https://github.com/s3fs-fuse/s3fs-fuse
https://blog.csdn.net/miaodichiyou/article/details/76050361
http://mathslinux.org/?p=717
http://elf8848.iteye.com/blog/2089055

測試目標(biāo)

使用rbd映射掛載塊存儲(chǔ)并測試性能
使用rbd-nbd映射掛載條帶塊存儲(chǔ)并測試性能
使用s3brower測試對象存儲(chǔ)讀寫
使用s3fs掛載掛載對象存儲(chǔ)
使用對象存儲(chǔ)寫使用塊存儲(chǔ)讀

一简逮，使用rbd映射掛載塊存儲(chǔ)并測試性能

1、創(chuàng)建image

[root@idcv-ceph0 cluster]# ceph osd pool create test_pool 100
pool 'test_pool' created
[root@idcv-ceph0 cluster]# rados lspools
rbd
.rgw.root
default.rgw.control
default.rgw.data.root
default.rgw.gc
default.rgw.log
default.rgw.users.uid
default.rgw.users.keys
default.rgw.buckets.index
default.rgw.buckets.data
test_pool
[root@idcv-ceph0 cluster]# rbd list
[root@idcv-ceph0 cluster]# rbd create test_pool/testimage1 --size 40960
[root@idcv-ceph0 cluster]# rbd create test_pool/testimage2 --size 40960
[root@idcv-ceph0 cluster]# rbd create test_pool/testimage3 --size 40960
[root@idcv-ceph0 cluster]# rbd create test_pool/testimage4 --size 40960
[root@idcv-ceph0 cluster]# rbd list
[root@idcv-ceph0 cluster]# rbd list test_pool
testimage1
testimage2
testimage3
testimage4

2牺汤、映射image

[root@idcv-ceph0 cluster]# rbd map test_pool/testimage1
rbd: sysfs write failed
RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable".
In some cases useful info is found in syslog - try "dmesg | tail" or so.
rbd: map failed: (6) No such device or address
[root@idcv-ceph0 cluster]# dmesg |tail
[113320.926463] rbd: loaded (major 252)
[113320.931044] libceph: mon2 172.20.1.141:6789 session established
[113320.931364] libceph: client4193 fsid 812d3acb-eaa8-4355-9a74-64f2cd5209b3
[113320.936922] rbd: image testimage1: image uses unsupported features: 0x38
[113339.870548] libceph: mon1 172.20.1.140:6789 session established
[113339.870906] libceph: client4168 fsid 812d3acb-eaa8-4355-9a74-64f2cd5209b3
[113339.877109] rbd: image testimage1: image uses unsupported features: 0x38
[113381.405453] libceph: mon2 172.20.1.141:6789 session established
[113381.405784] libceph: client4202 fsid 812d3acb-eaa8-4355-9a74-64f2cd5209b3
[113381.411625] rbd: image testimage1: image uses unsupported features: 0x38

報(bào)錯(cuò)處理方法：disable新特性

[root@idcv-ceph0 cluster]# rbd info test_pool/testimage1
rbd image 'testimage1':
size 40960 MB in 10240 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.10802ae8944a
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
flags:
[root@idcv-ceph0 cluster]# rbd feature disable test_pool/testimage1
rbd: at least one feature name must be specified
[root@idcv-ceph0 cluster]# rbd feature disable test_pool/testimage1 fast-diff
[root@idcv-ceph0 cluster]# rbd feature disable test_pool/testimage1 object-map
[root@idcv-ceph0 cluster]# rbd feature disable test_pool/testimage1 exclusive-lock
[root@idcv-ceph0 cluster]# rbd feature disable test_pool/testimage1 deep-flatten
[root@idcv-ceph0 cluster]# rbd info test_pool/testimage1
rbd image 'testimage1':
size 40960 MB in 10240 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.10802ae8944a
format: 2
features: layering
flags:
[root@idcv-ceph0 cluster]# rbd map test_pool/testimage1
/dev/rbd0

同理操作testimage2\3\4澳腹，最終如下

[root@idcv-ceph0 cluster]# rbd showmapped
id pool image snap device
0 test_pool testimage1 - /dev/rbd0
1 test_pool testimage2 - /dev/rbd1
2 test_pool testimage3 - /dev/rbd2
3 test_pool testimage4 - /dev/rbd3

備注收縮image大小

[root@idcv-ceph0 ceph-disk0]# rbd resize -p test_pool --image testimage1 -s 10240 --allow-shrink
Resizing image: 100% complete...done.
[root@idcv-ceph0 ceph-disk0]# rbd resize -p test_pool --image testimage2 -s 10240 --allow-shrink
Resizing image: 100% complete...done.
[root@idcv-ceph0 ceph-disk0]# rbd resize -p test_pool --image testimage3 -s 10240 --allow-shrink
Resizing image: 100% complete...done.
[root@idcv-ceph0 ceph-disk0]# rbd resize -p test_pool --image testimage4 -s 10240 --allow-shrink
Resizing image: 100% complete...done.

3填帽、格式化掛載

[root@idcv-ceph0 ceph-disk0]# mkfs.xfs /dev/rbd0
[root@idcv-ceph0 ceph-disk0]# mount /mnt/ceph-disk0 /dev/rbd0

4攘蔽、DD測試

[root@idcv-ceph0 ceph-disk0]# dd if=/dev/zero of=/mnt/ceph-disk0/file0 count=1000 bs=4M conv=fsync
1000+0 records in
1000+0 records out
4194304000 bytes (4.2 GB) copied, 39.1407 s, 107 MB/s

二句狼、使用rbd-nbd映射掛載條帶塊存儲(chǔ)并測試性能

1、創(chuàng)建image
根據(jù)官網(wǎng)文檔條帶化測試需要帶參數(shù)--stripe-unit及--stripe-count
計(jì)劃測試object-size為4M十偶、4K且count為1時(shí)菩鲜，object-szie為32M且count為8、16時(shí)塊存儲(chǔ)性能

[root@idcv-ceph0 ceph-disk0]# rbd create test_pool/testimage5 --size 10240 --stripe-unit 2097152 --stripe-count 16
[root@idcv-ceph0 ceph-disk0]# rbd info test_pool/testimage5
rbd image 'testimage5':
size 10240 MB in 2560 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.10c52ae8944a
format: 2
features: layering, striping, exclusive-lock, object-map, fast-diff, deep-flatten
flags:
stripe unit: 2048 kB
stripe count: 16
[root@idcv-ceph0 ceph-disk0]# rbd create test_pool/testimage6 --size 10240 --stripe-unit 4096 --stripe-count 4
[root@idcv-ceph0 ceph-disk0]# rbd info test_pool/testimage6
rbd image 'testimage6':
size 10240 MB in 2560 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.10c82ae8944a
format: 2
features: layering, striping, exclusive-lock, object-map, fast-diff, deep-flatten
flags:
stripe unit: 4096 bytes
stripe count: 4

[root@idcv-ceph0 ceph-disk0]# rbd create test_pool/testimage7 --size 10240 --object-size 32M --stripe-unit 4194304 --stripe-count 4
[root@idcv-ceph0 ceph-disk0]# rbd info test_pool/testimage7
rbd image 'testimage7':
size 10240 MB in 320 objects
order 25 (32768 kB objects)
block_name_prefix: rbd_data.107e238e1f29
format: 2
features: layering, striping, exclusive-lock, object-map, fast-diff, deep-flatten
flags:
stripe unit: 4096 kB
stripe count: 4
[root@idcv-ceph0 ceph-disk0]# rbd create test_pool/testimage8 --size 10240 --object-size 32M --stripe-unit 2097152 --stripe-count 16
[root@idcv-ceph0 ceph-disk0]# rbd info test_pool/testimage8
rbd image 'testimage8':
size 10240 MB in 320 objects
order 25 (32768 kB objects)
block_name_prefix: rbd_data.109d2ae8944a
format: 2
features: layering, striping, exclusive-lock, object-map, fast-diff, deep-flatten
flags:
stripe unit: 2048 kB
stripe count: 16

[root@idcv-ceph0 ceph-disk0]# rbd create test_pool/testimage11 --size 10240 --object-size 4M
[root@idcv-ceph0 ceph-disk0]# rbd create test_pool/testimage12 --size 10240 --object-size 4K
[root@idcv-ceph0 ceph-disk0]# rbd info test_pool/testimage11
rbd image 'testimage11':
size 10240 MB in 2560 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.10ac238e1f29
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
flags:
[root@idcv-ceph0 ceph-disk0]# rbd info test_pool/testimage12
rbd image 'testimage12':
size 10240 MB in 2621440 objects
order 12 (4096 bytes objects)
block_name_prefix: rbd_data.10962ae8944a
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
flags:

2惦积、映射image

[root@idcv-ceph2 mnt]# rbd map test_pool/testimage8
rbd: sysfs write failed
In some cases useful info is found in syslog - try "dmesg | tail" or so.
rbd: map failed: (22) Invalid argument
[root@idcv-ceph2 mnt]# dmesg |tail
[118760.024660] XFS (rbd0): Log I/O Error Detected. Shutting down filesystem
[118760.024710] XFS (rbd0): Please umount the filesystem and rectify the problem(s)
[118760.024766] XFS (rbd0): Unable to update superblock counters. Freespace may not be correct on next mount.
[118858.837102] XFS (rbd0): Mounting V5 Filesystem
[118858.872345] XFS (rbd0): Ending clean mount
[173522.968410] rbd: rbd0: encountered watch error: -107
[176701.031429] rbd: image testimage8: unsupported stripe unit (got 2097152 want 33554432)
[176827.317008] rbd: image testimage8: unsupported stripe unit (got 2097152 want 33554432)
[177423.107103] rbd: image testimage8: unsupported stripe unit (got 2097152 want 33554432)
[177452.820032] rbd: image testimage8: unsupported stripe unit (got 2097152 want 33554432)

3接校、排錯(cuò)發(fā)現(xiàn)rbd不支持條帶特性需要需要使用rbd-nbd
rbd-nbd支持所有的新特性，后續(xù)map時(shí)也不需要disable新特性狮崩，但是linux內(nèi)核默認(rèn)沒有nbd模塊蛛勉，需要編譯內(nèi)核安裝，可以參考下面鏈接https://blog.csdn.net/miaodichiyou/article/details/76050361

[root@idcv-ceph2 ~]# wget http://vault.centos.org/7.5.1804/updates/Source/SPackages/kernel-3.10.0-862.2.3.el7.src.rpm
[root@idcv-ceph2 ~]# rpm -ivh kernel-3.10.0-862.2.3.el7.src.rpm
[root@idcv-ceph2 ~]# cd /root/rpmbuild/
[root@idcv-ceph0 rpmbuild]# cd SOURCES/
[root@idcv-ceph0 SOURCES]# tar Jxvf linux-3.10.0-862.2.3.el7.tar.xz -C /usr/src/kernels/
[root@idcv-ceph0 SOURCES]# cd /usr/src/kernels/
[root@idcv-ceph0 kernels]# mv 3.10.0-862.6.3.el7.x86_64 3.10.0-862.6.3.el7.x86_64-old
[root@idcv-ceph0 kernels]# mv linux-3.10.0-862.2.3.el7 3.10.0-862.6.3.el7.x86_64
[root@idcv-ceph0 3.10.0-862.6.3.el7.x86_64]# cd 3.10.0-862.6.3.el7.x86_64
[root@idcv-ceph0 3.10.0-862.6.3.el7.x86_64]# mkdir mrproper
[root@idcv-ceph0 3.10.0-862.6.3.el7.x86_64]# cp ../3.10.0-862.6.3.el7.x86_64-old/Module.symvers ./
[root@idcv-ceph0 3.10.0-862.6.3.el7.x86_64]# cp /boot/config-3.10.0-862.2.3.el7.x86_64 ./.config
[root@idcv-ceph0 3.10.0-862.6.3.el7.x86_64]# yum install elfutils-libelf-devel
[root@idcv-ceph0 3.10.0-862.6.3.el7.x86_64]# make prepare
[root@idcv-ceph0 3.10.0-862.6.3.el7.x86_64]# make scripts
[root@idcv-ceph0 3.10.0-862.6.3.el7.x86_64]# make CONFIG_BLK_DEV_NBD=m M=drivers/block
[root@idcv-ceph0 3.10.0-862.6.3.el7.x86_64]# modinfo nbd
[root@idcv-ceph0 3.10.0-862.6.3.el7.x86_64]# cp drivers/block/nbd.ko /lib/modules/3.10.0-862.2.3.el7.x86_64/kernel/drivers/block/
[root@idcv-ceph0 3.10.0-862.6.3.el7.x86_64]# depmod -a
[root@idcv-ceph0 3.10.0-862.6.3.el7.x86_64]# modprobe nbd
[root@idcv-ceph0 3.10.0-862.6.3.el7.x86_64]# lsmod |grep nbd
nbd 17554 5

4睦柴、使用rbd-nbd映射image

[root@idcv-ceph0 ~]# rbd-nbd map test_pool/testimage17
/dev/nbd0
[root@idcv-ceph0 ~]# rbd info test_pool/testimage17
rbd image 'testimage17':
size 10240 MB in 1280 objects
order 23 (8192 kB objects)
block_name_prefix: rbd_data.112d74b0dc51
format: 2
features: layering, striping
flags:
stripe unit: 1024 kB
stripe count: 8
[root@idcv-ceph0 ~]# mkfs.xfs /dev/nbd0
meta-data=/dev/nbd0 isize=512 agcount=4, agsize=655360 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=2621440, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[root@idcv-ceph0 ~]# mount /dev/nbd0 /mnt/ceph-8M/
[root@idcv-ceph0 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 100G 3.5G 96G 4% /
devtmpfs 7.8G 0 7.8G 0% /dev
tmpfs 7.8G 0 7.8G 0% /dev/shm
tmpfs 7.8G 12M 7.8G 1% /run
tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
/dev/sda1 497M 150M 348M 31% /boot
tmpfs 1.6G 0 1.6G 0% /run/user/0
/dev/sdb1 95G 40G 56G 42% /var/lib/ceph/osd/ceph-0
/dev/rbd0 10G 7.9G 2.2G 79% /mnt/ceph-disk0
/dev/rbd1 10G 7.9G 2.2G 79% /mnt/ceph-4M
/dev/nbd0 10G 33M 10G 1% /mnt/ceph-8M

5诽凌、dd測試性能
object-size為8M

[root@idcv-ceph0 ~]# dd if=/dev/zero of=/mnt/ceph-8M/file0-1 count=800 bs=10M conv=fsync
800+0 records in
800+0 records out
8388608000 bytes (8.4 GB) copied, 50.964 s, 165 MB/s
[root@idcv-ceph0 ~]# dd if=/dev/zero of=/mnt/ceph-8M/file0-1 count=80 bs=100M conv=fsync
80+0 records in
80+0 records out
8388608000 bytes (8.4 GB) copied, 26.3178 s, 319 MB/s

object-size為32M

[root@idcv-ceph0 ceph-32M]# rbd info test_pool/testimage18
rbd image 'testimage18':
size 40960 MB in 1280 objects
order 25 (32768 kB objects)
block_name_prefix: rbd_data.11052ae8944a
format: 2
features: layering, striping, exclusive-lock, object-map, fast-diff, deep-flatten
flags:
stripe unit: 2048 kB
stripe count: 8

[root@idcv-ceph0 ceph-32M]# dd if=/dev/zero of=/mnt/ceph-32M/file0-1 count=2000 bs=10M conv=fsync
2000+0 records in
2000+0 records out
20971520000 bytes (21 GB) copied, 67.4266 s, 311 MB/s
[root@idcv-ceph0 ceph-32M]# dd if=/dev/zero of=/mnt/ceph-32M/file0-1 count=20000 bs=1M conv=fsync
20000+0 records in
20000+0 records out
20971520000 bytes (21 GB) copied, 61.7757 s, 339 MB/s

6、測試方法匯總
4m cnt=1
4k cnt=1
32M cnt=8坦敌，16
dd測試
1M
100M

32M /mnt/ceph-32M-8 /mnt/ceph-32M-16

rbd create test_pool/testimage8 --size 10240 --object-size 32M --stripe-unit 2097152 --stripe-count 16
dd if=/dev/zero of=/mnt/ceph-32M-16/file32M count=80 bs=100M conv=fsync
dd if=/dev/zero of=/mnt/ceph-32M-16/file32M count=8000 bs=1M conv=fsync

rbd create test_pool/testimage19 --size 10240 --object-size 32M --stripe-unit 4194304 --stripe-count 8
dd if=/dev/zero of=/mnt/ceph-32M-8/file32M count=80 bs=100M conv=fsync
dd if=/dev/zero of=/mnt/ceph-32M-8/file32M count=8000 bs=1M conv=fsync

4M /mnt/ceph-4M

rbd create test_pool/testimage11 --size 10240 --object-size 4M
dd if=/dev/zero of=/mnt/ceph-4M/file4M count=80 bs=100M conv=fsync
dd if=/dev/zero of=/mnt/ceph-4M/file4M count=8000 bs=1M conv=fsync

4K /mnt/ceph-4K

rbd create test_pool/testimage12 --size 10240 --object-size 4K
dd if=/dev/zero of=/mnt/ceph-4K/file4K count=80 bs=100M conv=fsync
dd if=/dev/zero of=/mnt/ceph-4K/file4K count=8000 bs=1M conv=fsync

7侣诵、dd測試結(jié)果匯總

8、使用fio隨機(jī)寫測試
先安裝fio

yum install libaio-devel
wget http://brick.kernel.dk/snaps/fio-2.1.10.tar.gz
tar zxf fio-2.1.10.tar.gz
cd fio-2.1.10/
make
make install

32M-8

fio -ioengine=libaio -bs=1m -direct=1 -thread -rw=randwrite -size=4G -filename=/dev/nbd4 -name="EBS 1m randwrite test" -iodepth=1 -runtime=60
Run status group 0 (all jobs):
WRITE: io=4096.0MB, aggrb=272729KB/s, minb=272729KB/s, maxb=272729KB/s, mint=15379msec, maxt=15379msec
Disk stats (read/write):
nbd4: ios=0/32280, merge=0/0, ticks=0/36624, in_queue=36571, util=97.61%

fio -ioengine=libaio -bs=100m -direct=1 -thread -rw=randwrite -size=4G -filename=/dev/nbd4 -name="EBS 100m randwrite test" -iodepth=1 -runtime=60
Run status group 0 (all jobs):
WRITE: io=4000.0MB, aggrb=326504KB/s, minb=326504KB/s, maxb=326504KB/s, mint=12545msec, maxt=12545msec
Disk stats (read/write):
nbd4: ios=0/31391, merge=0/0, ticks=0/1592756, in_queue=1597878, util=97.04%

32M-16

fio -ioengine=libaio -bs=1m -direct=1 -thread -rw=randwrite -size=4G -filename=/dev/nbd3 -name="EBS 1m randwrite test" -iodepth=1 -runtime=60
fio -ioengine=libaio -bs=100m -direct=1 -thread -rw=randwrite -size=4G -filename=/dev/nbd3 -name="EBS 100m randwrite test" -iodepth=1 -runtime=60

4M

fio -ioengine=libaio -bs=1m -direct=1 -thread -rw=randwrite -size=4G -filename=/dev/rbd1 -name="EBS 1m randwrite test" -iodepth=1 -runtime=60
fio -ioengine=libaio -bs=100m -direct=1 -thread -rw=randwrite -size=4G -filename=/dev/rbd1 -name="EBS 100m randwrite test" -iodepth=1 -runtime=60

4K

fio -ioengine=libaio -bs=1m -direct=1 -thread -rw=randwrite -size=400M -filename=/dev/rbd2 -name="EBS 1m randwrite test" -iodepth=1 -runtime=60
fio -ioengine=libaio -bs=100m -direct=1 -thread -rw=randwrite -size=400M -filename=/dev/rbd2 -name="EBS 100m randwrite test" -iodepth=1 -runtime=60

9狱窘、fio測試結(jié)果匯總

三杜顺、使用s3brower測試對象存儲(chǔ)讀寫

1、創(chuàng)建對象存儲(chǔ)賬號密碼

[root@idcv-ceph0 cluster]# radosgw-admin user create --uid=test --display-name="test" --access-key=123456 --secret=123456
[root@idcv-ceph0 cluster]# radosgw-admin user info --uid=test
{
"user_id": "test",
"display_name": "test",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [],
"keys": [
{
"user": "test",
"access_key": "123456",
"secret_key": "123456"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
},
"temp_url_keys": []
}

2训柴、安裝配置s3brower

3哑舒、創(chuàng)建bucket上傳下載測試

四、使用s3fs掛載掛載對象存儲(chǔ)讀寫

測試對象存儲(chǔ)方式寫入文件幻馁，從rbd方式讀目錄
1洗鸵、安裝部署
https://github.com/s3fs-fuse/s3fs-fuse/releases

安裝
查看README
On CentOS 7:

sudo yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel

Then compile from master via the following commands:

git clone https://github.com/s3fs-fuse/s3fs-fuse.git
cd s3fs-fuse
./autogen.sh
./configure
make
sudo make install

[root@idcv-ceph0 ~]# wget https://github.com/s3fs-fuse/s3fs-fuse/archive/v1.83.tar.gz
[root@idcv-ceph0 ~]# ls
[root@idcv-ceph0 ~]# tar zxvf v1.83.tar.gz
[root@idcv-ceph0 s3fs-fuse-1.83]# cd s3fs-fuse-1.83/
[root@idcv-ceph0 s3fs-fuse-1.83]# ls
[root@idcv-ceph0 s3fs-fuse-1.83]# vi README.md
[root@idcv-ceph0 s3fs-fuse-1.83]# yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel
[root@idcv-ceph0 s3fs-fuse-1.83]# ./autogen.sh
[root@idcv-ceph0 s3fs-fuse-1.83]# ls
[root@idcv-ceph0 s3fs-fuse-1.83]# ./configure
[root@idcv-ceph0 s3fs-fuse-1.83]# make
[root@idcv-ceph0 s3fs-fuse-1.83]# make install
[root@idcv-ceph0 s3fs-fuse-1.83]# mkdir /mnt/s3
[root@idcv-ceph0 s3fs-fuse-1.83]# vi /root/.passwd-s3fs
[root@idcv-ceph0 s3fs-fuse-1.83]# chmod 600 /root/.passwd-s3fs

2越锈、掛載

[root@idcv-ceph0 ~]# s3fs testbucket /mnt/s3 -o url=http://172.20.1.139:7480 -o umask=0022 -o use_path_request_style
[root@idcv-ceph0 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 95G 75G 21G 79% /var/lib/ceph/osd/ceph-0
/dev/rbd1 10G 7.9G 2.2G 79% /mnt/ceph-4M
/dev/rbd2 10G 814M 9.2G 8% /mnt/ceph-4K
/dev/nbd3 10G 7.9G 2.2G 79% /mnt/ceph-32M-16
/dev/nbd4 10G 33M 10G 1% /mnt/ceph-32M-8
s3fs 256T 0 256T 0% /mnt/s3

3、驗(yàn)證讀寫

[root@idcv-ceph0 ~]# ls /mnt/s3/images/
kernel-3.10.0-862.2.3.el7.src.rpm nbd.ko test.jpg
[root@idcv-ceph0 ~]# cp /etc/hosts
hosts hosts.allow hosts.deny
[root@idcv-ceph0 ~]# cp /etc/hosts /mnt/s3/images/
[root@idcv-ceph0 ~]# ls /mnt/s3/images/
hosts kernel-3.10.0-862.2.3.el7.src.rpm nbd.ko test.jpg

最后編輯于 ：
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請聯(lián)系作者
  • 人面猴
    序言：七十年代末膘滨，一起剝皮案震驚了整個(gè)濱河市甘凭，隨后出現(xiàn)的幾起案子，更是在濱河造成了極大的恐慌火邓，老刑警劉巖丹弱，帶你破解...
    沈念sama閱讀 211,290評論 6贊 491
  • 死咒
    序言：濱河連續(xù)發(fā)生了三起死亡事件，死亡現(xiàn)場離奇詭異铲咨，居然都是意外死亡躲胳，警方通過查閱死者的電腦和手機(jī)，發(fā)現(xiàn)死者居然都...
    沈念sama閱讀 90,107評論 2贊 385
  • 救了他兩次的神仙讓他今天三更去死
    文/潘曉璐 我一進(jìn)店門纤勒，熙熙樓的掌柜王于貴愁眉苦臉地迎上來坯苹，“玉大人，你說我怎么就攤上這事摇天〈馀龋” “怎么了？”我有些...
    開封第一講書人閱讀 156,872評論 0贊 347
  • 道士緝兇錄：失蹤的賣姜人
    文/不壞的土叔 我叫張陵泉坐，是天一觀的道長为鳄。 經(jīng)常有香客問我，道長腕让，這世上最難降的妖魔是什么孤钦？ 我笑而不...
    開封第一講書人閱讀 56,415評論 1贊 283
  • ?港島之戀（遺憾婚禮）
    正文 為了忘掉前任，我火速辦了婚禮记某，結(jié)果婚禮上司训，老公的妹妹穿的比我還像新娘。我一直安慰自己液南，他們只是感情好壳猜，可當(dāng)我...
    茶點(diǎn)故事閱讀 65,453評論 6贊 385
  • 惡毒庶女頂嫁案：這布局不是一般人想出來的
    文/花漫 我一把揭開白布。 她就那樣靜靜地躺著滑凉，像睡著了一般统扳。 火紅的嫁衣襯著肌膚如雪。 梳的紋絲不亂的頭發(fā)上畅姊，一...
    開封第一講書人閱讀 49,784評論 1贊 290
  • 城市分裂傳說
    那天咒钟，我揣著相機(jī)與錄音，去河邊找鬼若未。 笑死朱嘴，一個(gè)胖子當(dāng)著我的面吹牛，可吹牛的內(nèi)容都是我干的。 我是一名探鬼主播萍嬉，決...
    沈念sama閱讀 38,927評論 3贊 406
  • 雙鴛鴦連環(huán)套：你想象不到人心有多黑
    文/蒼蘭香墨 我猛地睜開眼乌昔，長吁一口氣：“原來是場噩夢啊……” “哼！你這毒婦竟也來了壤追？” 一聲冷哼從身側(cè)響起磕道，我...
    開封第一講書人閱讀 37,691評論 0贊 266
  • 萬榮殺人案實(shí)錄
    序言：老撾萬榮一對情侶失蹤，失蹤者是張志新（化名）和其女友劉穎行冰，沒想到半個(gè)月后溺蕉，有當(dāng)?shù)厝嗽跇淞掷锇l(fā)現(xiàn)了一具尸體，經(jīng)...
    沈念sama閱讀 44,137評論 1贊 303
  • ?護(hù)林員之死
    正文 獨(dú)居荒郊野嶺守林人離奇死亡悼做，尸身上長有42處帶血的膿包…… 初始之章·張勛 以下內(nèi)容為張勛視角 年9月15日...
    茶點(diǎn)故事閱讀 36,472評論 2贊 326
  • ?白月光啟示錄
    正文 我和宋清朗相戀三年疯特，在試婚紗的時(shí)候發(fā)現(xiàn)自己被綠了。 大學(xué)時(shí)的朋友給我發(fā)了我未婚夫和他白月光在一起吃飯的照片肛走。...
    茶點(diǎn)故事閱讀 38,622評論 1贊 340
  • 活死人
    序言：一個(gè)原本活蹦亂跳的男人離奇死亡辙芍，死狀恐怖，靈堂內(nèi)的尸體忽然破棺而出羹与，到底是詐尸還是另有隱情，我是刑警寧澤庶灿，帶...
    沈念sama閱讀 34,289評論 4贊 329
  • ?日本核電站爆炸內(nèi)幕
    正文 年R本政府宣布纵搁，位于F島的核電站，受9級特大地震影響往踢，放射性物質(zhì)發(fā)生泄漏腾誉。R本人自食惡果不足惜，卻給世界環(huán)境...
    茶點(diǎn)故事閱讀 39,887評論 3贊 312
  • 男人毒藥：我在死后第九天來索命
    文/蒙蒙 一峻呕、第九天 我趴在偏房一處隱蔽的房頂上張望利职。 院中可真熱鬧，春花似錦瘦癌、人聲如沸猪贪。這莊子的主人今日做“春日...
    開封第一講書人閱讀 30,741評論 0贊 21
  • 一樁弒父案讯私，背后竟有這般陰謀
    文/蒼蘭香墨 我抬頭看了看天上的太陽热押。三九已至，卻和暖如春斤寇，著一層夾襖步出監(jiān)牢的瞬間桶癣，已是汗流浹背。 一陣腳步聲響...
    開封第一講書人閱讀 31,977評論 1贊 265
  • 情欲美人皮
    我被黑心中介騙來泰國打工娘锁， 沒想到剛下飛機(jī)就差點(diǎn)兒被人妖公主榨干…… 1. 我叫王不留牙寞，地道東北人。 一個(gè)月前我還...
    沈念sama閱讀 46,316評論 2贊 360
  • 代替公主和親
    正文 我出身青樓莫秆，卻偏偏與公主長得像间雀，于是被迫代替她去往敵國和親悔详。 傳聞我的和親對象是個(gè)殘疾皇子，可洞房花燭夜當(dāng)晚...
    茶點(diǎn)故事閱讀 43,490評論 2贊 348

推薦閱讀更多精彩內(nèi)容