概述
我們知道ceph是統(tǒng)一的分布式存儲系統(tǒng)务漩，其可以提供給應用三種訪問數(shù)據(jù)的接口：對象(RGW)温赔、塊(RBD)和文件(CEPHFS)接口蛤奢。我們在使用對象接口的時候通常都是基于HTTP協(xié)議來存取數(shù)據(jù)。
下面介紹另外一種方式來使用ceph的對象(RGW)接口 -- nfs-ganesha陶贼。這種方式可以通過基于文件的訪問協(xié)議（如NFSv3和NFSv4）來訪問Ceph對象網(wǎng)關命名空間啤贩。詳細的介紹可以去ceph官網(wǎng)看看：http://docs.ceph.com/docs/master/radosgw/nfs/。
1拜秧、環(huán)境準備
1.1痹屹、準備虛擬機
單純?yōu)榱藴y試可用性，所以我這里就是使用一個虛擬機搭建ceph集群枉氮，然后配置nfs-ganesha將rgw導出為nfs文件接口
[root@ceph01 tmp]# cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)
[root@ceph01 tmp]# uname -a
Linux ceph01 4.17.3-1.el7.elrepo.x86_64 #1 SMP Tue Jun 26 10:14:25 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
1.2志衍、配置yum源
[root@ceph01 tmp]# ll /etc/yum.repos.d/
total 12
-rw-r--r--. 1 root root 2523 Jun 16 06:22 CentOS-Base.repo
-rw-r--r--. 1 root root 180 Jul 3 11:38 ceph.repo
-rw-r--r--. 1 root root 664 May 11 11:35 epel.repo
1.2.1暖庄、base源
[root@ceph01 tmp]# cat /etc/yum.repos.d/CentOS-Base.repo
[base]
name=CentOS-$releasever - Base - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/os/$basearch/
http://mirrors.aliyuncs.com/centos/$releasever/os/$basearch/
http://mirrors.cloud.aliyuncs.com/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7
#released updates
[updates]
name=CentOS-$releasever - Updates - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/updates/$basearch/
http://mirrors.aliyuncs.com/centos/$releasever/updates/$basearch/
http://mirrors.cloud.aliyuncs.com/centos/$releasever/updates/$basearch/
gpgcheck=1
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7
#additional packages that may be useful
[extras]
name=CentOS-$releasever - Extras - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/extras/$basearch/
http://mirrors.aliyuncs.com/centos/$releasever/extras/$basearch/
http://mirrors.cloud.aliyuncs.com/centos/$releasever/extras/$basearch/
gpgcheck=1
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7
#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-$releasever - Plus - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/centosplus/$basearch/
http://mirrors.aliyuncs.com/centos/$releasever/centosplus/$basearch/
http://mirrors.cloud.aliyuncs.com/centos/$releasever/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7
#contrib - packages by Centos Users
[contrib]
name=CentOS-$releasever - Contrib - mirrors.aliyun.com
failovermethod=priority
baseurl=http://mirrors.aliyun.com/centos/$releasever/contrib/$basearch/
http://mirrors.aliyuncs.com/centos/$releasever/contrib/$basearch/
http://mirrors.cloud.aliyuncs.com/centos/$releasever/contrib/$basearch/
gpgcheck=1
enabled=0
gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7
1.2.2、epel源
[root@ceph01 tmp]# cat /etc/yum.repos.d/epel.repo
[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
baseurl=http://mirrors.aliyun.com/epel/7/$basearch
failovermethod=priority
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
[epel-debuginfo]
name=Extra Packages for Enterprise Linux 7 - $basearch - Debug
baseurl=http://mirrors.aliyun.com/epel/7/$basearch/debug
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=0
[epel-source]
name=Extra Packages for Enterprise Linux 7 - $basearch - Source
baseurl=http://mirrors.aliyun.com/epel/7/SRPMS
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=0
[root@ceph01 tmp]# cat /etc/yum.repos.d/epel.repo
[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
baseurl=http://mirrors.aliyun.com/epel/7/$basearch
failovermethod=priority
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
[epel-debuginfo]
name=Extra Packages for Enterprise Linux 7 - $basearch - Debug
baseurl=http://mirrors.aliyun.com/epel/7/$basearch/debug
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=0
[epel-source]
name=Extra Packages for Enterprise Linux 7 - $basearch - Source
baseurl=http://mirrors.aliyun.com/epel/7/SRPMS
failovermethod=priority
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
gpgcheck=0
1.2.3楼肪、ceph源
我這里配置的Jewel的ceph源
[root@ceph01 tmp]# cat /etc/yum.repos.d/ceph.repo
[ceph]
name=ceph
baseurl=http://download.ceph.com/rpm-jewel/el7/x86_64/
gpgcheck=0
[ceph-noarch]
name=cephnoarch
baseurl=http://download.ceph.com/rpm-jewel/el7/noarch/
gpgcheck=0
2培廓、相關軟件包準備
2.1、安裝ceph相關包
注意這里的 librgw2-devel 包后面編譯nfs-ganesha時需要春叫，所以一定要安裝
[root@ceph01 tmp]# yum install ceph librgw2-devel -y
[root@ceph01 tmp]# ceph -v
ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e)
部署ceph集群肩钠，下面是我部署好的單機ceph集群
[root@ceph01 tmp]# ceph -s
cluster 29978a95-87f1-4a0c-96fa-2369e985bfbe
health HEALTH_OK
monmap e1: 1 mons at {ceph01=192.168.10.20:6789/0}
election epoch 4, quorum 0 ceph01
osdmap e26: 2 osds: 2 up, 2 in
flags sortbitwise,require_jewel_osds
pgmap v61: 128 pgs, 9 pools, 2390 bytes data, 177 objects
217 MB used, 30480 MB / 30697 MB avail
128 active+clean
2.2、編譯安裝nfs-ganesha
下載nfs-ganesha源代碼：https://github.com/nfs-ganesha/nfs-ganesha/tree/3cf14c95d0a6efb5d366f97333d07a08d6f3b2a5
下載nfs-ganesha依賴的模塊：https://github.com/nfs-ganesha/ntirpc/tree/5575ddb022b996c318d263e504273ee6a34499a3
因為我使用的ceph是Jewel 10.2.11暂殖，所以這里下載的nfs-ganesha是V2.5.0价匠，高版本或低版本都會有點問題。
將上面下載好的兩個包放到/tmp目錄下
[root@ceph01 tmp]# ll
···
-rw-r--r--. 1 root root 3936969 Jul 3 15:08 nfs-ganesha.zip
-rw-r--r--. 1 root root 632189 Jul 3 15:08 ntirpc.zip
解壓
[root@ceph01 tmp]# unzip nfs-ganesha.zip
[root@ceph01 tmp]# unzip ntirpc.zip
因為ntirpc是nfs-ganesha所依賴的子模塊央星，所以這里將ntirpc源代碼放到nfs-ganasha對應的目錄中去
[root@ceph01 tmp]# cp -r ntirpc/* nfs-ganesha/src/libntirpc/
編譯安裝nfs-nanesha
[root@ceph01 tmp]# mkdir mybuild && cd mybuild
[root@ceph01 mybuild]# cmake -DUSE_FSAL_RGW=ON /tmp/nfs-ganesha/src/
[root@ceph01 mybuild]# make && make install
如果有報錯霞怀，根據(jù)報錯信息去處理下，每個環(huán)境配置不一樣莉给，報錯可能不一樣毙石。
3、開始配置
3.1颓遏、準備rgw用戶
[root@ceph01 mybuild]# radosgw-admin user create --uid=testuser --display-name="testuser " --access-key=testuser --secret=testuser
3.2徐矩、準備nfs-ganesha配置文件
配置起來還是比較簡單的（不過是最簡配置··· 哈哈）
[root@ceph01 mybuild]# cat /etc/ganesha/ganesha.conf
EXPORT
{
Export_ID=1;
Path = "/";
Pseudo = "/";
Access_Type = RW;
NFS_Protocols = 4;
Transport_Protocols = TCP;
FSAL {
# 下面的值就是開始創(chuàng)建的rgw用戶信息
Name = RGW;
User_Id = "testuser";
Access_Key_Id ="testuser";
Secret_Access_Key = "testuser";
}
}
RGW {
ceph_conf = "/etc/ceph/ceph.conf";
}
3.3、啟動nfs-ganesha服務
[root@ceph01 mybuild]# ganesha.nfsd -L /var/log/ganesha.log
3.4叁幢、查看nfs-ganesha服務是否正常啟動
[root@ceph01 tmp]# ps aux|grep ganesha
root 16055 0.2 1.0 5222380 42292 ? Ssl 15:22 0:06 ganesha.nfsd -L /var/log/ganesha.log
可以看到nfs-ganesha服務已經(jīng)正常啟動了
4滤灯、使用
現(xiàn)在來到另外一臺服務器上面
4.1、安裝nfs-utils
[root@ceph07 ~]# yum install -y nfs-utils
4.2曼玩、掛載nfs
這里的192.168.10.20就是我們上面的服務器ip
[root@ceph07 ~]# mount -t nfs4 192.168.10.20:/ /mnt/
[root@ceph07 ~]# mount |grep mnt
192.168.10.20:/ on /mnt type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.10.32,local_lock=none,addr=192.168.10.20)
4.3鳞骤、使用nfs客戶端掛載目錄
在創(chuàng)建之前查看ceph rgw池里面的對象數(shù)
[root@ceph01 tmp]# ceph df
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
30697M 30480M 217M 0.71
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
rbd 0 0 0 14472M 0
.rgw.root 1 1588 0 14472M 4
default.rgw.control 2 0 0 14472M 8
default.rgw.data.root 3 0 0 14472M 0
default.rgw.gc 4 0 0 14472M 32
default.rgw.log 5 0 0 14472M 127
default.rgw.users.uid 6 202 0 14472M 1
default.rgw.users.keys 7 9 0 14472M 1
可以看到現(xiàn)在rgw data池里面的對象數(shù)是0個，現(xiàn)在在nfs掛載目錄下創(chuàng)建一個目錄試試
[root@ceph07 ~]# mkdir /mnt/bk001
再次查看rgw data池里面的對象數(shù)
[root@ceph01 ceph01 ]# ceph df
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
30697M 30480M 217M 0.71
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
rbd 0 0 0 14472M 0
.rgw.root 1 1588 0 14472M 4
default.rgw.control 2 0 0 14472M 8
default.rgw.data.root 3 591 0 14472M 2
default.rgw.gc 4 0 0 14472M 32
default.rgw.log 5 0 0 14472M 127
default.rgw.users.uid 6 202 0 14472M 2
default.rgw.users.keys 7 9 0 14472M 1
default.rgw.buckets.index 8 0 0 14472M 1
可以看到rgw data池里面的對象數(shù)已經(jīng)由0增加到2了黍判，其實這里的目錄bk001就是對應到rgw里面的bucket
總結(jié)
我們上面主要做了這些步驟：
- 安裝ceph并搭建ceph集群
- 使用radosgw-admin創(chuàng)建rgw用戶（在創(chuàng)建rgw用戶的時候豫尽，會自動創(chuàng)建rgw所需的存儲池）
- 獲取nfs-ganesha及其依賴的ntirpc模塊
- 編譯、安裝和配置nfs-ganesha
- 最后啟動nfs-ganesha服務并在其客戶端掛載顷帖、測試使用
