codis架構(gòu)
Codis組件
Codis 3.x 由以下組件組成：
-
Codis FE：集群管理界面
- 多個集群實(shí)例共享可以共享同一個前端展示頁面佑力；
- 通過配置文件管理后端 codis-dashboard 列表，配置文件可自動更新袖裕。
-
Codis Dashboard：集群管理工具漱凝，支持 codis-proxy烈疚、codis-server 的添加定躏、刪除，以及據(jù)遷移等操作谴麦。在集群狀態(tài)發(fā)生改變時，codis-dashboard 維護(hù)集群下所有 codis-proxy 的狀態(tài)的一致性伸头。
- 對于同一個業(yè)務(wù)集群而言匾效，同一個時刻 codis-dashboard 只能有 0個或者1個；
- 所有對集群的修改都必須通過 codis-dashboard 完成恤磷。
redis-sentinel:redis類似的高可用機(jī)制支持自動主從切換（HA）
Storage: 外部存儲（注冊中心）例如FileSystem面哼、ETCD、Zookeeper
-
Codis Proxy：客戶端連接的 Redis 代理服務(wù), 實(shí)現(xiàn)了 Redis 協(xié)議扫步。 除部分命令不支持以外魔策，表現(xiàn)的和原生的 Redis 沒有區(qū)別（就像 Twemproxy）。
- 對于同一個業(yè)務(wù)集群而言河胎，可以同時部署多個 codis-proxy 實(shí)例闯袒；
- 不同 codis-proxy 之間由 codis-dashboard 保證狀態(tài)同步。
Codis-Group: 沒有對應(yīng)的組件,而是一個虛擬的節(jié)點(diǎn)機(jī)制通過槽位映射機(jī)制實(shí)現(xiàn)proxy與server的映射
- Codis Server(Codis Redis)：基于 redis-3.2.8 分支開發(fā)游岳。增加了額外的數(shù)據(jù)結(jié)構(gòu)政敢，以支持 slot 有關(guān)的操作以及數(shù)據(jù)遷移指令。
-
Codis Admin：集群管理的命令行工具胚迫。
- 可用于控制 codis-proxy喷户、codis-dashboard 狀態(tài)以及訪問外部存儲。
Codis集群啟動
Codis集群啟動需要zookeeper或者etcd,我們以zookeeper為例访锻，搭建Codis集群褪尝。
關(guān)于zookeeper的配置，可以參考zookeeper安裝部署
我們使用3臺主機(jī)搭建集群環(huán)境
配置如下：
| hostName | ip |
|---|---|
| vc0 | 192.168.4.50 |
| vc1 | 192.168.4.51 |
| vc2 | 192.168.4.52 |
vc0-1分別配置環(huán)境變量期犬，在~/.bashrc中增加
export CODIS_HOME=/storage/dep/go/src/github.com/CodisLabs/codis
export PATH=$PATH:$CODIS_HOME/bin
啟動 Codis Dashboard
使用vc0作為主機(jī),修改vc0的Codis Dashboard配置文件vim /storage/dep/go/src/github.com/CodisLabs/codis/config/dashboard.toml
##################################################
# #
# Codis-Dashboard #
# #
##################################################
# Set Coordinator, only accept "zookeeper" & "etcd" & "filesystem".
# for zookeeper/etcd, coorinator_auth accept "user:password"
# Quick Start
coordinator_name = "zookeeper"
coordinator_addr = "192.168.4.50:2181,192.168.4.51:2181,192.168.4.52:2181"
#coordinator_auth = ""
# Set Codis Product Name/Auth.
product_name = "codis-nasa"
product_auth = ""
# Set bind address for admin(rpc), tcp only.
admin_addr = "0.0.0.0:18080"
# Set arguments for data migration (only accept 'sync' & 'semi-async').
migration_method = "semi-async"
migration_parallel_slots = 100
migration_async_maxbulks = 200
migration_async_maxbytes = "32mb"
migration_async_numkeys = 500
migration_timeout = "30s"
# Set configs for redis sentinel.
sentinel_client_timeout = "10s"
sentinel_quorum = 2
sentinel_parallel_syncs = 1
sentinel_down_after = "30s"
sentinel_failover_timeout = "5m"
sentinel_notification_script = ""
sentinel_client_reconfig_script = ""
重點(diǎn)修改的參數(shù)有
-
coordinator_name
- 外部存儲類型河哑，接受 zookeeper/etcd,本例使用zookeeper
-
coordinator_addr
- 外部存儲地址，使用zookeeper集群的客戶端暴露訪問地址龟虎，以逗號分開
-
product_name
- 區(qū)分不同的集群灾馒，注冊到zookeeper也是用這個區(qū)分 ,根據(jù)自己需求起名
-
admin_addr
- 用于dashboard與codas-proxy,codis-server進(jìn)行tcp通信
使用命令nohup codis-dashboard --ncpu=1 --config=/storage/dep/go/src/github.com/CodisLabs/codis/config/dashboard.toml --log=/storage/dep/go/src/github.com/CodisLabs/codis/log/dashboard.log --log-level=WARN & 啟動
其中：
--ncpu=N 最大使用 CPU 個數(shù)
-c CONF, --config=CONF 指定啟動配置文件
-l FILE, --log=FILE 設(shè)置 log 輸出文件
--log-level=LEVEL 設(shè)置 log 輸出等級：INFO,WARN,DEBUG,ERROR；默認(rèn)INFO遣总，推薦WARN
執(zhí)行后查看日志 tail -100f /storage/dep/go/src/github.com/CodisLabs/codis/log/dashboard.log.2018-06-25,得到輸出
2018/06/25 17:12:36 main.go:78: [WARN] set ncpu = 1
2018/06/25 17:12:36 topom.go:119: [WARN] create new topom:
{
"token": "cf7a714f4756eec383928d11c2899196",
"start_time": "2018-06-25 17:12:36.761816028 +0800 CST",
"admin_addr": "vc0:18080",
"product_name": "codis-nsa",
"pid": 5364,
"pwd": "/storage/dep/go/src/github.com/CodisLabs/codis/log",
"sys": "Linux vc0 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux"
}
2018/06/25 17:12:36 main.go:142: [WARN] create topom with config
coordinator_name = "zookeeper"
coordinator_addr = "192.168.4.50:2181,192.168.4.51:2181,192.168.4.52:2181"
coordinator_auth = ""
admin_addr = "0.0.0.0:18080"
product_name = "codis-nsa"
product_auth = ""
......
......
2018/06/25 17:12:37 topom_sentinel.go:169: [WARN] rewatch sentinels = []
2018/06/25 17:12:37 main.go:179: [WARN] [0xc4202bed80] dashboard is working ...
2018/06/25 17:12:37 topom.go:429: [WARN] admin start service on [::]:18080
觀察日志輸出發(fā)現(xiàn)Codis Dashboard已經(jīng)按照配置正常啟動
啟動 Codis PROXY
配置文件如下：
product_name = "codis-nasa" # 集群名稱睬罗，同codis-dashboard中配置
product_auth = "" #集群密碼
session_auth = ""
# Set bind address for admin(rpc), tcp only.
admin_addr = "0.0.0.0:11080" # 用于和codis-dashboard進(jìn)行tcp通信同步信息
# Set bind address for proxy, proto_type can be "tcp", "tcp4", "tcp6", "unix" or "unixpacket".
proto_type = "tcp4" #端口協(xié)議類型
proxy_addr = "0.0.0.0:19000" #用于redis客戶端連接,支持redis協(xié)議，端口唯一
jodis_name = ""
jodis_addr = ""
jodis_auth = ""
jodis_timeout = "20s"
jodis_compatible = false
# Set datacenter of proxy.
proxy_datacenter = ""
# Set max number of alive sessions.
proxy_max_clients = 1000
# Set max offheap memory size. (0 to disable)
proxy_max_offheap_size = "1024mb"
# Set heap placeholder to reduce GC frequency.
proxy_heap_placeholder = "256mb"
# Proxy will ping backend redis (and clear 'MASTERDOWN' state) in a predefined interval. (0 to disable)
backend_ping_period = "5s"
# Set backend recv buffer size & timeout.
backend_recv_bufsize = "128kb"
backend_recv_timeout = "30s"
# Set backend send buffer & timeout.
backend_send_bufsize = "128kb"
backend_send_timeout = "30s"
# Set backend pipeline buffer size.
backend_max_pipeline = 20480
# Set backend never read replica groups, default is false
backend_primary_only = false
# Set backend parallel connections per server
backend_primary_parallel = 1
backend_replica_parallel = 1
# Set backend tcp keepalive period. (0 to disable)
backend_keepalive_period = "75s"
# Set number of databases of backend.
backend_number_databases = 16
# If there is no request from client for a long time, the connection will be closed. (0 to disable)
# Set session recv buffer size & timeout.
session_recv_bufsize = "128kb"
session_recv_timeout = "30m"
# Set session send buffer size & timeout.
session_send_bufsize = "64kb"
session_send_timeout = "30s"
# Make sure this is higher than the max number of requests for each pipeline request, or your client may be blocked.
# Set session pipeline buffer size.
session_max_pipeline = 10000
# Set session tcp keepalive period. (0 to disable)
session_keepalive_period = "75s"
# Set session to be sensitive to failures. Default is false, instead of closing socket, proxy will send an error response to client.
session_break_on_failure = false
# Set metrics server (such as http://localhost:28000), proxy will report json formatted metrics to specified server in a predefined period.
metrics_report_server = ""
metrics_report_period = "1s"
# Set influxdb server (such as http://localhost:8086), proxy will report metrics to influxdb.
metrics_report_influxdb_server = ""
metrics_report_influxdb_period = "1s"
metrics_report_influxdb_username = ""
metrics_report_influxdb_password = ""
metrics_report_influxdb_database = ""
# Set statsd server (such as localhost:8125), proxy will report metrics to statsd.
metrics_report_statsd_server = ""
metrics_report_statsd_period = "1s"
metrics_report_statsd_prefix = ""
使用命令nohup codis-proxy --ncpu=1 --config=/storage/dep/go/src/github.com/CodisLabs/codis/config/proxy.toml --log=/storage/dep/go/src/github.com/CodisLabs/codis/log/proxy.log --log-level=WARN & 啟動
查看日志tail -f /storage/dep/go/src/github.com/CodisLabs/codis/log/proxy.log.2018-06-25
2018/06/25 18:18:40 main.go:229: [WARN] [0xc4200eb8c0] proxy waiting online ...
2018/06/25 18:18:41 main.go:229: [WARN] [0xc4200eb8c0] proxy waiting online ...
2018/06/25 18:18:43 main.go:229: [WARN] [0xc4200eb8c0] proxy waiting online ...
2018/06/25 18:18:44 main.go:229: [WARN] [0xc4200eb8c0] proxy waiting online ...
可以觀察到codis-proxy已經(jīng)啟動旭斥，但是游離的容达，沒有加入codis-dashboard管理，也沒有與任何codis-server建立聯(lián)系,我們可以稍后通過codis-fe將proxy納入管理
啟動 Codis Server(Redis實(shí)例)
我們可以將vc0,vc1,vc2三個主機(jī)上分別啟動一個redis,vc0使用端口6380,vc1使用端口6381,vc2使用端口6382,
對vc0-2分別添加配置文件
vc0中 /storage/dep/go/src/github.com/CodisLabs/codis/config/redis6380.conf
vc1中 /storage/dep/go/src/github.com/CodisLabs/codis/config/redis6381.conf
vc2中 /storage/dep/go/src/github.com/CodisLabs/codis/config/redis6382.conf
vc0中主要修改port 6380和pidfile /tmp/redis_6380.pid以及logfile "/tmp/redis_6380.log"以及bind 127.0.0.1 192.168.4.50
vc1中主要修改port 6381和pidfile /tmp/redis_6381.pid以及logfile "/tmp/redis_6381.log"以及bind 127.0.0.1 192.168.4.51
vc0中主要修改port 6382和pidfile /tmp/redis_6382.pid以及logfile "/tmp/redis_6382.log"以及bind 127.0.0.1 192.168.4.52
在vc0-2中分別執(zhí)行以下命令啟動redis實(shí)例
codis-server /storage/dep/go/src/github.com/CodisLabs/codis/config/redis6380.conf
codis-server /storage/dep/go/src/github.com/CodisLabs/codis/config/redis6381.conf
codis-server /storage/dep/go/src/github.com/CodisLabs/codis/config/redis6382.conf
啟動 Codis FE(管理平臺)
vc0中啟動Codis FE nohup ./bin/codis-fe --ncpu=1 --log=/storage/dep/go/src/github.com/CodisLabs/codis/log/fe.log --log-level=WARN --zookeeper=127.0.0.1:2181 --listen=192.168.4.50:8080 &
注意上面的命令必須要在CODIS_HOME下執(zhí)行
其中:
zookeeper=127.0.0.1:2181為zk的連接地址
listen=192.168.4.50:8080為管理界面的訪問地址和端口
通過fe管理集群
通過fe增加Proxy
打開FE的控制頁面在Proxy欄中增加之前配置好的Proxy垂券，輸入Proxy的連接地址192.168.4.50:11080
如圖：
[圖片上傳失敗...(image-74f13d-1530000375805)]
通過fe增加Group(redis組)
