10. Redis Cluster
10.1 Redis Cluster工作原理
在哨兵sentinel機(jī)制中，可以解決redis高可用問題饵筑，即當(dāng)master故障后可以自動(dòng)將slave提升為master，從而可以保證redis服務(wù)的正常使用，但是無法解決redis單機(jī)寫入的瓶頸問題墙贱，即單機(jī)redis寫入性能受限于單機(jī)的內(nèi)存大小、并發(fā)數(shù)量贱傀、網(wǎng)卡速率等因素惨撇。
主從解決單點(diǎn)失敗, 哨兵實(shí)現(xiàn)自動(dòng)切換, cluster實(shí)現(xiàn)多個(gè)主節(jié)點(diǎn), 提高性能, 內(nèi)置主從復(fù)制, 也實(shí)現(xiàn)了高可用
早期Redis 分布式集群部署方案:
客戶端分區(qū)：由客戶端程序決定key的寫入分配和寫入的redis 節(jié)點(diǎn)(比如對(duì)key進(jìn)行取模運(yùn)算, 和預(yù)先定義好的存放規(guī)則做匹配)，但是需要客戶端自己處理寫入分配府寒、高可用管理和故障轉(zhuǎn)移等
代理方案：基于第三方軟件實(shí)現(xiàn)redis proxy魁衙，客戶端先連接到代理層，由代理層實(shí)現(xiàn)key的寫入分配株搔，對(duì)客戶端來說是比較簡(jiǎn)單剖淀，但是對(duì)于集群管理節(jié)點(diǎn)增減相對(duì)比較麻煩，而且代理本身也是單點(diǎn)并且有性能瓶頸
客戶端分區(qū):
圖片.png
代理方案:
圖片.png
因此redis 3.0版本之后推出了無中心架構(gòu)的redis cluster機(jī)制纤房，在無中心的redis集群當(dāng)中纵隔，其每個(gè)節(jié)點(diǎn)保存當(dāng)前節(jié)點(diǎn)數(shù)據(jù)和整個(gè)集群狀態(tài),每個(gè)節(jié)點(diǎn)都和其他所有節(jié)點(diǎn)連接
Redis Cluster特點(diǎn)如下:
所有Redis節(jié)點(diǎn)使用(PING機(jī)制)互聯(lián)
集群中某個(gè)節(jié)點(diǎn)的數(shù)據(jù)是否失效，需要整個(gè)集群中超過半數(shù)的節(jié)點(diǎn)監(jiān)測(cè)都失效炮姨，才能算真正的失效
客戶端不需要proxy即可直接連接redis捌刮，應(yīng)用程序需要寫全部的redis服務(wù)器IP
redis cluster把所有的redis主節(jié)點(diǎn)平均映射到 0-16383個(gè)槽位(slot)上，讀寫需要到指定的redis 節(jié)點(diǎn)上進(jìn)行
操作舒岸，因此有多少個(gè)redis 節(jié)點(diǎn)相當(dāng)于redis 并發(fā)擴(kuò)展了多少倍绅作，每個(gè)redis 節(jié)點(diǎn)承擔(dān)16384/N個(gè)槽位
Redis cluster預(yù)先分配16384個(gè)(slot)槽位，當(dāng)需要在redis集群中寫入一個(gè)key -value的時(shí)候蛾派，會(huì)使用
CRC16(key) 模 16384之后的值俄认，決定將key寫入值哪一個(gè)槽位從而決定寫入哪一個(gè)Redis節(jié)點(diǎn)上，從而有效解決單機(jī)瓶頸
cluster中, 每套主從,負(fù)責(zé)存放不同的數(shù)據(jù), 實(shí)現(xiàn)了分布式, 提高了redis的性能, 實(shí)現(xiàn)了橫向擴(kuò)展
配合主從, 每一份數(shù)據(jù)都是放在一套主從中, 從主節(jié)點(diǎn)寫入, 然后同步給自己對(duì)應(yīng)的從節(jié)點(diǎn), 實(shí)現(xiàn)數(shù)據(jù)備份
一旦任何一個(gè)主節(jié)點(diǎn)宕機(jī), 其對(duì)應(yīng)的從節(jié)點(diǎn)會(huì)自動(dòng)被提升為主節(jié)點(diǎn), 無需額外配置
槽位是在主節(jié)點(diǎn)分配
10.2 Redis Cluster架構(gòu)
圖片.png
10.2.1 Redis Cluster基本架構(gòu)
假如三個(gè)主節(jié)點(diǎn)分別是：A, B, C 三個(gè)節(jié)點(diǎn)洪乍，采用哈希槽 (hash slot)的方式來分配16384個(gè)slot 的話
它們?nèi)齻€(gè)節(jié)點(diǎn)分別承擔(dān)的slot 區(qū)間是：
節(jié)點(diǎn)A覆蓋 0－5460
節(jié)點(diǎn)B覆蓋 5461－10922
節(jié)點(diǎn)C覆蓋 10923－16383
10.2.2 Redis Cluster主從架構(gòu)
Redis cluster的架構(gòu)雖然解決了并發(fā)的問題眯杏，但是又引入了一個(gè)新的問題，每個(gè)Redis master的高可用如何解決壳澳？- 對(duì)每個(gè)master節(jié)點(diǎn), 都實(shí)現(xiàn)主從復(fù)制, 從而實(shí)現(xiàn)Redis高可用性, cluster會(huì)自動(dòng)監(jiān)控每個(gè)master節(jié)點(diǎn), 一旦master宕機(jī), 其對(duì)應(yīng)的從節(jié)點(diǎn)會(huì)自動(dòng)代替主節(jié)點(diǎn)成為新的主節(jié)點(diǎn), 該過程無需額外配置, cluster內(nèi)置功能
10.2.3 cluster部署架構(gòu)選擇
環(huán)境A: 節(jié)約成本, 測(cè)試環(huán)境
三臺(tái)服務(wù)器, 每臺(tái)服務(wù)器啟動(dòng)兩個(gè)redis實(shí)例,比如6379和6380
把一套主從的主節(jié)點(diǎn)和從節(jié)點(diǎn)配置在不同的服務(wù)器上, 避免單點(diǎn)失敗
圖片.png
10.0.0.81:6379/6380
10.0.0.82:6379/6380
10.0.0.83:6379/6380
# 另外預(yù)留一臺(tái)服務(wù)器做集群添加節(jié)點(diǎn)測(cè)試
環(huán)境B: 需要硬件投資, 生產(chǎn)環(huán)境
六臺(tái)服務(wù)器, 分別是三組主從
圖片.png
集群節(jié)點(diǎn)
10.0.0.81
10.0.0.82
10.0.0.83
10.0.0.84
10.0.0.85
10.0.0.86
預(yù)留服務(wù)器擴(kuò)展使用
10.0.0.87
10.0.0.88
Redis 5.X和之前的版本相比, 有很多變化, 以下介紹版本5.X的配置
10.2.4 部署方式介紹
- 原生命令安裝
可以幫助理解cluster架構(gòu), 但是生產(chǎn)環(huán)境一般不使用, 比較繁瑣
- 官方工具安裝
高效, 準(zhǔn)確, 一般在生產(chǎn)環(huán)境使用
- 自主開發(fā)
實(shí)現(xiàn)可視化的自動(dòng)化部署
實(shí)驗(yàn)案例: Redis 5 搭建 cluster
實(shí)驗(yàn)環(huán)境:
6臺(tái)redis服務(wù)器, 實(shí)現(xiàn)三套主從, 每套主從關(guān)系會(huì)有集群自動(dòng)設(shè)定
要求所有redis節(jié)點(diǎn), 必須沒有任何數(shù)據(jù)
10.0.0.81
10.0.0.82
10.0.0.83
10.0.0.84
10.0.0.85
10.0.0.86
1. 所有節(jié)點(diǎn)安裝redis, 備份redis配置文件
yum -y install redis
cp -a /etc/redis.conf /etc/redis.conf.bak
2. 修改配置文件
bind 0.0.0.0
masterauth redis #必須配置, 否則后期master和slave主從無法切換, 還要配置
requirepass redis
cluster-enabled yes #取消此行注釋, 必須開啟集群模式, 開啟后redis進(jìn)行會(huì)顯示為 [cluster]
cluster-config-file nodes-6379.conf #取消此行注釋, 此為集群狀態(tài)文件, 記錄主從關(guān)系及slot范圍信息, 由redis cluster集群自動(dòng)創(chuàng)建和維護(hù)
cluster-require-full-coverage no #默認(rèn)值為yes, 設(shè)為no可以防止當(dāng)一套主從節(jié)點(diǎn)全不可用而導(dǎo)致整個(gè)cluster不可用. 因?yàn)閏luster中每套主從都是存放和為維護(hù)固定的槽位, 固定的數(shù)據(jù), 不能因?yàn)橐徊糠植畚徊豢捎镁蛯?dǎo)致整個(gè)cluster不可用
利用sed修改配置文件, 在每一臺(tái)redis節(jié)點(diǎn)都要執(zhí)行
[20:17:37 root@81 ~]#sed -i.bak -e 's/^bind 127.0.0.1/bind 0.0.0.0/' -e '/masterauth/a masterauth redis' -e '/# requirepass/a requirepass redis' -e '/# cluster-enabled yes/a cluster-enabled yes' -e '/cluster-config-file nodes-6379.conf/a cluster-config-file nodes-6379.conf' -e '/cluster-require-full-coverage yes/c cluster-require-full-coverage no' /etc/redis.conf
[20:19:11 root@81 ~]#diff /etc/redis.conf /etc/redis.conf.bak
69c69
< bind 0.0.0.0
---
> bind 127.0.0.1
294d293
< masterauth redis
509d507
< requirepass redis
841d838
< cluster-enabled yes
850d846
< cluster-config-file nodes-6379.conf
933c929
< cluster-require-full-coverage no
---
> # cluster-require-full-coverage yes
3. 所有節(jié)點(diǎn)啟動(dòng)redis服務(wù)
systemctl enable --now redis
驗(yàn)證端口綁定和運(yùn)行進(jìn)行
- 啟動(dòng)cluster后, redis會(huì)額外監(jiān)聽在16379端口
- 并且, 進(jìn)程是以[cluster]模式運(yùn)行
[20:25:25 root@81 ~]#ss -ntlp
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 0.0.0.0:6379 0.0.0.0:* users:(("redis-server",pid=22806,fd=6))
LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=823,fd=3))
LISTEN 0 128 0.0.0.0:16379 0.0.0.0:* users:(("redis-server",pid=22806,fd=8))
LISTEN 0 128 [::]:22 [::]:* users:(("sshd",pid=823,fd=4))
redis 22806 0.1 0.5 53520 10060 ? Ssl 20:24 0:00 /usr/bin/redis-server 0.0.0.0:6379 [cluster]
連接到redis, 查看變化
# Cluster #開啟集群后, 會(huì)多一個(gè)Cluster字段
cluster_enabled:1
此時(shí), 無法立即寫輸入, 因?yàn)檫€沒有分配slot槽位, 所以數(shù)據(jù)是不知道往哪個(gè)節(jié)點(diǎn)存放的
[20:28:06 root@81 ~]#redis-cli -a redis set course linux
(error) CLUSTERDOWN Hash slot not served
4. 創(chuàng)建集群
只需在任意cluster節(jié)點(diǎn)執(zhí)行創(chuàng)建集群的命令即可
該命令會(huì)創(chuàng)建集群, 并且自動(dòng)分配好主從關(guān)系, 一般寫在前面的節(jié)點(diǎn)為主節(jié)點(diǎn), 寫在后面的節(jié)點(diǎn)為從節(jié)點(diǎn), 并且分配好槽位給每個(gè)主節(jié)點(diǎn)
[20:34:17 root@81 ~]#redis-cli -a redis --cluster create 10.0.0.81:6379 10.0.0.82:6379 10.0.0.83:6379 10.0.0.84:6379 10.0.0.85:6379 10.0.0.86:6379 --cluster-replicas 1 #1表示, 每個(gè)主帶一個(gè)從
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460 # 給每個(gè)master分配槽位
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 10.0.0.84:6379 to 10.0.0.81:6379 # 設(shè)定每套主從的主從關(guān)系
Adding replica 10.0.0.85:6379 to 10.0.0.82:6379
Adding replica 10.0.0.86:6379 to 10.0.0.83:6379
M: 0599da2e785b53626bb1292b371845d943563c33 10.0.0.81:6379 # 主節(jié)點(diǎn)信息和槽位分配
slots:[0-5460] (5461 slots) master
M: fc9198a6ce8e06dd78f97a766bb067a0d2d43d7b 10.0.0.82:6379
slots:[5461-10922] (5462 slots) master
M: 4ee223748925c7a081746bbf7088eb6c8ae3ba2e 10.0.0.83:6379
slots:[10923-16383] (5461 slots) master
S: d72390a3603a459a5a7c0cf652a187b73a785802 10.0.0.84:6379 # 從節(jié)點(diǎn)信息, 以及指向的主節(jié)點(diǎn)
replicates 0599da2e785b53626bb1292b371845d943563c33
S: ec0e9c4d56abbc604718ab4f611782e533f1eb8c 10.0.0.85:6379
replicates fc9198a6ce8e06dd78f97a766bb067a0d2d43d7b
S: 584d6494b4fcb5cdee237a61a3b80cb2fd20ffd5 10.0.0.86:6379
replicates 4ee223748925c7a081746bbf7088eb6c8ae3ba2e
Can I set the above configuration? (type 'yes' to accept): yes # 輸入yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
......
>>> Performing Cluster Check (using node 10.0.0.81:6379)
M: 0599da2e785b53626bb1292b371845d943563c33 10.0.0.81:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: d72390a3603a459a5a7c0cf652a187b73a785802 10.0.0.84:6379
slots: (0 slots) slave
replicates 0599da2e785b53626bb1292b371845d943563c33
S: ec0e9c4d56abbc604718ab4f611782e533f1eb8c 10.0.0.85:6379
slots: (0 slots) slave
replicates fc9198a6ce8e06dd78f97a766bb067a0d2d43d7b
S: 584d6494b4fcb5cdee237a61a3b80cb2fd20ffd5 10.0.0.86:6379
slots: (0 slots) slave
replicates 4ee223748925c7a081746bbf7088eb6c8ae3ba2e
M: 4ee223748925c7a081746bbf7088eb6c8ae3ba2e 10.0.0.83:6379
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
M: fc9198a6ce8e06dd78f97a766bb067a0d2d43d7b 10.0.0.82:6379
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration. # 節(jié)點(diǎn)槽位分配成功
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered. # 節(jié)點(diǎn)槽位分配成功
驗(yàn)證cluster配置
[20:37:37 root@81 ~]#redis-cli -a redis
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> cluster info
cluster_state:ok # cluster狀態(tài)
cluster_slots_assigned:16384 # 一共分配了多少個(gè)槽位
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6 # cluster一共有多少個(gè)節(jié)點(diǎn)
cluster_size:3 # cluster有幾套主從
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:161
cluster_stats_messages_pong_sent:155
cluster_stats_messages_sent:316
cluster_stats_messages_ping_received:150
cluster_stats_messages_pong_received:161
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:316
查看nodes-6379.conf文件
[20:40:11 root@81 ~]#cat /var/lib/redis/nodes-6379.conf # 每個(gè)節(jié)點(diǎn)上的文件內(nèi)容一致, 但是前后行的順序不一定相同
d72390a3603a459a5a7c0cf652a187b73a785802 10.0.0.84:6379@16379 slave 0599da2e785b53626bb1292b371845d943563c33 0 1615184835000 4 connected
ec0e9c4d56abbc604718ab4f611782e533f1eb8c 10.0.0.85:6379@16379 slave fc9198a6ce8e06dd78f97a766bb067a0d2d43d7b 0 1615184838000 5 connected
584d6494b4fcb5cdee237a61a3b80cb2fd20ffd5 10.0.0.86:6379@16379 slave 4ee223748925c7a081746bbf7088eb6c8ae3ba2e 0 1615184837000 6 connected
4ee223748925c7a081746bbf7088eb6c8ae3ba2e 10.0.0.83:6379@16379 master - 0 1615184836000 3 connected 10923-16383
fc9198a6ce8e06dd78f97a766bb067a0d2d43d7b 10.0.0.82:6379@16379 master - 0 1615184838000 2 connected 5461-10922
0599da2e785b53626bb1292b371845d943563c33 10.0.0.81:6379@16379 myself,master - 0 1615184837000 1 connected 0-5460 # myself表示在本機(jī)查看的該文件, 后面的master是表示主節(jié)點(diǎn)
vars currentEpoch 6 lastVoteEpoch 0
到這里, 集群創(chuàng)建完畢, 主從關(guān)系確定, 槽位分配完畢
主節(jié)點(diǎn) |||| 從節(jié)點(diǎn)
10.0.0.81 10.0.0.84
10.0.0.82 10.0.0.85
10.0.0.83 10.0.0.86
5. 連接redis創(chuàng)建數(shù)據(jù)
此時(shí), 由于搭建了集群, 因此, 每次插入數(shù)據(jù)時(shí), redis會(huì)計(jì)算key的hash值, 與key對(duì)應(yīng)的value無關(guān), 根據(jù)結(jié)果, 得到屬于的槽位, 存到相應(yīng)的節(jié)點(diǎn)上, 因此, 如果連接到節(jié)點(diǎn)A執(zhí)行添加數(shù)據(jù), 但是經(jīng)過redis計(jì)算, 該數(shù)據(jù)屬于的槽位是在B節(jié)點(diǎn)上, 那么添加數(shù)據(jù)時(shí), A就會(huì)返還客戶端消息, 告訴redis客戶端要添加的數(shù)據(jù)屬于的槽位在B節(jié)點(diǎn)上. 之后客戶端會(huì)直接向B重新發(fā)送命令. 該過程由redis內(nèi)部自動(dòng)完成.
注意: redis會(huì)告訴redis客戶端去哪存, 但是不會(huì)幫助客戶端到正確的節(jié)點(diǎn)存數(shù)據(jù), 客戶端需要重新連到正確的節(jié)點(diǎn)去存數(shù)據(jù).
圖片.png
127.0.0.1:6379> set key value
(error) MOVED 12539 10.0.0.83:6379 #redis經(jīng)過計(jì)算, 發(fā)現(xiàn)key的哈希值應(yīng)該在slot-12539, 而12539在10.0.0.83上. 因此會(huì)告訴客戶端去向10.0.0.83發(fā)送命令存數(shù)據(jù). 之后, 客戶端要重新向10.0.0.83發(fā)起連接, 并且set數(shù)據(jù), 這一步(error)MOVED并不會(huì)把數(shù)據(jù)key和value寫到10.0.0.83中
另外, 如果在A機(jī)器上查一個(gè)屬于B機(jī)器slot的key, 那么查詢時(shí), redis也會(huì)返還給客戶端, 這個(gè)key屬于哪個(gè)節(jié)點(diǎn)的哪個(gè)slot. 但是, 這里是基于A要查詢的key的slot位置來告訴客戶端, 但是在目標(biāo)的redis節(jié)點(diǎn), 不一定有這些key.
比如在A節(jié)點(diǎn)查看name, A節(jié)點(diǎn)經(jīng)過計(jì)算, 發(fā)現(xiàn)name屬于的槽位應(yīng)該在B節(jié)點(diǎn), 那么A就會(huì)告訴客戶端去連接到B節(jié)點(diǎn)查看name的值, 但是此時(shí), B節(jié)點(diǎn)并不一定存在name這個(gè)key. 所以, A只是告訴客戶端某個(gè)key應(yīng)該是在哪個(gè)節(jié)點(diǎn)上, 但自己并不知道這個(gè)key是否真的存在
6. 利用-c集群模式操作redis
-c: enable cluster mode (follow -ASK and -MOVED redirections)
使用-c選項(xiàng), redis會(huì)直接幫客戶端完成查找或者添加的數(shù)據(jù)操作, 不會(huì)只是提醒正確的slots槽位在哪個(gè)redis節(jié)點(diǎn)上.
[21:56:34 root@83 ~]#redis-cli -a redis -h 10.0.0.82 set key2 haha
(error) MOVED 4998 10.0.0.81:6379
[21:56:35 root@83 ~]#redis-cli -a redis -h 10.0.0.81 get key2 # 默認(rèn)情況下, 并不會(huì)負(fù)責(zé)幫助客戶端完成數(shù)據(jù)的寫入
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
(nil)
[21:56:40 root@83 ~]#redis-cli -c -a redis -h 10.0.0.82 set key2 haha
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
OK
[21:56:55 root@83 ~]#redis-cli -a redis -h 10.0.0.81 get key2
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
"haha"
注意: 集群cluser是不支持keys *的, 只能查到本機(jī)redis上所有的key, 無法查詢整個(gè)集群的key
7. 利用python腳本實(shí)現(xiàn)集群數(shù)據(jù)導(dǎo)入
yum -y install python3
pip3 install --upgrade pip
pip3 install redis-py-cluster
vim redis_cluster_test.py
#! /usr/bin/env python3
from rediscluster import RedisCluster
startup_nodes = [
{"host":"10.0.0.81","port":6379},
{"host":"10.0.0.82","port":6379},
{"host":"10.0.0.83","port":6379},
{"host":"10.0.0.84","port":6379},
{"host":"10.0.0.85","port":6379},
{"host":"10.0.0.86","port":6379}
]
redis_conn = RedisCluster(startup_nodes = startup_nodes, password = 'redis', decode_responses = True)
for i in range(0,10000):
redis_conn.set('key'+str(i), 'value'+str(i))
print('key'+str(i)+':', redis_conn.get('key'+str(i)))
chmod +x redis_cluster_test.py
./redis_cluster_test.py
...
key9996: value9996
key9997: value9997
key9998: value9998
key9999: value9999
這10000個(gè)key會(huì)被按照槽點(diǎn), 平均分配到3個(gè)主節(jié)點(diǎn)上, 并且同步到各自的從節(jié)點(diǎn)
主: 10.0.0.83
# Keyspace
db0:keys=3329,expires=0,avg_ttl=0
從: 10.0.0.86
# Keyspace
db0:keys=3329,expires=0,avg_ttl=0
8. 測(cè)試故障切換
目前10.0.0.81是主節(jié)點(diǎn), 10.0.0.84是其從節(jié)點(diǎn), 構(gòu)成一套主從
現(xiàn)在, 停止10.0.0.81上的redis服務(wù), 觀察是否會(huì)提升10.0.0.84為主節(jié)點(diǎn)
10.0.0.81
systemctl stop redis
觀察10.0.0.84上redis日志, 可以看到10.0.0.84提升為了新的主節(jié)點(diǎn)
停止10.0.0.81上redis服務(wù)器后, 10.0.0.84提升為主節(jié)點(diǎn)的整個(gè)日志
可以看到, 從節(jié)點(diǎn)經(jīng)過若干次的嘗試連接后, 默認(rèn)15秒超時(shí)時(shí)間岂贩，其余主節(jié)點(diǎn)會(huì)通知該從節(jié)點(diǎn)其主已經(jīng)宕機(jī)
1777:S 08 Mar 2021 15:51:36.237 # Connection with master lost.
1777:S 08 Mar 2021 15:51:36.237 * Caching the disconnected master state.
1777:S 08 Mar 2021 15:51:36.788 * Connecting to MASTER 10.0.0.81:6379
1777:S 08 Mar 2021 15:51:36.788 * MASTER <-> REPLICA sync started
1777:S 08 Mar 2021 15:51:36.790 # Error condition on socket for SYNC: Connection refused
1777:S 08 Mar 2021 15:51:37.807 * Connecting to MASTER 10.0.0.81:6379
1777:S 08 Mar 2021 15:51:37.807 * MASTER <-> REPLICA sync started
1777:S 08 Mar 2021 15:51:37.807 # Error condition on socket for SYNC: Connection refused
1777:S 08 Mar 2021 15:51:38.828 * Connecting to MASTER 10.0.0.81:6379
1777:S 08 Mar 2021 15:51:38.828 * MASTER <-> REPLICA sync started
1777:S 08 Mar 2021 15:51:38.828 # Error condition on socket for SYNC: Connection refused
1777:S 08 Mar 2021 15:51:39.857 * Connecting to MASTER 10.0.0.81:6379
1777:S 08 Mar 2021 15:51:39.858 * MASTER <-> REPLICA sync started
1777:S 08 Mar 2021 15:51:39.860 # Error condition on socket for SYNC: Connection refused
1777:S 08 Mar 2021 15:51:40.887 * Connecting to MASTER 10.0.0.81:6379
1777:S 08 Mar 2021 15:51:40.888 * MASTER <-> REPLICA sync started
1777:S 08 Mar 2021 15:51:40.889 # Error condition on socket for SYNC: Connection refused
1777:S 08 Mar 2021 15:51:41.911 * Connecting to MASTER 10.0.0.81:6379
1777:S 08 Mar 2021 15:51:41.912 * MASTER <-> REPLICA sync started
1777:S 08 Mar 2021 15:51:41.912 # Error condition on socket for SYNC: Connection refused
1777:S 08 Mar 2021 15:51:42.941 * Connecting to MASTER 10.0.0.81:6379
1777:S 08 Mar 2021 15:51:42.943 * MASTER <-> REPLICA sync started
1777:S 08 Mar 2021 15:51:42.945 # Error condition on socket for SYNC: Connection refused
1777:S 08 Mar 2021 15:51:43.967 * Connecting to MASTER 10.0.0.81:6379
1777:S 08 Mar 2021 15:51:43.969 * MASTER <-> REPLICA sync started
1777:S 08 Mar 2021 15:51:43.970 # Error condition on socket for SYNC: Connection refused
1777:S 08 Mar 2021 15:51:45.004 * Connecting to MASTER 10.0.0.81:6379
1777:S 08 Mar 2021 15:51:45.004 * MASTER <-> REPLICA sync started
1777:S 08 Mar 2021 15:51:45.005 # Error condition on socket for SYNC: Connection refused
1777:S 08 Mar 2021 15:51:46.022 * Connecting to MASTER 10.0.0.81:6379
1777:S 08 Mar 2021 15:51:46.024 * MASTER <-> REPLICA sync started
1777:S 08 Mar 2021 15:51:46.024 # Error condition on socket for SYNC: Connection refused
1777:S 08 Mar 2021 15:51:47.045 * Connecting to MASTER 10.0.0.81:6379
1777:S 08 Mar 2021 15:51:47.046 * MASTER <-> REPLICA sync started
1777:S 08 Mar 2021 15:51:47.046 # Error condition on socket for SYNC: Connection refused
1777:S 08 Mar 2021 15:51:48.069 * Connecting to MASTER 10.0.0.81:6379
1777:S 08 Mar 2021 15:51:48.072 * MASTER <-> REPLICA sync started
1777:S 08 Mar 2021 15:51:48.073 # Error condition on socket for SYNC: Connection refused
c1777:S 08 Mar 2021 15:51:49.102 * Connecting to MASTER 10.0.0.81:6379
1777:S 08 Mar 2021 15:51:49.103 * MASTER <-> REPLICA sync started
1777:S 08 Mar 2021 15:51:49.103 # Error condition on socket for SYNC: Connection refused
1777:S 08 Mar 2021 15:51:50.121 * Connecting to MASTER 10.0.0.81:6379
1777:S 08 Mar 2021 15:51:50.122 * MASTER <-> REPLICA sync started
1777:S 08 Mar 2021 15:51:50.122 # Error condition on socket for SYNC: Connection refused
1777:S 08 Mar 2021 15:51:51.149 * Connecting to MASTER 10.0.0.81:6379
1777:S 08 Mar 2021 15:51:51.149 * MASTER <-> REPLICA sync started
1777:S 08 Mar 2021 15:51:51.149 # Error condition on socket for SYNC: Connection refused
^C1777:S 08 Mar 2021 15:51:52.164 * Connecting to MASTER 10.0.0.81:6379
1777:S 08 Mar 2021 15:51:52.164 * MASTER <-> REPLICA sync started
1777:S 08 Mar 2021 15:51:52.165 # Error condition on socket for SYNC: Connection refused
1777:S 08 Mar 2021 15:51:52.422 * FAIL message received from fc9198a6ce8e06dd78f97a766bb067a0d2d43d7b(10.0.0.82的masterid) about 0599da2e785b53626bb1292b371845d943563c33(10.0.0.81的masterid), 說明主節(jié)點(diǎn)宕機(jī), 會(huì)由其余主節(jié)點(diǎn)發(fā)現(xiàn)
1777:S 08 Mar 2021 15:51:52.469 # Start of election delayed for 505 milliseconds (rank #0, offset 142833). # 開啟選舉
1777:S 08 Mar 2021 15:51:52.977 # Starting a failover election for epoch 7.
1777:S 08 Mar 2021 15:51:53.015 # Failover election won: I'm the new master. # 此從節(jié)點(diǎn)選舉為新的主節(jié)點(diǎn)
1777:S 08 Mar 2021 15:51:53.015 # configEpoch set to 7 after successful failover
1777:M 08 Mar 2021 15:51:53.015 # Setting secondary replication ID to eb61dac029ac9a57d5f2396f91cac13529764b92, valid up to offset: 142834. New replication ID is 71a7f74e8666f718ed8f0503d63989ab61edc59d
1777:M 08 Mar 2021 15:51:53.015 * Discarding previously cached master state. # 此從節(jié)點(diǎn)清除此前的主節(jié)點(diǎn)信息緩存
查看10.0.0.82上的日志
[15:46:05 root@CentOS-8-2 ~]#tail /var/log/redis/redis.log
2157:C 08 Mar 2021 15:37:08.163 * DB saved on disk
2157:C 08 Mar 2021 15:37:08.164 * RDB: 4 MB of memory used by copy-on-write
2102:M 08 Mar 2021 15:37:08.236 * Background saving terminated with success
2102:M 08 Mar 2021 15:42:09.046 * 10 changes in 300 seconds. Saving...
2102:M 08 Mar 2021 15:42:09.047 * Background saving started by pid 2159
2159:C 08 Mar 2021 15:42:09.050 * DB saved on disk
2159:C 08 Mar 2021 15:42:09.050 * RDB: 2 MB of memory used by copy-on-write
2102:M 08 Mar 2021 15:42:09.152 * Background saving terminated with success
2102:M 08 Mar 2021 15:51:53.310 * Marking node 0599da2e785b53626bb1292b371845d943563c33 as failing (quorum reached). # 其余主節(jié)點(diǎn)會(huì)把故障主節(jié)點(diǎn)標(biāo)記為failling, 并且為故障主從提升新的主節(jié)點(diǎn)
2102:M 08 Mar 2021 15:51:53.874 # Failover auth granted to d72390a3603a459a5a7c0cf652a187b73a785802 for epoch 7
查看10.0.0.83上的日志
[15:46:05 root@CentOS-8-3 ~]#tail /var/log/redis/redis.log
2163:C 08 Mar 2021 15:37:08.103 * DB saved on disk
2163:C 08 Mar 2021 15:37:08.103 * RDB: 4 MB of memory used by copy-on-write
2102:M 08 Mar 2021 15:37:08.196 * Background saving terminated with success
2102:M 08 Mar 2021 15:42:09.054 * 10 changes in 300 seconds. Saving...
2102:M 08 Mar 2021 15:42:09.055 * Background saving started by pid 2165
2165:C 08 Mar 2021 15:42:09.065 * DB saved on disk
2165:C 08 Mar 2021 15:42:09.065 * RDB: 2 MB of memory used by copy-on-write
2102:M 08 Mar 2021 15:42:09.157 * Background saving terminated with success
2102:M 08 Mar 2021 15:51:53.301 * Marking node 0599da2e785b53626bb1292b371845d943563c33 as failing (quorum reached).
2102:M 08 Mar 2021 15:51:53.861 # Failover auth granted to d72390a3603a459a5a7c0cf652a187b73a785802 for epoch 7
查看10.0.0.85上的日志
[15:46:04 root@CentOS-8-5 ~]#tail /var/log/redis/redis.log
1785:S 08 Mar 2021 15:37:07.449 * Background saving started by pid 1841
1841:C 08 Mar 2021 15:37:07.502 * DB saved on disk
1841:C 08 Mar 2021 15:37:07.506 * RDB: 4 MB of memory used by copy-on-write
1785:S 08 Mar 2021 15:37:07.567 * Background saving terminated with success
1785:S 08 Mar 2021 15:42:08.012 * 10 changes in 300 seconds. Saving...
1785:S 08 Mar 2021 15:42:08.013 * Background saving started by pid 1843
1843:C 08 Mar 2021 15:42:08.019 * DB saved on disk
1843:C 08 Mar 2021 15:42:08.019 * RDB: 2 MB of memory used by copy-on-write
1785:S 08 Mar 2021 15:42:08.117 * Background saving terminated with success
1785:S 08 Mar 2021 15:51:52.505 * FAIL message received from fc9198a6ce8e06dd78f97a766bb067a0d2d43d7b about 0599da2e785b53626bb1292b371845d943563c33 # 其余從節(jié)點(diǎn)也會(huì)受到某個(gè)主節(jié)點(diǎn)宕機(jī)的信息
查看node文件
10.0.0.81因?yàn)橐呀?jīng)停止服務(wù), 所以node文件不會(huì)跟新
[15:51:36 root@CentOS-8-1 ~]#cat /var/lib/redis/nodes-6379.conf
d72390a3603a459a5a7c0cf652a187b73a785802 10.0.0.84:6379@16379 slave 0599da2e785b53626bb1292b371845d943563c33 0 1615184835000 4 connected
ec0e9c4d56abbc604718ab4f611782e533f1eb8c 10.0.0.85:6379@16379 slave fc9198a6ce8e06dd78f97a766bb067a0d2d43d7b 0 1615184838000 5 connected
584d6494b4fcb5cdee237a61a3b80cb2fd20ffd5 10.0.0.86:6379@16379 slave 4ee223748925c7a081746bbf7088eb6c8ae3ba2e 0 1615184837000 6 connected
4ee223748925c7a081746bbf7088eb6c8ae3ba2e 10.0.0.83:6379@16379 master - 0 1615184836000 3 connected 10923-16383
fc9198a6ce8e06dd78f97a766bb067a0d2d43d7b 10.0.0.82:6379@16379 master - 0 1615184838000 2 connected 5461-10922
0599da2e785b53626bb1292b371845d943563c33 10.0.0.81:6379@16379 myself,master - 0 1615184837000 1 connected 0-5460
vars currentEpoch 6 lastVoteEpoch 0
其余所有節(jié)點(diǎn)的node文件都會(huì)顯示10.0.0.81故障
[15:58:44 root@CentOS-8-5 ~]#cat /var/lib/redis/nodes-6379.conf
ec0e9c4d56abbc604718ab4f611782e533f1eb8c 10.0.0.85:6379@16379 myself,slave fc9198a6ce8e06dd78f97a766bb067a0d2d43d7b 0 1615189908000 5 connected
0599da2e785b53626bb1292b371845d943563c33 10.0.0.81:6379@16379 master,fail - 1615189896412 1615189893000 1 disconnected
4ee223748925c7a081746bbf7088eb6c8ae3ba2e 10.0.0.83:6379@16379 master - 0 1615189911388 3 connected 10923-16383
d72390a3603a459a5a7c0cf652a187b73a785802 10.0.0.84:6379@16379 master - 0 1615189912408 7 connected 0-5460
fc9198a6ce8e06dd78f97a766bb067a0d2d43d7b 10.0.0.82:6379@16379 master - 0 1615189912000 2 connected 5461-10922
584d6494b4fcb5cdee237a61a3b80cb2fd20ffd5 10.0.0.86:6379@16379 slave 4ee223748925c7a081746bbf7088eb6c8ae3ba2e 0 1615189911000 6 connected
vars currentEpoch 7 lastVoteEpoch 0
恢復(fù)10.0.0.81 redis服務(wù)
10.0.0.81
systemctl start redis
10.0.0.81觀察redis日志, 可以看到其會(huì)被自動(dòng)加入到集群中, 以10.0.0.84為其主節(jié)點(diǎn)
2674:M 08 Mar 2021 15:59:42.789 * Ready to accept connections
2674:M 08 Mar 2021 15:59:42.796 # Configuration change detected. Reconfiguring myself as a replica of d72390a3603a459a5a7c0cf652a187b73a785802
2674:S 08 Mar 2021 15:59:42.796 * Before turning into a replica, using my master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
2674:S 08 Mar 2021 15:59:42.796 # Cluster state changed: ok
2674:S 08 Mar 2021 15:59:43.813 * Connecting to MASTER 10.0.0.84:6379 # 以10.0.0l.84為主節(jié)點(diǎn)
2674:S 08 Mar 2021 15:59:43.813 * MASTER <-> REPLICA sync started
2674:S 08 Mar 2021 15:59:43.813 * Non blocking connect for SYNC fired the event.
2674:S 08 Mar 2021 15:59:43.814 * Master replied to PING, replication can continue...
2674:S 08 Mar 2021 15:59:43.815 * Trying a partial resynchronization (request e45c9c57b27adf0c536f056286e3ed099569b2d2:1).
2674:S 08 Mar 2021 15:59:43.817 * Full resync from master: 71a7f74e8666f718ed8f0503d63989ab61edc59d:142833 # 重新加入到cluster后會(huì)進(jìn)行全量復(fù)制， 因?yàn)楣收系闹鞴?jié)點(diǎn)原先記錄的master_replid是它自己的钾埂， 因此發(fā)送給10.0.0.84新的主節(jié)點(diǎn)后河闰， 會(huì)不匹配
2674:S 08 Mar 2021 15:59:43.817 * Discarding previously cached master state.
2674:S 08 Mar 2021 15:59:43.863 * MASTER <-> REPLICA sync: receiving 62735 bytes from master
2674:S 08 Mar 2021 15:59:43.877 * MASTER <-> REPLICA sync: Flushing old data
2674:S 08 Mar 2021 15:59:43.882 * MASTER <-> REPLICA sync: Loading DB in memory
2674:S 08 Mar 2021 15:59:43.888 * MASTER <-> REPLICA sync: Finished with success
10.0.0.84觀察redis日志, 可以看到10.0.0.81在服務(wù)恢復(fù)后成為了84的從節(jié)點(diǎn)
1777:M 08 Mar 2021 15:59:42.729 * Clear FAIL state for node 0599da2e785b53626bb1292b371845d943563c33: master without slots is reachable again. # 清除fail信息
1777:M 08 Mar 2021 15:59:43.702 * Replica 10.0.0.81:6379 asks for synchronization
1777:M 08 Mar 2021 15:59:43.702 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for 'e45c9c57b27adf0c536f056286e3ed099569b2d2', my replication IDs are '71a7f74e8666f718ed8f0503d63989ab61edc59d' and 'eb61dac029ac9a57d5f2396f91cac13529764b92')
1777:M 08 Mar 2021 15:59:43.702 * Starting BGSAVE for SYNC with target: disk
1777:M 08 Mar 2021 15:59:43.704 * Background saving started by pid 1861
1861:C 08 Mar 2021 15:59:43.708 * DB saved on disk
1861:C 08 Mar 2021 15:59:43.709 * RDB: 2 MB of memory used by copy-on-write
1777:M 08 Mar 2021 15:59:43.749 * Background saving terminated with success
1777:M 08 Mar 2021 15:59:43.762 * Synchronization with replica 10.0.0.81:6379 succeeded
再次停止10.0.0.84上redis服務(wù), 使其成為10.0.0.81從節(jié)點(diǎn), 恢復(fù)初始主從關(guān)系
10.0.0.84
systemctl stop redis
手動(dòng)切換主節(jié)點(diǎn)時(shí), 不要直接重啟, 或者關(guān)閉后馬上啟動(dòng)redis, 要等待超時(shí)時(shí)間到達(dá), 集群完成新的主節(jié)點(diǎn)提升, 在開啟故障的redis節(jié)點(diǎn)服務(wù)
systemctl start redis
注意:
在集群模式下, 所有的讀寫操作都是發(fā)生在主節(jié)點(diǎn), 主節(jié)點(diǎn)寫入數(shù)據(jù)后, 會(huì)同步給從節(jié)點(diǎn). 在沒有實(shí)現(xiàn)讀寫分離的情況下, 從節(jié)點(diǎn)只起到冗余備份作用.
在集群模式下, 在從節(jié)點(diǎn)想要讀寫數(shù)據(jù),都會(huì)被重定向到主節(jié)點(diǎn). 可以在從節(jié)點(diǎn)看到所有的key, 但是想讀的話, 就會(huì)被重定向的主節(jié)點(diǎn).
10.0.0.86上執(zhí)行keys *可以看到本從節(jié)點(diǎn)存放的key
127.0.0.1:6379> keys *
...
3327) "key6300"
3328) "key3461"
3329) "key8906"
但是直接在從節(jié)點(diǎn)讀數(shù)據(jù), 會(huì)被重定向到其主節(jié)點(diǎn)
127.0.0.1:6379> get key8906
(error) MOVED 12488 10.0.0.83:6379
可以指定-c集群模式, 完成自動(dòng)的讀取和寫入
redis-cli -a redis -c get key8906
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
"value8906"
在從節(jié)點(diǎn)執(zhí)行寫操作, 也會(huì)被重定向到對(duì)應(yīng)的槽位的主節(jié)點(diǎn)
[15:49:37 root@CentOS-8-6 ~]#redis-cli -a redis set haha lala
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
(error) MOVED 3662 10.0.0.81:6379
在主從或者主從+哨兵模式下, 可以直接在從節(jié)點(diǎn)讀數(shù)據(jù).
9. cluster相關(guān)配置參數(shù)
cluster-enabled yes #是否開啟集群模式，默認(rèn)是單機(jī)模式
cluster-config-file nodes-6379.conf #由node節(jié)點(diǎn)自動(dòng)生成的集群配置文件名稱
cluster-node-timeout 15000 #集群中node節(jié)點(diǎn)連接超時(shí)時(shí)間褥紫，超過此時(shí)間姜性，會(huì)踢出集群, 默認(rèn)15秒, 從節(jié)點(diǎn)在15秒內(nèi)無法連接主節(jié)點(diǎn), 那么其余主節(jié)點(diǎn)會(huì)通知該從節(jié)點(diǎn),其主節(jié)點(diǎn)已宕機(jī), 之后該從節(jié)點(diǎn)會(huì)被提升為主節(jié)點(diǎn)
cluster-replica-validity-factor 10 #在執(zhí)行故障轉(zhuǎn)移的時(shí)候可能有些節(jié)點(diǎn)和master斷開一段時(shí)間數(shù)據(jù)比較舊，這些節(jié)點(diǎn)就不適用于選舉為master髓考，超過這個(gè)時(shí)間的就不會(huì)被進(jìn)行故障轉(zhuǎn)移部念，計(jì)算公式：(node-timeout * replica-validity-factor) + repl-ping-replica-period
cluster-migration-barrier 1 #集群遷移屏障，一個(gè)主節(jié)點(diǎn)至少擁有一個(gè)正常工作的從節(jié)點(diǎn)，即如果主節(jié)點(diǎn)的slave節(jié)點(diǎn)故障后會(huì)將多余的從節(jié)點(diǎn)分配到當(dāng)前主節(jié)點(diǎn)成為其新的從節(jié)點(diǎn)儡炼。
cluster-require-full-coverage yes #集群請(qǐng)求槽位全部覆蓋妓湘，如果一個(gè)主庫宕機(jī)且沒有備庫就會(huì)出現(xiàn)集群槽位不全，那么yes情況下redis集群槽位驗(yàn)證不全就不再對(duì)外提供服務(wù)乌询，而no則可以繼續(xù)使用但是會(huì)出現(xiàn)查詢數(shù)據(jù)查不到的情況(因?yàn)橛袛?shù)據(jù)丟失)榜贴。建議為no
cluster-replica-no-failover no #如果為yes,此選項(xiàng)阻止在主服務(wù)器發(fā)生故障時(shí)嘗試對(duì)其主服務(wù)器進(jìn)行故障轉(zhuǎn)移。 但是妹田，主服務(wù)器仍然可以執(zhí)行手動(dòng)強(qiáng)制故障轉(zhuǎn)移唬党，一般為no
10. 一套主從全部宕機(jī)的結(jié)果
cluster中, 如果一套主從的所有節(jié)點(diǎn)都宕機(jī), 那么原本由該主從負(fù)責(zé)存入的數(shù)據(jù)槽位, 會(huì)被重定向到其他的主從架構(gòu), 不影響新的數(shù)據(jù)的寫操作, 但是該主從現(xiàn)有的數(shù)據(jù)就沒法查看了, 無法讀取已存的數(shù)據(jù), 影響讀操作
10.3 Redis Cluster 集群節(jié)點(diǎn)維護(hù)
集群運(yùn)行時(shí)間長(zhǎng)久之后，難免由于硬件故障鬼佣、網(wǎng)絡(luò)規(guī)劃驶拱、業(yè)務(wù)增長(zhǎng)等原因?qū)σ延屑哼M(jìn)行相應(yīng)的調(diào)整， 比如增加Redis節(jié)點(diǎn)晶衷、減少節(jié)點(diǎn)蓝纲、節(jié)點(diǎn)遷移、更換服務(wù)器等
無論是增加還是減少主從數(shù)量, 最后都要確保是有奇數(shù)個(gè)主從架構(gòu), 因?yàn)楣?jié)點(diǎn)之間要進(jìn)行選舉, 防止出現(xiàn)偶數(shù)平票情況的腦裂現(xiàn)象
增加節(jié)點(diǎn)和刪除節(jié)點(diǎn)會(huì)涉及到已有的槽位重新分配及數(shù)據(jù)遷移
10.3.1 Redis Cluster 集群節(jié)點(diǎn)維護(hù)之動(dòng)態(tài)擴(kuò)容
這里只增加一套主從, 因?yàn)殡娔X內(nèi)存不夠了...
10.0.0.87為主, 10.0.0.88為從
- 配置新的節(jié)點(diǎn)設(shè)備
#配置node7節(jié)點(diǎn), 10.0.0.87
[root@redis-node7 ~]#dnf -y install redis
[root@redis-node7 ~]#sed -i.bak -e 's/bind 127.0.0.1/bind 0.0.0.0/' -e '/masterauth/a masterauth redis' -e '/# requirepass/a requirepass redis' -e '/# cluster-enabled yes/a cluster-enabled yes' -e '/# cluster-config-file nodes-6379.conf/a cluster-config-file nodes-6379.conf' -e '/appendonly no/c appendonly yes' /etc/redis.conf
[root@redis-node7 ~]#systemctl start redis
#配置node8節(jié)點(diǎn), 10.0.0.88
[root@redis-node8 ~]#dnf -y install redis
[root@redis-node8 ~]#sed -i.bak -e 's/bind 127.0.0.1/bind 0.0.0.0/' -e '/masterauth/a masterauth redis' -e '/# requirepass/a requirepass redis' -e '/# cluster-enabled yes/a cluster-enabled yes' -e '/# cluster-config-file nodes-6379.conf/a cluster-config-file nodes-6379.conf' -e '/appendonly no/c appendonly yes' /etc/redis.conf
[root@redis-node8 ~]#systemctl start redis
分別啟動(dòng)redis
systemctl enable --now redis
- 添加新的master(node7-10.0.0.87)到集群
使用以下命令添加新節(jié)點(diǎn)晌纫，要添加新redis節(jié)點(diǎn)IP和端口到已有的集群中任意節(jié)點(diǎn)的IP:端口
add-node new_host:new_port existing_host:existing_port
說明：
new_host:new_port #為新添加的主機(jī)的IP和端口
existing_host:existing_port #為已有的集群中任意節(jié)點(diǎn)的IP和端口
Redis 5 添加方式
#將一臺(tái)新的主機(jī)10.0.0.87加入集群, 以下示例中的10.0.0.85可以是任意存在的集群節(jié)點(diǎn), 是主還是從無所謂
[23:13:05 root@81 ~]#redis-cli -a redis --cluster add-node 10.0.0.87:6379 10.0.0.85:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Adding node 10.0.0.87:6379 to cluster 10.0.0.85:6379
>>> Performing Cluster Check (using node 10.0.0.85:6379)
S: 34f27507a24e436a121527344fab181e9877404a 10.0.0.85:6379
slots: (0 slots) slave
replicates 104844ad87a3e145884b520980cedc70232986f7
M: 104844ad87a3e145884b520980cedc70232986f7 10.0.0.82:6379
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 609ff8b268daa75a2fe0061d38f732b5ef859fb6 10.0.0.84:6379
slots: (0 slots) slave
replicates 6a789b9a447c400581df3c63071104f16032f2c6
M: 6a789b9a447c400581df3c63071104f16032f2c6 10.0.0.81:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: bd496dc81e17ccfdf8431d136254d083ab4bb145 10.0.0.86:6379
slots: (0 slots) slave
replicates 1e4c9a3e6fe5fce70617bf417ef76463a33e334f
M: 1e4c9a3e6fe5fce70617bf417ef76463a33e334f 10.0.0.83:6379
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 10.0.0.87:6379 to make it join the cluster.
[OK] New node added correctly.
#觀察到10.0.0.87已經(jīng)加入成功税迷，但沒有slot位，而且新的主機(jī)是master
[23:21:55 root@81 ~]#redis-cli -a redis --cluster info 10.0.0.81:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
10.0.0.81:6379 (6a789b9a...) -> 3331 keys | 5461 slots | 1 slaves.
10.0.0.87:6379 (7ccff9f7...) -> 0 keys | 0 slots | 0 slaves.
10.0.0.83:6379 (1e4c9a3e...) -> 3329 keys | 5461 slots | 1 slaves.
10.0.0.82:6379 (104844ad...) -> 3340 keys | 5462 slots | 1 slaves.
[OK] 10000 keys in 4 masters.
0.61 keys per slot on average.
[23:22:08 root@81 ~]#redis-cli -a redis --cluster check 10.0.0.81:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
10.0.0.81:6379 (6a789b9a...) -> 3331 keys | 5461 slots | 1 slaves.
10.0.0.87:6379 (7ccff9f7...) -> 0 keys | 0 slots | 0 slaves.
10.0.0.83:6379 (1e4c9a3e...) -> 3329 keys | 5461 slots | 1 slaves.
10.0.0.82:6379 (104844ad...) -> 3340 keys | 5462 slots | 1 slaves.
[OK] 10000 keys in 4 masters.
0.61 keys per slot on average.
>>> Performing Cluster Check (using node 10.0.0.81:6379)
M: 6a789b9a447c400581df3c63071104f16032f2c6 10.0.0.81:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
M: 7ccff9f724e4508303a8184df5b0903789979759 10.0.0.87:6379
slots: (0 slots) master
S: bd496dc81e17ccfdf8431d136254d083ab4bb145 10.0.0.86:6379
slots: (0 slots) slave
replicates 1e4c9a3e6fe5fce70617bf417ef76463a33e334f
S: 34f27507a24e436a121527344fab181e9877404a 10.0.0.85:6379
slots: (0 slots) slave
replicates 104844ad87a3e145884b520980cedc70232986f7
M: 1e4c9a3e6fe5fce70617bf417ef76463a33e334f 10.0.0.83:6379
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
S: 609ff8b268daa75a2fe0061d38f732b5ef859fb6 10.0.0.84:6379
slots: (0 slots) slave
replicates 6a789b9a447c400581df3c63071104f16032f2c6
M: 104844ad87a3e145884b520980cedc70232986f7 10.0.0.82:6379
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[23:22:58 root@81 ~]#cat /var/lib/redis/nodes-6379.conf
7ccff9f724e4508303a8184df5b0903789979759 10.0.0.87:6379@16379 master - 0 1621524090262 0 connected
6a789b9a447c400581df3c63071104f16032f2c6 10.0.0.81:6379@16379 myself,master - 0 1621524089000 8 connected 0-5460
bd496dc81e17ccfdf8431d136254d083ab4bb145 10.0.0.86:6379@16379 slave 1e4c9a3e6fe5fce70617bf417ef76463a33e334f 0 1621524088000 6 connected
34f27507a24e436a121527344fab181e9877404a 10.0.0.85:6379@16379 slave 104844ad87a3e145884b520980cedc70232986f7 0 1621524090259 5 connected
1e4c9a3e6fe5fce70617bf417ef76463a33e334f 10.0.0.83:6379@16379 master - 0 1621524088208 3 connected 10923-16383
609ff8b268daa75a2fe0061d38f732b5ef859fb6 10.0.0.84:6379@16379 slave 6a789b9a447c400581df3c63071104f16032f2c6 0 1621524089233 8 connected
104844ad87a3e145884b520980cedc70232986f7 10.0.0.82:6379@16379 master - 0 1621524089000 2 connected 5461-10922
vars currentEpoch 8 lastVoteEpoch 0
#和上面顯示結(jié)果一樣
[23:23:28 root@81 ~]#redis-cli -a redis CLUSTER NODES
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
7ccff9f724e4508303a8184df5b0903789979759 10.0.0.87:6379@16379 master - 0 1621524261243 0 connected
6a789b9a447c400581df3c63071104f16032f2c6 10.0.0.81:6379@16379 myself,master - 0 1621524260000 8 connected 0-5460
bd496dc81e17ccfdf8431d136254d083ab4bb145 10.0.0.86:6379@16379 slave 1e4c9a3e6fe5fce70617bf417ef76463a33e334f 0 1621524261000 6 connected
34f27507a24e436a121527344fab181e9877404a 10.0.0.85:6379@16379 slave 104844ad87a3e145884b520980cedc70232986f7 0 1621524261000 5 connected
1e4c9a3e6fe5fce70617bf417ef76463a33e334f 10.0.0.83:6379@16379 master - 0 1621524259000 3 connected 10923-16383
609ff8b268daa75a2fe0061d38f732b5ef859fb6 10.0.0.84:6379@16379 slave 6a789b9a447c400581df3c63071104f16032f2c6 0 1621524263291 8 connected
104844ad87a3e145884b520980cedc70232986f7 10.0.0.82:6379@16379 master - 0 1621524262259 2 connected 5461-10922
#查看集群狀態(tài)
[23:24:23 root@81 ~]#redis-cli -a redis CLUSTER INFO
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:7
cluster_size:3
cluster_current_epoch:8
cluster_my_epoch:8
cluster_stats_messages_ping_sent:1927
cluster_stats_messages_pong_sent:847
cluster_stats_messages_auth-req_sent:5
cluster_stats_messages_update_sent:2
cluster_stats_messages_sent:2781
cluster_stats_messages_ping_received:842
cluster_stats_messages_pong_received:845
cluster_stats_messages_meet_received:1
cluster_stats_messages_fail_received:2
cluster_stats_messages_auth-ack_received:2
cluster_stats_messages_received:1692
- 在新的master-10.0.0.87上重新分配槽位
新的node節(jié)點(diǎn)加到集群之后, 默認(rèn)是master節(jié)點(diǎn), 但是沒有槽位, 需要重新分配
添加主機(jī)之后需要對(duì)添加至集群中的新主機(jī)重新分配槽位, 否則其沒有槽位也就無法寫入數(shù)據(jù)
重新分配槽位,需要清空數(shù)據(jù), 所以需要先備份數(shù)據(jù), 擴(kuò)容完畢在恢復(fù)數(shù)據(jù). 需要讓開發(fā)把數(shù)據(jù)導(dǎo)出來, 擴(kuò)容后再導(dǎo)入, 因?yàn)閞db和aof只保存的是一個(gè)節(jié)點(diǎn)的數(shù)據(jù), 而cluster集群所有的數(shù)據(jù)會(huì)分布在每個(gè)節(jié)點(diǎn), 并且是按照槽位分配的, 因此不能像傳統(tǒng)的rdb和aof一樣導(dǎo)出導(dǎo)入
此外, 增加了槽位后, 原本master節(jié)點(diǎn)的原有槽位以及存儲(chǔ)的數(shù)據(jù)會(huì)被均分到新的master節(jié)點(diǎn), 造成數(shù)據(jù)遷移
Redis 5:
[23:26:00 root@81 ~]#redis-cli -a redis --cluster reshard 10.0.0.87:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing Cluster Check (using node 10.0.0.87:6379)
M: 7ccff9f724e4508303a8184df5b0903789979759 10.0.0.87:6379
slots: (0 slots) master
S: 34f27507a24e436a121527344fab181e9877404a 10.0.0.85:6379
slots: (0 slots) slave
replicates 104844ad87a3e145884b520980cedc70232986f7
M: 6a789b9a447c400581df3c63071104f16032f2c6 10.0.0.81:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
S: bd496dc81e17ccfdf8431d136254d083ab4bb145 10.0.0.86:6379
slots: (0 slots) slave
replicates 1e4c9a3e6fe5fce70617bf417ef76463a33e334f
M: 104844ad87a3e145884b520980cedc70232986f7 10.0.0.82:6379
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 609ff8b268daa75a2fe0061d38f732b5ef859fb6 10.0.0.84:6379
slots: (0 slots) slave
replicates 6a789b9a447c400581df3c63071104f16032f2c6
M: 1e4c9a3e6fe5fce70617bf417ef76463a33e334f 10.0.0.83:6379
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)?4096 #新分配多少個(gè)槽位=16384/master個(gè)數(shù)
What is the receiving node ID? 7ccff9f724e4508303a8184df5b0903789979759 #新的master,10.0.0.87的ID
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
Source node #1: all # 將哪些源主機(jī)的槽位分配給新的節(jié)點(diǎn)缸匪，all是自動(dòng)在所有的redis 節(jié)點(diǎn)選擇劃分翁狐，如果是從redis cluster刪除某個(gè)主機(jī)可以使用此方式指定某個(gè)節(jié)點(diǎn), 將指定主機(jī)上的槽位全部移動(dòng)到別的redis主機(jī)
......
Do you want to proceed with the proposed reshard plan (yes/no)? yes #確認(rèn)分配
......
Moving slot 5606 from 10.0.0.82:6379 to 10.0.0.87:6379:
Moving slot 5607 from 10.0.0.82:6379 to 10.0.0.87:6379: ..
Moving slot 5608 from 10.0.0.82:6379 to 10.0.0.87:6379:
Moving slot 5609 from 10.0.0.82:6379 to 10.0.0.87:6379:
Moving slot 5610 from 10.0.0.82:6379 to 10.0.0.87:6379: .
Moving slot 5611 from 10.0.0.82:6379 to 10.0.0.87:6379: ..
Moving slot 5612 from 10.0.0.82:6379 to 10.0.0.87:6379:
...
#確定slot分配成功
[23:31:35 root@81 ~]#redis-cli -a redis --cluster check 10.0.0.81:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
10.0.0.81:6379 (6a789b9a...) -> 2511 keys | 4096 slots | 1 slaves.
10.0.0.87:6379 (7ccff9f7...) -> 2474 keys | 4096 slots | 0 slaves.
10.0.0.83:6379 (1e4c9a3e...) -> 2500 keys | 4096 slots | 1 slaves.
10.0.0.82:6379 (104844ad...) -> 2515 keys | 4096 slots | 1 slaves.
[OK] 10000 keys in 4 masters.
0.61 keys per slot on average.
>>> Performing Cluster Check (using node 10.0.0.81:6379)
M: 6a789b9a447c400581df3c63071104f16032f2c6 10.0.0.81:6379
slots:[1365-5460] (4096 slots) master
1 additional replica(s)
M: 7ccff9f724e4508303a8184df5b0903789979759 10.0.0.87:6379
slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master # 可以看到分配了4096個(gè)槽位, 雖然不是連續(xù)的但沒關(guān)系
S: bd496dc81e17ccfdf8431d136254d083ab4bb145 10.0.0.86:6379
slots: (0 slots) slave
replicates 1e4c9a3e6fe5fce70617bf417ef76463a33e334f
S: 34f27507a24e436a121527344fab181e9877404a 10.0.0.85:6379
slots: (0 slots) slave
replicates 104844ad87a3e145884b520980cedc70232986f7
M: 1e4c9a3e6fe5fce70617bf417ef76463a33e334f 10.0.0.83:6379
slots:[12288-16383] (4096 slots) master
1 additional replica(s)
S: 609ff8b268daa75a2fe0061d38f732b5ef859fb6 10.0.0.84:6379
slots: (0 slots) slave
replicates 6a789b9a447c400581df3c63071104f16032f2c6
M: 104844ad87a3e145884b520980cedc70232986f7 10.0.0.82:6379
slots:[6827-10922] (4096 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
- 為新的master添加新的slave節(jié)點(diǎn), 10.0.0.88
需要再向當(dāng)前Redis集群中添加一個(gè)Redis單機(jī)服務(wù)器10.0.0.88, 用于解決當(dāng)前10.0.0.87單機(jī)的潛在宕機(jī)問題, 即實(shí)現(xiàn)高可用功能
在新加節(jié)點(diǎn)到集群時(shí)，直接將之設(shè)置為slave
Redis 5
#查看當(dāng)前狀態(tài)
[23:31:41 root@81 ~]#redis-cli -a redis --cluster check 10.0.0.81:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
10.0.0.81:6379 (6a789b9a...) -> 2511 keys | 4096 slots | 1 slaves.
10.0.0.87:6379 (7ccff9f7...) -> 2474 keys | 4096 slots | 0 slaves.
10.0.0.83:6379 (1e4c9a3e...) -> 2500 keys | 4096 slots | 1 slaves.
10.0.0.82:6379 (104844ad...) -> 2515 keys | 4096 slots | 1 slaves.
[OK] 10000 keys in 4 masters.
0.61 keys per slot on average.
>>> Performing Cluster Check (using node 10.0.0.81:6379)
M: 6a789b9a447c400581df3c63071104f16032f2c6 10.0.0.81:6379
slots:[1365-5460] (4096 slots) master
1 additional replica(s)
M: 7ccff9f724e4508303a8184df5b0903789979759 10.0.0.87:6379
slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master
S: bd496dc81e17ccfdf8431d136254d083ab4bb145 10.0.0.86:6379
slots: (0 slots) slave
replicates 1e4c9a3e6fe5fce70617bf417ef76463a33e334f
S: 34f27507a24e436a121527344fab181e9877404a 10.0.0.85:6379
slots: (0 slots) slave
replicates 104844ad87a3e145884b520980cedc70232986f7
M: 1e4c9a3e6fe5fce70617bf417ef76463a33e334f 10.0.0.83:6379
slots:[12288-16383] (4096 slots) master
1 additional replica(s)
S: 609ff8b268daa75a2fe0061d38f732b5ef859fb6 10.0.0.84:6379
slots: (0 slots) slave
replicates 6a789b9a447c400581df3c63071104f16032f2c6
M: 104844ad87a3e145884b520980cedc70232986f7 10.0.0.82:6379
slots:[6827-10922] (4096 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
#直接加為slave節(jié)點(diǎn)
[23:33:04 root@81 ~]#redis-cli -a redis --cluster add-node 10.0.0.88:6379 10.0.0.81:6379 --cluster-slave --cluster-master-id 7ccff9f724e4508303a8184df5b0903789979759 # 這里只需指明新添加的從節(jié)點(diǎn)的主節(jié)點(diǎn)id
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Adding node 10.0.0.88:6379 to cluster 10.0.0.81:6379
>>> Performing Cluster Check (using node 10.0.0.81:6379)
M: 6a789b9a447c400581df3c63071104f16032f2c6 10.0.0.81:6379
slots:[1365-5460] (4096 slots) master
1 additional replica(s)
M: 7ccff9f724e4508303a8184df5b0903789979759 10.0.0.87:6379
slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master
S: bd496dc81e17ccfdf8431d136254d083ab4bb145 10.0.0.86:6379
slots: (0 slots) slave
replicates 1e4c9a3e6fe5fce70617bf417ef76463a33e334f
S: 34f27507a24e436a121527344fab181e9877404a 10.0.0.85:6379
slots: (0 slots) slave
replicates 104844ad87a3e145884b520980cedc70232986f7
M: 1e4c9a3e6fe5fce70617bf417ef76463a33e334f 10.0.0.83:6379
slots:[12288-16383] (4096 slots) master
1 additional replica(s)
S: 609ff8b268daa75a2fe0061d38f732b5ef859fb6 10.0.0.84:6379
slots: (0 slots) slave
replicates 6a789b9a447c400581df3c63071104f16032f2c6
M: 104844ad87a3e145884b520980cedc70232986f7 10.0.0.82:6379
slots:[6827-10922] (4096 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
>>> Send CLUSTER MEET to node 10.0.0.88:6379 to make it join the cluster.
Waiting for the cluster to join
>>> Configure node as replica of 10.0.0.87:6379.
[OK] New node added correctly.
#驗(yàn)證是否成功
[23:33:25 root@81 ~]#redis-cli -a redis --cluster check 10.0.0.81:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
10.0.0.81:6379 (6a789b9a...) -> 2511 keys | 4096 slots | 1 slaves.
10.0.0.87:6379 (7ccff9f7...) -> 2474 keys | 4096 slots | 1 slaves.
10.0.0.83:6379 (1e4c9a3e...) -> 2500 keys | 4096 slots | 1 slaves.
10.0.0.82:6379 (104844ad...) -> 2515 keys | 4096 slots | 1 slaves.
[OK] 10000 keys in 4 masters.
0.61 keys per slot on average.
>>> Performing Cluster Check (using node 10.0.0.81:6379)
M: 6a789b9a447c400581df3c63071104f16032f2c6 10.0.0.81:6379
slots:[1365-5460] (4096 slots) master
1 additional replica(s)
S: 0c9c433017111a3a20af6583840a824c1c020630 10.0.0.88:6379
slots: (0 slots) slave
replicates 7ccff9f724e4508303a8184df5b0903789979759
M: 7ccff9f724e4508303a8184df5b0903789979759 10.0.0.87:6379
slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master
1 additional replica(s)
S: bd496dc81e17ccfdf8431d136254d083ab4bb145 10.0.0.86:6379
slots: (0 slots) slave
replicates 1e4c9a3e6fe5fce70617bf417ef76463a33e334f
S: 34f27507a24e436a121527344fab181e9877404a 10.0.0.85:6379
slots: (0 slots) slave
replicates 104844ad87a3e145884b520980cedc70232986f7
M: 1e4c9a3e6fe5fce70617bf417ef76463a33e334f 10.0.0.83:6379
slots:[12288-16383] (4096 slots) master
1 additional replica(s)
S: 609ff8b268daa75a2fe0061d38f732b5ef859fb6 10.0.0.84:6379
slots: (0 slots) slave
replicates 6a789b9a447c400581df3c63071104f16032f2c6
M: 104844ad87a3e145884b520980cedc70232986f7 10.0.0.82:6379
slots:[6827-10922] (4096 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
10.3.2 Redis Cluster 集群節(jié)點(diǎn)維護(hù)之動(dòng)態(tài)縮容
刪除節(jié)點(diǎn)過程:
添加節(jié)點(diǎn)的時(shí)候是先添加node節(jié)點(diǎn)到集群凌蔬，然后分配槽位，刪除節(jié)點(diǎn)的操作與添加節(jié)點(diǎn)的操作正好相反诞帐，是先將要被刪除的Redis 節(jié)點(diǎn)上的槽位遷移到集群中的其他Redis節(jié)點(diǎn)上唇聘，然后再將其刪除
如果一個(gè)Redis節(jié)點(diǎn)上的槽位沒有被完全遷移柬泽，刪除該node的時(shí)候會(huì)提示有數(shù)據(jù)且無法刪除
Redis 5版本
刪除主節(jié)點(diǎn)10.0.0.81和對(duì)應(yīng)的從節(jié)點(diǎn)10.0.0.84
- 遷移10.0.0.81-master的槽位至其他的master節(jié)點(diǎn)
被遷移Redis master原服務(wù)器必須保證沒有數(shù)據(jù)
#查看當(dāng)前狀態(tài)
[23:34:06 root@81 ~]#redis-cli -a redis --cluster check 10.0.0.81:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
10.0.0.81:6379 (6a789b9a...) -> 2511 keys | 4096 slots | 1 slaves.
10.0.0.87:6379 (7ccff9f7...) -> 2474 keys | 4096 slots | 1 slaves.
10.0.0.83:6379 (1e4c9a3e...) -> 2500 keys | 4096 slots | 1 slaves.
10.0.0.82:6379 (104844ad...) -> 2515 keys | 4096 slots | 1 slaves.
[OK] 10000 keys in 4 masters.
0.61 keys per slot on average.
>>> Performing Cluster Check (using node 10.0.0.81:6379)
M: 6a789b9a447c400581df3c63071104f16032f2c6 10.0.0.81:6379
slots:[1365-5460] (4096 slots) master
1 additional replica(s)
S: 0c9c433017111a3a20af6583840a824c1c020630 10.0.0.88:6379
slots: (0 slots) slave
replicates 7ccff9f724e4508303a8184df5b0903789979759
M: 7ccff9f724e4508303a8184df5b0903789979759 10.0.0.87:6379
slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master
1 additional replica(s)
S: bd496dc81e17ccfdf8431d136254d083ab4bb145 10.0.0.86:6379
slots: (0 slots) slave
replicates 1e4c9a3e6fe5fce70617bf417ef76463a33e334f
S: 34f27507a24e436a121527344fab181e9877404a 10.0.0.85:6379
slots: (0 slots) slave
replicates 104844ad87a3e145884b520980cedc70232986f7
M: 1e4c9a3e6fe5fce70617bf417ef76463a33e334f 10.0.0.83:6379
slots:[12288-16383] (4096 slots) master
1 additional replica(s)
S: 609ff8b268daa75a2fe0061d38f732b5ef859fb6 10.0.0.84:6379
slots: (0 slots) slave
replicates 6a789b9a447c400581df3c63071104f16032f2c6
M: 104844ad87a3e145884b520980cedc70232986f7 10.0.0.82:6379
slots:[6827-10922] (4096 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
#連接到任意集群節(jié)點(diǎn)，#前1365個(gè)slot從10.0.0.81移動(dòng)到第二個(gè)master節(jié)點(diǎn)10.0.0.83上
[23:38:29 root@81 ~]#redis-cli -a redis --cluster reshard 10.0.0.81:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing Cluster Check (using node 10.0.0.81:6379)
M: 6a789b9a447c400581df3c63071104f16032f2c6 10.0.0.81:6379
slots:[1365-5460] (4096 slots) master
1 additional replica(s)
S: 0c9c433017111a3a20af6583840a824c1c020630 10.0.0.88:6379
slots: (0 slots) slave
replicates 7ccff9f724e4508303a8184df5b0903789979759
M: 7ccff9f724e4508303a8184df5b0903789979759 10.0.0.87:6379
slots:[0-1364],[5461-6826],[10923-12287] (4096 slots) master
1 additional replica(s)
S: bd496dc81e17ccfdf8431d136254d083ab4bb145 10.0.0.86:6379
slots: (0 slots) slave
replicates 1e4c9a3e6fe5fce70617bf417ef76463a33e334f
S: 34f27507a24e436a121527344fab181e9877404a 10.0.0.85:6379
slots: (0 slots) slave
replicates 104844ad87a3e145884b520980cedc70232986f7
M: 1e4c9a3e6fe5fce70617bf417ef76463a33e334f 10.0.0.83:6379
slots:[12288-16383] (4096 slots) master
1 additional replica(s)
S: 609ff8b268daa75a2fe0061d38f732b5ef859fb6 10.0.0.84:6379
slots: (0 slots) slave
replicates 6a789b9a447c400581df3c63071104f16032f2c6
M: 104844ad87a3e145884b520980cedc70232986f7 10.0.0.82:6379
slots:[6827-10922] (4096 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 1365 #共4096/3分別給其它三個(gè)master節(jié)點(diǎn)
What is the receiving node ID? 1e4c9a3e6fe5fce70617bf417ef76463a33e334f # master 10.0.0.83
Please enter all the source node IDs.
Type 'all' to use all the nodes as source nodes for the hash slots.
Type 'done' once you entered all the source nodes IDs.
Source node #1: 6a789b9a447c400581df3c63071104f16032f2c6 #輸入要?jiǎng)h除10.0.0.81節(jié)點(diǎn)ID
Source node #2: done
Ready to move 1356 slots.
Source nodes:
M: cb028b83f9dc463d732f6e76ca6bbcd469d948a7 10.0.0.81:6379
slots:[1365-5460] (4096 slots) master
1 additional replica(s)
Destination node:
M: d34da8666a6f587283a1c2fca5d13691407f9462 10.0.0.83:6379
slots:[12288-16383] (4096 slots) master
1 additional replica(s)
Resharding plan:
Moving slot 1365 from cb028b83f9dc463d732f6e76ca6bbcd469d948a7
......
Moving slot 2719 from cb028b83f9dc463d732f6e76ca6bbcd469d948a7
Moving slot 2720 from cb028b83f9dc463d732f6e76ca6bbcd469d948a7
Do you want to proceed with the proposed reshard plan (yes/no)? yes #確定
......
Moving slot 2718 from 10.0.0.81:6379 to 10.0.0.83:6379: ..
Moving slot 2719 from 10.0.0.81:6379 to 10.0.0.83:6379: .
Moving slot 2720 from 10.0.0.81:6379 to 10.0.0.83:6379: ..
#非交互式方式
#再將1365個(gè)slot從10.0.0.81移動(dòng)到第一個(gè)master節(jié)點(diǎn)10.0.0.82上
[23:46:42 root@81 ~]#redis-cli -a redis --cluster reshard 10.0.0.81:6379 --cluster-slots 1365 --cluster-from 6a789b9a447c400581df3c63071104f16032f2c6 --cluster-to 104844ad87a3e145884b520980cedc70232986f7 --cluster-yes
#最后的slot從10.0.0.81移動(dòng)到第三個(gè)master節(jié)點(diǎn)10.0.0.87上
[23:46:42 root@81 ~]#redis-cli -a redis --cluster reshard 10.0.0.81:6379 --cluster-slots 1366 --cluster-from 6a789b9a447c400581df3c63071104f16032f2c6 --cluster-to 7ccff9f724e4508303a8184df5b0903789979759 --cluster-yes
#確認(rèn)10.0.0.81的所有slot都移走了辩诞，上面的slave也自動(dòng)刪除，成為其它master的slave
[23:48:15 root@81 ~]#redis-cli -a redis --cluster check 10.0.0.81:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
10.0.0.81:6379 (6a789b9a...) -> 0 keys | 0 slots | 0 slaves.
10.0.0.87:6379 (7ccff9f7...) -> 3306 keys | 5462 slots | 2 slaves.
10.0.0.83:6379 (1e4c9a3e...) -> 3341 keys | 5461 slots | 1 slaves.
10.0.0.82:6379 (104844ad...) -> 3353 keys | 5461 slots | 1 slaves.
[OK] 10000 keys in 4 masters.
0.61 keys per slot on average.
>>> Performing Cluster Check (using node 10.0.0.81:6379)
M: 6a789b9a447c400581df3c63071104f16032f2c6 10.0.0.81:6379
slots: (0 slots) master
S: 0c9c433017111a3a20af6583840a824c1c020630 10.0.0.88:6379
slots: (0 slots) slave
replicates 7ccff9f724e4508303a8184df5b0903789979759
M: 7ccff9f724e4508303a8184df5b0903789979759 10.0.0.87:6379
slots:[0-1364],[4095-6826],[10923-12287] (5462 slots) master
2 additional replica(s)
S: bd496dc81e17ccfdf8431d136254d083ab4bb145 10.0.0.86:6379
slots: (0 slots) slave
replicates 1e4c9a3e6fe5fce70617bf417ef76463a33e334f
S: 34f27507a24e436a121527344fab181e9877404a 10.0.0.85:6379
slots: (0 slots) slave
replicates 104844ad87a3e145884b520980cedc70232986f7
M: 1e4c9a3e6fe5fce70617bf417ef76463a33e334f 10.0.0.83:6379
slots:[1365-2729],[12288-16383] (5461 slots) master
1 additional replica(s)
S: 609ff8b268daa75a2fe0061d38f732b5ef859fb6 10.0.0.84:6379
slots: (0 slots) slave
replicates 7ccff9f724e4508303a8184df5b0903789979759
M: 104844ad87a3e145884b520980cedc70232986f7 10.0.0.82:6379
slots:[2730-4094],[6827-10922] (5461 slots) master
1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
#原有的10.0.0.84自動(dòng)成為10.0.0.87的slave
[23:48:30 root@81 ~]#redis-cli -a redis -h 10.0.0.87 INFO replication
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
# Replication
role:master
connected_slaves:2
slave0:ip=10.0.0.88,port=6379,state=online,offset=61891,lag=0
slave1:ip=10.0.0.84,port=6379,state=online,offset=61891,lag=0
master_replid:b1403e0730cf01c6fa4ac7d95841c4d67a3afde4
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:61891
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:61891
- 從集群中刪除服務(wù)器
雖然槽位已經(jīng)遷移完成纺涤，但是服務(wù)器IP信息還在集群當(dāng)中译暂，因此還需要將IP信息從集群刪除
# 移除主節(jié)點(diǎn)10.0.0.81
[23:50:00 root@81 ~]#redis-cli -a redis --cluster del-node 10.0.0.81:6379 6a789b9a447c400581df3c63071104f16032f2c6
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Removing node 6a789b9a447c400581df3c63071104f16032f2c6 from cluster 10.0.0.81:6379
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
#刪除節(jié)點(diǎn)后, redis進(jìn)程自動(dòng)關(guān)閉
[23:51:13 root@81 ~]#ss -ntl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 128 [::]:22 [::]:*
[23:51:34 root@81 ~]#ps aux | grep redis
root 1936 0.0 0.1 12108 1100 pts/0 R+ 23:51 0:00 grep --color=auto redis
#刪除節(jié)點(diǎn)信息文件
[23:51:41 root@81 ~]#rm -f /var/lib/redis/nodes-6379.conf
#驗(yàn)證刪除成功
[root@redis-node1 ~]#ss -ntl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 0.0.0.0:22 0.0.0.0:*
LISTEN 0 128 [::]:22 [::]:*
[23:52:05 root@81 ~]#redis-cli -a redis --cluster check 10.0.0.82:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
10.0.0.82:6379 (104844ad...) -> 3353 keys | 5461 slots | 1 slaves.
10.0.0.87:6379 (7ccff9f7...) -> 3306 keys | 5462 slots | 2 slaves.
10.0.0.83:6379 (1e4c9a3e...) -> 3341 keys | 5461 slots | 1 slaves.
[OK] 10000 keys in 3 masters.
0.61 keys per slot on average.
>>> Performing Cluster Check (using node 10.0.0.82:6379)
M: 104844ad87a3e145884b520980cedc70232986f7 10.0.0.82:6379
slots:[2730-4094],[6827-10922] (5461 slots) master
1 additional replica(s)
S: bd496dc81e17ccfdf8431d136254d083ab4bb145 10.0.0.86:6379
slots: (0 slots) slave
replicates 1e4c9a3e6fe5fce70617bf417ef76463a33e334f
M: 7ccff9f724e4508303a8184df5b0903789979759 10.0.0.87:6379
slots:[0-1364],[4095-6826],[10923-12287] (5462 slots) master
2 additional replica(s)
S: 0c9c433017111a3a20af6583840a824c1c020630 10.0.0.88:6379
slots: (0 slots) slave
replicates 7ccff9f724e4508303a8184df5b0903789979759
M: 1e4c9a3e6fe5fce70617bf417ef76463a33e334f 10.0.0.83:6379
slots:[1365-2729],[12288-16383] (5461 slots) master
1 additional replica(s)
S: 34f27507a24e436a121527344fab181e9877404a 10.0.0.85:6379
slots: (0 slots) slave
replicates 104844ad87a3e145884b520980cedc70232986f7
S: 609ff8b268daa75a2fe0061d38f732b5ef859fb6 10.0.0.84:6379
slots: (0 slots) slave
replicates 7ccff9f724e4508303a8184df5b0903789979759
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
#刪除多余的從節(jié)點(diǎn)
[23:52:37 root@81 ~]#redis-cli -a redis --cluster del-node 10.0.0.84:6379 609ff8b268daa75a2fe0061d38f732b5ef859fb6
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Removing node 609ff8b268daa75a2fe0061d38f732b5ef859fb6 from cluster 10.0.0.84:6379
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.
#刪除集群文件
[23:54:42 root@84 ~]#rm -f /var/lib/redis/nodes-6379.conf
[23:54:18 root@81 ~]#redis-cli -a redis --cluster check 10.0.0.82:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
10.0.0.82:6379 (104844ad...) -> 3353 keys | 5461 slots | 1 slaves.
10.0.0.87:6379 (7ccff9f7...) -> 3306 keys | 5462 slots | 1 slaves.
10.0.0.83:6379 (1e4c9a3e...) -> 3341 keys | 5461 slots | 1 slaves.
[OK] 10000 keys in 3 masters.
0.61 keys per slot on average.
>>> Performing Cluster Check (using node 10.0.0.82:6379)
M: 104844ad87a3e145884b520980cedc70232986f7 10.0.0.82:6379
slots:[2730-4094],[6827-10922] (5461 slots) master
1 additional replica(s)
S: bd496dc81e17ccfdf8431d136254d083ab4bb145 10.0.0.86:6379
slots: (0 slots) slave
replicates 1e4c9a3e6fe5fce70617bf417ef76463a33e334f
M: 7ccff9f724e4508303a8184df5b0903789979759 10.0.0.87:6379
slots:[0-1364],[4095-6826],[10923-12287] (5462 slots) master
1 additional replica(s)
S: 0c9c433017111a3a20af6583840a824c1c020630 10.0.0.88:6379
slots: (0 slots) slave
replicates 7ccff9f724e4508303a8184df5b0903789979759
M: 1e4c9a3e6fe5fce70617bf417ef76463a33e334f 10.0.0.83:6379
slots:[1365-2729],[12288-16383] (5461 slots) master
1 additional replica(s)
S: 34f27507a24e436a121527344fab181e9877404a 10.0.0.85:6379
slots: (0 slots) slave
replicates 104844ad87a3e145884b520980cedc70232986f7
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
[23:55:04 root@81 ~]#redis-cli -a redis --cluster info 10.0.0.82:6379
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
10.0.0.82:6379 (104844ad...) -> 3353 keys | 5461 slots | 1 slaves.
10.0.0.87:6379 (7ccff9f7...) -> 3306 keys | 5462 slots | 1 slaves.
10.0.0.83:6379 (1e4c9a3e...) -> 3341 keys | 5461 slots | 1 slaves.
[OK] 10000 keys in 3 masters.
0.61 keys per slot on average.
#查看集群信息
[23:55:45 root@81 ~]#redis-cli -a redis -h 10.0.0.82 CLUSTER INFO
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:12
cluster_my_epoch:11
cluster_stats_messages_ping_sent:8248
cluster_stats_messages_pong_sent:3768
cluster_stats_messages_meet_sent:4
cluster_stats_messages_fail_sent:8
cluster_stats_messages_auth-ack_sent:2
cluster_stats_messages_update_sent:45
cluster_stats_messages_sent:12075
cluster_stats_messages_ping_received:3764
cluster_stats_messages_pong_received:3744
cluster_stats_messages_meet_received:4
cluster_stats_messages_fail_received:2
cluster_stats_messages_auth-req_received:2
cluster_stats_messages_update_received:2
cluster_stats_messages_received:7518
10.3.3 Cluster的局限性
大多數(shù)情況, 客戶端性能會(huì)降低, 因?yàn)闊o論查詢還是寫入, 都要涉及hash計(jì)算, 找到槽位和節(jié)點(diǎn)信息
命令無法跨節(jié)點(diǎn)使用: 比如, mget, keys, scan, flush, sinter等
客戶端維護(hù)更復(fù)雜, SDK和應(yīng)用本身消耗(例如更多的連接池)
不支持多個(gè)數(shù)據(jù)庫: 集群模式只有一個(gè)db 0
復(fù)制只支持一層: 不支持樹形復(fù)制結(jié)構(gòu), 不支持級(jí)聯(lián)復(fù)制, 只可以一主多從
key事務(wù)和Lua支持有限: 操作的key必須是在一個(gè)節(jié)點(diǎn), Lua和事務(wù)無法跨節(jié)點(diǎn)使用
架構(gòu)更改時(shí), 比如從主從, 變成哨兵, 或者從哨兵變成cluster, 應(yīng)用程序的代碼需要更改, 比如python程序在不同集群下使用的庫是不一樣的
多個(gè)節(jié)點(diǎn)長(zhǎng)時(shí)間運(yùn)行后, 可能出現(xiàn)某個(gè)節(jié)點(diǎn)的數(shù)據(jù)量大, 消耗內(nèi)存, 并且要接受更多的請(qǐng)求訪問, 出現(xiàn)集群傾斜,
發(fā)生傾斜的原因:
1. 節(jié)點(diǎn)和槽位分配布局
2. 不同槽位對(duì)應(yīng)的鍵值數(shù)量差異較大
3. 包含bigkey, 建議少用
4. 內(nèi)存相關(guān)配置不一致
5. 熱點(diǎn)數(shù)據(jù)不均衡: 一致性不高時(shí), 可以使用本地緩存和MQ
