RedisShake使用
RedisShake可以用于數據的遷移和同步蹋笼。
遷移的場景很好理解堪置，比如原先的時候多個業(yè)務使用一個redis殃饿，然后為了避免相互影響饮寞，這個時候需要把緩存也拆分出來(特別在業(yè)務初創(chuàng)時期)孝扛。
同步的場景的話列吼，可以用在主備雙活。因為一般調用的鏈路是vip->proxy->redis集群苦始，如果vip發(fā)生了網絡抖動可能會導致redis讀取失敗寞钥，如果做純緩存應用的話可以有數據庫兜底還好(雖然可能預熱期間會導致數據庫壓力增加)，如果用redis當持久存儲的業(yè)務來說陌选，就會導致業(yè)務方調用失敗理郑，造成業(yè)務不可用。這個時候如果有備集群的話咨油，可以很好的切換(最簡單的直接修改vip的轉發(fā)您炉，nginx是可以無損重啟的)
也可以用在異地多活，具體設計可以看下阿里云的redis災備介紹中有一小節(jié)介紹跨地域容災役电。
github地址
項目github的地址 https://github.com/alibaba/RedisShake/releases
可以下載source code進行編譯赚爵，也可以直接下載二進制版。
說明文檔
第一次使用法瑟，如何進行配置
通過Redis-shake將自建Redis遷移至阿里云
配置文件說明
sync模式配置以下幾個內容, 包括source.type冀膝，source.address，source.password_raw霎挟，target.type窝剖，target.address，target.password_raw 6個配置酥夭。
具體配置見 第一次使用赐纱，如何進行配置
啟動命令
./redis-shake -conf=redis-shake.conf -type=sync
單個節(jié)點到單個節(jié)點配置舉例
- 下載redis源碼，地址 https://github.com/redis/redis/releases/tag/6.0.9
- 編譯源碼 make
不一定需要make install
- 啟動兩個服務端
./redis-server --port 6380
./redis-server --port 6381
- 下載redis-shake的二進制包
wget https://github.com/alibaba/RedisShake/releases/download/release-v2.0.3-20200724/redis-shake-v2.0.3.tar.gz
- 配置redis-shake文件采郎，然后啟動redis-shake
沒有設置密碼的話不用填寫
source.type = standalone
source.address = 127.0.0.1:6380
source.password_raw =
target.type = standalone
target.address = 127.0.0.1:6381
target.password_raw =
- 啟動redis-shake
這邊看你的環(huán)境如果是mac os就用darwin后綴的千所，如果是linux用linux后綴
cp redis-shake.conf redis-shake-standalone.conf
./redis-shake.darwin -conf=redis-shake-standalone.conf -type=sync
- 在source的redis db (端口號6380)中輸入一些命令
redis有5種類型的數據結構狂魔，都試下
redis-cli -h 127.0.0.1 -p 6380
set hello world
hset user:zihao name zihao age 28 sex male
hset user:xmz name xmz age 26 sex female
SADD phone apple sanxing huawei
lpush city hangzhou suzhou chengdu
ZADD stu_score 100 zihao 99 xmz 90 zpy
- 查看target的redis db中數據
edis-cli -h 127.0.0.1 -p 6381
localhost:6381> keys *
1) "city"
2) "user:xmz"
3) "stu_score"
4) "hello"
5) "phone"
6) "user:zihao"
集群cluster到集群cluster配置舉例
集群直接使用docker運行蒜埋，redis-cluster鏡像地址
1.啟動1個redis集群，命名為cluster1
docker run -p 7000-7005:7000-7005 -d --name cluster1 grokzen/redis-cluster:6.0.9
2.啟動另1個redis集群最楷，命名為cluster2
docker run -p 8000-8005:7000-7005 -d --name cluster2 grokzen/redis-cluster:6.0.9
3.獲取cluster1和cluster2的集群信息
redis-cli -h localhost -p 7000 -c
localhost:7000> CLUSTER NODES
redis-cli -h localhost -p 8000 -c
localhost:7000> CLUSTER NODES
4.redis-shake文件配置
這邊有兩種配置一種是自動發(fā)現的整份，一種把所有master或者slave寫上。具體看上面的github文檔
source.type = cluster
source.address = master@127.0.0.1:7000
source.password_raw =
target.type = cluster
target.address = master@127.0.0.1:8000
target.password_raw =
5.啟動redis-shake
cp redis-shake.conf redis-shake-cluster.conf
./redis-shake.linux -conf=redis-shake-cluster.conf -type=sync
6.在source的redis-cluster(cluster1)輸入一些命令
set hello world
hset user:zihao name zihao age 28 sex male
hset user:xmz name xmz age 26 sex female
SADD phone apple sanxing huawei
lpush city hangzhou suzhou chengdu
ZADD stu_score 100 zihao 99 xmz 90 zpy
- 在target的redis-cluster(2)查看
172.17.0.3:7002> get hello
-> Redirected to slot [866] located at 172.17.0.3:7000
"world"
172.17.0.3:7002> hgetall user:zihao
1) "name"
2) "zihao"
3) "age"
4) "28"
5) "sex"
6) "male"
172.17.0.3:7000> SMEMBERS phone
-> Redirected to slot [8939] located at 172.17.0.3:7001
1) "huawei"
2) "sanxing"
3) "apple"
172.17.0.3:7001> LRANGE city 0 -1
-> Redirected to slot [11479] located at 172.17.0.3:7002
1) "chengdu"
2) "suzhou"
3) "hangzhou"
172.17.0.3:7001> ZRANGE stu_score 0 -1
1) "zpy"
2) "xmz"
3) "zihao"
集群版cluster到proxy配置舉例
這邊proxy使用的是codis (阿里云的集群版redis也是使用proxy模式的籽孙，阿里云的redis架構)烈评。
搭建一個codis集群環(huán)境
- 獲取codis的二進制安裝包 https://github.com/CodisLabs/codis/releases/download/3.2.2/codis3.2.2-go1.8.5-linux.tar.gz
- 依次啟動各個組件
具體的配置細節(jié)本文不做過多描述，詳情見 https://github.com/CodisLabs/codis/blob/release3.2/doc/tutorial_zh.md
- 啟動zk
docker run --name "Codis-Z2181" -d --read-only -p 2181:2181 jplock/zookeeper
- 啟動dashboard
#dashboard.toml https://github.com/CodisLabs/codis/blob/release3.2/config/dashboard.toml
#coordinator_name和coordinator_addr改成zk
nohup ./codis-dashboard --ncpu=4 --config=dashboard.toml --log=dashboard.log --log-level=WARN &
- 啟動proxy
#proxy.toml https://github.com/CodisLabs/codis/blob/release3.2/config/proxy.toml
nohup ./codis-proxy --ncpu=4 --config=proxy.toml --log=proxy.log --log-level=WARN &
- 啟動codis server
這邊啟動4臺
nohup ./codis-server --port 6379 &
nohup ./codis-server --port 6380 &
nohup ./codis-server --port 6381 &
nohup ./codis-server --port 6382 &
可以通過redis-cli分別連接測試下連接通不通
redis-cli -h localhost -p 6379
- 啟動codis fe
nohup ./codis-fe --ncpu=4 --log=fe.log --log-level=WARN --zookeeper=127.0.0.1:2181 --listen=0.0.0.0:8080 &
3.通過fe配置codis集群
打開localhost:8080的頁面
? 添加proxy
通過proxy.toml的admin_addr地址添加
添加proxy.png
-
添加group
添加group.png -
添加server
添加server.png -
分配slot
分配slot.png
codis連接測試
[root@JD codis3.2.2-go1.8.5-linux]# redis-cli -h localhost -p 19000
localhost:19000> set hello world
OK
localhost:19000> get hello
"world"
啟動redis-shake將redis-cluster的數據同步到codis
- 配置redis-shake
source.type = cluster
source.address = master@127.0.0.1:7000
source.password_raw =
target.type = proxy
target.address = 127.0.0.1:19000
target.password_raw =
#源端的redis版本是6.0.9,target版本的redis版本比較低 所以需要設置這個 不然會報restore的錯誤
big_key_threshold=1
- 啟動redis-shake
cp redis-shake.conf redis-shake-proxy.conf
./redis-shake.linux -conf=redis-shake-proxy.conf -type=sync
RedisFullCheck
github地址
https://github.com/alibaba/RedisFullCheck
說明文檔
使用
./redis-full-check -s "172.17.0.2:7000;172.17.0.2:7001;172.17.0.2:7002" -t "172.17.0.4:7000;172.17.0.4:7001;172.17.0.4:7002" --comparemode=1 --comparetimes=1 --qps=10 --batchcount=100 --sourcedbtype=1 --targetdbtype=1 --targetdbfilterlist=0
比對結果輸出
[root@JD redis-full-check-1.4.8]# ./redis-full-check -s "172.17.0.2:7000;172.17.0.2:7001;172.17.0.2:7002" -t "172.17.0.4:7000;172.17.0.4:7001;172.17.0.4:7002" --comparemode=1 --comparetimes=1 --qps=10 --batchcount=100 --sourcedbtype=1 --targetdbtype=1 --targetdbfilterlist=0
[INFO 2021-04-08-17:45:41 main.go:65]: init log success
[INFO 2021-04-08-17:45:41 main.go:168]: configuration: {172.17.0.2:7000;172.17.0.2:7001;172.17.0.2:7002 auth 1 -1 172.17.0.4:7000;172.17.0.4:7001;172.17.0.4:7002 auth 1 0 result.db 1 1 unknown unknown unknown 10 5 100 5 false 16384 20445 false}
[INFO 2021-04-08-17:45:41 main.go:170]: ---------
[INFO 2021-04-08-17:45:41 full_check.go:238]: sourceDbType=1, p.sourcePhysicalDBList=[172.17.0.2:7000 172.17.0.2:7001 172.17.0.2:7002]
[INFO 2021-04-08-17:45:41 full_check.go:241]: db=0:keys=0(inaccurate for type cluster)
[INFO 2021-04-08-17:45:41 full_check.go:253]: ---------------- start 1th time compare
[INFO 2021-04-08-17:45:41 full_check.go:278]: start compare db 0
[INFO 2021-04-08-17:45:41 scan.go:20]: build connection[source redis addr: [172.17.0.2:7000]]
[INFO 2021-04-08-17:45:41 scan.go:20]: build connection[source redis addr: [172.17.0.2:7001]]
[INFO 2021-04-08-17:45:41 scan.go:20]: build connection[source redis addr: [172.17.0.2:7002]]
[INFO 2021-04-08-17:45:42 full_check.go:203]: stat:
times:1, db:0, dbkeys:0, finish:-1%, finished:true
KeyScan:{6 6 0}
KeyEqualAtLast|string|equal|{1 1 0}
KeyEqualAtLast|hash|equal|{2 2 0}
KeyEqualAtLast|list|equal|{1 1 0}
KeyEqualAtLast|set|equal|{1 1 0}
KeyEqualAtLast|zset|equal|{1 1 0}
FieldEqualAtLast|hash|equal|{6 6 0}
FieldEqualAtLast|set|equal|{3 3 0}
FieldEqualAtLast|zset|equal|{3 3 0}
[INFO 2021-04-08-17:45:42 full_check.go:328]: --------------- finished! ----------------
all finish successfully, totally 0 key(s) and 0 field(s) conflict
