redis-shake簡(jiǎn)介
redis-shake是阿里開源的用于redis數(shù)據(jù)同步的工具遂庄，基本功能有：
- 恢復(fù)restore：將RDB文件恢復(fù)到目的redis數(shù)據(jù)庫(kù)责嚷。
- 備份dump：將源redis的全量數(shù)據(jù)通過RDB文件備份起來儡蔓。
- 解析decode：對(duì)RDB文件進(jìn)行讀取抛蚤，并以json格式解析存儲(chǔ)旅掂。
- 同步sync：支持源redis和目的redis的數(shù)據(jù)同步搁骑，支持全量和增量數(shù)據(jù)的遷移悲龟，支持從云下到阿里云云上的同步屋讶，也支持云下到云下不同環(huán)境的同步，支持單節(jié)點(diǎn)须教、主從版皿渗、集群版之間的互相同步。需要注意的是轻腺，如果源端是集群版乐疆，可以啟動(dòng)一個(gè)RedisShake，從不同的db結(jié)點(diǎn)進(jìn)行拉取贬养，同時(shí)源端不能開啟move slot功能挤土；對(duì)于目的端，如果是集群版误算，寫入可以是1個(gè)或者多個(gè)db結(jié)點(diǎn)仰美。
- 同步rump：支持源redis和目的redis的數(shù)據(jù)同步，僅支持全量的遷移儿礼。采用scan和restore命令進(jìn)行遷移咖杂，支持不同云廠商不同redis版本的遷移。
基本原理
如果源端是集群模式蚊夫，只需要啟動(dòng)一個(gè)redis-shake進(jìn)行拉取诉字，同時(shí)不能開啟源端的move slot操作。如果目的端是集群模式知纷，可以寫入到一個(gè)結(jié)點(diǎn)壤圃，然后再進(jìn)行slot的遷移，當(dāng)然也可以多對(duì)多寫入琅轧。
目前埃唯，redis-shake到目的端采用單鏈路實(shí)現(xiàn)，對(duì)于正常情況下鹰晨，這不會(huì)成為瓶頸墨叛，但對(duì)于極端情況止毕，qps比較大的時(shí)候，此部分性能可能成為瓶頸漠趁。另外扁凛，redis-shake到目的端的數(shù)據(jù)同步采用異步的方式，讀寫分離在2個(gè)線程操作闯传，降低因?yàn)榫W(wǎng)絡(luò)時(shí)延帶來的同步性能下降谨朝。
全量同步階段并發(fā)執(zhí)行，增量同步階段異步執(zhí)行甥绿，能夠達(dá)到毫秒級(jí)別延遲（取決于網(wǎng)絡(luò)延遲）字币。同時(shí)，對(duì)大key同步進(jìn)行分批拉取共缕，優(yōu)化同步性能洗出。
用戶可以通過我們提供的restful拉取metric來對(duì)redis-shake進(jìn)行實(shí)時(shí)監(jiān)控：curl 127.0.0.1:9320/metric。
支持
- 支持2.8-5.0版本的同步图谷。
- 支持codis翩活。
- 支持云下到云上，云上到云上便贵，云上到云下（阿里云目前支持主從版）菠镇，其他云到阿里云等鏈路，幫助用戶靈活構(gòu)建混合云場(chǎng)景承璃。
使用
1利耍、直接下載二進(jìn)制包這里
2、修改配置文件
]# tar -zxf release-v2.1.0-20210819.tar.gz
]# cd redis-shake
]# ll
total 41252
-rw-r--r-- 1 502 games 12542 Aug 25 17:50 redis-shake.conf
-rwxr-xr-x 1 502 games 11853504 Aug 19 10:11 redis-shake.darwin
-rwxr-xr-x 1 502 games 11918954 Aug 19 10:11 redis-shake.linux
-rwxr-xr-x 1 502 games 12227584 Aug 19 10:11 redis-shake.windows
修改redis-shake.conf盔粹，我的環(huán)境是從兩套redis主從之間同步數(shù)據(jù)隘梨，配置文件如下（刪減過部分默認(rèn)配置）
conf.version = 1
id = redis-shake
log.file = /opt/redis-shake/redis-shake.log
log.level = info
pid_path =
system_profile = 9310
http_profile = 9320
parallel = 32
source.type = standalone
source.address = 10.26.31.172:6379
source.password_raw =
source.auth_type = auth
source.tls_enable = false
source.rdb.input =
source.rdb.parallel = 0
source.rdb.special_cloud =
target.type = standalone
target.address = 10.26.29.235:6389
target.password_raw =
target.auth_type = auth
target.db = 0
key_exists = rewrite
filter.db.whitelist = 0
說明：
a、主要需要修改源redis和目的redis的信息
b玻佩、當(dāng)源和目的有重復(fù) key 時(shí)是否進(jìn)行覆寫, 可選值:
1. rewrite: 源端覆蓋目的端
2. none: 一旦發(fā)生進(jìn)程直接退出
3. ignore: 保留目的端key，忽略源端的同步 key. 該值在 rump 模式下不會(huì)生效.
key_exists = rewrite
c席楚、指定的db被通過咬崔，比如0;5;10將會(huì)使db0, db5, db10通過, 其他的被過濾
filter.db.whitelist = 0
更多詳細(xì)參數(shù)參見官網(wǎng)
3、啟動(dòng)
用戶配置完配置文件烦秩，然后以不同的模式啟動(dòng)即可：
./redis-shake -conf=redis-shake.conf -type=sync &
然后可以查看配置的日子路徑垮斯，查看服務(wù)運(yùn)行狀態(tài),一般正常情況下日志如下
2021/08/25 17:19:54 [WARN] source.auth_type[auth] != auth
2021/08/25 17:19:54 [WARN] target.auth_type[auth] != auth
2021/08/25 17:19:54 [INFO] input password is empty, skip auth address[10.26.29.235:6389] with type[auth].
2021/08/25 17:19:54 [INFO] input password is empty, skip auth address[10.26.31.172:6379] with type[auth].
2021/08/25 17:19:54 [INFO] input password is empty, skip auth address[10.26.31.172:6379] with type[auth].
2021/08/25 17:19:54 [INFO] source rdb[10.26.31.172:6379] checksum[yes]
2021/08/25 17:19:54 [WARN]
______________________________
\ \ _ ______ |
\ \ / \___-=O'/|O'/__|
\ RedisShake, here we go !! \_______\ / | / )
/ / '/-==__ _/__|/__=-| -GM
/ Alibaba Cloud / * \ | |
/ / (o)
------------------------------
if you have any problem, please visit https://github.com/alibaba/RedisShake/wiki/FAQ
2021/08/25 17:19:54 [INFO] redis-shake configuration: {"ConfVersion":1,"Id":"redis-shake","LogFile":"/opt/redis-shake/redis-shake.log","LogLevel":"info","SystemProfile":9310,"HttpProfile":9320,"Parallel":32,"SourceType":"standalone","SourceAddress":"10.26.31.172:6379","SourcePasswordRaw":"***","SourcePasswordEncoding":"***","SourceAuthType":"auth","SourceTLSEnable":false,"SourceRdbInput":[],"SourceRdbParallel":1,"SourceRdbSpecialCloud":"","TargetAddress":"10.26.29.235:6389","TargetPasswordRaw":"***","TargetPasswordEncoding":"***","TargetDBString":"0","TargetDBMapString":"","TargetAuthType":"auth","TargetType":"standalone","TargetTLSEnable":false,"TargetRdbOutput":"local_dump","TargetVersion":"6.0.9","FakeTime":"","KeyExists":"none","FilterDBWhitelist":[],"FilterDBBlacklist":[],"FilterKeyWhitelist":[],"FilterKeyBlacklist":[],"FilterSlot":[],"FilterLua":false,"BigKeyThreshold":524288000,"Metric":true,"MetricPrintLog":false,"SenderSize":104857600,"SenderCount":4095,"SenderDelayChannelSize":65535,"KeepAlive":0,"PidPath":"","ScanKeyNumber":50,"ScanSpecialCloud":"","ScanKeyFile":"","Qps":200000,"ResumeFromBreakPoint":false,"Psync":true,"NCpu":0,"HeartbeatUrl":"","HeartbeatInterval":10,"HeartbeatExternal":"","HeartbeatNetworkInterface":"","ReplaceHashTag":false,"ExtraInfo":false,"SockFileName":"","SockFileSize":0,"FilterKey":null,"FilterDB":"","Rewrite":false,"SourceAddressList":["10.26.31.172:6379"],"TargetAddressList":["10.26.29.235:6389"],"SourceVersion":"4.0.11","HeartbeatIp":"127.0.0.1","ShiftTime":0,"TargetReplace":false,"TargetDB":0,"Version":"tidy-code-for-release,ba0f3e1aa6e0956bfa2973a4c894cbc1a8d3d732,go1.16.7,2021-08-19_10:11:39","Type":"sync","TargetDBMap":null}
2021/08/25 17:19:54 [INFO] DbSyncer[0] starts syncing data from 10.26.31.172:6379 to [10.26.29.235:6389] with http[9321], enableResumeFromBreakPoint[false], slot boundary[-1, -1]
2021/08/25 17:19:54 [INFO] input password is empty, skip auth address[10.26.31.172:6379] with type[auth].
2021/08/25 17:19:54 [INFO] DbSyncer[0] psync connect '10.26.31.172:6379' with auth type[auth] OK!
2021/08/25 17:19:54 [INFO] DbSyncer[0] psync send listening port[9320] OK!
2021/08/25 17:19:54 [INFO] DbSyncer[0] try to send 'psync' command: run-id[?], offset[-1]
2021/08/25 17:22:30 [INFO] Event:FullSyncStart Id:redis-shake
2021/08/25 17:22:30 [INFO] DbSyncer[0] psync runid = 52aa8214f9f8238c32f172badbb4f4ea325c6cc4, offset = 9452286340998, fullsync
2021/08/25 17:22:30 [INFO] DbSyncer[0] +
2021/08/25 17:22:31 [INFO] DbSyncer[0] -
2021/08/25 17:22:31 [INFO] DbSyncer[0] +
日志信息
同步分為三個(gè)階段：
1、等待源端save rdb完畢只祠，日志如下
2021/08/25 17:22:30 [INFO] DbSyncer[0] +
2021/08/25 17:22:31 [INFO] DbSyncer[0] -
2021/08/25 17:22:31 [INFO] DbSyncer[0] +
2兜蠕、全量同步階段，顯示百分比
2021/08/25 17:28:09 [INFO] DbSyncer[0] total = 9.06GB - 3.82MB [ 0%] entry=56515
2021/08/25 17:28:10 [INFO] DbSyncer[0] total = 9.06GB - 7.67MB [ 0%] entry=114522
2021/08/25 17:28:11 [INFO] DbSyncer[0] total = 9.06GB - 11.28MB [ 0%] entry=168877
3抛寝、增量同步熊杨，出現(xiàn)字樣sync rdb done后曙旭，當(dāng)前dbSyncer進(jìn)入增量同步
2021/08/26 09:40:58 [INFO] DbSyncer[0] sync: +forwardCommands=3 +filterCommands=3 +writeBytes=33
2021/08/26 09:40:59 [INFO] DbSyncer[0] sync: +forwardCommands=2 +filterCommands=0 +writeBytes=23
2021/08/26 09:41:00 [INFO] DbSyncer[0] sync: +forwardCommands=1 +filterCommands=3 +writeBytes=11
注意事項(xiàng)
1、如果目標(biāo)庫(kù)的數(shù)據(jù)逐出策略（maxmemory-policy）配置為noeviction以外的值晶府，可能導(dǎo)致目標(biāo)庫(kù)的數(shù)據(jù)與源庫(kù)不一致桂躏。關(guān)于數(shù)據(jù)逐出策略詳情，請(qǐng)參見Redis數(shù)據(jù)逐出策略介紹川陆。
2剂习、如果源庫(kù)中的某些Key使用了過期（expire）機(jī)制，由于可能存在Key已過期但未被及時(shí)刪除的情形较沪，所以在目標(biāo)庫(kù)中查看（如通過info命令）到的Key數(shù)量會(huì)比源庫(kù)的Key數(shù)量少鳞绕。
