alibaba canal 項目本地搭建過程和踩坑記錄 mac環(huán)境

前言：為了做一個博客檢索系統(tǒng)强品，需要將數(shù)據(jù)從數(shù)據(jù)庫同步到elasticsearch

anal [k?'n?l]蛮粮，譯意為水道/管道/溝渠，主要用途是基于 MySQL 數(shù)據(jù)庫增量日志解析祖能，提供增量數(shù)據(jù)訂閱和消費 早期阿里巴巴因為杭州和美國雙機房部署澳迫，存在跨機房同步的業(yè)務需求，實現(xiàn)方式主要是基于業(yè)務 trigger 獲取增量變更是鬼。從 2010 年開始肤舞，業(yè)務逐步嘗試數(shù)據(jù)庫日志解析獲取增量變更進行同步，由此衍生出了大量的數(shù)據(jù)庫增量訂閱和消費業(yè)務均蜜。

mysql下載安裝配置

  • 當前的canal開源版本支持5.7及以下的版本(阿里內(nèi)部mysql 5.7.13, 5.6.10, mysql 5.5.18和5.1.40/48) 我現(xiàn)在的第一個版本為 5.7.29一直無法成功李剖，后面換成 5.7.13 此處有坑
  • mysql官網(wǎng)地址
  • 配置 首先查看mysql安裝目錄下support-files文件夾下是否有my-default.cnf這個文件
MacBook-Pro:local f$ cd  /usr/local/mysql/support-files
MacBook-Pro:support-files f$ ls
magic           mysql.server
mysql-log-rotate    mysqld_multi.server

ps:當然我這里沒有是因為我已經(jīng)移動了。

  • 如果有囤耳，就好辦了篙顺，直接轉(zhuǎn)移到/etc下偶芍，并且修改名字為my.cnf
mv /usr/local/mysql/support-files/my-default.cnf /etc/my.cnf
  • 如果沒有則需要自己創(chuàng)建一個文件。
vim /etc/my.cnf  #會打開創(chuàng)建一個新的文件德玫，將下邊的內(nèi)容復制進去

# Example MySQL config file for small systems.
#
# This is for a system with little memory (<= 64M) where MySQL is only used
# from time to time and it's important that the mysqld daemon
# doesn't use much resources.
#
# MySQL programs look for option files in a set of
# locations which depend on the deployment platform.
# You can copy this option file to one of those
# locations. For information about these locations, see:
# http://dev.mysql.com/doc/mysql/en/option-files.html
#
# In this file, you can use all long options that a program supports.
# If you want to know which options a program supports, run the program
# with the "--help" option.
 
# The following options will be passed to all MySQL clients
[client]
default-character-set=utf8
#password   = your_password
port        = 3306
socket      = /tmp/mysql.sock
 
# Here follows entries for some specific programs
 
# The MySQL server
[mysqld]
#解決only_full_group_by的問題
#sql_mode=STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
 
default-storage-engine=INNODB
character-set-server=utf8
collation-server=utf8_general_ci
port        = 3306
socket      = /tmp/mysql.sock
skip-external-locking
key_buffer_size = 16K
max_allowed_packet = 1M
table_open_cache = 4
sort_buffer_size = 64K
read_buffer_size = 256K
read_rnd_buffer_size = 256K
net_buffer_length = 2K
thread_stack = 128K
 
# Don't listen on a TCP/IP port at all. This can be a security enhancement,
# if all processes that need to connect to mysqld run on the same host.
# All interaction with mysqld must be made via Unix sockets or named pipes.
# Note that using this option without enabling named pipes on Windows
# (using the "enable-named-pipe" option) will render mysqld useless!
#
#skip-networking
server-id   = 1
 
# Uncomment the following if you want to log updates
#log-bin=mysql-bin
 
# binary logging format - mixed recommended
#binlog_format=mixed
 
# Causes updates to non-transactional engines using statement format to be
# written directly to binary log. Before using this option make sure that
# there are no dependencies between transactional and non-transactional
# tables such as in the statement INSERT INTO t_myisam SELECT * FROM
# t_innodb; otherwise, slaves may diverge from the master.
#binlog_direct_non_transactional_updates=TRUE
 
# Uncomment the following if you are using InnoDB tables
#innodb_data_home_dir = /usr/local/mysql/data
#innodb_data_file_path = ibdata1:10M:autoextend
#innodb_log_group_home_dir = /usr/local/mysql/data
# You can set .._buffer_pool_size up to 50 - 80 %
# of RAM but beware of setting memory usage too high
#innodb_buffer_pool_size = 16M
#innodb_additional_mem_pool_size = 2M
# Set .._log_file_size to 25 % of buffer pool size
#innodb_log_file_size = 5M
#innodb_log_buffer_size = 8M
#innodb_flush_log_at_trx_commit = 1
#innodb_lock_wait_timeout = 50
 
[mysqldump]
quick
max_allowed_packet = 16M
 
[mysql]
no-auto-rehash
# Remove the next comment character if you are not familiar with SQL
#safe-updates
 
[myisamchk]
key_buffer_size = 8M
sort_buffer_size = 8M
 
[mysqlhotcopy]
interactive-timeout
  • 配置完成之后匪蟀，還需要在my.cnf 文件中添加以下參數(shù)
[mysqld]
log-bin=mysql-bin # 開啟 binlog
binlog-format=ROW # 選擇 ROW 模式
server_id=1 # 配置 MySQL replaction 需要定義，不要和 canal 的 slaveId 重復

mac mysql命令行

MacBook-Pro:support-files f$ /usr/local/MySQL/bin/mysql -u root -p
Enter password:
mysql>

創(chuàng)建mysql的canal用戶

mysql> CREATE USER 'canal'@'localhost' IDENTIFIED BY 'canal';
Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON *.* TO 'canal'@'localhost' WITH GRANT OPTION;
Query OK, 0 rows affected (0.01 sec)
mysql> CREATE USER 'canal'@'%' IDENTIFIED BY 'canal';
Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' WITH GRANT OPTION;
Query OK, 0 rows affected (0.00 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

canal下載和服務端配置

  • 下載服務這里
  • 解壓之后
image
  • 這里需要配置 log/canal.properties文件

我的配置如下

  • 這里有一個坑就是
    下面的需要改成自己的ip地址宰僧，而不能用 localhost 或者127.0.0.1.
canal.instance.master.address = 10.0.40.153:3306 

#################################################
#########       common argument     #############
#################################################
# tcp bind ip
canal.ip =
# register ip to zookeeper
canal.register.ip =
canal.port = 11111
canal.metrics.pull.port = 11112
# canal instance user/passwd
# canal.user = canal
# canal.passwd = E3619321C1A937C46A0D8BD1DAC39F93B27D4458

# canal admin config
#canal.admin.manager = 127.0.0.1:8089
canal.admin.port = 11110
canal.admin.user = admin
canal.admin.passwd = 4ACFE3202A5FF5CF467898FC58AAB1D615029441

canal.zkServers =
# flush data to zk
canal.zookeeper.flush.period = 1000
canal.withoutNetty = false
# tcp, kafka, RocketMQ
canal.serverMode = tcp
# flush meta cursor/parse position to file
canal.file.data.dir = ${canal.conf.dir}
canal.file.flush.period = 1000
## memory store RingBuffer size, should be Math.pow(2,n)
canal.instance.memory.buffer.size = 16384
## memory store RingBuffer used memory unit size , default 1kb
canal.instance.memory.buffer.memunit = 1024 
## meory store gets mode used MEMSIZE or ITEMSIZE
canal.instance.memory.batch.mode = MEMSIZE
canal.instance.memory.rawEntry = true

## detecing config
canal.instance.detecting.enable = false
#canal.instance.detecting.sql = insert into retl.xdual values(1,now()) on duplicate key update x=now()
canal.instance.detecting.sql = select 1
canal.instance.detecting.interval.time = 3
canal.instance.detecting.retry.threshold = 3
canal.instance.detecting.heartbeatHaEnable = false

# support maximum transaction size, more than the size of the transaction will be cut into multiple transactions delivery
canal.instance.transaction.size =  1024
# mysql fallback connected to new master should fallback times
canal.instance.fallbackIntervalInSeconds = 60

# network config
canal.instance.network.receiveBufferSize = 16384
canal.instance.network.sendBufferSize = 16384
canal.instance.network.soTimeout = 30

# binlog filter config
canal.instance.filter.druid.ddl = true
canal.instance.filter.query.dcl = false
canal.instance.filter.query.dml = false
canal.instance.filter.query.ddl = false
canal.instance.filter.table.error = false
canal.instance.filter.rows = false
canal.instance.filter.transaction.entry = false

# binlog format/image check
canal.instance.binlog.format = ROW,STATEMENT,MIXED 
canal.instance.binlog.image = FULL,MINIMAL,NOBLOB

# binlog ddl isolation
canal.instance.get.ddl.isolation = false

# parallel parser config
canal.instance.parser.parallel = false
## concurrent thread number, default 60% available processors, suggest not to exceed Runtime.getRuntime().availableProcessors()
#canal.instance.parser.parallelThreadSize = 16
## disruptor ringbuffer size, must be power of 2
canal.instance.parser.parallelBufferSize = 256

# table meta tsdb info
canal.instance.tsdb.enable = true
canal.instance.tsdb.dir = ${canal.file.data.dir:../conf}/${canal.instance.destination:}
canal.instance.tsdb.url = jdbc:h2:${canal.instance.tsdb.dir}/h2;CACHE_SIZE=1000;MODE=MYSQL;
canal.instance.tsdb.dbUsername = canal
canal.instance.tsdb.dbPassword = canal
# dump snapshot interval, default 24 hour
canal.instance.tsdb.snapshot.interval = 24
# purge snapshot expire , default 360 hour(15 days)
canal.instance.tsdb.snapshot.expire = 360

# aliyun ak/sk , support rds/mq
canal.aliyun.accessKey =
canal.aliyun.secretKey =

#################################################
#########       destinations        #############
#################################################
canal.destinations = example
# conf root dir
canal.conf.dir = ../conf
# auto scan instance dir add/remove and start/stop instance
canal.auto.scan = true
canal.auto.scan.interval = 5

canal.instance.tsdb.spring.xml = classpath:spring/tsdb/h2-tsdb.xml
#canal.instance.tsdb.spring.xml = classpath:spring/tsdb/mysql-tsdb.xml

canal.instance.global.mode = spring
canal.instance.global.lazy = false
canal.instance.global.manager.address = ${canal.admin.manager}
#canal.instance.global.spring.xml = classpath:spring/memory-instance.xml
canal.instance.global.spring.xml = classpath:spring/file-instance.xml
#canal.instance.global.spring.xml = classpath:spring/default-instance.xml

##################################################
#########            MQ              #############
##################################################
canal.mq.servers = 127.0.0.1:6667
canal.mq.retries = 0
canal.mq.batchSize = 16384
canal.mq.maxRequestSize = 1048576
canal.mq.lingerMs = 100
canal.mq.bufferMemory = 33554432
canal.mq.canalBatchSize = 50
canal.mq.canalGetTimeout = 100
canal.mq.flatMessage = true
canal.mq.compressionType = none
canal.mq.acks = all
#canal.mq.properties. =
canal.mq.producerGroup = test
# Set this value to "cloud", if you want open message trace feature in aliyun.
canal.mq.accessChannel = local
# aliyun mq namespace
#canal.mq.namespace =

##################################################
#########     Kafka Kerberos Info    #############
##################################################
canal.mq.kafka.kerberos.enable = false
canal.mq.kafka.kerberos.krb5FilePath = "../conf/kerberos/krb5.conf"
canal.mq.kafka.kerberos.jaasFilePath = "../conf/kerberos/jaas.conf"
## mysql serverId
canal.instance.mysql.slaveId = 1234
#position info材彪，需要改成自己的數(shù)據(jù)庫信息
canal.instance.master.address = 10.0.40.153:3306 
canal.instance.master.journal.name = 
canal.instance.master.position = 
canal.instance.master.timestamp = 
#canal.instance.standby.address = 
#canal.instance.standby.journal.name =
#canal.instance.standby.position = 
#canal.instance.standby.timestamp = 
#username/password，需要改成自己的數(shù)據(jù)庫信息
canal.instance.dbUsername = canal  
canal.instance.dbPassword = canal
canal.instance.defaultDatabaseName =
canal.instance.connectionCharset = UTF-8
#table regex
canal.instance.filter.regex = .\*\\\\..\*

服務運行

  • 項目代碼很簡單琴儿，直接使用阿里的給的示例即可 阿里示例
  • 個人也新建了一個springboot項目上傳到github 個人代碼
image
  • 當我在 navicat客戶端在數(shù)據(jù)中添加如下參數(shù)
image
  • 控制臺打出如下命令
empty count : 69
empty count : 70
empty count : 71
empty count : 72
================&gt; binlog[mysql-bin.000005:2110] , name[test,user] , eventType : INSERT
id : 1    update=true
name : 小明    update=true
empty count : 1

搜索公眾號”會講歷史的程序員"段化，關(guān)注并回復“谷歌"學習獲取外網(wǎng)訪問教程。我們一起學習成長造成。

?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請聯(lián)系作者
  • 人面猴
    序言：七十年代末显熏，一起剝皮案震驚了整個濱河市，隨后出現(xiàn)的幾起案子晒屎，更是在濱河造成了極大的恐慌喘蟆，老刑警劉巖，帶你破解...
    沈念sama閱讀 222,378評論 6贊 516
  • 死咒
    序言：濱河連續(xù)發(fā)生了三起死亡事件鼓鲁，死亡現(xiàn)場離奇詭異蕴轨，居然都是意外死亡，警方通過查閱死者的電腦和手機骇吭，發(fā)現(xiàn)死者居然都...
    沈念sama閱讀 94,970評論 3贊 399
  • 救了他兩次的神仙讓他今天三更去死
    文/潘曉璐 我一進店門尺棋，熙熙樓的掌柜王于貴愁眉苦臉地迎上來，“玉大人绵跷，你說我怎么就攤上這事〕筛＃” “怎么了碾局？”我有些...
    開封第一講書人閱讀 168,983評論 0贊 362
  • 道士緝兇錄：失蹤的賣姜人
    文/不壞的土叔 我叫張陵，是天一觀的道長奴艾。 經(jīng)常有香客問我净当，道長，這世上最難降的妖魔是什么蕴潦？ 我笑而不...
    開封第一講書人閱讀 59,938評論 1贊 299
  • ?港島之戀（遺憾婚禮）
    正文 為了忘掉前任像啼，我火速辦了婚禮，結(jié)果婚禮上潭苞，老公的妹妹穿的比我還像新娘忽冻。我一直安慰自己，他們只是感情好此疹，可當我...
    茶點故事閱讀 68,955評論 6贊 398
  • 惡毒庶女頂嫁案：這布局不是一般人想出來的
    文/花漫 我一把揭開白布僧诚。 她就那樣靜靜地躺著遮婶，像睡著了一般。 火紅的嫁衣襯著肌膚如雪湖笨。 梳的紋絲不亂的頭發(fā)上旗扑，一...
    開封第一講書人閱讀 52,549評論 1贊 312
  • 城市分裂傳說
    那天，我揣著相機與錄音慈省，去河邊找鬼臀防。 笑死，一個胖子當著我的面吹牛边败，可吹牛的內(nèi)容都是我干的袱衷。 我是一名探鬼主播，決...
    沈念sama閱讀 41,063評論 3贊 422
  • 雙鴛鴦連環(huán)套：你想象不到人心有多黑
    文/蒼蘭香墨 我猛地睜開眼放闺，長吁一口氣：“原來是場噩夢啊……” “哼祟昭！你這毒婦竟也來了？” 一聲冷哼從身側(cè)響起怖侦，我...
    開封第一講書人閱讀 39,991評論 0贊 277
  • 萬榮殺人案實錄
    序言：老撾萬榮一對情侶失蹤篡悟，失蹤者是張志新（化名）和其女友劉穎，沒想到半個月后匾寝，有當?shù)厝嗽跇淞掷锇l(fā)現(xiàn)了一具尸體搬葬，經(jīng)...
    沈念sama閱讀 46,522評論 1贊 319
  • ?護林員之死
    正文 獨居荒郊野嶺守林人離奇死亡，尸身上長有42處帶血的膿包…… 初始之章·張勛 以下內(nèi)容為張勛視角 年9月15日...
    茶點故事閱讀 38,604評論 3贊 342
  • ?白月光啟示錄
    正文 我和宋清朗相戀三年艳悔，在試婚紗的時候發(fā)現(xiàn)自己被綠了急凰。 大學時的朋友給我發(fā)了我未婚夫和他白月光在一起吃飯的照片。...
    茶點故事閱讀 40,742評論 1贊 353
  • 活死人
    序言：一個原本活蹦亂跳的男人離奇死亡猜年，死狀恐怖抡锈，靈堂內(nèi)的尸體忽然破棺而出，到底是詐尸還是另有隱情乔外，我是刑警寧澤床三，帶...
    沈念sama閱讀 36,413評論 5贊 351
  • ?日本核電站爆炸內(nèi)幕
    正文 年R本政府宣布，位于F島的核電站杨幼，受9級特大地震影響撇簿，放射性物質(zhì)發(fā)生泄漏。R本人自食惡果不足惜差购，卻給世界環(huán)境...
    茶點故事閱讀 42,094評論 3贊 335
  • 男人毒藥：我在死后第九天來索命
    文/蒙蒙 一四瘫、第九天 我趴在偏房一處隱蔽的房頂上張望。 院中可真熱鬧欲逃，春花似錦找蜜、人聲如沸。這莊子的主人今日做“春日...
    開封第一講書人閱讀 32,572評論 0贊 25
  • 一樁弒父案锹杈，背后竟有這般陰謀
    文/蒼蘭香墨 我抬頭看了看天上的太陽撵孤。三九已至，卻和暖如春竭望，著一層夾襖步出監(jiān)牢的瞬間邪码，已是汗流浹背。 一陣腳步聲響...
    開封第一講書人閱讀 33,671評論 1贊 274
  • 情欲美人皮
    我被黑心中介騙來泰國打工咬清， 沒想到剛下飛機就差點兒被人妖公主榨干…… 1. 我叫王不留闭专，地道東北人。 一個月前我還...
    沈念sama閱讀 49,159評論 3贊 378
  • 代替公主和親
    正文 我出身青樓旧烧，卻偏偏與公主長得像影钉，于是被迫代替她去往敵國和親。 傳聞我的和親對象是個殘疾皇子掘剪，可洞房花燭夜當晚...
    茶點故事閱讀 45,747評論 2贊 361

推薦閱讀更多精彩內(nèi)容