環(huán)境

• centos7.2
• php5.5.7
• mysql5.7

安裝elasticsearch

下載地址：https://www.elastic.co/downloads/elasticsearch

image

安裝2.4.6

wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/rpm/elasticsearch/2.4.6/elasticsearch-2.4.6.rpm

yum -y install elasticsearch-2.4.6.rpm

環(huán)境：

• ubuntu 18.0.4
• php7.2.13
• mysql5.7
• java1.8+
• go1.9+

安裝elasticsearch

下載地址：https://www.elastic.co/downloads/elasticsearch

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.5.0.deb

gdebi elasticsearch-6.5.0.deb

安裝完成后茫打，配置elasticsearch

vim /etc/elasticsearch/elasticsearch.yml

//配置內容
.
.
.
cluster.name: zpdx-search    //集群名稱
.
.
.
node.name: zpdx-1      //節(jié)點名稱
.
.
.
path.data: /var/lib/elasticsearch    //數(shù)據(jù)路徑
.
.
.
path.log: /var/log/elasticsearch      //日志路徑
.
.
.
network.host:  127.0.0.1      //主機地址撕捍，默認
.
.
.
http:port:  9200      //端口钧舌，默認

開啟elasticsearch

systemctl start elasticsearch.service

將elasticsearch設置開機自啟動

sudo /bin/systemctl daemon-reload

sudo /bin/systemctl enable elasticsearch.service

安裝mvn

sudo apt update

sudo apt install maven 

安裝 ik

github下載地址： elasticsearch-analysis-ik

git clone https://github.com/medcl/elasticsearch-analysis-ik.git

cd lasticsearch-analysis-ik

git checkout tags/1.10.6

mvn clean

mvn compile

mvn package

在./elasticsearch-analysis-ik/target/releases目錄會下載了一個對應版本的插件包elasticsearch-analysis-ik-1.10.6.zip

將插件包復制到elaticsearch的插件目錄下

cp elasticsearch-analysis-ik-1.10.6.zip /usr/share/elasticsearch/plugins/

cd /usr/share/elasticsearch/plugins/

unzip elasticsearch-analysis-ik-1.10.6.zip

mkdir ik

mv ./* ik

重啟elasticsearch

systemctl restart elasticsearch.service

測試肝陪，使用curl 'http://127.0.0.1:9200'歉糜，或者用瀏覽器訪問http://127.0.0.1:9200```，顯示

{
  "name" : "zpdx-1",
  "cluster_name" : "zpdxshop-search",
  "cluster_uuid" : "g1wO4CR8TGOwfrDHY-vcGw",
  "version" : {
    "number" : "6.5.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "816e6f6",
    "build_date" : "2018-11-09T18:58:36.352602Z",
    "build_snapshot" : false,
    "lucene_version" : "7.5.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

elasticsearch安裝完成并成功啟動

安裝go

sudo apt update

sudo apt install golang

查看go 版本

go version

查看go 環(huán)境

go env

設置go環(huán)境苫纤，vim /etc/profile

//將go環(huán)境路徑設置到/usr/local/go方便管理
//在最后添加
.
.
.
#go環(huán)境
export GOPATH=/usr/local/go

保存并退出，使/etc/profile文件生效

source /etc/profile

安裝go-mysql-elasticsearch

go-mysql-elasticsearch的基本原理是：如果是第一次啟動該程序流妻，首先使用mysqldump工具對源mysql數(shù)據(jù)庫進行一次全量同步，通過elasticsearch client執(zhí)行操作寫入數(shù)據(jù)到ES笆制；然后實現(xiàn)了一個mysql client,作為slave連接到源mysql,源mysql作為master會將所有數(shù)據(jù)的更新操作通過binlog event同步給slave绅这， 通過解析binlog event就可以獲取到數(shù)據(jù)的更新內容，之后寫入到ES.

使用限制

1. mysql binlog必須是ROW模式

2. 要同步的mysql數(shù)據(jù)表必須包含主鍵项贺，否則直接忽略君躺，這是因為如果數(shù)據(jù)表沒有主鍵，UPDATE和DELETE操作就會因為在ES中找不到對應的document而無法進行同步

3. 不支持程序運行過程中修改表結構

4. 要賦予用于連接mysql的賬戶RELOAD權限以及REPLICATION權限, SUPER權限：
       GRANT REPLICATION SLAVE ON *.* TO 'admin'@'127.0.0.1';
       GRANT RELOAD ON *.* TO 'admin'@'27.0.0.1';
       UPDATE mysql.user SET Super_Priv='Y' WHERE user='admin' AND host='127.0.0.1';

查看mysql的binlog开缎、server_id棕叫，需要設置

//進入mysql

mysql> show global variables like '%binlog_format%'; 
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| binlog_format | ROW   |
+---------------+-------+
1 row in set (0.02 sec)

mysql> show variables like '%server_id%';
+----------------+-------+
| Variable_name  | Value |
+----------------+-------+
| server_id      | 100   |
| server_id_bits | 32    |
+----------------+-------+
2 rows in set (0.01 sec)

mysql> 

設置mysql，編輯/etc/mysql/mysql.conf.d/mysqld.cnf

vim /etc/mysql/mysql.conf.d/mysqld.cnf

.
.
.
[mysql]
log-bin=mysql-bin
binlog_format="ROW"
server-id=100

重啟mysql

systemctl restart mysql

安裝go-mysql-elasticsearch

github下載地址：go-mysql-elasticsearch

go get github.com/siddontang/go-mysql-elasticsearch

cd $GOPATH/src/github.com/siddontang/go-mysql-elasticsearch

make

go-elasticsearch不能自動創(chuàng)建索引index奕删，需手動創(chuàng)建

vim create_index.json

//添加內容
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0
  },
  "mappings": {
    "content": {
      "properties": {
        "title": {
          "type": "text",
          "analyzer": "ik_smart",     
          "search_analyzer": "ik_smart"
        },
        "content": {
          "type": "text",
          "analyzer": "ik_smart",
          "search_analyzer": "ik_smart"
        }
      }
    }
  }
}

curl執(zhí)行

chase@chase-MACH-WX9:~$ curl -XPUT 'localhost:9200/test?pretty' -H 'Content-Type:application/json' -d'@create_index.json'
{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "test"
}
chase@chase-MACH-WX9:~$ 

創(chuàng)建成功俺泣，查看索引

chase@chase-MACH-WX9:~$ curl localhost:9200/test?pretty
{
  "test" : {
    "aliases" : { },
    "mappings" : {
      "content" : {
        "properties" : {
          "content" : {
            "type" : "text",
            "analyzer" : "ik_smart"
          },
          "title" : {
            "type" : "text",
            "analyzer" : "ik_smart"
          }
        }
      }
    },
    "settings" : {
      "index" : {
        "creation_date" : "1548855659726",
        "number_of_shards" : "1",
        "number_of_replicas" : "0",
        "uuid" : "F3ze_A2gQo6mE88aEQzKNA",
        "version" : {
          "created" : "6050099"
        },
        "provided_name" : "test"
      }
    }
  }
}
chase@chase-MACH-WX9:~$ 

配置go-mysql-elasticsearch，編輯$GOPATH/src/github.com/siddontang/go-mysql-elasticsearch/etc/river.toml

# MySQL address, user and password
# user must have replication privilege in MySQL.
my_addr = "127.0.0.1:3306"
my_user = "admin"
my_pass = "123456"
my_charset = "utf8"

# Set true when elasticsearch use https
#es_https = false
# Elasticsearch address
es_addr = "127.0.0.1:9200"
# Elasticsearch user and password, maybe set by shield, nginx, or x-pack
es_user = ""
es_pass = ""

# Path to store data, like master.info, if not set or empty,
# we must use this to support breakpoint resume syncing. 
# TODO: support other storage, like etcd. 
data_dir = "./var"

# Inner Http status address
stat_addr = "127.0.0.1:12800"

# pseudo server id like a slave 
server_id = 100

# mysql or mariadb
flavor = "mysql"

# mysqldump execution path
# if not set or empty, ignore mysqldump.
mysqldump = "mysqldump"

# if we have no privilege to use mysqldump with --master-data,
# we must skip it.
#skip_master_data = false

# minimal items to be inserted in one bulk
bulk_size = 128

# force flush the pending requests if we don't have enough items >= bulk_size
flush_bulk_time = "200ms"

# Ignore table without primary key
skip_no_pk_table = false

# MySQL data source
[[source]]
schema = "test"

# Only below tables will be synced into Elasticsearch.
# "t_[0-9]{4}" is a wildcard table format, you can use it if you have many sub tables, like table_0000 - table_1023
# I don't think it is necessary to sync all tables in a database.
tables = ["node"]

# Below is for special rule mapping

# Very simple example
# 
# desc t;
# +-------+--------------+------+-----+---------+-------+
# | Field | Type         | Null | Key | Default | Extra |
# +-------+--------------+------+-----+---------+-------+
# | id    | int(11)      | NO   | PRI | NULL    |       |
# | name  | varchar(256) | YES  |     | NULL    |       |
# +-------+--------------+------+-----+---------+-------+
# 
# The table `t` will be synced to ES index `test` and type `t`.
[[rule]]
schema = "test"
table = "node"
index = "test"
type = "content"

啟動go-mysql-elasticsearch

cd $GOPATH/src/github.com/siddontang/go-mysql-elasticsearch

./bin/go-mysql-elasticsearch -config=./etc/river.toml

同步數(shù)據(jù)信息

[2019/01/30 21:48:31] [info] binlogsyncer.go:111 create BinlogSyncer with config {100 mysql 127.0.0.1 3306 admin   utf8 false false <nil> false false 0 0s 0s 0}
[2019/01/30 21:48:31] [info] dump.go:164 skip dump, use last binlog replication pos (mysql-bin.000001, 790) or GTID set %!s(<nil>)
[2019/01/30 21:48:31] [info] status.go:53 run status http server 127.0.0.1:12800
[2019/01/30 21:48:31] [info] binlogsyncer.go:323 begin to sync binlog from position (mysql-bin.000001, 790)
[2019/01/30 21:48:31] [info] binlogsyncer.go:172 register slave for master server 127.0.0.1:3306
[2019/01/30 21:48:31] [info] sync.go:31 start sync binlog at binlog file (mysql-bin.000001, 790)
[2019/01/30 21:48:31] [info] binlogsyncer.go:692 rotate to (mysql-bin.000001, 790)
[2019/01/30 21:48:31] [info] binlogsyncer.go:692 rotate to (mysql-bin.000002, 4)
[2019/01/30 21:48:31] [info] sync.go:73 rotate binlog to (mysql-bin.000001, 790)
[2019/01/30 21:48:31] [info] master.go:54 save position (mysql-bin.000001, 790)
[2019/01/30 21:48:31] [info] sync.go:73 rotate binlog to (mysql-bin.000002, 4)
[2019/01/30 21:48:31] [info] master.go:54 save position (mysql-bin.000002, 4)

自此完残，數(shù)據(jù)全量導入到elasticsearch

設置go-mysql-elasticsearch開機自啟動

ubuntu安裝sysv-rc-conf

sudo apt update

sudo apt install sysv-rc-conf

如果報錯：

sudo apt-get install sysv-rc-conf
正在讀取軟件包列表... 完成
正在分析軟件包的依賴關系樹       
正在讀取狀態(tài)信息... 完成       
E: 無法定位軟件包 sysv-rc-conf

解決方法：添加鏡像源

vim /etc/apt/sources.list

//添加如下內容
deb http://archive.ubuntu.com/ubuntu/ trusty main universe restricted multiverse

sudo apt update

sudo apt install sysv-rc-con

創(chuàng)建go-mysql-elasticsearch腳本

vim /etc/init.d/go-mysql-elasticsearch

//添加內容如下：
#!/bin/bash
#go-mysql-elasticsearch start
$GOPATH/src/github.com/siddontang/go-mysql-elasticsearch/bin/go-mysql-elasticsearch -config=$GOPATH/src/github.com/siddontang/go-mysql-elasticsearch/etc/river.toml

執(zhí)行

sysv-rc-conf go-mysql-elasticsearch on

service go-mysql-elasticsearch start

centos安裝chkconfig

yum -y install chkconfig

安裝elasticsearch-php

composer require elasticsearch/elasticsearch

elasticsearch官方文檔

elasticsearch-php官方文檔

Mysql與Elasticsearch數(shù)據(jù)實時同步

elasticsearch之實例篇

阮一峰 -- 全文搜索引擎 Elasticsearch 入門教程