利用Logstash同步MySql數(shù)據(jù)至ElasticSearch
安裝Logstash
參考鏈接Installing Logstash
Download and install the public signing key:
rpm --importhttps://artifacts.elastic.co/GPG-KEY-elasticsearch
Add the following in your /etc/yum.repos.d/ directory in a file with a .repo suffix, for example logstash.repo
[logstash-7.x]name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1autorefresh=1type=rpm-md
And your repository is ready for use. You can install it with:
sudo yum install logstash
yum方式安裝好后，程序目錄位于/usr/share/logstash/, 配置文件位于/etc/logstash/啃洋，
配置MySql數(shù)據(jù)同步
在/etc/logstash中創(chuàng)建mysql.conf配置文件,參考鏈接jdbc input plugin
input {
jdbc {
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://xxx.mysql.rds.aliyuncs.com/xxx"
jdbc_user => "root"
jdbc_password => "******"
schedule => "* * * * *"
statement => "SELECT * from biz_goods where create_time > :sql_last_value AND create_time < NOW() AND is_delete=0 ORDER BY create_time desc"
}
}
output {
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "goods"
document_id => "%{id}"
}
}
在官方文檔中input->jdbc中需要指定jdbc_driver_library => "mysql-connector-java-5.1.36-bin.jar"谐区，此處如果將mysq-connector-java.jar放在常規(guī)目錄通過(guò)路徑指定椭员，將會(huì)出現(xiàn)如下錯(cuò)誤
LogStash::ConfigurationError
com.mysql.jdbc.Driver not loaded. Are you sure you've included the correct jdbc driver in :jdbc_driver_library?
解決方式為在mvnrepository下載jar文件后皂吮，將該文件拷貝至/usr/share/logstash/logstash-core/lib/jars 中，然后無(wú)需在input->jdbc中配置jdbc_driver_library
運(yùn)行l(wèi)ogstash開啟同步
./logstash -f /etc/logstash/mysql.conf
