1狗热、我使用的是windows 版本的logstash-7.5.1 進(jìn)入 logstash-7.5.1\bin饵沧,直接 點(diǎn)擊logstash.bat運(yùn)行即可即可孩灯。
2、在bin 文件夾中建立pgsql 文件夾(隨意位置碉怔,隨意名字)
3烘贴、首先要連接postgreSQL 數(shù)據(jù)庫(kù),需要pg引擎撮胧,我使用的是postgresql-42.2.8.jar
放到pgsql文件夾中
然后建立logstas-pgsql.conf
input {
stdin {
}
jdbc {
jdbc_connection_string => "jdbc:postgresql://127.0.0.1:5432/testdata"
jdbc_user => "***"
jdbc_password => "******" 更換自己的數(shù)據(jù)庫(kù)用戶名和密碼
jdbc_driver_library => "******\logstash-7.5.1\bin\pgsql\postgresql-42.2.8.jar" 更換自己的驅(qū)動(dòng)地址
jdbc_driver_class => "org.postgresql.Driver"
jdbc_paging_enabled => "true"
jdbc_page_size => "300000"
use_column_value => "true"
tracking_column => "id"
statement_filepath => "******\logstash-7.5.1\bin\pgsql\logstash-pgsql.sql" 這是要操作的sql 表桨踪,下面會(huì)提供
schedule => "* * * * *"
type => "jdbc"
jdbc_default_timezone =>"Asia/Shanghai"
}
}
filter {
json {
source => "message"
remove_field => ["message"]
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "test_out"
template => "****\logstash-7.5.1\bin\pgsql\es-template.json" es 索引模板,可不定義
template_name => "t-statistic-out-logstash"
template_overwrite => true
document_type => "out"
document_id => "%{id}"
}
stdout {
codec => json_lines
}
}
logstash-pgsql.sql 文件內(nèi)容非常簡(jiǎn)單
select * from tb_table
es-template.json
{
"template" : "t-statistis-out-template",
"order":1,
"settings": {
"index": {
"refresh_interval": "5s"
}
},
"mappings": {
"_default_": {
"_all" : {"enabled":false},
"dynamic_templates": [
{
"message_field" : {
"match" : "message",
"match_mapping_type" : "string",
"mapping" : { "type" : "string", "index" : "not_analyzed" }
}
}, {
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : { "type" : "string", "index" : "not_analyzed" }
}
}
],
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "keyword"
},
"id": {
"type": "keyword"
},
"name": {
"type": "keyword"
},
"pp": {
"type": "keyword"
}
}
}
},
"aliases": {}
}
4芹啥、進(jìn)入 bin 目錄
運(yùn)行命令
logstash -f ./pgsql/logstash-pgsql.conf
注意:?jiǎn)?dòng)時(shí)因?yàn)槭峭_(tái)機(jī)器運(yùn)行多個(gè)logstash實(shí)例锻离, 需要指定不同的數(shù)據(jù)存儲(chǔ)目錄 path.Data
執(zhí)行命令:
logstash -f ./pgsql/logstash-pgsql.conf --path.data=/pgconfig/