filebeat
官網(wǎng)
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-overview.html
下載
https://www.elastic.co/cn/downloads/beats/filebeat
這里我們選擇匹配的版本：Linux aarch64
filebeat-8.6.2-linux-arm64.tar.gz
安裝
tar -zxvf filebeat-8.6.2-linux-arm64.tar.gz filebeat-8.6.2-linux-arm64
啟動(dòng)
./filebeat -e -c filebeat.yaml
配置項(xiàng)
輸入項(xiàng) Inputs
https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html
輸出項(xiàng) Output
https://www.elastic.co/guide/en/beats/filebeat/current/configuring-output.html
部分配置項(xiàng)
通用設(shè)置
https://www.elastic.co/guide/en/beats/filebeat/8.7/filtering-and-enhancing-data.html
processors: # 處理器 必須在配置的頂層
# 在這里可以設(shè)置要去除的字段
- drop_fields:
# when: 可以設(shè)置去除的條件
# condition
fields: ["log","host","input","agent","ecs"]
ignore_missing: false #如果當(dāng)指定的字段不存在時(shí) 處理器不會(huì)返回錯(cuò)誤
name: "my-beat" #beat名稱丢早，無(wú)設(shè)置則取hostname agent.name
tags: ["my-service","hardware","test"]
實(shí)例
- 用于監(jiān)聽(tīng)
pd_cd_server.log文件的的數(shù)據(jù)梗夸，將發(fā)送到控制臺(tái)console
- 定義配置文件
filebeta.yaml
filebeat.inputs: #定義輸入配置
- type: log #文件類型為log
paths: #文件路徑
- /opt/cd/log/pd_cd_server.log
- /var/log/supervisor/*.log
multiline:
type: pattern #定義要使用的聚合方法。正則表達(dá)式
pattern: '^\[' #正則表達(dá)式
negate: true #默認(rèn)是false驯妄，匹配pattern的行合并到上一行
match: after #指定 Filebeat 如何將匹配的行合并到事件中把多行合并成一個(gè)事件
output.console: #定義輸出
pretty: true
enable: true
- 啟動(dòng)該配置文件
./filebeat -e -c filebeat.yaml
- 返回打印控制臺(tái)
{
"@timestamp": "2023-04-06T03:18:28.411Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "8.6.2"
},
"log": {
"file": {
"path": "/opt/cd/log/pd_cd_server.log"
},
"offset": 6374
},
"message": "INFO: 192.168.0.181:10173 - \"GET /docs HTTP/1.1\" 200 OK",
"input": {
"type": "log"
},
"ecs": {
"version": "8.0.0"
},
"host": {
"name": "Test_007"
},
"agent": {
"name": "Test_007",
"type": "filebeat",
"version": "8.6.2",
"ephemeral_id": "e9e3e7f9-dfea-4d15-9abf-fd5e2d93e0cb",
"id": "05f3a3cf-de63-447d-aa38-985a81983e26"
}
}
logstash
官網(wǎng)
https://www.elastic.co/guide/en/logstash/current/getting-started-with-logstash.html
tar -zxvf logstash-7.14.0-linux-x86_64.tar.gz
運(yùn)行
./bin/logstash -f config/logstash-sample1.conf
配置
#接收beat數(shù)據(jù)窖张，標(biāo)準(zhǔn)輸出控制臺(tái)
input { #從filebeat取數(shù)據(jù)椭岩，端口與filebeat配置文件一致
beats {
port => 9022
}
}
filter {
if [filetype] == "log_pd_cd"{
json {
source => "message"
remove_field => ["log","offset","tags","instance_id"] #移除字段践叠，不需要采集
}
}
}
output {
stdout {}
}
output {
elasticsearch {
hosts => [ "192.168.0.3:9200" ]
index => "first-9an--%{+YYYY.MM.dd}"
}
}
