背景:
在網(wǎng)上看過filebeat的很多文檔,可能因為時間原因或者filebeat使用者較少吧,總之不能滿足我的需求,所以記錄下,希望能幫到有需要的人吧.奧力給!
一共分為兩部分 第一部分為基礎(chǔ)說明,第二部分為真實演示流程,第三部分為碰到過的問題收集
一 基礎(chǔ)說明:
1 下載安裝filebeat,可以看之前的記錄
2 默認配置文件為filebeat.reference.yml,正式使用配置文件為filebeat.yml
3 配置文件基礎(chǔ)解讀:
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
#每收集一個日志,需要配置一個type
- type: log
enabled: true
paths:
- /home/work/dsp/log/webserver/access_log #日志路徑
fields:
type: "php-nginx-access" #自定義字段
close_renamed: true #日志替換名字時,停止采集
close_removed: true #日志移動時.停止采集
scan_frequency: 10s #頻率 10s去檢測日志更新
#整塊注釋如上#
- type: log
enabled: true
paths:
- /home/work/dsp/log/service/service.log
fields:
type: "service-log"
close_renamed: true
close_removed: true
scan_frequency: 10s
#整塊注釋如上#
#- type: log
# enabled: true
# paths:
# - /home/work/dsp/log/webserver/error_log.2019111217
# fields:
# type: "nginx-error"
#============================= Filebeat modules ===============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
#==================== Elasticsearch template setting ==========================
#setup.template.settings:
# index.number_of_shards: 3
#index.number_of_replicas: 2
# index:
# number_of_shards: 3
# #codec: best_compression
# #number_of_routing_shards: 30
#setup.template.settings:
#index.number_of_shards: 3
#index.codec: best_compression
#_source.enabled: false
#============================== Kibana =====================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
host: "10.19.145.2:5601"
username: "YouName"
password: "YourPassword"
# Kibana Host
#-------------------------- Elasticsearch output ------------------------------
setup.ilm.enabled: false
setup.template.name: "php-nginx" # 給咱的模板起個名字,隨便喊
setup.template.pattern: "php-nginx-*" #調(diào)取的正則名稱
setup.template.settings: # 配置生成索引的分片與副本數(shù)
index.number_of_shards: 3
index.number_of_replicas: 1
setup.template.overwrite: true
setup.template.enabled: true
setup.template.name: "php-service"
setup.template.pattern: "php-service-*"
setup.template.settings:
index.number_of_shards: 3
index.number_of_replicas: 1
setup.template.overwrite: true
setup.template.enabled: true
#es 集群配置
output.elasticsearch:
hosts: ["ip1:9200", "ip2.15:9200", "ip3:9200"]
username: "youName"
password: "YouPassword"
index: "php-nginx-%{+yyyy.MM.dd}"
indices:
- index: "php-nginx-%{+yyyy.MM.dd}"
when.equals:
fields.type: "php-nginx-access"
- index: "php-service-%{+yyyy.MM.dd}"
when.equals:
fields.type: "service-log"
pipelines:
- pipeline: "php-nginx-access"
when.equals:
fields.type: "php-nginx-access"
- pipeline: "service-log"
when.equals:
fields.type: "service-log"
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
二 真實演示流程:
完整流程舉例: 添加nginx接入層日志采集
1 在filebeat.yml 添加type
- type: log
enabled: true
paths:
- /home/work/nginx/logs/access_log
fields:
type: "nginx-access"
close_renamed: true
close_removed: true
scan_frequency: 10s
exclude_lines: ['\/static\/img\/', '\/static\/js\/', '\/static\/css\/', '\/static\/fonts\/'] #目的是為了過濾css,js等你想過濾的東西
2 在filebeat.yml 添加template
setup.template.overwrite: true
setup.template.name: "nginx-flow"
setup.template.pattern: "nginx-flow-*"
setup.template.fields: ${path.config}/nginx-template.yml
setup.template.enabled: false
3 構(gòu)建es管道
PUT _ingest/pipeline/nginx-flow
參數(shù)為:
{
"nginx-access" : {
"description" : "nginx-flow",
"processors" : [
{
"grok" : {
"ignore_failure" : true,
"field" : "message",
"patterns" : [
"""%{IPV4:remote_addr_ip} - (%{USERNAME:user_name}|-) \[%{HTTPDATE:log_timestamp:date}\] \"(?<method>[A-Z]+) (?<request_uri>[\s\S]*) (?<proto>[A-Za-z]+([A-Za-z0-9+\-./]+)+)\" (?<status>\d+) (?<body_size>\d+) \"(?<refer>(.*?))\" \"(?<cookie>(.*?))\" \"(?<user_agent_info>(.*?)*)\" %{NUMBER:cost_time} %{IPV4:remote_addr} %{IPV4:server_addr} (?<sock_path>[a-z0-9\.]+:(/?[a-z0-9\-]+)+(\.sock)?|-) (?<service_name>[a-z\d\.]+) \"(?<forwarded_foo>(.*?))\" (?<log_id>[\w\-]*) (?<log_id>[\w\-]*) %{NUMBER:time} %{NUMBER:time}"""
]
}
},
{
"date" : {
"ignore_failure" : true,
"field" : "log_timestamp",
"formats" : [
"dd/MMM/yyyy:HH:mm:ss Z"
],
"timezone" : "Asia/Shanghai"
}
}
]
}
}
4 在filebeat.yml 添加索引與es管道配置
output.elasticsearch:
hosts: ["ip1:9200", "ip2:9200", "ip3:9200"]
username: "YouName"
password: "YouPassword"
index: "php-nginx-%{+yyyy.MM.dd}"
indices:
- index: "php-nginx-%{+yyyy.MM.dd}"
when.equals:
fields.type: "php-nginx-access"
- index: "php-service-%{+yyyy.MM.dd}"
when.equals:
fields.type: "service-log"
#在原有配置上的新增行1 start ###
- index: "nginx-flow-%{+yyyy.MM.dd}"
when.equals:
fields.type: "nginx-access"
#在原有配置上的新增行1 end ###
pipelines:
- pipeline: "php-nginx-access"
when.equals:
fields.type: "php-nginx-access"
- pipeline: "service-log"
when.equals:
fields.type: "service-log"
#在原有配置上的新增行2 start ###
- pipeline: "nginx-access"
when.equals:
fields.type: "nginx-access"
#在原有配置上的新增行2 end ###
5 重啟filebeat服務(wù)
supervisorctl restart filebeat
# 或者
./filebeat -c filebeat.yml -e -d '*'
三 碰到過的問題:
1 es 集群 cpu負載過高,集群內(nèi)每個節(jié)點的cpu負載都接近100%
解決手段: 1 在es管道配置中新增如下圖1中標紅參數(shù).
2 調(diào)整es的jvm gc參數(shù),如下圖2中
1.png
2.png
2 當采集日志數(shù)量>=2時,filebeat設(shè)置es索引分片與副本數(shù)失敗
解決手段: 采集日志數(shù)量>=2時,es的分片與副本數(shù)量收集需要在不同的模板下分別設(shè)置,否則會使設(shè)置參數(shù)失效.如下圖所示:
3.png
?
