一.Filebeat介紹
Filebeat附帶預(yù)構(gòu)建的模塊边翼，這些模塊包含收集鱼响、解析、充實(shí)和可視化各種日志文件格式數(shù)據(jù)所需的配置组底，每個(gè)Filebeat模塊由一個(gè)或多個(gè)文件集組成丈积，這些文件集包含攝取節(jié)點(diǎn)管道、Elasticsearch模板债鸡、Filebeat勘探者配置和Kibana儀表盤(pán)江滨。
Filebeat模塊很好的入門(mén)，它是輕量級(jí)單用途的日志收集工具厌均，用于在沒(méi)有安裝java的服務(wù)器上專(zhuān)門(mén)收集日志唬滑，可以將日志轉(zhuǎn)發(fā)到logstash、elasticsearch或redis等場(chǎng)景中進(jìn)行下一步處理
為什么使用filebeat?
filebeat比logstash占用更少的系統(tǒng)資源棺弊，特別是內(nèi)存晶密。
二.使用filebeat收集nginx日志
2.1使用filebeat收集普通的nginx日志
1.安裝Nginx
cat >/etc/yum.repos.d/nginx.repo <<EOF
[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/\$releasever/\$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true
[nginx-mainline]
name=nginx mainline repo
baseurl=http://nginx.org/packages/mainline/centos/\$releasever/\$basearch/
gpgcheck=1
enabled=0
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true
EOF
yum install nginx -y
systemctl start nginx
curl 127.0.0.1
2.配置Nginx并創(chuàng)建測(cè)試頁(yè)面
rm -rf /etc/nginx/conf.d/default.conf
cat >/etc/nginx/conf.d/www.conf<<EOF
server {
listen 80;
server_name localhost;
location / {
root /code/www;
index index.html index.htm;
}
}
EOF
mkdir /code/www/ -p
echo "db01-www" > /code/www/index.html
nginx -t
systemctl restart nginx
curl 127.0.0.1
tail -f /var/log/nginx/access.log
3.安裝filebet
rpm -ivh filebeat-6.6.0-x86_64.rpm
4.配置filebeat
cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
EOF
5.啟動(dòng)測(cè)試
systemctl start filebeat
6.檢查結(jié)果
tail -f /var/log/filebeat/filebeat
curl -s 127.0.0.1:9200/_cat/indices|awk '{print $3}'
7.es-head查看
8.kabana查看
說(shuō)明：這樣收集的日志，信息全是在messge這個(gè)字段模她，還是無(wú)法分離我們想要查看的內(nèi)容稻艰。
2.2.filebeat收集Nginx的json格式日志
1.修改nginx配置文件使日志轉(zhuǎn)換成json
log_format json '{ "time_local": "$time_local", '
'"remote_addr": "$remote_addr", '
'"referer": "$http_referer", '
'"request": "$request", '
'"status": $status, '
'"bytes": $body_bytes_sent, '
'"agent": "$http_user_agent", '
'"x_forwarded": "$http_x_forwarded_for", '
'"up_addr": "$upstream_addr",'
'"up_host": "$upstream_http_host",'
'"upstream_time": "$upstream_response_time",'
'"request_time": "$request_time"'
' }';
access_log /var/log/nginx/access.log json;
2.清除舊日志
> /var/log/nginx/access.log
3.檢查并重啟nginx
nginx -t
systemctl restart nginx
4.修改filebeat配置文件支持json解析
說(shuō)明：由于filebeat是go語(yǔ)言開(kāi)發(fā)的，所以默認(rèn)的是不支持json解析的侈净，需要額外配置尊勿。配置如下：
cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
json.keys_under_root: true
json.overwrite_keys: true
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
EOF
5.刪除ES里以前的索引
es-head >> filebeat-6.6.0-2019.11.15 >> 動(dòng)作 >>刪除
6.重啟filebeat
systemctl restart filebeat
7.es-head查看
8.kabana查看
沒(méi)有配置filebeat的結(jié)果為：
配置的結(jié)果為：
2.3.filebeat自定義ES索引名稱(chēng)
1.理想的索引的名稱(chēng)要與收集的對(duì)象的日志相關(guān)，以便區(qū)分畜侦。
例如 nginx-6.6.0-2020.02
2.filebeat配置
cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
json.keys_under_root: true
json.overwrite_keys: true
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
index: "nginx-%{[beat.version]}-%{+yyyy.MM}"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF
3.測(cè)試訪問(wèn)后es-head查看
4.kabana添加新的模板查看
2.4.filebeat按照服務(wù)類(lèi)型拆分索引
1.兩種配置方法
1.第一種寫(xiě)法
cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
json.keys_under_root: true
json.overwrite_keys: true
- type: log
enabled: true
paths:
- /var/log/nginx/error.log
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
indices:
- index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
source: "/var/log/nginx/access.log"
- index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
source: "/var/log/nginx/error.log"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF
2.第二種寫(xiě)法：
cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
json.keys_under_root: true
json.overwrite_keys: true
tags: ["access"]
- type: log
enabled: true
paths:
- /var/log/nginx/error.log
tags: ["error"]
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
indices:
- index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
tags: "access"
- index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
tags: "error"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF
2.重啟filebeat
systemctl restart filebeat
3.es-head查看
4.kabana查看結(jié)果
2.5.多服務(wù)器收集相同的日志合并
說(shuō)明：默認(rèn)是會(huì)自動(dòng)合并
新開(kāi)一臺(tái)服務(wù)器元扔，配置與之前的一樣。
1.es-head查看結(jié)果
2.kabana查看結(jié)果
一起查詢
分離查詢所需查看的結(jié)果
2.6.使用filebeat模塊收集nginx日志
官方配置說(shuō)說(shuō)明請(qǐng)參考：https://www.elastic.co/guide/en/beats/filebeat/6.6/filebeat-module-nginx.html
說(shuō)明：之前的收集需要修改nginx日志格式為json以及配置filebeat支持解析json格式夏伊，這樣的使用適合新的環(huán)境進(jìn)行使用，如果后期才上線日志收集這一塊吻氧，使用filebeat模塊收集日志可以不用修改原有的普通日志的格式和修改filebeat的配置文件溺忧。
0.配置es支持nginx模塊的插件
cd /usr/share/elasticsearch/
./bin/elasticsearch-plugin install file:///root/ingest-geoip-6.6.0.zip
./bin/elasticsearch-plugin install file:///root/ingest-user-agent-6.6.0.zip
systemctl restart elasticsearch
1.配置filebeat配置文件咏连，配置支持模塊功能
#默認(rèn)配置是faslse
============================= Filebeat modules ===============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
#精簡(jiǎn)并修改配置文件為ture
[root@nginx ~]# vim /etc/filebeat/filebeat.yml
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
reload.period: 10s
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
indices:
- index: "nginx-access-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
event.dataset: "nginx.access"
- index: "nginx-error-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
event.dataset: "nginx.error"
setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true
2.查看支持的模塊
[root@nginx ~]# filebeat modules list
Enabled: #表示已經(jīng)開(kāi)啟的模塊
Disabled: #表示所有支持的模塊或未開(kāi)啟的模塊
apache2
auditd
elasticsearch
haproxy
icinga
iis
kafka
kibana
logstash
mongodb
mysql
nginx
osquery
postgresql
redis
suricata
system
traefik
3.開(kāi)啟nginx模塊
[root@nginx ~]# filebeat modules enable nginx
Enabled nginx
[root@nginx ~]# filebeat modules list
Enabled:
nginx
Disabled:
apache2
auditd
elasticsearch
haproxy
icinga
iis
kafka
kibana
logstash
mongodb
mysql
osquery
postgresql
redis
suricata
system
traefik
4.配置nginx模塊
#默認(rèn)格式
[root@nginx /etc/filebeat]# vim modules.d/nginx.yml
- module: nginx
# Access logs
access:
denabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
# Error logs
error:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
#var.paths:
#修改并簡(jiǎn)化配置文件
[root@nginx /etc/filebeat/modules.d]# vim nginx.yml
- module: nginx
access:
enabled: true
var.paths: ["/var/log/nginx/access.log"]
error:
enabled: true
var.paths: ["/var/log/nginx/error.log"]
5.啟動(dòng)filebeat
systemctl start filebeat.service
6.es-head查看結(jié)果
7.kabana界面創(chuàng)建索引模板并查看
說(shuō)明：模塊中字段拆分的更加詳細(xì)
三.使用filebeat收集tomccat日志
1.修改tomact的日志文件問(wèn)json
#默認(rèn)格式
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="localhost_access_log." suffix=".txt" pattern="%h %l %u %t "%r" %s %b" />
#修改為json格式
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="localhost_access_log." suffix=".txt" pattern="{"clientip":"%h",&quo
t;ClientUser":"%l","authenticated":"%u","AccessTime&quo
t;:"%t","method":"%r","status":"%s","Sen
dBytes":"%b","Query?string":"%q","partner":"%
{Referer}i","AgentVersion":"%{User-Agent}i"}"/>
2.filebeat配置文件設(shè)置
[root@tomcat ~]# vim /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
enabled: true
paths:
- /usr/local/tomcat/logs/localhost_access_log.*.txt
json.keys_under_root: true
json.overwrite_keys: true
tags: ["tomcat"]
output.elasticsearch:
hosts: ["10.0.0.72:9200"]
index: "tomcat_access-%{[beat.version]}-%{+yyyy.MM}"
setup.template.name: "tomcat"
setup.template.pattern: "tomcat_*"
setup.template.enabled: false
setup.template.overwrite: true
3.重啟filebeat
[root@tomcat ~]# systemctl restart filebeat
4.測(cè)試訪問(wèn)查看es-head
5.kabana
四. filebeat收集java多行匹配模式
官方配置請(qǐng)參考鏈接：https://www.elastic.co/guide/en/beats/filebeat/6.6/multiline-examples.html
1.filebeat配置文件
cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/elasticsearch/elasticsearch.log
multiline.pattern: '^\['
multiline.negate: true
multiline.match: after
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
index: "es-%{[beat.version]}-%{+yyyy.MM}"
setup.template.name: "es"
setup.template.pattern: "es-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF
2.重啟filebeat
systemctl restart filebeat
五.使用filebeat模塊收集mysql慢日志和錯(cuò)誤日志
1.配置mysql錯(cuò)誤日志和慢日志路徑
編輯my.cnf
vim /etc/my.cnf
[mysqld]
slow_query_log=ON
slow_query_log_file=/data/mysql/data/slow.log
long_query_time=1
2.重啟mysql并制造慢日志
systemctl restart mysql
慢日志制造語(yǔ)句
select sleep(2) user,host from mysql.user ;
3.確認(rèn)慢日志和錯(cuò)誤日志確實(shí)有生成
mysql -uroot -poldboy123 -e "show variables like '%slow_query_log%'"
4.激活filebeat的mysql模塊
filebeat module enable mysql
5.配置mysql的模塊
[root@db05 ~]# vim /etc/filebeat/modules.d/mysql.yml
- module: mysql
# Error logs
error:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: ["/application/mysql/data/error.log"]
# Slow logs
slowlog:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: ["/application/mysql/data/slow.log"]
6.配置filebeat根據(jù)日志類(lèi)型做判斷
cat >/etc/filebeat/filebeat.yml<<EOF
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
reload.period: 10s
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
indices:
- index: "mysql-slow-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
source: "/application/mysql/data/slow.log"
- index: "mysql-err-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
source: "/application/mysql/data/error.log"
setup.template.name: "mysql"
setup.template.pattern: "mysql-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF
7.重啟filebeat
systemctl restart filebeat
8.es-head查看
六.使用filebeat收集docker日志
filebeat收集docker日志終極殺人王火云邪神版
1.需求分析
json格式并且按照下列索引生成
docker-nginx-access-6.6.0-2020.02
docker-db-access-6.6.0-2020.02
docker-db-error-6.6.0-2020.02
docker-nginx-error-6.6.0-2020.02
3.創(chuàng)建新容器并掛載本地的目錄到容器的日志文件目錄下
docker run -d -p 80:80 -v /opt/nginx:/var/log/nginx nginx
docker run -d -p 8080:80 -v /opt/mysql:/var/log/nginx nginx
4.準(zhǔn)備json格式的nginx配置文件并拷貝到容器里并重啟
docker cp nginx.conf 5d62b35651e6:/etc/nginx/
docker cp nginx.conf 310e85addbcd:/etc/nginx/
docker stop $(docker ps -qa)
docker start Nginx容器的ID
docker start mysql容器的ID
5.配置filebeat配置文件
cat >/etc/filebeat/filebeat.yml <<EOF
filebeat.inputs:
- type: log
enabled: true
paths:
- /opt/nginx/access.log
json.keys_under_root: true
json.overwrite_keys: true
tags: ["nginx_access"]
- type: log
enabled: true
paths:
- /opt/nginx/error.log
tags: ["nginx_err"]
- type: log
enabled: true
paths:
- /opt/mysql/access.log
json.keys_under_root: true
json.overwrite_keys: true
tags: ["db_access"]
- type: log
enabled: true
paths:
- /opt/mysql/error.log
tags: ["db_err"]
output.elasticsearch:
hosts: ["10.0.0.51:9200"]
indices:
- index: "docker-nginx-access-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
tags: "nginx_access"
- index: "docker-nginx-error-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
tags: "nginx_err"
- index: "docker-db-access-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
tags: "db_access"
- index: "docker-db-error-%{[beat.version]}-%{+yyyy.MM}"
when.contains:
tags: "db_err"
setup.template.name: "docker"
setup.template.pattern: "docker-*"
setup.template.enabled: false
setup.template.overwrite: true
EOF
6.啟動(dòng)filebeat
systemctl restart filebeat
7.訪問(wèn)并測(cè)試
curl 127.0.0.1
curl 127.0.0.1:8080/
cat /opt/nginx/access.log
cat /opt/mysql/access.log
8.es-head查看
