日志收集平臺 ELK stack方案學(xué)習(xí)筆記

ELK Stack解決方案

Elastic Search + LogStash + Kibana =ELK Stack

https://elastic.co
https://www.elastic.co/guide/index.html

Elasticsearch 概念

https://baike.baidu.com/item/elasticsearch/3411206?fr=aladdin

Elasticsearch安裝方法

https://www.elastic.co/guide/en/elasticsearch/reference/5.5/rpm.html

yum install java -y
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
cat > /etc/yum.repos.d/elasticsearch.repo << EOF
[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

yum install -y elasticsearch

[root@jump elasticsearch]# grep ^[a-z] elasticsearch.yml 
cluster.name: fbo
node.name: linux-node-1
path.data: /data/es-data
path.logs: /var/log/elasticsearch/
bootstrap.memory_lock: true
network.host: 0.0.0.0
http.port: 9200
transport.host: localhost
transport.tcp.port: 9300

Elasticsearch查詢

支持 java api，RESTful API

RESTful API 查詢

# curl -i -XGET 'http://192.168.57.100:9200/_count?pretty' -d'{ 
"query":{"match_all":{}}}'
HTTP/1.1 200 OK
Warning: 299 Elasticsearch-5.5.1-19c13d0 "Content type detection for rest requests is deprecated. Specify the content type using the [Content-Type] header." "Tue, 08 Aug 2017 03:57:56 GMT"
content-type: application/json; charset=UTF-8
content-length: 95

{
  "count" : 0,
  "_shards" : {
    "total" : 0,
    "successful" : 0,
    "failed" : 0
  }
}

安裝插件工具M(jìn)arvel插件

cd /usr/share/elasticsearch
sudo bin/plugin install license
sudo bin/plugin install marvel-agent
sudo bin/plugin install mobz/elasticsearch

這個插件5.0之后的版本包括在x-pack下，安裝方法如下：
https://www.elastic.co/guide/en/elasticsearch/reference/5.5/installing-xpack-es.html

安裝開源插件elasticsearch-head
https://github.com/mobz/elasticsearch-head#running-with-built-in-server

安裝開源插件bigdesk
https://github.com/lukas-vlcek/bigdesk
安裝開源插件kopf
https://github.com/lmenezes/elasticsearch-kopf

logstash

安裝logstash

https://www.elastic.co/guide/en/logstash/5.5/installing-logstash.html

bin/logstash -e 'input { stdin { } } output { stdout {} }'
bin/logstash -e 'input {stdin{}} output {elasticsearch{hosts => ["192.168.57.100:9200"]} stdout{codec => rubydebug}}'

配置logstash

https://www.elastic.co/guide/en/logstash/5.5/configuration.html

input 插件

https://www.elastic.co/guide/en/logstash/5.5/input-plugins.html

codec插件

https://www.elastic.co/guide/en/logstash/5.5/codec-plugins.html

file java nginx syslog tcp

json格式收集nginx日志

nc 192.168.57.100 6666 < /etc/resolv.conf
偽終端
echo "oldboy" > /dev/tcp/192.168.57.100/6666

filter插件

https://www.elastic.co/guide/en/logstash/5.5/filter-plugins.html
grok插件
https://www.elastic.co/guide/en/logstash/5.5/plugins-filters-grok.html
https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns

ELK架構(gòu)設(shè)計

解耦之消息隊(duì)列

image.png

經(jīng)典設(shè)計架構(gòu)