ELK分布式日志收集原理
- 每臺(tái)服務(wù)器集群節(jié)點(diǎn)安裝Logstash日志收集系統(tǒng)插件
- 每臺(tái)服務(wù)器節(jié)點(diǎn)將日志輸出到Logstash中
- Logstash將該日志格式化為json格式邻薯，根據(jù)每天創(chuàng)建不同的索引酱吝，輸出到Elasticsearch中
- 瀏覽器使用歐冠Kibana查詢?nèi)罩拘畔?/li>
環(huán)境安裝
- 安裝Elasticsearch
- 安裝Logstash
- 安裝Kibana
Logstash操作流程
把每臺(tái)服務(wù)器的日志文件轉(zhuǎn)換成json格式存放在ES里面，包括創(chuàng)建索引和文檔等番枚。
一、搭建elasticsearch
tar -zxvf elasticsearch-6.4.3.tar.gz
cd elasticsearch-6.4.3/config/
vi elasticsearch.yml
修改
-
network.host: 203.75.156.116狈谊，改為自己的ip地址 -
http.port: 9200,
cd ../bin/
#啟動(dòng) elasticsearch
./elasticsearch
PS:root用戶是不能正常啟動(dòng)的
若報(bào)一下錯(cuò)誤：
vi /etc/sysctl.conf
vm.max_map_count=655360
sysctl -p
若又報(bào)4096錯(cuò)誤
vi /etc/security/limits.conf
#添加以下四行
* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096
重啟服務(wù)器
訪問(wèn) ip:9200登刺，得到以下響應(yīng)內(nèi)容，就elasticsearch就算搭建成功了爱葵。
二施戴、搭建Logstash
- 配置
tar -zxvf logstash-6.4.3.tar.gz
cd logstash-6.4.3/config/
# 新建conf文件
vim myconf.conf
input {
# 從文件讀取日志信息 輸送到控制臺(tái)
file {
path => "/home/zhanhao/software/ELK/elasticsearch-6.4.3/logs/elasticsearch.log"
codec => "json" ## 以JSON格式讀取日志
type => "elasticsearch"
start_position => "beginning"
}
}
# filter {
#
# }
output {
# 標(biāo)準(zhǔn)輸出
# stdout {}
# 輸出進(jìn)行格式化，采用Ruby庫(kù)來(lái)解析日志
stdout { codec => rubydebug }
elasticsearch {
hosts => ["203.75.156.116:9200"]
index => "es-%{+YYYY.MM.dd}"
}
}
- 啟動(dòng)
cd logstash-6.4.3/bin/
./logstash -f ../config/myconf.conf
出現(xiàn)以下日志就算ok啦~(當(dāng)然首先得啟動(dòng)elasticsearch)
...
{
"tags" => [
[0] "_jsonparsefailure"
],
"@version" => "1",
"path" => "/home/zhanhao/software/ELK/elasticsearch-6.4.3/logs/elasticsearch.log",
"message" => "[2019-08-01T15:08:17,819][INFO ][o.e.g.GatewayService ] [node-1] recovered [3] indices into cluster_state",
"@timestamp" => 2019-08-01T07:08:17.998Z,
"type" => "elasticsearch",
"host" => "203-75-156-116.HINET-IP.hinet.net"
}
{
"tags" => [
[0] "_jsonparsefailure"
],
"@version" => "1",
"path" => "/home/zhanhao/software/ELK/elasticsearch-6.4.3/logs/elasticsearch.log",
"message" => "[2019-08-01T15:08:45,399][INFO ][o.e.c.m.MetaDataIndexTemplateService] [node-1] adding template [logstash] for index patterns [logstash-*]",
"@timestamp" => 2019-08-01T07:08:46.055Z,
"type" => "elasticsearch",
"host" => "203-75-156-116.HINET-IP.hinet.net"
}
{
"tags" => [
[0] "_jsonparsefailure"
],
"@version" => "1",
"path" => "/home/zhanhao/software/ELK/elasticsearch-6.4.3/logs/elasticsearch.log",
"message" => "[2019-08-01T15:08:47,960][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-1] [es-2019.08.01] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings []",
"@timestamp" => 2019-08-01T07:08:48.065Z,
"type" => "elasticsearch",
"host" => "203-75-156-116.HINET-IP.hinet.net"
}
{
"tags" => [
[0] "_jsonparsefailure"
],
"@version" => "1",
"path" => "/home/zhanhao/software/ELK/elasticsearch-6.4.3/logs/elasticsearch.log",
"message" => "[2019-08-01T15:08:48,339][INFO ][o.e.c.m.MetaDataMappingService] [node-1] [es-2019.08.01/hstmRsyvT5qG6NSIf-474w] create_mapping [doc]",
"@timestamp" => 2019-08-01T07:08:49.069Z,
"type" => "elasticsearch",
"host" => "203-75-156-116.HINET-IP.hinet.net"
}
三萌丈、搭建Kibana
tar -zxvf kibana-6.4.3-linux-x86_64.tar.gz
cd kibana-6.4.3-linux-x86_64/config/
vi kibana.yml
# 修改
server.port: 5601
server.host: "自己ip"
elasticsearch.url: "http://自己ip:9200"
cd ../bin/
# 啟動(dòng)
./kibana
訪問(wèn) ip:5601
四赞哗、視圖化界面
以時(shí)間戳創(chuàng)建
創(chuàng)建完成效果
沒(méi)有找到日志數(shù)據(jù)，擴(kuò)大查詢范圍
