1. ELK收集Nginx普通格式的日志

1.1 測(cè)試服務(wù)器架構(gòu)

1.2 ab工具使用

yum install httpd-tools -y

# -n 總共發(fā)送多少條請(qǐng)求琼稻，注意，最后"/"一定要寫(xiě)饶囚，否則命令無(wú)法執(zhí)行
# -c 多少條請(qǐng)求發(fā)送一次
ab -c 10 -n 100 http://10.0.0.100:80/

[root@node01 log]# tail -f /var/log/nginx/access.log 
10.0.0.100 - - [16/Apr/2020:19:03:40 +0800] "GET / HTTP/1.0" 200 612 "-" "ApacheBench/2.3" "-"
10.0.0.100 - - [16/Apr/2020:19:03:40 +0800] "GET / HTTP/1.0" 200 612 "-" "ApacheBench/2.3" "-"
10.0.0.100 - - [16/Apr/2020:19:03:40 +0800] "GET / HTTP/1.0" 200 612 "-" "ApacheBench/2.3" "-"
10.0.0.100 - - [16/Apr/2020:19:03:40 +0800] "GET / HTTP/1.0" 200 612 "-" "ApacheBench/2.3" "-"
10.0.0.100 - - [16/Apr/2020:19:03:40 +0800] "GET / HTTP/1.0" 200 612 "-" "ApacheBench/2.3" "-"
10.0.0.100 - - [16/Apr/2020:19:03:40 +0800] "GET / HTTP/1.0" 200 612 "-" "ApacheBench/2.3" "-"
10.0.0.100 - - [16/Apr/2020:19:03:40 +0800] "GET / HTTP/1.0" 200 612 "-" "ApacheBench/2.3" "-"
10.0.0.100 - - [16/Apr/2020:19:03:40 +0800] "GET / HTTP/1.0" 200 612 "-" "ApacheBench/2.3" "-"
10.0.0.100 - - [16/Apr/2020:19:03:40 +0800] "GET / HTTP/1.0" 200 612 "-" "ApacheBench/2.3" "-"
10.0.0.100 - - [16/Apr/2020:19:03:40 +0800] "GET / HTTP/1.0" 200 612 "-" "ApacheBench/2.3" "-"

ab工具用于批量發(fā)送HTTP請(qǐng)求到指定的URL帕翻，是一個(gè)壓力測(cè)試工具，這里使用它來(lái)生成Nginx的日志

1.3 filebeat配置

• 配置文件：/etc/filebeat/filebeat.yml

  # 我們只留下最精簡(jiǎn)的部分
  # 定義數(shù)據(jù)源
  filebeat.inputs:
  # 數(shù)據(jù)源為普通日志文件
  - type: log
    # 啟用
    enabled: true
    # 日志文件的位置
    paths:
      - /var/log/nginx/access.log
  
  # 定義輸出類(lèi)型
  # 輸出到elasitcsearch
  output.elasticsearch:
    hosts: ["10.0.0.100:9200","10.0.0.101:9200","10.0.0.102:9200"]
  

• 啟動(dòng)filebeat：systemctl start filebeat

• 查看ES的index

  GET _cat/indices
  
  green open filebeat-6.6.0-2020.04.16 Y9pmNuEoTW2lGdxq40wsqg 3 1 100 0 225.1kb 106.3kb
  
  GET filebeat-6.6.0-2020.04.16/_search
  
  {
    "took" : 6,
    "timed_out" : false,
    "_shards" : {
      "total" : 3,
      "successful" : 3,
      "skipped" : 0,
      "failed" : 0
    },
    "hits" : {
      "total" : 100,
      "max_score" : 1.0,
      "hits" : [
        {
          "_index" : "filebeat-6.6.0-2020.04.15",
          "_type" : "doc",
          "_id" : "9GaVfXEBcWrWjTbD1Bo0",
          "_score" : 1.0,
          "_source" : {
            "@timestamp" : "2020-04-16T11:25:01.369Z",
            "beat" : {
              "version" : "6.6.0",
              "name" : "node01",
              "hostname" : "node01"
            },
            "host" : {
              "name" : "node01",
              "architecture" : "x86_64",
              "os" : {
                "family" : "redhat",
                "name" : "CentOS Linux",
                "codename" : "Core",
                "platform" : "centos",
                "version" : "7 (Core)"
              },
              "id" : "ea70b3ad93714ed2be82e374ec284fe6",
              "containerized" : true
            },
            "log" : {
              "file" : {
                "path" : "/var/log/nginx/access.log"
              }
            },
            # Nginx日志
            "message" : """10.0.0.100 - - [16/Apr/2020:19:03:40 +0800] "GET / HTTP/1.0" 200 612 "-" "ApacheBench/2.3" "-"""",
            "source" : "/var/log/nginx/access.log",
            "offset" : 4940,
            "prospector" : {
              "type" : "log"
            },
            "input" : {
              "type" : "log"
            }
          }
        }
        ......
      ]
    }
  }
  

1.4 Kibana WEB-UI 配置

2. ELK收集Nginx Json格式的日志

• 關(guān)閉filebeat服務(wù)：systemctl start filebeat

• 刪除kibana管理的Index Pattern

• 刪除ES的index：DELETE filebeat-6.6.0-2020.04.16

• 清空Nginx日志：> /var/log/nginx/access.log

• 修改Nginx配置文件萝风，重啟Nginx

  log_format json  '{"time_local": "$time_local", '
                            '"remote_addr": "$remote_addr", '
                            '"referer": "$http_referer", '
                            '"request": "$request", '
                            '"status": $status, '
                            '"bytes": $body_bytes_sent, '
                            '"agent": "$http_user_agent", '
                            '"x_forwarded": "$http_x_forwarded_for", '
                            '"up_addr": "$upstream_addr", '
                            '"up_host": "$upstream_http_host", '
                            '"upstream_time": "$upstream_response_time", '
                            '"request_time": "$request_time"}';
  
  access_log  /var/log/nginx/access.log  json;
  

• 修改/etc/filebeat/filebeat.yml

  filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/nginx/access.log
    # 以下兩行設(shè)置將nginx日志存儲(chǔ)為json格式
    json.keys_under_root: true
    json.overwrite_keys: true
  
  output.elasticsearch:
    hosts: ["10.0.0.100:9200","10.0.0.101:9200","10.0.0.102:9200"]
    # 設(shè)置index名嘀掸，通常按月滾動(dòng)
    index: "nginx-%{+yyyy.MM}"
  
  # 當(dāng)index被重寫(xiě)后，以下4個(gè)配置也必須重寫(xiě)
  # 設(shè)置自定義的配置模板的名稱(chēng)
  setup.template.name: "nginx"
  # 保存到哪個(gè)index的時(shí)候使用此模板
  setup.template.pattern: "nginx-*"
  # 設(shè)置默認(rèn)配置模板不可用
  setup.template.enabled: false
  # 設(shè)置自定義的配置模板可用
  setup.template.overwrite: true
  

• 啟動(dòng)filebeat：systemctl start filebeat

• 發(fā)送測(cè)試數(shù)據(jù)

  # 使用3個(gè)服務(wù)器發(fā)送請(qǐng)求
  [root@node01 ~]# ab -c 100 -n 100 http://10.0.0.100:80/jingdong
  [root@node01 ~]# ab -c 100 -n 100 http://10.0.0.100:80/
  [root@node02 ~]# ab -c 100 -n 100 http://10.0.0.100:80/baidu
  [root@node02 ~]# ab -c 100 -n 100 http://10.0.0.100:80/
  [root@node03 ~]# ab -c 100 -n 100 http://10.0.0.100:80/taobao
  [root@node03 ~]# ab -c 100 -n 100 http://10.0.0.100:80/
  

• 查看ES index

  GET _cat/indices
  green open nginx-2020.04             2l7iUDU9SpWDxN96ui2DhQ 5 1 600 0     1mb   502kb
  
  GET nginx-2020.04/_search
  {
    "took" : 4,
    "timed_out" : false,
    "_shards" : {
      "total" : 5,
      "successful" : 5,
      "skipped" : 0,
      "failed" : 0
    },
    "hits" : {
      "total" : 600,
      "max_score" : 1.0,
      "hits" : [
        {
          "_index" : "nginx-2020.04",
          "_type" : "doc",
          "_id" : "7KN_gXEB3XeAWkvtHPjB",
          "_score" : 1.0,
          "_source" : {
            "@timestamp" : "2020-04-16T05:38:42.359Z",
            "request_time" : "0.000",
            "up_host" : "-",
            "time_local" : "16/Apr/2020:13:34:01 +0800",
            "request" : "GET /baidu HTTP/1.0",
            "input" : {
              "type" : "log"
            },
            "beat" : {
              "version" : "6.6.0",
              "name" : "node01",
              "hostname" : "node01"
            },
            # nginx日志存儲(chǔ)成了json格式
            "bytes" : 153,
            "remote_addr" : "10.0.0.101",
            "up_addr" : "-",
            "upstream_time" : "-",
            "x_forwarded" : "-",
            "referer" : "-",
            "agent" : "ApacheBench/2.3",
            "host" : {
              "name" : "node01",
              "os" : {
                "family" : "redhat",
                "name" : "CentOS Linux",
                "codename" : "Core",
                "platform" : "centos",
                "version" : "7 (Core)"
              },
              "id" : "ea70b3ad93714ed2be82e374ec284fe6",
              "containerized" : true,
              "architecture" : "x86_64"
            },
            "source" : "/var/log/nginx/access.log",
            "status" : 404,
            "offset" : 277900,
            "log" : {
              "file" : {
                "path" : "/var/log/nginx/access.log"
              }
            },
            "prospector" : {
              "type" : "log"
            }
          }
        }
        ......
      ]
    }
  }
  

• Kibana WEB-UI的配置

  重復(fù)步驟不再列出

  添加頁(yè)面顯示的字段

3. ELK收集多臺(tái)Nginx服務(wù)器的日志

3.1 測(cè)試服務(wù)器架構(gòu)

3.2 部署過(guò)程

• 3個(gè)節(jié)點(diǎn)的Nginx的配置同步后规惰，啟動(dòng)Nginx服務(wù)
• 3個(gè)節(jié)點(diǎn)的filebeat配置同步后睬塌，啟動(dòng)filebeat
• 發(fā)送測(cè)試請(qǐng)求

# 使用3個(gè)服務(wù)器發(fā)送請(qǐng)求
[root@node01 ~]# ab -c 5 -n 5 http://10.0.0.101:80/test
[root@node01 ~]# ab -c 5 -n 5 http://10.0.0.102:80/test
[root@node02 ~]# ab -c 5 -n 5 http://10.0.0.100:80/test
[root@node02 ~]# ab -c 5 -n 5 http://10.0.0.102:80/test
[root@node03 ~]# ab -c 5 -n 5 http://10.0.0.100:80/test
[root@node03 ~]# ab -c 5 -n 5 http://10.0.0.101:80/test

• 檢查數(shù)據(jù)

GET _cat/indices
# 數(shù)據(jù)增加了30條
green open nginx-2020.04             2l7iUDU9SpWDxN96ui2DhQ 5 1 630 0   1.8mb 921.4kb

• 顯示數(shù)據(jù)，添加host.name歇万，并過(guò)濾出指定的主機(jī)收集到的日志

4. Nginx正常日志與錯(cuò)誤日志拆分

• 修改filebeat配置并同步

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  json.keys_under_root: true
  json.overwrite_keys: true
  tags: ["access"]
- type: log
  enabled: true
  # 錯(cuò)誤日志不需要使用json格式揩晴，因?yàn)槲覀兒苌賹?duì)錯(cuò)誤日志進(jìn)行聚合分析
  paths:
    - /var/log/nginx/error.log
  tags: ["error"]

output.elasticsearch:
  hosts: ["10.0.0.100:9200","10.0.0.101:9200","10.0.0.102:9200"]
  indices:
  - index: "nginx-access-%{+yyyy.MM}"
    when.contains:
      tags: "access"
  - index: "nginx-error-%{+yyyy.MM}"
    when.contains:
      tags: "error"

setup.template.name: "nginx"
setup.template.pattern: "nginx-*"
setup.template.enabled: false
setup.template.overwrite: true

setup.template.settings:
  # 設(shè)置目標(biāo)index的shard個(gè)數(shù)
  index.number_of_shards: 3
# 設(shè)置kibana的IP和端口
setup.kibana:
  host: "10.0.0.100:5601"

• 重啟filebeat
• 查看索引

GET _cat/indices
green open nginx-error-2020.04       723oaOL3SamTcJId6E--9Q 5 1 1011 0   1.5mb 738.8kb
green open nginx-access-2020.04      v-9G7VLeREKvfh9kg-Wi3g 5 1   30 0 394.6kb 197.3kb

5. 使用filebeat自帶的nginx module收集nginx日志

filebeat配置

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s

output.elasticsearch:
  hosts: ["10.0.0.100:9200","10.0.0.101:9200","10.0.0.102:9200"]
  indices:
    - index: "nginx_access-%{+yyyy.MM}"
      when.contains:
        fileset.name: "access"
    - index: "nginx_error-%{+yyyy.MM}"
      when.contains:
        fileset.name: "error"
setup.template.name: "nginx"
setup.template.pattern: "nginx_*"
setup.template.enabled: false
setup.template.overwrite: true
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
  host: "10.0.0.100:5601"

查看filebeat自帶的模塊

[root@node01 ~]# filebeat modules list
Enabled:

Disabled:
apache2
auditd
elasticsearch
haproxy
icinga
iis
kafka
kibana
logstash
mongodb
mysql
nginx
osquery
postgresql
redis
suricata
system
traefik

修改nginx模塊的配置

[root@node01 ~]# cat /etc/filebeat/modules.d/nginx.yml.disabled 
- module: nginx
  access:
    enabled: true
    var.paths: ["/var/log/nginx/access.log"]
  error:
    enabled: true
    var.paths: ["/var/log/nginx/error.log"]

激活nginx模塊

激活后原來(lái)的配置文件nginx.yml.disabled變?yōu)榱?code>nginx.yml

[root@node01 ~]# filebeat modules enable nginx
Enabled nginx
[root@node01 ~]# filebeat modules list
Enabled:
nginx

Disabled:
apache2
auditd
elasticsearch
haproxy
icinga
iis
kafka
kibana
logstash
mongodb
mysql
osquery
postgresql
redis
suricata
system
traefik

nginx還是使用默認(rèn)的日志格式

access_log  /var/log/nginx/access.log main;

安裝ingest-user-agent插件和ingest-geoip插件

• 在線安裝

/usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-user-agent
/usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-geoip

• 離線安裝

wget https://artifacts.elastic.co/downloads/elasticsearch-plugins/ingest-user-agent/ingest-user-agent-6.6.0.zip
wget https://artifacts.elastic.co/downloads/elasticsearch-plugins/ingest-geoip/ingest-geoip-6.6.0.zip
/usr/share/elasticsearch/bin/elasticsearch-plugin install file:///root/ingest-geoip-6.6.0.zip 
/usr/share/elasticsearch/bin/elasticsearch-plugin install file:///root/ingest-user-agent-6.6.0.zip

[root@node03 ~]# /usr/share/elasticsearch/bin/elasticsearch-plugin install file:///root/ingest-user-agent-6.6.0.zip
-> Downloading file:///root/ingest-user-agent-6.6.0.zip
[=================================================] 100%   
-> Installed ingest-user-agent
[root@node03 ~]# /usr/share/elasticsearch/bin/elasticsearch-plugin install file:///root/ingest-geoip-6.6.0.zip
-> Downloading file:///root/ingest-geoip-6.6.0.zip
[=================================================] 100%   
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessDeclaredMembers
* java.lang.reflect.ReflectPermission suppressAccessChecks
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
-> Installed ingest-geoip

說(shuō)明：

• ES集群中的所有節(jié)點(diǎn)都需要安裝這兩個(gè)插件，安裝完之后重啟ES服務(wù)
• ES6.7之后這兩個(gè)插件默認(rèn)集成到了elasticsearch贪磺，不需要單獨(dú)安裝了

測(cè)試

• 清空原有的index和index pattern

• 清空nginx日志

• 重啟nginx

• 啟動(dòng)filebeat

• 使用ab工具發(fā)送幾條測(cè)試數(shù)據(jù)

GET _cat/indices

green open nginx_access-2020.04 7ibKAbFGQx66-a86s_53SQ 5 1 25 0 568.9kb 284.4kb
green open nginx_error-2020.04  bt-yYMQBTbqyZdBvmAzkRQ 5 1 15 0 275.9kb   145kb

注意硫兰，給nginx_error創(chuàng)建index pattern時(shí)，Time Filter field name 選擇read_timestamp寒锚，而nginx_access選擇@timestamp

可以看到劫映，filebeat內(nèi)置的nginx模塊配合解析User-agent的插件ingest-user-agent-6.6.0.zip以及解析IP的插件ingest-geoip-6.6.0.zip幫我們把nginx的普通日志做了很細(xì)力度的解析违孝，并且自動(dòng)保存成JSON格式，但是error日志還是使用message來(lái)表示一整行日志