開(kāi)始入門Logstash(上)

2017.11.28

寫(xiě)這篇文章的時(shí)候,我的Logstash,Filebeat,Elasticsearch,kibana都是6.0版本.本文的適用對(duì)象:像我一樣沒(méi)有學(xué)過(guò)運(yùn)維的小白,另外,這篇文章其實(shí)就是翻譯官網(wǎng)上的文檔+我自己在使用中的備注,本人英語(yǔ)六級(jí)水平,有錯(cuò)誤的地方請(qǐng)輕噴.

Elk技術(shù)棧是分別由Elasticsearch,Logstash,Kibana組成,隨著技術(shù)的推進(jìn)也衍生了更多性能更好的工具,比如收集文件更快的Filebeat,在有生之年我希望能把整個(gè)技術(shù)棧都展現(xiàn)在大家眼前..

本文的內(nèi)容將會(huì)指引你入門Logstash,從最簡(jiǎn)單的應(yīng)用到創(chuàng)建多條管道(pipeline)將apcahe的日志作為輸入,然后解析日志,最后將日志輸出到elasticsearch.最后你將組裝多個(gè)輸入以及輸出插件從不同的源頭處最終獲取到統(tǒng)一的數(shù)據(jù).

本文將循序漸進(jìn)的帶你了解Logstash

• 安裝Logstash
• 運(yùn)行第一個(gè)實(shí)例
• 通過(guò)Logstash解析日志
• 整合多輸入,輸出插件(下篇文章)

安裝Logstash

Logstash 是動(dòng)態(tài)數(shù)據(jù)收集管道，擁有可擴(kuò)展的插件生態(tài)系統(tǒng)茬腿，能夠與 Elasticsearch 產(chǎn)生強(qiáng)大的協(xié)同作用肩豁。

官網(wǎng)上有全部軟件的下載地址:The Elastic Stack Download,我的安裝方式很簡(jiǎn)單,首先,你需要安裝java環(huán)境,然后再下載軟件

yum install yum install java-1.8.0
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.0.0.tar.gz
tar zxf logstash-6.0.0.tar.gz

之后我會(huì)改名,然后刪除壓縮包,對(duì)于其他軟件也是這樣處理:

mv logstash-6.0.0 logstash
rm logstash-6.0.0.tar.gz

這樣做完后,我的用戶目錄會(huì)比較清晰:

>ll
drwxr-xr-x  7 j j      4096 Nov 11 02:42 elastic
drwxr-xr-x  6 j j      4096 Nov 28 17:36 filebeat
drwxrwxr-x 12 j j      4096 Nov 26 22:39 kibana
drwxrwxr-x 12 j j      4096 Nov 29 09:59 logstash

另外官網(wǎng)上還提供了其他各種各樣的安裝方式:Installing Logstash,如果你跟我一樣是個(gè)初學(xué)者,可以就用我的方法來(lái)安裝

運(yùn)行你的第一個(gè)實(shí)例

一個(gè)Logstash需要兩個(gè)最近基本的元素,輸入以及輸出,以及一個(gè)可選的元素,過(guò)濾.輸入組件負(fù)責(zé)從數(shù)據(jù)源獲取數(shù)據(jù),過(guò)濾組件負(fù)責(zé)按需修飾數(shù)據(jù)(過(guò)濾,格式化等),輸出組件負(fù)責(zé)將數(shù)據(jù)寫(xiě)入指定的目的地.

模型圖

cd logstash
bin/logstash -e 'input { stdin { } } output { stdout {} }'

如果你看到Pipeline stared的日志輸出在命令行中,就代表運(yùn)行成功啦
> bin目錄的地址會(huì)根據(jù)不同的安裝環(huán)境(.zip,.tar.gz,rpm,Docker,)而改變,具體可以看這里=>
Logstash目錄布局

-e參數(shù)可以直接在命令行中配置參數(shù),可以讓你快速的測(cè)試配置而不用編輯一個(gè)配置文件.實(shí)例中的管道使用標(biāo)準(zhǔn)輸入stdin作為輸入,并將其以一定的格式作為標(biāo)準(zhǔn)輸出stdou

在你看到"Pipeline main started"后,在命令行中輸入Hello world

Hello world
2017-11-28T08:11:35.811Z localhost Hello world

Logstash在日志中添加了時(shí)間戳以及IP地址的信息,想要退出的話可以執(zhí)行CTRL-D來(lái)退出.

恭喜您吶!你已經(jīng)創(chuàng)建并且跑了一個(gè)基礎(chǔ)的Logstash管道.準(zhǔn)備一下迎接真實(shí)世界的黑暗吧.

用Logstash來(lái)修飾日志

在上一節(jié)中,你跑了一個(gè)最基礎(chǔ)的Logstash管道來(lái)測(cè)試你的Logstash.在真實(shí)環(huán)境中,一個(gè)Logstash管道會(huì)復(fù)雜一些:往往會(huì)有不止一個(gè)的輸入,過(guò)濾,以及輸出的組件.

在這一節(jié)中,你將使用Filebeat獲取Apache的網(wǎng)絡(luò)日志并將其作為輸入,將這些日志解析成明確的信息,并將解析后的數(shù)據(jù)輸入Elasticsearch集群.這次你將使用文件來(lái)配置管道.

開(kāi)始前點(diǎn)擊這里來(lái)下載本次教程的示例數(shù)據(jù).解壓下載后得到文件.

配置Filebeat將日志傳輸給Logstash

在你創(chuàng)建Logstash管道前,你將配置Filebeat將日志傳輸給Logstash.Filebeat是一個(gè)輕量,資源友好的從服務(wù)器生成的文件中收集日志的工具,并將日志發(fā)送給你的Logstash實(shí)例進(jìn)行處理.Filebeat被設(shè)計(jì)成穩(wěn)定,高效(低延遲),在主機(jī)上占用很少的資源,Beats input插件最小化了對(duì)Logstash實(shí)例所需要的資源.

在一個(gè)典型的部署中,Filebeat將在你運(yùn)行Logstash實(shí)例以外的多個(gè)機(jī)器中收集日志.為了這次示例,Logstash與Filebeat將會(huì)在同一臺(tái)機(jī)器上運(yùn)行

Logstash默認(rèn)安裝后會(huì)帶有Beats input插件,該組件使Logstash可以從Elastic Beats 框架中接收事件.這意味著任何一個(gè)使用Beat框架寫(xiě)的服務(wù)都能將事件數(shù)據(jù)傳輸給Logstash,比如Packetbeat和Metricbeat

首先呢,你需要一個(gè)Filebeat.

安裝完Filebeat后,你需要配置它.打開(kāi)安裝目錄的filebeat.yml文件,替換如下的內(nèi)容,要保證paths指向了之前下載的示例數(shù)據(jù)logstash-tutorial.log(文件名可能不一樣,自行改名):

filebeat.prospectors:

• type: log
  paths:
  • /path/to/file/logstash-tutorial.log
    output.logstash:
    hosts: ["localhost:5043"]

請(qǐng)使用絕對(duì)路徑,output.logstash前不能有空格.

保存你的編輯
為了簡(jiǎn)化這次的配置,不用指定TLS/SSL這些在真實(shí)環(huán)境中需要的配置

在數(shù)據(jù)源機(jī)器,用如下命令啟動(dòng)Filebeat

sudo ./filebeat -e -c filebeat.yml -d "publish"

如果使用root賬號(hào)運(yùn)行Filebeat,需要修改配置文件的擁有者,這里建議為elk創(chuàng)建新的用戶以及用戶組,因?yàn)镋lasticsearch也是不能使用root用戶運(yùn)行的.

Filebeat會(huì)嘗試去連接5043端口,直到Logstash與Beats插件啟動(dòng)前都不會(huì)在該端口上有響應(yīng),所以現(xiàn)在你看到的應(yīng)該都是連接端口失敗的消息.

配置Logstash使用Filebeat作為輸入

接下來(lái),你會(huì)創(chuàng)建一個(gè)使用Beats輸入插件來(lái)接收Beats事件的Logstash管道.

下面的文字代表了這次管道配置的骨架信息:

The # character at the beginning of a line indicates a comment.Use # comments to describe your configuration.
input {
}
# The filter part of this file is commented out to indicate that it is
# optional.
# filter {
#
# }
output {
}

這個(gè)骨架配置不會(huì)起效,因?yàn)闆](méi)有任何有效的配置

開(kāi)始前,將這段骨架配置復(fù)制到你自己的配置中,并命名文件為first-pipeline.conf,保存在你的Logstash目錄中

接下來(lái),配置你的Logstash實(shí)例使用Beats input插件,在first-pipeline.conf文件的input部分加入如下文本

beats {
port => "5043"
}

在未來(lái)你會(huì)使用Logstash推送到Elasticsearch,不過(guò)現(xiàn)在,先使用標(biāo)準(zhǔn)輸出output來(lái)打印信息:

stdout { codec => rubydebug }

現(xiàn)在,你的first-pipeline.conf文件應(yīng)該像這樣:

input {
beats {
port => "5043"
}
}
# The filter part of this file is commented out to indicate that it is
# optional.
# filter {
#
# }
output {
stdout { codec => rubydebug }
}

用這個(gè)命令來(lái)驗(yàn)證你的配置是否正確:

bin/logstash -f first-pipeline.conf --config.test_and_exit

--config.test_and_exit選項(xiàng)將會(huì)解析并且提供配置文件的錯(cuò)誤信息,如果配置通過(guò)了配置測(cè)試,運(yùn)行這個(gè)命令來(lái)啟動(dòng)Logstash:

bin/logstash -f first-pipeline.conf --config.reload.automatic

--config.reload.automatic選項(xiàng)將會(huì)自動(dòng)讀取配置文件,這樣你就不用再每次修改配置后都要停止再啟動(dòng)Logstash了

Logstash啟動(dòng)后,你可能會(huì)看到多條關(guān)于Logstash忽略了pipelines.yml文件的警告信息,你可以放心的忽略這些告警.pipelines.yml文件是用來(lái)配置一個(gè)Logstash實(shí)例運(yùn)行多個(gè)管道,本次示例中,你只運(yùn)行一個(gè)管道.

如果你的管道正常工作,你將會(huì)收到多條事件:

{
    "@timestamp" => 2017-11-09T01:44:20.071Z,
        "offset" => 325,
      "@version" => "1",
          "beat" => {
            "name" => "My-MacBook-Pro.local",
        "hostname" => "My-MacBook-Pro.local",
         "version" => "6.0.0"
    },
          "host" => "My-MacBook-Pro.local",
    "prospector" => {
        "type" => "log"
    },
        "source" => "/path/to/file/logstash-tutorial.log",
       "message" => "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
          "tags" => [
        [0] "beats_input_codec_plain_applied"
    ]
}
...

通過(guò)Grok Filter插件來(lái)解析網(wǎng)絡(luò)日志

現(xiàn)在呢,你有一條通過(guò)Filebeat來(lái)讀取日志的管道了.然后應(yīng)該注意到現(xiàn)在的日志信息并不是理想中的格式.你想講日志解析成一條清晰明了的信息,這樣的話,就需要使用到grok過(guò)濾插件

grok過(guò)濾插件是Logstash支持的默認(rèn)插件之一,想知道管理Logstash插件的更多情況,可以來(lái)這里:reference documentation

grok插件讓你能夠?qū)⒖勺x性差,非結(jié)構(gòu)化的日志數(shù)據(jù)解析成某種結(jié)構(gòu)可查詢的數(shù)據(jù).

一條具有代表性的網(wǎng)絡(luò)服務(wù)日志如下:

83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png
HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"

開(kāi)頭的ip地址很容易識(shí)別,括號(hào)里的時(shí)間戳也是.為了解析數(shù)據(jù),你可以使用%{COMBINEDAPACHELOG}grok 表達(dá)式,講會(huì)通過(guò)如下表格來(lái)組織文本:

如果字構(gòu)建grok表達(dá)式時(shí)遇到問(wèn)題,可以嘗試使用Grok Debugger.

修改first-pipeline.conf文件替換filter的內(nèi)容:

filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
}

修改完后,你的first-pipeline.conf應(yīng)該是這個(gè)樣子的:

input {
    beats {
        port => "5043"
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
}
output {
    stdout { codec => rubydebug }
}

保存文件,因?yàn)橹澳愦蜷_(kāi)了自動(dòng)重載配置文件的設(shè)置,所以不需要重啟Logstash以生效.然而,還是要強(qiáng)制Filebeat重頭開(kāi)始讀取日志.你需要做的事很簡(jiǎn)單,找到運(yùn)行著Filebeat的命令行,按下Ctrl-C關(guān)閉Filebeat,刪除Filebeat的注冊(cè)文件:

sudo rm data/registry

正是因?yàn)镕ilebeat講文件的收錄狀態(tài)記錄在了注冊(cè)文件中,所以當(dāng)你刪除了注冊(cè)文件后,就能強(qiáng)制Filebeat從頭讀取這些文件啦

接下來(lái),重新運(yùn)行Filebeat:

sudo ./filebeat -e -c filebeat.yml -d "publish"

Logstash重新載入config時(shí)會(huì)有些許的延遲,當(dāng)你清除注冊(cè)文件后可能得等一會(huì)兒

在Logstash的grok表達(dá)式生效后,會(huì)收到如下的JSON格式的事件:

{
        "request" => "/presentations/logstash-monitorama-2013/images/kibana-search.png",
          "agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
         "offset" => 325,
           "auth" => "-",
          "ident" => "-",
           "verb" => "GET",
     "prospector" => {
        "type" => "log"
    },
         "source" => "/path/to/file/logstash-tutorial.log",
        "message" => "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
           "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
       "referrer" => "\"http://semicomplete.com/presentations/logstash-monitorama-2013/\"",
     "@timestamp" => 2017-11-09T02:51:12.416Z,
       "response" => "200",
          "bytes" => "203023",
       "clientip" => "83.149.9.216",
       "@version" => "1",
           "beat" => {
            "name" => "My-MacBook-Pro.local",
        "hostname" => "My-MacBook-Pro.local",
         "version" => "6.0.0"
    },
           "host" => "My-MacBook-Pro.local",
    "httpversion" => "1.1",
      "timestamp" => "04/Jan/2015:05:13:42 +0000"
}

注意到事件同時(shí)包括了原始信息,日志被細(xì)分成了特定的字段.

用Geoip Filter插件里優(yōu)化你的數(shù)據(jù)

為了使你的數(shù)據(jù)更加便于搜索,過(guò)濾組件可以從現(xiàn)有的數(shù)據(jù)中獲取額外的信息.例如,geoip插件查詢ip,從地址獲取地理位置,并添加位置信息到日志中.

來(lái)吧,添加geoip插件的設(shè)置到你的配置文件first-pipeline.conf中去:

 geoip {
        source => "clientip"
    }

保存設(shè)置,像之前一樣強(qiáng)制Filebeat讀取日志.停止運(yùn)行Filebeat(按下Ctrl-C),刪除注冊(cè)文件,然后運(yùn)行Filebeat:

sudo ./filebeat -e -c filebeat.yml -d "publish"

注意到推送的事件中已經(jīng)包含了地理信息:

{
        "request" => "/presentations/logstash-monitorama-2013/images/kibana-search.png",
          "agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
          "geoip" => {
              "timezone" => "Europe/Moscow",
                    "ip" => "83.149.9.216",
              "latitude" => 55.7485,
        "continent_code" => "EU",
             "city_name" => "Moscow",
          "country_name" => "Russia",
         "country_code2" => "RU",
         "country_code3" => "RU",
           "region_name" => "Moscow",
              "location" => {
            "lon" => 37.6184,
            "lat" => 55.7485
        },
           "postal_code" => "101194",
           "region_code" => "MOW",
             "longitude" => 37.6184
    },
    ...

將你的數(shù)據(jù)推給Elasticsearch

經(jīng)過(guò)上述的步驟,你的日志已經(jīng)被細(xì)分成很多字段了,Logstash管道可以將數(shù)據(jù)編入Elasticsearch集群.修改first-pipeline.conf文件,替換如下內(nèi)容:

output {
    elasticsearch {
        hosts => [ "localhost:9200" ]
    }
}

設(shè)置了這個(gè)內(nèi)容后,Logstash會(huì)使用http協(xié)議去連接Elasticsearch.上面的示例假設(shè)Logstash以及Elasticsearch在同一臺(tái)設(shè)備上運(yùn)行,你可以通過(guò)hosts設(shè)置指定一臺(tái)遠(yuǎn)程的Elasticsearch實(shí)例,比如hosts => ["es-machine:9092"].

現(xiàn)在,你的first-pipeline.conf配置文件的輸入,過(guò)濾,輸出都有了正確的設(shè)置,看起來(lái)是這個(gè)樣子的:

input {
    beats {
        port => "5043"
    }
}
 filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
    geoip {
        source => "clientip"
    }
}
output {
    elasticsearch {
        hosts => [ "localhost:9200" ]
    }
}

我還是保存了標(biāo)準(zhǔn)輸出的output,為了檢測(cè)是否有正常的輸出

保存設(shè)置,初始化你的Filebeat.Ctrl-C,刪除注冊(cè)文件,啟動(dòng)Filebeat

sudo ./filebeat -e -c filebeat.yml -d "publish"

測(cè)試你的管♂道

現(xiàn)在Logstash管道已經(jīng)設(shè)置成講數(shù)據(jù)添加至Elasticsearch集群的索引了,你可以查詢Elasticsearch

試著通過(guò)grok插件創(chuàng)建的字段來(lái)查詢Elasticsearch.講命令中的$DATE以YYYY.MM.DD的格式替換成當(dāng)前的日期:

curl -XGET 'localhost:9200/logstash-$DATE/_search?pretty&q=response=200'

比如我得url就是這樣的:curl -XGET 'localhost:9200/logstash-2017.11.28/_search?pretty&q=response=200'

你應(yīng)該會(huì)得到多條反饋:

{
  "took": 50,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 98,
    "max_score": 2.793642,
    "hits": [
      {
        "_index": "logstash-2017.11.09",
        "_type": "doc",
        "_id": "3IzDnl8BW52sR0fx5wdV",
        "_score": 2.793642,
        "_source": {
          "request": "/presentations/logstash-monitorama-2013/images/frontend-response-codes.png",
          "agent": """"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"""",
          "geoip": {
            "timezone": "Europe/Moscow",
            "ip": "83.149.9.216",
            "latitude": 55.7485,
            "continent_code": "EU",
            "city_name": "Moscow",
            "country_name": "Russia",
            "country_code2": "RU",
            "country_code3": "RU",
            "region_name": "Moscow",
            "location": {
              "lon": 37.6184,
              "lat": 55.7485
            },
            "postal_code": "101194",
            "region_code": "MOW",
            "longitude": 37.6184
          },
          "offset": 2932,
          "auth": "-",
          "ident": "-",
          "verb": "GET",
          "prospector": {
            "type": "log"
          },
          "source": "/path/to/file/logstash-tutorial.log",
          "message": """83.149.9.216 - - [04/Jan/2015:05:13:45 +0000] "GET /presentations/logstash-monitorama-2013/images/frontend-response-codes.png HTTP/1.1" 200 52878 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"""",
          "tags": [
            "beats_input_codec_plain_applied"
          ],
          "referrer": """"http://semicomplete.com/presentations/logstash-monitorama-2013/"""",
          "@timestamp": "2017-11-09T03:11:35.304Z",
          "response": "200",
          "bytes": "52878",
          "clientip": "83.149.9.216",
          "@version": "1",
          "beat": {
            "name": "My-MacBook-Pro.local",
            "hostname": "My-MacBook-Pro.local",
            "version": "6.0.0"
          },
          "host": "My-MacBook-Pro.local",
          "httpversion": "1.1",
          "timestamp": "04/Jan/2015:05:13:45 +0000"
        }
      },
    ...

現(xiàn)在試一下用ip獲取的地理位置來(lái)搜索.同樣是替換$DATE以YYYY.MM.DD的格式為當(dāng)前的時(shí)間:

curl -XGET 'localhost:9200/logstash-$DATE/_search?pretty&q=geoip.city_name=Buffalo'

只有幾條數(shù)據(jù)來(lái)自Buffalo,所以查詢結(jié)果日下:

{
  "took": 9,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 2,
    "max_score": 2.6390574,
    "hits": [
      {
        "_index": "logstash-2017.11.09",
        "_type": "doc",
        "_id": "L4zDnl8BW52sR0fx5whY",
        "_score": 2.6390574,
        "_source": {
          "request": "/blog/geekery/disabling-battery-in-ubuntu-vms.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+semicomplete%2Fmain+%28semicomplete.com+-+Jordan+Sissel%29",
          "agent": """"Tiny Tiny RSS/1.11 (http://tt-rss.org/)"""",
          "geoip": {
            "timezone": "America/New_York",
            "ip": "198.46.149.143",
            "latitude": 42.8864,
            "continent_code": "NA",
            "city_name": "Buffalo",
            "country_name": "United States",
            "country_code2": "US",
            "dma_code": 514,
            "country_code3": "US",
            "region_name": "New York",
            "location": {
              "lon": -78.8781,
              "lat": 42.8864
            },
            "postal_code": "14202",
            "region_code": "NY",
            "longitude": -78.8781
          },
          "offset": 22795,
          "auth": "-",
          "ident": "-",
          "verb": "GET",
          "prospector": {
            "type": "log"
          },
          "source": "/path/to/file/logstash-tutorial.log",
          "message": """198.46.149.143 - - [04/Jan/2015:05:29:13 +0000] "GET /blog/geekery/disabling-battery-in-ubuntu-vms.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+semicomplete%2Fmain+%28semicomplete.com+-+Jordan+Sissel%29 HTTP/1.1" 200 9316 "-" "Tiny Tiny RSS/1.11 (http://tt-rss.org/)"""",
          "tags": [
            "beats_input_codec_plain_applied"
          ],
          "referrer": """"-"""",
          "@timestamp": "2017-11-09T03:11:35.321Z",
          "response": "200",
          "bytes": "9316",
          "clientip": "198.46.149.143",
          "@version": "1",
          "beat": {
            "name": "My-MacBook-Pro.local",
            "hostname": "My-MacBook-Pro.local",
            "version": "6.0.0"
          },
          "host": "My-MacBook-Pro.local",
          "httpversion": "1.1",
          "timestamp": "04/Jan/2015:05:29:13 +0000"
        }
      },
     ...

如果你在使用kibana視覺(jué)化你的數(shù)據(jù),同樣可以在kibana上研究FIlebeat數(shù)據(jù)

kibana

至此,給自己鼓個(gè)掌吧,你已經(jīng)成功的創(chuàng)建了一個(gè)將apache的網(wǎng)絡(luò)日志作為輸入,經(jīng)過(guò)插件解析成指定的字段,并且將其寫(xiě)入了Elasticsearch群.接下來(lái),你還會(huì)學(xué)習(xí)到如何創(chuàng)建一個(gè)有多輸入以及輸出組件的管道.

也給自己鼓個(gè)掌..