所謂ELKK 是指 Elasticsearch, Logstash, Kibana, Kafka 這4個開源日志分析，收集, 分析展現(xiàn)和傳輸工具捞镰。

ELKK

由于 Logstash 是基于 JRuby 來實現(xiàn)的狮含，所以為避免對服務器的資源占用，現(xiàn)在大多推薦用 FileBeat 之類的 beats 來收集 metrics

Beats + ELK

下面我們分別介紹一下這幾個組件曼振，并搭建一個數(shù)據(jù)傳輸管道 Metrics data pipeline:

1. ElasticSearch

基本概念

• Cluster 集群
  存儲索引數(shù)據(jù)的節(jié)點的集合

• Node 節(jié)點
  一個 ElasticSearch 的運行實例, 根據(jù) node 的 master 屬性和 data 屬性不同, 可以分為以下三種類型的節(jié)點

  • 主節(jié)點: node.master=true, node.data=false
  • 數(shù)據(jù)節(jié)點: node.master=false, node.data=true
  • 路由節(jié)點: node.master=false, node.data=false

• Document 文檔
  被索引的信息的基本單位,它可表示一個 Json 文檔

• Mapping 映射
  文檔中字段的定義稱為為映射

• Index 索引
  具有共同特征的數(shù)據(jù)集, 包含許多個映射

• Type 類型
  對索引的邏輯分區(qū), 一個索引可以有多個類型, 在新的6.0之后的版本去除了這一概念

• Shards & Replicas 分片和副本
  索引文件會分別存儲在一些分片(shards) 中, 并且復制到不同節(jié)點上相應的副本(Replicas)中, 從而保證了 Elasticsearch 的高可用性

安裝

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.tar.gz

tar xvfz elasticsearch-6.2.4.tar.gz

cd elasticsearch-6.2.4

sed -i 's/#network.host: 192.168.0.1/network.host: 10.224.77.184/g'  ./config/elasticsearch.yml 

新版本的 ElasticSearch 不允許由 root 用戶啟動几迄，并且對文件句柄打開個數(shù)有所要求，所以要做些改動, 放開文件句柄數(shù)量限制冰评，且不能以 root 用戶啟動

ulimit -n 262144
sysctl -w vm.max_map_count=262144

groupadd elsearch

useradd elsearch -g elsearch -p elsearch

chown -R elsearch:elsearch /opt/elasticsearch
su elsearch 
cd /opt/elasticsearch
./bin/elasticsearch -d

啟動后可以直接訪問http://localhost:9200/來看是否正常服務以及角色映胁、集群名等信息：

{
    name: "node-client-001",
    cluster_name: "jiafu-es",
    cluster_uuid: "WzUQifOuRg28zlTfS33Rjg",
    version: 
    {
        number: "6.2.4",
        build_hash: "ccec39f",
        build_date: "2018-04-12T20:37:28.497551Z",
        build_snapshot: false,
        lucene_version: "7.2.1",
        minimum_wire_compatibility_version: "5.6.0",
        minimum_index_compatibility_version: "5.0.0"
    },
    tagline: "You Know, for Search"
}

以后臺方式啟動

./bin/elasticsearch -d

curl 'http://10.224.77.184:9200/?pretty'

如果發(fā)起一個查詢

curl -X GET "http://10.224.76.179:9200/_search" -H 'Content-Type: application/json' -d'
{
    "query": {
        "query_string" : {
            "default_field" : "content",
            "query" : "this AND that OR thus"
        }
    }
}
'

使用

ElasticSearch 提供了易于使用的 REST API

1. 查詢節(jié)點

$ curl -XGET 'http://10.224.77.184:9200/_cat/nodes?v'
ip            heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
10.224.76.179           32          94  12    0.47    0.26     0.23 mdi       *      czVTNso

2. 查詢集群健康狀態(tài)

$ curl -XGET 'http://10.224.77.184:9200/_cluster/health?pretty=true'
{
  "cluster_name" : "elasticsearch",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 51,
  "active_shards" : 51,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 51,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 50.0
}

3. 創(chuàng)建索引

curl -XPUT 'http://10.224.77.184:9200/booklib/?pretty’

4. 創(chuàng)建索引, 類型及文檔

# curl -XPUT 'http://10.224.77.184:9200/booklib/walter/1?pretty' -d '{ "title": "posa1"}'
{
  "_index" : "booklib",
  "_type" : "walter",
  "_id" : "1",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "created" : true
}

5. 獲取文檔

curl -XGET 'http://10.224.77.184:9200/booklib/walter/1?pretty'

6. 查詢文檔

curl -XPOST 'http://10.224.77.184:9200/booklib/_search' -d '{ "query": {"match_all": {}}, "size": 3}'
{"took":13,"timed_out":false,"_shards”:
{"total":5,"successful":5,"skipped":0,"failed":0},
"hits":{"total":1,"max_score":1.0,
"hits":[{"_index":"booklib","_type":"walter","_id":"1","_score":1.0,"_source":{ "title": "posa1"}}]}}

7 .刪除文檔

curl -XDELETE 'http://10.224.77.184:9200/booklib/walter/1?pretty'

實例

{
  "timestamp": "2018-02-20T01:36:30.255Z",    
  "version": "1.0",
  "ip": "10.224.11.22",
  "host": "potato01.fanyamin.com",
  "service": "potato",
  "feature": "create_potato",
  "pool": "china_east",
  "properties": {
        "url": "http://potato01/api/v1.0/meetings",
        "method": "post",
        "potatoId": "12345",
        "responseCode": 200,
        "responseTimeInMs": 1000,
        "isSuccess": true,
        "errorCode": 0,
        "errorReason": ""
      
  }
}

2. LogStash

Logstash是一個開源的服務器端數(shù)據(jù)處理管道工具，它可以同時從多個源中提取數(shù)據(jù)甲雅，對其進行轉(zhuǎn)換解孙，然后將其發(fā)送到您最喜歡的目的地, 最常見的就是 ElasticSearch, 在實際應用中, 為了避免在流量高峰對 ElasticSearch 的并發(fā)請求過多, 常用 Kafka 來中轉(zhuǎn)一下.

Logstash 的優(yōu)點就是插件豐富, 可以滿足你的大多數(shù)需求, 插件分為三類

1. Input: 輸入插件
2. Output: 輸出插件
3. Filter: 過濾器插件

• 安裝

wget https://artifacts.elastic.co/downloads/logstash/logstash-7.3.1.tar.gz
tar xvfz logstash-7.3.1.tar.gz
mv logstash-7.3.1 /opt/logstash
cd /opt/logstash/
vi logstash.conf
./bin/logstash -f logstash.conf &

在實際應用中，logstash 可以部署在應用服務器上抛人，這時應該盡量少做耗費CPU 的過濾解析弛姜，將日志收集并發(fā)送到 Kafka 上就好, 配置文件 logstash.conf 示例如下

• 1）在應用服務器上安裝的 logstash， 將日志文件中的日志數(shù)據(jù)收集并發(fā)送到 kafka 上

input {

  file {
    type => "potato-web-service"
    path => [ "/opt/potato/logs/potato-web.metrics*.log" ]
    add_field => [ "component", "potato-web" ]
    add_field => [ "eventtype", "metrics" ]
    start_position => "beginning"
  }

   file {
    type => "potato-scheduler-service"
    path => [ "/opt/potato/logs/potato-scheduler.metrics*.log" ]
    add_field => [ "component", "potato-scheduler" ]
    add_field => [ "eventtype", "metrics" ]
    start_position => "beginning"
  }

  file {
    type => "potato-service"
    path => [ "/opt/potato/logs//potato.metrics*.log" ]
    add_field => [ "component", "potato-service" ]
    add_field => [ "eventtype", "metrics" ]
    start_position => "beginning"
  }

}
filter {

       if [eventtype] == 'metrics'  {
        json {
            source => "message"
            target => "message"
        }
    }

}

output {
     if [eventtype] == 'metrics' {
          kafka {
            topic_id => "metrics_%{component}"
            client_id => "potato"
            retry_backoff_ms => 60000

            bootstrap_servers => "10.224.77.178:9092"

            codec => "json"
            reconnect_backoff_ms => 60000
         }
    }
    stdout {
    }
}

• 2）在日志分析服務器安裝的 logstash , 將日志從 Kafka 中取出妖枚，發(fā)送到 ElaticSearch中

input {
  kafka {
    bootstrap_servers => "10.224.77.178:9092"
    topics => ["metrics_potato-scheduler" ,"metrics_potato-service","metrics_potato-web"]
    auto_offset_reset => "earliest"
    enable_auto_commit => "false"
    codec => "json"
    consumer_threads => 1
    max_poll_records => "10"

    }
}

filter {
    json {
        source => "message"
        target => "message"
        skip_on_invalid_json => true
    }
}

output {


        elasticsearch {

          hosts =>"10.224.77.176:9200"

          codec => "json"
        }

        stdout {}
}

3. Kafka

Kafka 是現(xiàn)在最為流行的消息隊列系統(tǒng), 它稱自己為分布的數(shù)據(jù)流平臺.

一個數(shù)據(jù)流平臺具備三個關(guān)鍵能力:

1. 發(fā)布和訂閱數(shù)據(jù)流記錄廷臼，類似于消息隊列或企業(yè)消息傳遞系統(tǒng)。
2. 以容錯的持久方式存儲數(shù)據(jù)流記錄绝页。
3. 當記錄產(chǎn)生時處理數(shù)據(jù)流記錄荠商。

Kafka 一般用于兩大類應用程序

1. 構(gòu)建可在系統(tǒng)或應用程序之間, 可靠地獲取數(shù)據(jù)的實時數(shù)據(jù)流管道
2. 構(gòu)建可實時轉(zhuǎn)換或處理數(shù)據(jù)的實時數(shù)據(jù)流應用程序

Kafka 的核心概念

• Kafka 運行在一個或多個可跨多個數(shù)據(jù)中心的服務器所組成的集群上
• Kafka 集群分類存儲數(shù)據(jù)流的記錄, 這個分類稱為主題 topic.
• 每條記錄包含一個 key, 一個 value, 和一個 timestamp

Kafka 應用實例

1. 先安裝 Zookeeper

• 1）下載安裝包

wget http://apache.org/dist/zookeeper/stable/apache-zookeeper-3.5.5-bin.tar.gz

• 2）解壓

tar xvfz [apache-zookeeper-3.5.5-bin.tar.gz](http://apache.org/dist/zookeeper/stable/apache-zookeeper-3.5.5-bin.tar.gz)

• 3）創(chuàng)建相關(guān)目錄和配置

mkdir -p /opt

mv apache-zookeeper-3.5.5-bin /opt/zookeeper
cd /opt/zookeeper/conf
cp zoo_sample.cfg zoo.cfg
mkdir -p /opt/data/zookeeper
vi zoo.cfg

在 zoo.cfg 文件將數(shù)據(jù)目錄修改為 dataDir=/opt/zookeeper/data

• 4）啟動

cd zookeeper/bin/
./zkServer.sh start

安裝 Kafka

1. 下載

wget http://apache.claz.org/kafka/2.3.0/kafka_2.12-2.3.0.tgz

2. 解壓

tar xvfz kafka_2.12-2.3.0.tgz

3. 創(chuàng)建相關(guān)目錄和修改配置

mv kafka_2.12-2.3.0 /opt/kafka
cd /opt/kafka/
vi config/server.properties

## 修改配置文件如下

log.dirs=/opt/logs/kafka-logs
listeners = PLAINTEXT://10.224.77.178:9092

mkdir -p /opt/logs/kafka-logs

4. 啟動 Kafka

./bin/kafka-server-start.sh -daemon config/server.properties

5. 創(chuàng)建一個測試主題

./bin/kafka-topics.sh -create -zookeeper localhost:2181 --replication-factor 1 -partitions 1 -topic mlogs

6. 獲取這個測試主題

./bin/kafka-topics.sh -list -zookeeper localhost:2181

./kafka-topics.sh --bootstrap-server 10.224.77.178:9092 --list

4. Kibana

Kibana 是一個基于 Elastic Search 的可視化工具, 通過調(diào)用 Elastic Search 的 API , 可以方便地搜索存儲在 Elastic Search 的數(shù)據(jù), 并可以繪制各種圖表和儀表盤.

Kibana dashboard 的構(gòu)建主要可以分為以下三部分

• Usage 用量
• Performance 性能
• Error 錯誤

打開 http://10.224.76.176:5601

ELK的容器化

也可以用 docker 來安裝啟動一個 ELK 鏡像
鏡像地址 https://hub.docker.com/r/sebp/elk/
詳細文檔見 http://elk-docker.readthedocs.io/
源文件見 https://github.com/spujadas/elk-docker

看看它的 dockerfile , 這里包括了詳細的安裝配置過程, 我加了點中文注釋

# Dockerfile for ELK stack
# Elasticsearch, Logstash, Kibana 6.3.1

# Build with:
# docker build -t <repo-user>/elk .

# Run with: 
# docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk <repo-user>/elk
# phusion/baseimage 是基于Ubuntu 針對 docker 做了一些修改, 并修復了 PID 1 僵尸進程問題
FROM phusion/baseimage
MAINTAINER Sebastien Pujadas http://pujadas.net
ENV REFRESHED_AT 2017-02-28


###############################################################################
#                                INSTALLATION
###############################################################################
# 先安裝一些依賴工具, cURL, JDK 不用說了, gosu(高手) 一門新的JVM腳本語言, tzdata 是修改時區(qū)的命令
### install prerequisites (cURL, gosu, JDK, tzdata)

ENV GOSU_VERSION 1.10

ARG DEBIAN_FRONTEND=noninteractive
RUN set -x \
 && apt-get update -qq \
 && apt-get install -qqy --no-install-recommends ca-certificates curl \
 && rm -rf /var/lib/apt/lists/* \
 && curl -L -o /usr/local/bin/gosu "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$(dpkg --print-architecture)" \
 && curl -L -o /usr/local/bin/gosu.asc "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$(dpkg --print-architecture).asc" \
 && export GNUPGHOME="$(mktemp -d)" \
 && gpg --keyserver hkp://ha.pool.sks-keyservers.net:80 --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4 \
 && gpg --batch --verify /usr/local/bin/gosu.asc /usr/local/bin/gosu \
 && rm -r "$GNUPGHOME" /usr/local/bin/gosu.asc \
 && chmod +x /usr/local/bin/gosu \
 && gosu nobody true \
 && apt-get update -qq \
 && apt-get install -qqy openjdk-8-jdk tzdata \
 && apt-get clean \
 && set +x

# 首先都是一樣, 先從官網(wǎng)上下載壓縮包后解壓縮, 添加用戶及組, 創(chuàng)建相應目錄, 并設置權(quán)限

ENV ELK_VERSION 6.3.1

### install Elasticsearch 

ENV ES_VERSION ${ELK_VERSION}
ENV ES_HOME /opt/elasticsearch
ENV ES_PACKAGE elasticsearch-${ES_VERSION}.tar.gz
ENV ES_GID 991
ENV ES_UID 991
ENV ES_PATH_CONF /etc/elasticsearch
ENV ES_PATH_BACKUP /var/backups

RUN mkdir ${ES_HOME} \
 && curl -O https://artifacts.elastic.co/downloads/elasticsearch/${ES_PACKAGE} \
 && tar xzf ${ES_PACKAGE} -C ${ES_HOME} --strip-components=1 \
 && rm -f ${ES_PACKAGE} \
 && groupadd -r elasticsearch -g ${ES_GID} \
 && useradd -r -s /usr/sbin/nologin -M -c "Elasticsearch service user" -u ${ES_UID} -g elasticsearch elasticsearch \
 && mkdir -p /var/log/elasticsearch ${ES_PATH_CONF} ${ES_PATH_CONF}/scripts /var/lib/elasticsearch ${ES_PATH_BACKUP} \
 && chown -R elasticsearch:elasticsearch ${ES_HOME} /var/log/elasticsearch /var/lib/elasticsearch ${ES_PATH_CONF} ${ES_PATH_BACKUP}
# 自啟動
ADD ./elasticsearch-init /etc/init.d/elasticsearch
RUN sed -i -e 's#^ES_HOME=$#ES_HOME='$ES_HOME'#' /etc/init.d/elasticsearch \
 && chmod +x /etc/init.d/elasticsearch


### install Logstash

ENV LOGSTASH_VERSION ${ELK_VERSION}
ENV LOGSTASH_HOME /opt/logstash
ENV LOGSTASH_PACKAGE logstash-${LOGSTASH_VERSION}.tar.gz
ENV LOGSTASH_GID 992
ENV LOGSTASH_UID 992
ENV LOGSTASH_PATH_CONF /etc/logstash
ENV LOGSTASH_PATH_SETTINGS ${LOGSTASH_HOME}/config

RUN mkdir ${LOGSTASH_HOME} \
 && curl -O https://artifacts.elastic.co/downloads/logstash/${LOGSTASH_PACKAGE} \
 && tar xzf ${LOGSTASH_PACKAGE} -C ${LOGSTASH_HOME} --strip-components=1 \
 && rm -f ${LOGSTASH_PACKAGE} \
 && groupadd -r logstash -g ${LOGSTASH_GID} \
 && useradd -r -s /usr/sbin/nologin -d ${LOGSTASH_HOME} -c "Logstash service user" -u ${LOGSTASH_UID} -g logstash logstash \
 && mkdir -p /var/log/logstash ${LOGSTASH_PATH_CONF}/conf.d \
 && chown -R logstash:logstash ${LOGSTASH_HOME} /var/log/logstash ${LOGSTASH_PATH_CONF}

ADD ./logstash-init /etc/init.d/logstash
RUN sed -i -e 's#^LS_HOME=$#LS_HOME='$LOGSTASH_HOME'#' /etc/init.d/logstash \
 && chmod +x /etc/init.d/logstash


### install Kibana

ENV KIBANA_VERSION ${ELK_VERSION}
ENV KIBANA_HOME /opt/kibana
ENV KIBANA_PACKAGE kibana-${KIBANA_VERSION}-linux-x86_64.tar.gz
ENV KIBANA_GID 993
ENV KIBANA_UID 993

RUN mkdir ${KIBANA_HOME} \
 && curl -O https://artifacts.elastic.co/downloads/kibana/${KIBANA_PACKAGE} \
 && tar xzf ${KIBANA_PACKAGE} -C ${KIBANA_HOME} --strip-components=1 \
 && rm -f ${KIBANA_PACKAGE} \
 && groupadd -r kibana -g ${KIBANA_GID} \
 && useradd -r -s /usr/sbin/nologin -d ${KIBANA_HOME} -c "Kibana service user" -u ${KIBANA_UID} -g kibana kibana \
 && mkdir -p /var/log/kibana \
 && chown -R kibana:kibana ${KIBANA_HOME} /var/log/kibana

ADD ./kibana-init /etc/init.d/kibana
RUN sed -i -e 's#^KIBANA_HOME=$#KIBANA_HOME='$KIBANA_HOME'#' /etc/init.d/kibana \
 && chmod +x /etc/init.d/kibana


###############################################################################
#                               CONFIGURATION
###############################################################################
# 配置比較麻煩, 預先在`  
### configure Elasticsearch

ADD ./elasticsearch.yml ${ES_PATH_CONF}/elasticsearch.yml
ADD ./elasticsearch-default /etc/default/elasticsearch
RUN cp ${ES_HOME}/config/log4j2.properties ${ES_HOME}/config/jvm.options \
    ${ES_PATH_CONF} \
 && chown -R elasticsearch:elasticsearch ${ES_PATH_CONF} \
 && chmod -R +r ${ES_PATH_CONF}

### configure Logstash

# certs/keys for Beats and Lumberjack input
RUN mkdir -p /etc/pki/tls/certs && mkdir /etc/pki/tls/private
ADD ./logstash-beats.crt /etc/pki/tls/certs/logstash-beats.crt
ADD ./logstash-beats.key /etc/pki/tls/private/logstash-beats.key

# filters
ADD ./02-beats-input.conf ${LOGSTASH_PATH_CONF}/conf.d/02-beats-input.conf
ADD ./10-syslog.conf ${LOGSTASH_PATH_CONF}/conf.d/10-syslog.conf
ADD ./11-nginx.conf ${LOGSTASH_PATH_CONF}/conf.d/11-nginx.conf
ADD ./30-output.conf ${LOGSTASH_PATH_CONF}/conf.d/30-output.conf

# patterns
ADD ./nginx.pattern ${LOGSTASH_HOME}/patterns/nginx
RUN chown -R logstash:logstash ${LOGSTASH_HOME}/patterns

# Fix permissions
RUN chmod -R +r ${LOGSTASH_PATH_CONF}

### configure logrotate

ADD ./elasticsearch-logrotate /etc/logrotate.d/elasticsearch
ADD ./logstash-logrotate /etc/logrotate.d/logstash
ADD ./kibana-logrotate /etc/logrotate.d/kibana
RUN chmod 644 /etc/logrotate.d/elasticsearch \
 && chmod 644 /etc/logrotate.d/logstash \
 && chmod 644 /etc/logrotate.d/kibana


### configure Kibana

ADD ./kibana.yml ${KIBANA_HOME}/config/kibana.yml


###############################################################################
#                                   START
###############################################################################

ADD ./start.sh /usr/local/bin/start.sh
RUN chmod +x /usr/local/bin/start.sh

EXPOSE 5601 9200 9300 5044
VOLUME /var/lib/elasticsearch

CMD [ "/usr/local/bin/start.sh" ]

用法

sudo docker pull sebp/elk
sudo docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk sebp/elk

啟動之后, 可以看到

也可以用 docker-compose 來啟動

$vi docker-compose.yml 

elk:
  image: sebp/elk
  ports:
    - "5601:5601"
    - "9200:9200"
    - "5044:5044"

參考資料

• ElasticSearch 官方文檔
• ELK document
• Docker PID1 僵尸進程問題