準(zhǔn)備

環(huán)境規(guī)劃

有了上篇文章中的ElasticSearch集群，我們接下來(lái)就可以準(zhǔn)備日志數(shù)據(jù)采集的工作澎剥。業(yè)界推薦的最流行的有兩種：LogStash锡溯，F(xiàn)luentd。此文中哑姚，我們采用Kubernetes官方采用的Fluent體系中的組件：Fluent Bit 和 Fluentd.
所有的組件有：

部署架構(gòu)

image.png

此圖中，僅作描述Flunt-Bit 和Fluentd的采集集成魄懂，ES集群的部署架構(gòu)沿侈，和Kubernetes微服務(wù)整體的集群架構(gòu)，不在此圖詳述市栗，有興趣缀拭，可參考本人的其它文章。

日志集中中轉(zhuǎn)代理中心

當(dāng)服務(wù)節(jié)點(diǎn)比較多的時(shí)候填帽，推薦使用集中中轉(zhuǎn)代理中心進(jìn)行初步的匯集轉(zhuǎn)送蛛淋，如果節(jié)點(diǎn)數(shù)沒那么多，可以直接由Node發(fā)送到ES或者S3篡腌。

Docker鏡像準(zhǔn)備

由于我們的日志集中中轉(zhuǎn)代理中心需要將日志采集點(diǎn)采集過來(lái)的日志轉(zhuǎn)發(fā)到兩個(gè)地方：ElasticSearch和Amazon S3褐荷，所以需要對(duì)我們創(chuàng)建的Fluentd的鏡像進(jìn)行再次處理。
鑒于Kubernetes官網(wǎng)提供的有EFK的現(xiàn)成方案嘹悼，我們就在這個(gè)方案基礎(chǔ)上進(jìn)行調(diào)整诚卸，使其滿足我們自身的需求葵第。

獲取Kubernetes官網(wǎng)的Fluentd的配置信息

[centos@master1 efk]$ wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/fluentd-elasticsearch/fluentd-es-ds.yaml
--2018-06-08 17:46:35--  https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/fluentd-elasticsearch/fluentd-es-ds.yaml
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.72.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.72.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2774 (2.7K) [text/plain]
Saving to: ‘fluentd-es-ds.yaml’

fluentd-es-ds.yaml   100%[===================>]   2.71K  --.-KB/s    in 0s

2018-06-08 17:46:36 (8.56 MB/s) - ‘fluentd-es-ds.yaml’ saved [2774/2774]

[centos@master1 efk]$

打開該文件fluentd-es-ds.yaml，獲得官方使用的Docker鏡像合溺。

......
      - name: fluentd-es
        image: k8s.gcr.io/fluentd-elasticsearch:v2.0.4
        env:
......

在本地將該鏡像pull下來(lái)卒密，tag到本地Repo，并推送到私有Repo棠赛，以便方便獲取哮奇。此處需要科學(xué)上網(wǎng)，并將docker的代理加上睛约。

[centos@master1 efk]$ docker pull k8s.gcr.io/fluentd-elasticsearch:v2.0.4
Trying to pull repository k8s.gcr.io/fluentd-elasticsearch ... 
v2.0.4: Pulling from k8s.gcr.io/fluentd-elasticsearch
e7bb522d92ff: Pull complete 
92e6b816bc34: Pull complete 
ffb38dbddc64: Pull complete 
4900a3591877: Pull complete 
812a2bf6252f: Pull complete 
f8d5892f0b74: Pull complete 
e6736dda51ce: Pull complete 
Digest: sha256:b8c94527b489fb61d3d81ce5ad7f3ddbb7be71e9620a3a36e2bede2f2e487d73
Status: Downloaded newer image for k8s.gcr.io/fluentd-elasticsearch:v2.0.4
[centos@master1 efk]$ docker tag k8s.gcr.io/fluentd-elasticsearch:v2.0.4 hub.***.***/google_containers/fluentd-elasticsearch:v2.1.0
[centos@master1 efk]$ docker push hub.***.***/google_containers/fluentd-elasticsearch:v2.1.0

增加Amazon S3的支持

基于Kubernetes鏡像添加對(duì)S3的支持鼎俘，新建Dockerfile，內(nèi)容如下：

FROM hub.***.***/google_containers/fluentd-elasticsearch:v2.1.0
MAINTAINER X.J CHEN
RUN \
    apt-get update -y && apt-get install ruby-dev -y && \
    gem install fluent-plugin-s3 && \
    apt-get clean

編譯該鏡像辩涝，并上傳到我們私庫(kù)贸伐。

[centos@master1 efk] docker build -t hub.***.***/google_containers/fluentd-s3:v2.1.0 .
[centos@master1 efk] docker push hub.***.***/google_containers/fluentd-s3:v2.1.0

至此，我們的鏡像已準(zhǔn)備完畢怔揩。

Server Yaml文件準(zhǔn)備

參考上小節(jié)獲取的fluentd-es-ds.yaml捉邢，創(chuàng)建我們的Fluentd Server Yaml文件fluentd-server-s3.yaml，具體內(nèi)容如：
調(diào)整內(nèi)容主要有：

• 更改ServiceAccount及相關(guān)的Rule商膊。
• 更改鏡像為我們新Build的鏡像伏伐。
• 刪除var log等讀取本地日志參數(shù)的路徑。
• Replicas設(shè)置為5晕拆，這個(gè)主要是由于我們的環(huán)境生成日志量太大藐翎，該值可以依據(jù)自身的實(shí)際情況來(lái)決定Pod的個(gè)數(shù)，推薦最少2個(gè)实幕。
• 增加Fluent Server的Service吝镣，以供Kubernetes集群內(nèi)日志采集節(jié)點(diǎn)訪問上傳自己的日志。

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd-server
  namespace: kube-system
  labels:
    k8s-app: fluentd-server
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: fluentd-server
  labels:
    k8s-app: fluentd-server
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups:
  - ""
  resources:
  - "namespaces"
  - "pods"
  verbs:
  - "get"
  - "watch"
  - "list"
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: fluentd-server
  labels:
    k8s-app: fluentd-server
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
subjects:
- kind: ServiceAccount
  name: fluentd-server
  namespace: kube-system
  apiGroup: ""
roleRef:
  kind: ClusterRole
  name: fluentd-server
  apiGroup: ""

---
apiVersion: v1
kind: Service
metadata:
  name: fluentd-server
  namespace: kube-system
  labels:
    k8s-app: fluentd-server
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "Flunetd"
spec:
  ports:
  - port: 24224
    protocol: TCP
    targetPort: server
  selector:
    k8s-app: fluentd-server
    
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fluentd-server-v2.0.4
  namespace: kube-system
  labels:
    k8s-app: fluentd-server
    version: v2.0.4
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    matchLabels:
      k8s-app: fluentd-server
      version: v2.0.4
  replicas: 5
  template:
    metadata:
      labels:
        k8s-app: fluentd-server
        kubernetes.io/cluster-service: "true"
        version: v2.0.4
    spec:
      serviceAccountName: fluentd-server
      containers:
      - name: fluentd-server
        #image: k8s.gcr.io/fluentd-elasticsearch:v2.0.4
        image: hub.***.***/google_containers/fluentd-s3:v2.1.0
        imagePullPolicy: Always
        env:
        - name: FLUENTD_ARGS
          value: --no-supervisor -q
        resources:
          limits:
            memory: 1024Mi
          requests:
            cpu: 1000m
            memory: 200Mi
        volumeMounts:
        - name: config-volume
          mountPath: /etc/fluent/config.d
        ports:
        - containerPort: 24224
          name: server
          protocol: TCP
      terminationGracePeriodSeconds: 160
      volumes:
      - name: config-volume
        configMap:
          name: fluentd-server-config-v0.1.4
      imagePullSecrets:
      - name: kube-sec

從Kubernetes官方獲取Fluentd的配置文件昆庇。

[centos@master1 efk]  wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/fluentd-elasticsearch/fluentd-es-configmap.yaml

并在此基礎(chǔ)上末贾，創(chuàng)建我們自己的配置文件fluentd-server-s3-configmap.yaml，具體內(nèi)容如下：
主要調(diào)整內(nèi)容：

• 刪除日志采集部分片段凰锡。
• 增加Forward Input片段，增加Server的Host和監(jiān)聽信息圈暗。
• 調(diào)整Output片段掂为。
• 請(qǐng)注意，ES和S3兩個(gè)Output我們均有采用Buffer员串，均為文件Buffer勇哗。由于我們的Deployment中移除了Host 的Var log，此處Buffer緩存將會(huì)存在Docker 容器中寸齐，容器掛掉后欲诺，緩存的數(shù)據(jù)將丟失抄谐，請(qǐng)依據(jù)自身的實(shí)際情況調(diào)整（如果對(duì)數(shù)據(jù)特別敏感，甚至可以用StatetefulSet來(lái)創(chuàng)建PV）扰法。

kind: ConfigMap
apiVersion: v1
metadata:
  name: fluentd-server-config-v0.1.4
  namespace: kube-system
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
data:
  system.conf: |-
    <system>
      root_dir /tmp/fluentd-buffers/
    </system>

  system.input.conf: |-
    # Listen to incoming data over SSL
    <source>
      @type forward
      port 24224
      bind 0.0.0.0
    </source>

  output.conf: |-
    # Enriches records with Kubernetes metadata
    <filter kubernetes.**>
      @type kubernetes_metadata
    </filter>
    # Store Data in Elasticsearch and S3
    <match *.**>
      @type copy
      <store>
        @id elasticsearch
        @type elasticsearch
        @log_level info
        host elasticsearch
        port 9200
        #include_tag_key true
        #tag_key @log_name
        logstash_format true
        request_timeout    30s
        slow_flush_log_threshold 30s
        <buffer>
          @type file
          path /var/log/fluentd-buffers/server.buffer
          flush_mode interval
          #retry_type exponential_backoff
          flush_thread_count 12 # 可以根據(jù)實(shí)際需要也需兼顧下ES的處理能力進(jìn)行調(diào)整
          flush_interval 8s  # 可以根據(jù)實(shí)際需要進(jìn)行調(diào)整
          retry_max_interval 30
          chunk_limit_size 32M # 可以根據(jù)ES實(shí)際處理能力進(jìn)行調(diào)整蛹含，務(wù)必<100MB 
          #queue_limit_length 64 #8
          total_limit_size 20G
          retry_wait 10s
        </buffer>
      </store>
      <store>
        @id s3
        @type s3
        @log_level info
        #include_tag_key true
        aws_key_id *********  # 請(qǐng)?zhí)顚懽陨淼膋ey id和sec key
        aws_sec_key **********
        s3_bucket ******** #請(qǐng)?zhí)顚懽陨淼腂ucket
        s3_region cn-north-1
        s3_object_key_format "%{path}dt=%{time_slice}_%{index}.%{file_extension}"
        #store_as json
        path hive/    
        time_slice_format %Y%m%d/%Y%m%d%H
        <buffer>
          @type file
          path /var/log/fluentd-buffers/s3.buffer
          timekey 3600 # 1 hour partition
          timekey_wait 10m
          timekey_use_utc true # use utc
          chunk_limit_size 256m
        </buffer>
      </store>
    </match>

拉起Aggregation Server

創(chuàng)建Server的配置聂儒。

[centos@master1 efk]$ kubectl create -f fluentd-server-s3-configmap.yaml

創(chuàng)建Server的Deployment脏里。

[centos@master1 efk]$ kubectl create -f fluentd-server-s3.yaml 

檢查Server的啟動(dòng)狀態(tài)：

[centos@master1 efk]$ kubectl get service -n kube-system -o wide | grep fluentd-server
fluentd-server            ClusterIP   10.104.52.1      <none>        24224/TCP        4d        k8s-app=fluentd-server
[centos@master1 efk]$ 
[centos@master1 efk]$ kubectl get pods -n kube-system -o wide | grep fluentd-server
fluentd-server-v2.0.4-855db7cfc5-4wn47   1/1       Running            0          2h        10.244.29.20     minion6
fluentd-server-v2.0.4-855db7cfc5-pfmvd   1/1       Running            0          2h        10.244.3.211     minion17
fluentd-server-v2.0.4-855db7cfc5-rjqxl   1/1       Running            0          2h        10.244.13.47     minion19
fluentd-server-v2.0.4-855db7cfc5-shjfm   1/1       Running            0          2h        10.244.23.141    minion12
fluentd-server-v2.0.4-855db7cfc5-w7m5f   1/1       Running            0          2h        10.244.30.233    minion5
[centos@master1 efk]$ 

也可以查看下日志：

image.png

日志Agent

日志的Agent，我們使用的是Fluent Bit制妄，原因還是那句：性能相較Fluentd稍好祠锣，消耗資源要少一些酷窥。但是鑒于Fluent Bit 的穩(wěn)定性，有部分節(jié)點(diǎn)無(wú)法正常運(yùn)行（有些是日志無(wú)法解析造成的伴网，也有其它原因蓬推，由于太久沒接觸過C和C++，有時(shí)只能等待官方補(bǔ)对杼凇）沸伏，也有部分節(jié)點(diǎn)可能會(huì)運(yùn)行一段時(shí)間崩潰的情況。所以對(duì)于日志要求比較高的場(chǎng)景蛋铆，還是推薦使用Fluentd馋评。
常見的Fluent Bit的異常，該異常是由于日志文件Json解析異常直接導(dǎo)致Fluent Bit崩潰刺啦，號(hào)稱在0.13.3版本中解決，問題依舊：

[centos@master1 fluent-bit]$ 
[centos@master1 fluent-bit]$ kubectl logs -f fluent-bit-5kvpl -n kube-system
[2018/06/11 03:11:49] [ info] [engine] started (pid=1)
[2018/06/11 03:11:49] [ info] [filter_kube] https=1 host=kubernetes.default.svc.cluster.local port=443
[2018/06/11 03:11:49] [ info] [filter_kube] local POD info OK
[2018/06/11 03:11:49] [ info] [filter_kube] testing connectivity with API server...
[2018/06/11 03:11:49] [ info] [filter_kube] API server connectivity OK
[2018/06/11 03:11:49] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[engine] caught signal (SIGSEGV)
Fluent-Bit v0.13.2
Copyright (C) Treasure Data

#0  0x7fcc07f6eff1      in  ???() at ???:0
#1  0x55b0b655dede      in  msgpack_sbuffer_write() at lib/msgpack-2.1.3/include/msgpack/sbuffer.h:84
#2  0x55b0b6771ca5      in  msgpack_pack_ext_body() at lib/msgpack-2.1.3/include/msgpack/pack_template.h:890
#3  0x55b0b6771ca5      in  msgpack_pack_object() at lib/msgpack-2.1.3/src/objectc.c:72
#4  0x55b0b655e8c0      in  pack_map_content() at plugins/filter_kubernetes/kubernetes.c:321
#5  0x55b0b655f129      in  cb_kube_filter() at plugins/filter_kubernetes/kubernetes.c:493
#6  0x55b0b64feaea      in  flb_filter_do() at src/flb_filter.c:86
#7  0x55b0b64fc53c      in  flb_input_dbuf_write_end() at include/fluent-bit/flb_input.h:642
#8  0x55b0b64fe09c      in  flb_input_dyntag_append_raw() at src/flb_input.c:894
#9  0x55b0b6522b1d      in  process_content() at plugins/in_tail/tail_file.c:290
#10 0x55b0b6523968      in  flb_tail_file_chunk() at plugins/in_tail/tail_file.c:651
#11 0x55b0b6521357      in  in_tail_collect_static() at plugins/in_tail/tail.c:129
#12 0x55b0b64fe5db      in  flb_input_collector_fd() at src/flb_input.c:995
#13 0x55b0b6505370      in  flb_engine_handle_event() at src/flb_engine.c:296
#14 0x55b0b6505370      in  flb_engine_start() at src/flb_engine.c:515
#15 0x55b0b64a5606      in  main() at src/fluent-bit.c:824
#16 0x7fcc07e662e0      in  ???() at ???:0
#17 0x55b0b64a3a89      in  ???() at ???:0
#18 0xffffffffffffffff  in  ???() at ???:0
[centos@master1 fluent-bit]$ 

我們下面對(duì)于Fluent Bit和Fluentd的使用都將描述玛瘸，以供大家參考蜕青。

Fluent Bit

準(zhǔn)備Fluent Bit Yaml文件

我們參考https://github.com/fluent/fluent-bit-kubernetes-logging來(lái)整理我們自己的Fluent Bit相關(guān)Yaml文件。

[centos@master1 kube-log]$ git clone https://github.com/fluent/fluent-bit-kubernetes-logging.git

Fluent Bit 配置文件

我們還將繼續(xù)使用fluent-bit-configmap.yaml來(lái)作為我們的Fluent Bit的配置文件糊渊，不過要增加Forward組件右核，以使得Fluent Bit能夠正常的將日志轉(zhuǎn)發(fā)至日志集中中轉(zhuǎn)代理中心。

• 增加output-fluentd.conf 片段渺绒，輸出到Fluent Server贺喝。
• 去掉輸出到ES的配置片段。

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: kube-system
  labels:
    k8s-app: fluent-bit
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-fluentd.conf
    # @INCLUDE output-elasticsearch.conf

  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  5

  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc.cluster.local:443
        Merge_Log           On
        K8S-Logging.Parser  On

  output-elasticsearch.conf: |
    [OUTPUT]
        Name            es
        Match           *
        Host            ${FLUENT_ELASTICSEARCH_HOST}
        Port            ${FLUENT_ELASTICSEARCH_PORT}
        Logstash_Format On
        Retry_Limit     False

  output-fluentd.conf: |
    [OUTPUT]
        Name          forward
        Match         *
        Host          ${FLUENTD_SERVER_HOST}
        Port          ${FLUENTD_SERVER_PORT}
        
  parsers.conf: |
    [PARSER]
        Name   apache
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache2
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache_error
        Format regex
        Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$

    [PARSER]
        Name   nginx
        Format regex
        Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On
        # Command      |  Decoder | Field | Optional Action
        # =============|==================|=================
        Decode_Field_As   escaped    log

    [PARSER]
        Name        syslog
        Format      regex
        Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
        Time_Key    time
        Time_Format %b %d %H:%M:%S

Fluent Bit Pod配置

我們將官方的相關(guān)Pod的各種配置（fluent-bit-service-account.yaml宗兼，fluent-bit-role.yaml躏鱼，fluent-bit-role-binding.yaml）整合到一個(gè)文件中（fluent-bit-ds.yaml）以便方便維護(hù)。

• 更改Docker鏡像的Repo到我們的私庫(kù)殷绍。我們將原有的鏡像pull下來(lái)染苛，不做任何更改，tag并push到私庫(kù)主到。
• 增加Fluentd Server的環(huán)境變量茶行。
• 請(qǐng)注意nodeSelector片段躯概，需要將具備日志搜集的節(jié)點(diǎn)加上該標(biāo)簽。推薦一個(gè)節(jié)點(diǎn)一個(gè)節(jié)點(diǎn)加（一兩個(gè)節(jié)點(diǎn)相隔幾分鐘畔师，也可以用腳本來(lái)實(shí)現(xiàn)娶靡，當(dāng)日志量比較大的時(shí)候，強(qiáng)烈推薦這么做茉唉，血淋淋的教訓(xùn)）固蛾，批量加可能會(huì)造成數(shù)據(jù)風(fēng)暴，導(dǎo)致中轉(zhuǎn)中心處理不過來(lái)度陆，也會(huì)導(dǎo)致ES處理不過來(lái)而拒掉Fluentd的鏈接艾凯。

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: fluent-bit-read
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit-read
subjects:
- kind: ServiceAccount
  name: fluent-bit
  namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: kube-system
  labels:
    k8s-app: fluent-bit-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  template:
    metadata:
      labels:
        k8s-app: fluent-bit-logging
        version: v1
        kubernetes.io/cluster-service: "true"
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "2020"
        prometheus.io/path: /api/v1/metrics/prometheus
    spec:
      nodeSelector:
        beta.kubernetes.io/fluentd-ds-ready: "true"
      containers:
      - name: fluent-bit
        image: hub.***.***/google_containers/fluent-bit:0.13.2
        imagePullPolicy: Always
        ports:
          - containerPort: 2020
        env:
        - name: FLUENTD_SERVER_HOST
          value: "fluentd-server"
        - name: FLUENTD_SERVER_PORT
          value: "24224"
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
      terminationGracePeriodSeconds: 10
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config
      serviceAccountName: fluent-bit
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      imagePullSecrets:
      - name: kube-sec

拉起Fluent Bit DaemonSet

創(chuàng)建Fluent Bit所需的配置。

[centos@master1 fluent-bit]$ ls -al
total 28
drwxrwxr-x 2 centos centos 4096 Jun  7 02:32 .
drwxrwxr-x 7 centos centos 4096 Jun 11 03:03 ..
-rw-rw-r-- 1 centos centos 3562 Jun  7 02:32 fluent-bit-configmap.yaml
-rw-rw-r-- 1 centos centos 2248 Jun  6 12:51 fluent-bit-ds.yaml
-rw-rw-r-- 1 centos centos  273 May 31 13:35 fluent-bit-role-binding.yaml
-rw-rw-r-- 1 centos centos  194 May 31 13:33 fluent-bit-role.yaml
-rw-rw-r-- 1 centos centos   90 May 31 13:35 fluent-bit-service-account.yaml
[centos@master1 fluent-bit]$ 
[centos@master1 fluent-bit]$ kubectl create -f fluent-bit-configmap.yaml

拉起Fluent Bit的Daemon Set懂傀。

[centos@master1 fluent-bit]$ kubectl create -f fluent-bit-ds.yaml

檢查Pod趾诗，此時(shí)會(huì)發(fā)現(xiàn)

[centos@master1 fluent-bit]$ kubectl get pods -n kube-system -o wide | grep fluent-bit
[centos@master1 fluent-bit]$

不要緊張，我們接下來(lái)就要對(duì)Node打label蹬蚁，這樣就會(huì)拉起來(lái)了恃泪，中間注意間隔點(diǎn)時(shí)間。

[centos@master1 fluent-bit]$ kubectl label node minion1 beta.kubernetes.io/fluentd-ds-ready=true
[centos@master1 fluent-bit]$ kubectl label node minion2 beta.kubernetes.io/fluentd-ds-ready=true
[centos@master1 fluent-bit]$ kubectl label node minion3 beta.kubernetes.io/fluentd-ds-ready=true
[centos@master1 fluent-bit]$ kubectl label node minion4 beta.kubernetes.io/fluentd-ds-ready=true
[centos@master1 fluent-bit]$ kubectl label node minion5 beta.kubernetes.io/fluentd-ds-ready=true
.....

我們?cè)贆z查節(jié)點(diǎn)被拉起狀態(tài)：

[centos@master1 fluent-bit]$ kubectl get pods -n kube-system -o wide | grep flu
fluent-bit-2sd9k                         1/1       Running   0          4d        10.244.30.213    minion5
fluent-bit-5jd4w                         1/1       Running   0          4d        10.244.32.12     minion3
fluent-bit-952fn                         1/1       Running   0          4d        10.244.15.14     minion13
fluent-bit-cz2xq                         1/1       Running   0          4d        10.244.29.250    minion6
fluent-bit-fx22k                         1/1       Running   0          4d        10.244.25.235    minion10
fluent-bit-g4fmw                         1/1       Running   0          4d        10.244.23.99     minion12
fluent-bit-gnfxg                         1/1       Running   0          4d        10.244.28.207    minion7
fluent-bit-h9t9l                         1/1       Running   0          4d        10.244.11.91     minion22
fluent-bit-ld9fx                         1/1       Running   0          4d        10.244.3.191     minion17
fluent-bit-pgc2f                         1/1       Running   0          4d        10.244.14.48     minion20
fluent-bit-st2qq                         1/1       Running   0          3d        10.244.16.3      minion11
fluent-bit-tm5hl                         1/1       Running   0          4d        10.244.12.46     minion18
fluent-bit-tt44q                         1/1       Running   0          4d        10.244.21.24     minion14
fluent-bit-vgptk                         1/1       Running   0          4d        10.244.31.9      minion4
fluent-bit-vptft                         1/1       Running   0          4d        10.244.34.93     minion1
fluent-bit-wpwl4                         1/1       Running   0          4d        10.244.13.35     minion19
fluent-bit-xdvbz                         1/1       Running   0          4d        10.244.9.99      minion2
fluent-bit-zrmsj                         1/1       Running   0          4d        10.244.20.33     minion15

我們查看Kibana來(lái)確定下日志傳輸情況：

image.png

我們發(fā)現(xiàn)日志已傳輸?shù)紼S犀斋，我們接下來(lái)對(duì)Kibana稍微調(diào)整下顯示的Fields贝乎，更能滿足我們查看日志的需要。

• 增加Host到顯示列表叽粹。
• 增加Pod Name到顯示列表览效。
• 增加Log到顯示列表。

image.png

我們接下來(lái)挨個(gè)主機(jī)查看下日志情況：

• 增加輸入條件虫几，查詢節(jié)點(diǎn)的日志锤灿。

image.png

我們會(huì)發(fā)現(xiàn)每個(gè)主機(jī)的日志已正常的傳遞到ES，那我們接下來(lái)再檢查下S3辆脸。
我們會(huì)發(fā)現(xiàn)但校，日志已按照我們既定的目錄規(guī)則創(chuàng)建出來(lái)。

image.png

點(diǎn)開某一個(gè)目錄啡氢，則可以發(fā)現(xiàn)文件已存在状囱，亦可以下載到本地，進(jìn)行再次查看倘是，不問不再做論述亭枷。

image.png

至此，F(xiàn)luent Bit部署已完成辨绊，在拉起Fluent Bit的過程中奶栖，會(huì)有部分節(jié)點(diǎn)Crash匹表，無(wú)法正常拉起门坷。接下來(lái)這些節(jié)點(diǎn)宣鄙，我們將采用Fluentd來(lái)進(jìn)行日志的采集。

Fluentd

我們參考Fluentd Server章節(jié)中的描述默蚌，來(lái)準(zhǔn)備Fluentd的拉起冻晤。Fluentd的Docker文件可以和Server章節(jié)中使用的一致。

準(zhǔn)備Fluentd Yaml文件

Fluentd 配置文件

我們新建一文件fluentd-standalone-configmap.yaml绸吸，用來(lái)Fluentd的獨(dú)立運(yùn)行鼻弧。

• 調(diào)整output.conf。
• 移除ES片段锦茁。
• 添加forward片段攘轩。

kind: ConfigMap
apiVersion: v1
metadata:
  name: fluentd-sa-config-v0.1.4
  namespace: kube-system
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
data:
  system.conf: |-
    <system>
      root_dir /tmp/fluentd-buffers/
    </system>

  containers.input.conf: |-
    # This configuration file for Fluentd / td-agent is used
    # to watch changes to Docker log files. The kubelet creates symlinks that
    # capture the pod name, namespace, container name & Docker container ID
    # to the docker logs for pods in the /var/log/containers directory on the host.
    # If running this fluentd configuration in a Docker container, the /var/log
    # directory should be mounted in the container.
    #
    # These logs are then submitted to Elasticsearch which assumes the
    # installation of the fluent-plugin-elasticsearch & the
    # fluent-plugin-kubernetes_metadata_filter plugins.
    # See https://github.com/uken/fluent-plugin-elasticsearch &
    # https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter for
    # more information about the plugins.
    #
    # Example
    # =======
    # A line in the Docker log file might look like this JSON:
    #
    # {"log":"2014/09/25 21:15:03 Got request with path wombat\n",
    #  "stream":"stderr",
    #   "time":"2014-09-25T21:15:03.499185026Z"}
    #
    # The time_format specification below makes sure we properly
    # parse the time format produced by Docker. This will be
    # submitted to Elasticsearch and should appear like:
    # $ curl 'http://elasticsearch-logging:9200/_search?pretty'
    # ...
    # {
    #      "_index" : "logstash-2014.09.25",
    #      "_type" : "fluentd",
    #      "_id" : "VBrbor2QTuGpsQyTCdfzqA",
    #      "_score" : 1.0,
    #      "_source":{"log":"2014/09/25 22:45:50 Got request with path wombat\n",
    #                 "stream":"stderr","tag":"docker.container.all",
    #                 "@timestamp":"2014-09-25T22:45:50+00:00"}
    #    },
    # ...
    #
    # The Kubernetes fluentd plugin is used to write the Kubernetes metadata to the log
    # record & add labels to the log record if properly configured. This enables users
    # to filter & search logs on any metadata.
    # For example a Docker container's logs might be in the directory:
    #
    #  /var/lib/docker/containers/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b
    #
    # and in the file:
    #
    #  997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b-json.log
    #
    # where 997599971ee6... is the Docker ID of the running container.
    # The Kubernetes kubelet makes a symbolic link to this file on the host machine
    # in the /var/log/containers directory which includes the pod name and the Kubernetes
    # container name:
    #
    #    synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log
    #    ->
    #    /var/lib/docker/containers/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b-json.log
    #
    # The /var/log directory on the host is mapped to the /var/log directory in the container
    # running this instance of Fluentd and we end up collecting the file:
    #
    #   /var/log/containers/synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log
    #
    # This results in the tag:
    #
    #  var.log.containers.synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log
    #
    # The Kubernetes fluentd plugin is used to extract the namespace, pod name & container name
    # which are added to the log message as a kubernetes field object & the Docker container ID
    # is also added under the docker field object.
    # The final tag is:
    #
    #   kubernetes.var.log.containers.synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log
    #
    # And the final log record look like:
    #
    # {
    #   "log":"2014/09/25 21:15:03 Got request with path wombat\n",
    #   "stream":"stderr",
    #   "time":"2014-09-25T21:15:03.499185026Z",
    #   "kubernetes": {
    #     "namespace": "default",
    #     "pod_name": "synthetic-logger-0.25lps-pod",
    #     "container_name": "synth-lgr"
    #   },
    #   "docker": {
    #     "container_id": "997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b"
    #   }
    # }
    #
    # This makes it easier for users to search for logs by pod name or by
    # the name of the Kubernetes container regardless of how many times the
    # Kubernetes pod has been restarted (resulting in a several Docker container IDs).

    # Json Log Example:
    # {"log":"[info:2016-02-16T16:04:05.930-08:00] Some log text here\n","stream":"stdout","time":"2016-02-17T00:04:05.931087621Z"}
    # CRI Log Example:
    # 2016-02-17T00:04:05.931087621Z stdout F [info:2016-02-16T16:04:05.930-08:00] Some log text here
    <source>
      @id fluentd-containers.log
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/es-containers.log.pos
      time_format %Y-%m-%dT%H:%M:%S.%NZ
      tag raw.kubernetes.*
      read_from_head true
      <parse>
        @type multi_format
        <pattern>
          format json
          time_key time
          time_format %Y-%m-%dT%H:%M:%S.%NZ
        </pattern>
        <pattern>
          format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/
          time_format %Y-%m-%dT%H:%M:%S.%N%:z
        </pattern>
      </parse>
    </source>

    # Detect exceptions in the log output and forward them as one log entry.
    <match raw.kubernetes.**>
      @id raw.kubernetes
      @type detect_exceptions
      remove_tag_prefix raw
      message log
      stream stream
      multiline_flush_interval 5
      max_bytes 500000
      max_lines 1000
    </match>

  system.input.conf: |-
    # Example:
    # 2015-12-21 23:17:22,066 [salt.state       ][INFO    ] Completed state [net.ipv4.ip_forward] at time 23:17:22.066081
    <source>
      @id minion
      @type tail
      format /^(?<time>[^ ]* [^ ,]*)[^\[]*\[[^\]]*\]\[(?<severity>[^ \]]*) *\] (?<message>.*)$/
      time_format %Y-%m-%d %H:%M:%S
      path /var/log/salt/minion
      pos_file /var/log/salt.pos
      tag salt
    </source>

    # Example:
    # Dec 21 23:17:22 gke-foo-1-1-4b5cbd14-node-4eoj startupscript: Finished running startup script /var/run/google.startup.script
    <source>
      @id startupscript.log
      @type tail
      format syslog
      path /var/log/startupscript.log
      pos_file /var/log/es-startupscript.log.pos
      tag startupscript
    </source>

    # Examples:
    # time="2016-02-04T06:51:03.053580605Z" level=info msg="GET /containers/json"
    # time="2016-02-04T07:53:57.505612354Z" level=error msg="HTTP Error" err="No such image: -f" statusCode=404
    # TODO(random-liu): Remove this after cri container runtime rolls out.
    <source>
      @id docker.log
      @type tail
      format /^time="(?<time>[^)]*)" level=(?<severity>[^ ]*) msg="(?<message>[^"]*)"( err="(?<error>[^"]*)")?( statusCode=($<status_code>\d+))?/
      path /var/log/docker.log
      pos_file /var/log/es-docker.log.pos
      tag docker
    </source>

    # Example:
    # 2016/02/04 06:52:38 filePurge: successfully removed file /var/etcd/data/member/wal/00000000000006d0-00000000010a23d1.wal
    <source>
      @id etcd.log
      @type tail
      # Not parsing this, because it doesn't have anything particularly useful to
      # parse out of it (like severities).
      format none
      path /var/log/etcd.log
      pos_file /var/log/es-etcd.log.pos
      tag etcd
    </source>

    # Multi-line parsing is required for all the kube logs because very large log
    # statements, such as those that include entire object bodies, get split into
    # multiple lines by glog.

    # Example:
    # I0204 07:32:30.020537    3368 server.go:1048] POST /stats/container/: (13.972191ms) 200 [[Go-http-client/1.1] 10.244.1.3:40537]
    <source>
      @id kubelet.log
      @type tail
      format multiline
      multiline_flush_interval 5s
      format_firstline /^\w\d{4}/
      format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
      time_format %m%d %H:%M:%S.%N
      path /var/log/kubelet.log
      pos_file /var/log/es-kubelet.log.pos
      tag kubelet
    </source>

    # Example:
    # I1118 21:26:53.975789       6 proxier.go:1096] Port "nodePort for kube-system/default-http-backend:http" (:31429/tcp) was open before and is still needed
    <source>
      @id kube-proxy.log
      @type tail
      format multiline
      multiline_flush_interval 5s
      format_firstline /^\w\d{4}/
      format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
      time_format %m%d %H:%M:%S.%N
      path /var/log/kube-proxy.log
      pos_file /var/log/es-kube-proxy.log.pos
      tag kube-proxy
    </source>

    # Example:
    # I0204 07:00:19.604280       5 handlers.go:131] GET /api/v1/nodes: (1.624207ms) 200 [[kube-controller-manager/v1.1.3 (linux/amd64) kubernetes/6a81b50] 127.0.0.1:38266]
    <source>
      @id kube-apiserver.log
      @type tail
      format multiline
      multiline_flush_interval 5s
      format_firstline /^\w\d{4}/
      format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
      time_format %m%d %H:%M:%S.%N
      path /var/log/kube-apiserver.log
      pos_file /var/log/es-kube-apiserver.log.pos
      tag kube-apiserver
    </source>

    # Example:
    # I0204 06:55:31.872680       5 servicecontroller.go:277] LB already exists and doesn't need update for service kube-system/kube-ui
    <source>
      @id kube-controller-manager.log
      @type tail
      format multiline
      multiline_flush_interval 5s
      format_firstline /^\w\d{4}/
      format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
      time_format %m%d %H:%M:%S.%N
      path /var/log/kube-controller-manager.log
      pos_file /var/log/es-kube-controller-manager.log.pos
      tag kube-controller-manager
    </source>

    # Example:
    # W0204 06:49:18.239674       7 reflector.go:245] pkg/scheduler/factory/factory.go:193: watch of *api.Service ended with: 401: The event in requested index is outdated and cleared (the requested history has been cleared [2578313/2577886]) [2579312]
    <source>
      @id kube-scheduler.log
      @type tail
      format multiline
      multiline_flush_interval 5s
      format_firstline /^\w\d{4}/
      format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
      time_format %m%d %H:%M:%S.%N
      path /var/log/kube-scheduler.log
      pos_file /var/log/es-kube-scheduler.log.pos
      tag kube-scheduler
    </source>

    # Example:
    # I1104 10:36:20.242766       5 rescheduler.go:73] Running Rescheduler
    <source>
      @id rescheduler.log
      @type tail
      format multiline
      multiline_flush_interval 5s
      format_firstline /^\w\d{4}/
      format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
      time_format %m%d %H:%M:%S.%N
      path /var/log/rescheduler.log
      pos_file /var/log/es-rescheduler.log.pos
      tag rescheduler
    </source>

    # Example:
    # I0603 15:31:05.793605       6 cluster_manager.go:230] Reading config from path /etc/gce.conf
    <source>
      @id glbc.log
      @type tail
      format multiline
      multiline_flush_interval 5s
      format_firstline /^\w\d{4}/
      format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
      time_format %m%d %H:%M:%S.%N
      path /var/log/glbc.log
      pos_file /var/log/es-glbc.log.pos
      tag glbc
    </source>

    # Example:
    # I0603 15:31:05.793605       6 cluster_manager.go:230] Reading config from path /etc/gce.conf
    <source>
      @id cluster-autoscaler.log
      @type tail
      format multiline
      multiline_flush_interval 5s
      format_firstline /^\w\d{4}/
      format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
      time_format %m%d %H:%M:%S.%N
      path /var/log/cluster-autoscaler.log
      pos_file /var/log/es-cluster-autoscaler.log.pos
      tag cluster-autoscaler
    </source>

    # Logs from systemd-journal for interesting services.
    # TODO(random-liu): Remove this after cri container runtime rolls out.
    <source>
      @id journald-docker
      @type systemd
      filters [{ "_SYSTEMD_UNIT": "docker.service" }]
      <storage>
        @type local
        persistent true
      </storage>
      read_from_head true
      tag docker
    </source>

    <source>
      @id journald-container-runtime
      @type systemd
      filters [{ "_SYSTEMD_UNIT": "{{ container_runtime }}.service" }]
      <storage>
        @type local
        persistent true
      </storage>
      read_from_head true
      tag container-runtime
    </source>

    <source>
      @id journald-kubelet
      @type systemd
      filters [{ "_SYSTEMD_UNIT": "kubelet.service" }]
      <storage>
        @type local
        persistent true
      </storage>
      read_from_head true
      tag kubelet
    </source>

    <source>
      @id journald-node-problem-detector
      @type systemd
      filters [{ "_SYSTEMD_UNIT": "node-problem-detector.service" }]
      <storage>
        @type local
        persistent true
      </storage>
      read_from_head true
      tag node-problem-detector
    </source>
    
    <source>
      @id kernel
      @type systemd
      filters [{ "_TRANSPORT": "kernel" }]
      <storage>
        @type local
        persistent true
      </storage>
      <entry>
        fields_strip_underscores true
        fields_lowercase true
      </entry>
      read_from_head true
      tag kernel
    </source>

  forward.input.conf: |-
    # Takes the messages sent over TCP
    <source>
      @type forward
    </source>

  monitoring.conf: |-
    # Prometheus Exporter Plugin
    # input plugin that exports metrics
    <source>
      @type prometheus
    </source>

    <source>
      @type monitor_agent
    </source>

    # input plugin that collects metrics from MonitorAgent
    <source>
      @type prometheus_monitor
      <labels>
        host ${hostname}
      </labels>
    </source>

    # input plugin that collects metrics for output plugin
    <source>
      @type prometheus_output_monitor
      <labels>
        host ${hostname}
      </labels>
    </source>

    # input plugin that collects metrics for in_tail plugin
    <source>
      @type prometheus_tail_monitor
      <labels>
        host ${hostname}
      </labels>
    </source>

  output.conf: |-
    # Enriches records with Kubernetes metadata
    <filter kubernetes.**>
      @type kubernetes_metadata
    </filter>

    <match **>
      @type forward
      require_ack_response true
      ack_response_timeout 30
      recover_wait 10s
      heartbeat_interval 1s
      phi_threshold 16
      send_timeout 10s
      hard_timeout 10s
      expire_dns_cache 15
      heartbeat_type tcp
      buffer_chunk_limit 2M
      buffer_queue_limit 64
      flush_interval 5s
      max_retry_wait 15
      disable_retry_limit
      num_threads 8
      
      <server>
        name fluentd-server
        host fluentd-server
        port 24224
        weight 100
        </server>
    </match>

Fluentd Pod Yaml

我們亦將新建fluentd-standalone.yaml，用來(lái)控制Fluentd Pod的啟動(dòng)码俩。
請(qǐng)注意此Yaml中度帮，我們亦使用了NodeSelector，和Fluent Bit不同的是稿存，我們使用的
beta.kubernetes.io/fluentd-ds-ready = "fluentd"笨篷。

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd-es
  namespace: kube-system
  labels:
    k8s-app: fluentd-es
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: fluentd-es
  labels:
    k8s-app: fluentd-es
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups:
  - ""
  resources:
  - "namespaces"
  - "pods"
  verbs:
  - "get"
  - "watch"
  - "list"
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: fluentd-es
  labels:
    k8s-app: fluentd-es
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
subjects:
- kind: ServiceAccount
  name: fluentd-es
  namespace: kube-system
  apiGroup: ""
roleRef:
  kind: ClusterRole
  name: fluentd-es
  apiGroup: ""
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd-es-v2.0.4
  namespace: kube-system
  labels:
    k8s-app: fluentd-es
    version: v2.0.4
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  selector:
    matchLabels:
      k8s-app: fluentd-es
      version: v2.0.4
  template:
    metadata:
      labels:
        k8s-app: fluentd-es
        kubernetes.io/cluster-service: "true"
        version: v2.0.4
      # This annotation ensures that fluentd does not get evicted if the node
      # supports critical pod annotation based priority scheme.
      # Note that this does not guarantee admission on the nodes (#40573).
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      priorityClassName: system-node-critical
      serviceAccountName: fluentd-es
      containers:
      - name: fluentd-es
        #image: k8s.gcr.io/fluentd-elasticsearch:v2.0.4
        image: hub.***.***/google_containers/fluentd-s3:v2.1.0
        imagePullPolicy: Always
        env:
        - name: FLUENTD_ARGS
          value: --no-supervisor -q
        resources:
          limits:
            memory: 500Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: config-volume
          mountPath: /etc/fluent/config.d
      nodeSelector:
        beta.kubernetes.io/fluentd-ds-ready: "fluentd"
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: config-volume
        configMap:
          name: fluentd-sa-config-v0.1.4
      imagePullSecrets:
      - name: kube-sec

拉起Fluentd DaemonSet

創(chuàng)建Fluentd所需的配置。

[centos@master1 efk]$ kubectl create -f fluentd-standalone-configmap.yaml 

拉起Fluentd的DaemonSet瓣履。

[centos@master1 efk]$ kubectl create -f fluentd-standalone.yaml 

和Fluent Bit 拉起時(shí)一樣率翅，此時(shí)我們檢查Pod，除了Fluentd Server的5個(gè)Pod外袖迎，此時(shí)亦發(fā)現(xiàn)沒有Pod被拉起冕臭。

[centos@master1 efk]$ kubectl get pods -n kube-system -o wide | grep fluentd
fluentd-server-v2.0.4-855db7cfc5-4wn47   1/1       Running   0          6h        10.244.29.20     minion6
fluentd-server-v2.0.4-855db7cfc5-pfmvd   1/1       Running   0          6h        10.244.3.211     minion17
fluentd-server-v2.0.4-855db7cfc5-rjqxl   1/1       Running   0          6h        10.244.13.47     minion19
fluentd-server-v2.0.4-855db7cfc5-shjfm   1/1       Running   0          6h        10.244.23.141    minion12
fluentd-server-v2.0.4-855db7cfc5-w7m5f   1/1       Running   0          6h        10.244.30.233    minion5
[centos@master1 efk]$ 

我們接下來(lái)就要對(duì)Node打label，這樣就會(huì)拉起來(lái)了瓢棒，中間注意間隔點(diǎn)時(shí)間浴韭。

[centos@master1 efk]$ kubectl label node minion8 beta.kubernetes.io/fluentd-ds-ready=true
[centos@master1 efk]$ kubectl label node minion21 beta.kubernetes.io/fluentd-ds-ready=true
[centos@master1 efk]$ kubectl label node minion9 beta.kubernetes.io/fluentd-ds-ready=true
[centos@master1 efk]$ kubectl label node minion16 beta.kubernetes.io/fluentd-ds-ready=true

再次檢查節(jié)點(diǎn)被拉起的狀態(tài)：

[centos@master1 efk]$ kubectl get pods -n kube-system -o wide | grep fluentd
fluentd-es-v2.0.4-75gdf                  1/1       Running   1          4d        10.244.27.245    minion8
fluentd-es-v2.0.4-kx5pz                  1/1       Running   0          2h        10.244.10.96     minion21
fluentd-es-v2.0.4-n89xj                  1/1       Running   6          4d        10.244.26.92     minion9
fluentd-es-v2.0.4-zsrln                  1/1       Running   0          6h        10.244.19.67     minion16
fluentd-server-v2.0.4-855db7cfc5-4wn47   1/1       Running   0          6h        10.244.29.20     minion6
fluentd-server-v2.0.4-855db7cfc5-pfmvd   1/1       Running   0          6h        10.244.3.211     minion17
fluentd-server-v2.0.4-855db7cfc5-rjqxl   1/1       Running   0          6h        10.244.13.47     minion19
fluentd-server-v2.0.4-855db7cfc5-shjfm   1/1       Running   0          6h        10.244.23.141    minion12
fluentd-server-v2.0.4-855db7cfc5-w7m5f   1/1       Running   0          6h        10.244.30.233    minion5
[centos@master1 efk]$ 

和Fluent Bit一樣，需要去Kibana那邊再次檢查下節(jié)點(diǎn)的日志是否有正常傳遞到ES和S3脯宿，在此就不再累述念颈。