有時(shí)候調(diào)試fluent-bit的配置迷捧，達(dá)到想要的輸出效果织咧，并不是件簡(jiǎn)單的事情，以下通過(guò)debug鏡像調(diào)試fluent-bit采集kubernetes Pod的IP漠秋。

fluent-bit官方文檔給出了用于調(diào)試的鏡像：
https://docs.fluentbit.io/manual/installation/docker

dockerhub倉(cāng)庫(kù)鏈接為：
https://hub.docker.com/r/fluent/fluent-bit/

部署fluent-bit-debug

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: fluent-bit-debug
  name: fluent-bit-debug
  namespace: kubesphere-logging-system
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/name: fluent-bit-debug
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/name: fluent-bit-debug
      name: fluent-bit-debug
    spec:
      containers:
      - env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        command:
        - /usr/local/bin/sh
        - -c
        - sleep 9999      
        image: fluent/fluent-bit:1.6.9-debug
        imagePullPolicy: IfNotPresent
        name: fluent-bit
        ports:
        - containerPort: 2020
          name: metrics
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/docker/containers
          name: varlibcontainers
          readOnly: true
        - mountPath: /fluent-bit/config
          name: config
          readOnly: true
        - mountPath: /var/log/
          name: varlogs
          readOnly: true
        - mountPath: /var/log/journal
          name: systemd
          readOnly: true
        - mountPath: /fluent-bit/tail
          name: positions
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: fluent-bit
      serviceAccountName: fluent-bit
      terminationGracePeriodSeconds: 30
      volumes:
      - hostPath:
          path: /var/lib/docker/containers
          type: ""
        name: varlibcontainers
      - name: config
        secret:
          defaultMode: 420
          secretName: fluent-bit-debug-config
      - hostPath:
          path: /var/log
          type: ""
        name: varlogs
      - hostPath:
          path: /var/log/journal
          type: ""
        name: systemd
      - emptyDir: {}
        name: positions    

用到的secret：fluent-bit-debug-config如下笙蒙，包含兩個(gè)key：

在這里插入圖片描述

第一個(gè)parsers.conf：為空即可；配置細(xì)節(jié)可以參考官方文檔：
Configuration File

第二個(gè)fluent-bit.conf配置需根據(jù)情況配置庆锦，以下主要在不同場(chǎng)景捅位，給出fluent-bit.conf。也會(huì)涉及到kubesphere搂抒、Filter CRD一起使用艇搀。

fluent-bit使用

exec進(jìn)入容器可以使用/fluent-bit/bin/fluent-bit調(diào)試：

/fluent-bit/bin #  ./fluent-bit -h
Usage: fluent-bit [OPTION]

Available Options
  -b  --storage_path=PATH       specify a storage buffering path
  -c  --config=FILE     specify an optional configuration file
  -d, --daemon          run Fluent Bit in background mode
  -f, --flush=SECONDS   flush timeout in seconds (default: 5)
  -F  --filter=FILTER    set a filter
  -i, --input=INPUT     set an input
  -m, --match=MATCH     set plugin match, same as '-p match=abc'
  -o, --output=OUTPUT   set an output
  -p, --prop="A=B"      set plugin configuration property
  -R, --parser=FILE     specify a parser configuration file
  -e, --plugin=FILE     load an external plugin (shared lib)
  -l, --log_file=FILE   write log info to a file
  -t, --tag=TAG         set plugin tag, same as '-p tag=abc'
  -T, --sp-task=SQL     define a stream processor task
  -v, --verbose         increase logging verbosity (default: info)
  -H, --http            enable monitoring HTTP server
  -P, --port            set HTTP server TCP port (default: 2020)
  -s, --coro_stack_size Set coroutines stack size in bytes (default: 24576)
  -q, --quiet           quiet mode
  -S, --sosreport       support report for Enterprise customers
  -V, --version         show version number
  -h, --help            print this help

Inputs
  cpu                   CPU Usage
  mem                   Memory Usage
  thermal               Thermal
  kmsg                  Kernel Log Buffer
  proc                  Check Process health
  disk                  Diskstats
  systemd               Systemd (Journal) reader
  netif                 Network Interface Usage
  docker                Docker containers metrics
  docker_events         Docker events
  tail                  Tail files
  dummy                 Generate dummy data
  head                  Head Input
  health                Check TCP server health
  collectd              collectd input plugin
  statsd                StatsD input plugin
  serial                Serial input
  stdin                 Standard Input
  syslog                Syslog
  exec                  Exec Input
  tcp                   TCP
  mqtt                  MQTT, listen for Publish messages
  forward               Fluentd in-forward
  random                Random

Filters
  alter_size            Alter incoming chunk size
  aws                   Add AWS Metadata
  record_modifier       modify record
  throttle              Throttle messages using sliding window algorithm
  kubernetes            Filter to append Kubernetes metadata
  modify                modify records by applying rules
  nest                  nest events by specified field values
  parser                Parse events
  expect                Validate expected keys and values
  grep                  grep events by specified field values
  rewrite_tag           Rewrite records tags
  lua                   Lua Scripting Filter
  stdout                Filter events to STDOUT

Outputs
  azure                 Send events to Azure HTTP Event Collector
  azure_blob            Azure Blob Storage
  bigquery              Send events to BigQuery via streaming insert
  counter               Records counter
  datadog               Send events to DataDog HTTP Event Collector
  es                    Elasticsearch
  exit                  Exit after a number of flushes (test purposes)
  file                  Generate log file
  forward               Forward (Fluentd protocol)
  http                  HTTP Output
  influxdb              InfluxDB Time Series
  logdna                LogDNA
  loki                  Loki
  kafka                 Kafka
  kafka-rest            Kafka REST Proxy
  nats                  NATS Server
  nrlogs                New Relic
  null                  Throws away events
  plot                  Generate data file for GNU Plot
  pgsql                 PostgreSQL
  slack                 Send events to a Slack channel
  splunk                Send events to Splunk HTTP Event Collector
  stackdriver           Send events to Google Stackdriver Logging
  stdout                Prints events to STDOUT
  syslog                Syslog
  tcp                   TCP Output
  td                    Treasure Data
  flowcounter           FlowCounter
  gelf                  GELF Output
  cloudwatch_logs       Send logs to Amazon CloudWatch
  kinesis_firehose      Send logs to Amazon Kinesis Firehose
  s3                    Send to S3

Internal
 Event Loop  = epoll
 Build Flags =  FLB_HAVE_HTTP_CLIENT_DEBUG FLB_HAVE_PARSER FLB_HAVE_RECORD_ACCESSOR FLB_HAVE_STREAM_PROCESSOR JSMN_PARENT_LINKS JSMN_STRICT FLB_HAVE_TLS FLB_HAVE_AWS FLB_HAVE_SIGNV4 FLB_HAVE_SQLDB FLB_HAVE_METRICS FLB_HAVE_HTTP_SERVER FLB_HAVE_SYSTEMD FLB_HAVE_FORK FLB_HAVE_TIMESPEC_GET FLB_HAVE_GMTOFF FLB_HAVE_UNIX_SOCKET FLB_HAVE_PROXY_GO FLB_HAVE_JEMALLOC JEMALLOC_MANGLE FLB_HAVE_LIBBACKTRACE FLB_HAVE_REGEX FLB_HAVE_UTF8_ENCODER FLB_HAVE_LUAJIT FLB_HAVE_C_TLS FLB_HAVE_ACCEPT4 FLB_HAVE_INOTIFY

簡(jiǎn)單配置文件

以下用一個(gè)簡(jiǎn)單的配置文件采集calico-node-* pod的日志：

[Service]
    Parsers_File    parsers.conf
[Input]
    Name    tail
    Path    /var/log/containers/*_kube-system_calico-node-*.log
    Refresh_Interval    10
    Skip_Long_Lines    true
    DB    /fluent-bit/bin/pos.db
    DB.Sync    Normal
    Mem_Buf_Limit    5MB
    Parser    docker
    Tag    kube.*
[Filter]
    Name    kubernetes
    Match    kube.*
    Kube_URL    https://kubernetes.default.svc:443
    Kube_CA_File    /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File    /var/run/secrets/kubernetes.io/serviceaccount/token
    Labels    false
    Annotations    true
[Output]
    Name    stdout
    Match_Regex    (?:kube|service)\.(.*)    

fluent-bit-debug容器內(nèi)使用以下命令啟動(dòng)測(cè)試：

/fluent-bit/bin # ./fluent-bit -c /fluent-bit/config/fluent-bit.conf

可以看到stdout日志輸出：

[0] kube.var.log.containers.calico-node-lp4lm_kube-system_calico-node-cca502a39695f7452fd999af97bfbca5d74d2a372d94e0cacf2045f5f9721a81.log: [1634870260.700108403, {"log"=>"{"log":"2021-10-22 02:37:40.699 [INFO][85] monitor-addresses/startup.go 774: Using autodetected IPv4 address on interface bond4: 172.24.248.50/30\n","stream":"stdout","time":"2021-10-22T02:37:40.700056471Z"}", "kubernetes"=>{"pod_name"=>"calico-node-lp4lm", "namespace_name"=>"kube-system", "pod_id"=>"5a829979-9830-4b9c-a3cb-eeb6eee38bdd", "annotations"=>{"kubectl.kubernetes.io/restartedAt"=>"2021-10-20T23:00:27+08:00"}, "host"=>"node02", "container_name"=>"calico-node", "docker_id"=>"cca502a39695f7452fd999af97bfbca5d74d2a372d94e0cacf2045f5f9721a81", "container_hash"=>"calico/node@sha256:bc4a631d553b38fdc169ea4cb8027fa894a656e80d68d513359a4b9d46836b55", "container_image"=>"calico/node:v3.19.1"}}]

截取重要部分，可以看到?jīng)]經(jīng)過(guò)處理采集到的k8s日志格式求晶。

[
    {
        "kubernetes"=>{
            "pod_name"=>"calico-node-lp4lm",
            "namespace_name"=>"kube-system",
            "pod_id"=>"5a829979-9830-4b9c-a3cb-eeb6eee38bdd",
            "annotations"=>{
                "kubectl.kubernetes.io/restartedAt"=>"2021-10-20T23:00:27+08:00"
            },
            "host"=>"node02",
            "container_name"=>"calico-node",
            "docker_id"=>"cca502a39695f7452fd999af97bfbca5d74d2a372d94e0cacf2045f5f9721a81",
            "container_hash"=>"calico/node@sha256:bc4a631d553b38fdc169ea4cb8027fa894a656e80d68d513359a4b9d46836b55",
            "container_image"=>"calico/node:v3.19.1"
        }
    }
]

增加nest Filter

將kubernetes塊展開(kāi)焰雕，并添加kubernetes_前綴：

[Filter]
    Name    nest
    Match    kube.*
    Operation    lift
    Nested_under    kubernetes
    Add_prefix    kubernetes_

這次測(cè)試輸出，截取重要部分：

{
    "kubernetes_pod_name"=>"calico-node-lp4lm",
    "kubernetes_namespace_name"=>"kube-system",
    "kubernetes_pod_id"=>"5a829979-9830-4b9c-a3cb-eeb6eee38bdd",
    "kubernetes_annotations"=>{
        "kubectl.kubernetes.io/restartedAt"=>"2021-10-20T23:00:27+08:00"
    },
    "kubernetes_host"=>"node02",
    "kubernetes_container_name"=>"calico-node",
    "kubernetes_docker_id"=>"cca502a39695f7452fd999af97bfbca5d74d2a372d94e0cacf2045f5f9721a81",
    "kubernetes_container_hash"=>"calico/node@sha256:bc4a631d553b38fdc169ea4cb8027fa894a656e80d68d513359a4b9d46836b55",
    "kubernetes_container_image"=>"calico/node:v3.19.1"
}

移除掉kubernetes_annotations塊

[Filter]
    Name    modify
    Match    kube.*
    Remove    kubernetes_annotations

移除掉kubernetes_annotations塊中的某字段

[Filter]
    Name    nest
    Match    kube.*
    Operation    lift
    Nested_under    kubernetes_annotations
    Add_prefix    kubernetes_annotations_
[Filter]
    Name    modify
    Match    kube.*
    Remove    kubernetes_annotations_kubectl.kubernetes.io/restartedAt

或者用正則：

[Filter]
    Name    nest
    Match    kube.*
    Operation    lift
    Nested_under    kubernetes_annotations
    Add_prefix    kubernetes_annotations_
[Filter]
    Name    modify
    Match    kube.*
    Remove_regex    kubernetes_annotations_kubectl*

修改kubernetes_annotations塊中key名稱

[Filter]
    Name    nest
    Match    kube.*
    Operation    lift
    Nested_under    kubernetes_annotations
    Add_prefix    kubernetes_annotations_
[Filter]
    Name    modify
    Match    kube.*
    Rename    kubernetes_annotations_kubectl.kubernetes.io/restartedAt podIPs

修改之后：

[
    {
        "kubernetes_pod_name"=>"calico-node-lp4lm",
        "kubernetes_namespace_name"=>"kube-system",
        "kubernetes_pod_id"=>"5a829979-9830-4b9c-a3cb-eeb6eee38bdd",
        "kubernetes_host"=>"node02",
        "kubernetes_container_name"=>"calico-node",
        "kubernetes_docker_id"=>"cca502a39695f7452fd999af97bfbca5d74d2a372d94e0cacf2045f5f9721a81",
        "kubernetes_container_hash"=>"calico/node@sha256:bc4a631d553b38fdc169ea4cb8027fa894a656e80d68d513359a4b9d46836b55",
        "kubernetes_container_image"=>"calico/node:v3.19.1",
        "podIPs"=>"2021-10-20T23:00:27+08:00"
    }
]

結(jié)合ks配置采集podIPs

結(jié)合kubesphere Filter CR芳杏，配置采集podIPs矩屁，并去掉其他不相關(guān)的annotations。

因使用calico作為CNI爵赵，所以在pod annotations中會(huì)被添加上podIP相關(guān)的注解吝秕。

需要保留注解中的某一個(gè)key（cni.projectcalico.org/podIPs），移除掉其他key空幻，所以下面將要保留的key修改名稱之后烁峭，移除掉整個(gè)annotations。

kubernetes Filter CR配置如下：

apiVersion: logging.kubesphere.io/v1alpha2
kind: Filter
metadata:
  labels:
    logging.kubesphere.io/component: logging
    logging.kubesphere.io/enabled: 'true'
  name: kubernetes
  namespace: kubesphere-logging-system
spec:
  filters:
    - kubernetes:
        annotations: true
        kubeCAFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        kubeTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubeURL: 'https://kubernetes.default.svc:443'
        labels: false
    - nest:
        addPrefix: kubernetes_
        nestedUnder: kubernetes
        operation: lift
    - nest:
        addPrefix: kubernetes_annotations_
        nestedUnder: kubernetes_annotations
        operation: lift
    - modify:
        rules:
          - remove: stream
          - remove: kubernetes_pod_id
          - remove: kubernetes_host
          - remove: kubernetes_container_hash
          - rename:
              kubernetes_annotations_cni.projectcalico.org/podIPs: kubernetes_podIPs
          - removeRegex: kubernetes_annotations*
    - nest:
        nestUnder: kubernetes_annotations
        operation: nest
        removePrefix: kubernetes_annotations_
        wildcard:
          - kubernetes_annotations_*
    - nest:
        nestUnder: kubernetes
        operation: nest
        removePrefix: kubernetes_
        wildcard:
          - kubernetes_*
  match: kube.*

由kubernetes Filter CR生成的fluent-bit config配置如下（只看Filter部分秕铛，Input则剃、Output CR被省略）

[Service]
    Parsers_File    parsers.conf
[Input]
    Name    tail
    Path    /var/log/containers/*.log
    Exclude_Path    /var/log/containers/*_kubesphere-logging-system_events-exporter*.log,/var/log/containers/kube-auditing-webhook*_kubesphere-logging-system_kube-auditing-webhook*.log
    Refresh_Interval    10
    Skip_Long_Lines    true
    DB    /fluent-bit/tail/pos.db
    DB.Sync    Normal
    Mem_Buf_Limit    5MB
    Parser    docker
    Tag    kube.*
[Filter]
    Name    kubernetes
    Match    kube.*
    Kube_URL    https://kubernetes.default.svc:443
    Kube_CA_File    /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File    /var/run/secrets/kubernetes.io/serviceaccount/token
    Labels    false
    Annotations    true
[Filter]
    Name    nest
    Match    kube.*
    Operation    lift
    Nested_under    kubernetes
    Add_prefix    kubernetes_
[Filter]
    Name    nest
    Match    kube.*
    Operation    lift
    Nested_under    kubernetes_annotations
    Add_prefix    kubernetes_annotations_
[Filter]
    Name    modify
    Match    kube.*
    Remove    stream
    Remove    kubernetes_pod_id
    Remove    kubernetes_host
    Remove    kubernetes_container_hash
    Rename    kubernetes_annotations_cni.projectcalico.org/podIPs    kubernetes_podIPs
    Remove_regex    kubernetes_annotations*
[Filter]
    Name    nest
    Match    kube.*
    Operation    nest
    Wildcard    kubernetes_annotations_*
    Nest_under    kubernetes_annotations
    Remove_prefix    kubernetes_annotations_
[Filter]
    Name    nest
    Match    kube.*
    Operation    nest
    Wildcard    kubernetes_*
    Nest_under    kubernetes
    Remove_prefix    kubernetes_
[Output]
    Name    es
    Match_Regex    (?:kube|service)\.(.*)
    Host    es-cdc-a-es-http.cdc.svc.xke.test.cn
    Port    9200
    HTTP_User    elastic
    HTTP_Passwd    elasticpwd
    Logstash_Format    true
    Logstash_Prefix    ks-logstash-log
    Time_Key    @timestamp

可以在kibana看到采集到的podIP：

在這里插入圖片描述

作者簡(jiǎn)潔

作者：小碗湯，一位熱愛(ài)如捅、認(rèn)真寫(xiě)作的小伙，目前維護(hù)原創(chuàng)公眾號(hào)：『我的小碗湯』调煎，專注于寫(xiě)go語(yǔ)言镜遣、docker、kubernetes、java等開(kāi)發(fā)悲关、運(yùn)維知識(shí)等提升硬實(shí)力的文章谎僻，期待你的關(guān)注。轉(zhuǎn)載說(shuō)明：務(wù)必注明來(lái)源（注明：來(lái)源于公眾號(hào)：我的小碗湯寓辱，作者：小碗湯）