說明
promtail 是loki的日志收集agent劫灶，也是類似于 promtheus 的服務(wù)發(fā)現(xiàn)機(jī)制划址，應(yīng)該是最云原生的日志agent了扔嵌。贊一個
安裝
# helm pull loki/promtail --version=2.0.1
# helm show values promtail-2.0.1.tgz
生成 promtail 配置文件：
配置參考：https://grafana.com/docs/loki/latest/clients/promtail/configuration/
# cat > promtail-config.yaml <<EOF
deploymentStrategy:
rollingUpdate:
maxUnavailable: 100%
type: RollingUpdate
image:
#repository: grafana/promtail
repository: ops-harbor.hupu.io/k8s/promtail
tag: 2.0.0
pullPolicy: IfNotPresent
hostNetwork: true
podSecurityPolicy:
hostNetwork: true
#livenessProbe: {}
readinessProbe:
failureThreshold: 5
httpGet:
path: /ready
port: http-metrics
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
loki:
serviceName: "distributor-loki-headless"
servicePort: 3100
serviceScheme: http
# user: user
# password: pass
pipelineStages:
- docker: {}
podAnnotations:
prometheus.io/scrape: "true"
prometheus.io/port: "http-metrics"
resources:
limits:
cpu: 4000m
memory: 4Gi
requests:
cpu: 100m
memory: 100Mi
# Custom scrape_configs to override the default ones in the configmap
scrapeConfigs: []
#- job_name: kubernetes-pods-name
# # 管道階段用于轉(zhuǎn)換日志條目及其標(biāo)簽。發(fā)現(xiàn)過程完成后夺颤，將執(zhí)行管道
# # 支持管道：docker痢缎、cri、regex世澜、json独旷、template、match寥裂、timestamp嵌洼、output、labels封恰、metrics麻养、tenant
# pipeline_stages:
# - docker: {}
# kubernetes_sd_configs:
# - role: pod
# relabel_configs:
# - source_labels:
# - __meta_kubernetes_pod_label_name
# target_label: __service__
# - source_labels:
# - __meta_kubernetes_pod_node_name
# target_label: __host__
# - action: drop
# regex: ''
# source_labels:
# - __service__
# - action: labelmap
# regex: __meta_kubernetes_pod_label_(.+)
# - action: replace
# replacement: $1
# separator: /
# source_labels:
# - __meta_kubernetes_namespace
# - __service__
# target_label: job
# - action: replace
# source_labels:
# - __meta_kubernetes_namespace
# target_label: namespace
# - action: replace
# source_labels:
# - __meta_kubernetes_pod_name
# target_label: pod
# - action: replace
# source_labels:
# - __meta_kubernetes_pod_container_name
# target_label: container
# # 示例日志路徑：/var/log/pods/kube-system_kube-proxy-master-w5jxb_f395f3be-67a3-43e7-ae99-9d099fe23486/kube-proxy-master/0.log
# - replacement: /var/log/pods/*$1/*.log
# # 指定分隔符為 /，而不是默認(rèn)的 ;
# separator: /
# source_labels:
# # 取到 pod uid
# - __meta_kubernetes_pod_uid
# # 取到 pod 中的容器名
# - __meta_kubernetes_pod_container_name
# target_label: __path__
# #journal:
# #syslog:
# #loki_push_api:
# #static_configs:
# #file_sd_configs:
# Custom scrape_configs together with the default ones in the configmap
extraScrapeConfigs:
# 收集類似 /data0/log-data/basketball-all-api/stg/basketball-all-api-20201026182255-8dd968e-75c9bc469f-wtfxh/e4179a7b-d505-4b7b-ae66-ed62906fa596/basketball-all-api/ 路徑的 msv-access.log
#
# 1. 標(biāo)準(zhǔn)輸出 log：
# {app="ad-direct-msv",filename="/var/log/pods/stg_ad-direct-msv-20201116143649-2355d8c-777d74c8b-pqvzn_3e121e4d-16f9-4257-8e35-b05137d60443/ad-direct-msv/0.log"}
#
# 2. 寫文件 log：
# {app="ad-direct-msv",filename="/data0/log-data/ad-direct-msv/stg/ad-direct-msv-20201116143649-2355d8c-777d74c8b-pqvzn/3e121e4d-16f9-4257-8e35-b05137d60443/ad-direct-msv/msv-access.log"}
#
# 對于標(biāo)準(zhǔn)輸出和寫文件的 log 都有以下一致的標(biāo)簽诺舔，這樣就解決了 pod 元數(shù)據(jù)問題：
# 1. pod 自帶的標(biāo)簽：
# ad-direct-msv -- app
# java-msv -- apptype
# 2020-11-16-14-42-38-362580 -- datetime
# stg -- env
# 777d74c8b -- pod-template-hash
# ad-direct-msv-20201116143649-2355d8c -- version
#
# 2. 額外添加的標(biāo)簽：
# stg -- namespace
# ad-direct-msv -- container
# stg/ad-direct-msv -- job
# ad-direct-msv-20201116143649-2355d8c-777d -- pod
#
- job_name: alllog
pipeline_stages:
static_configs:
- targets:
- localhost
labels:
job: alllog
__path__: /data0/log-data/*/*/*/*/*.log
- job_name: alllog-next
pipeline_stages:
static_configs:
- targets:
- localhost
labels:
job: alllog-next
__path__: /data0/log-data/*/*/*/*/*/*.log
- job_name: kubernetes-pods-name-msv-access
# loki_push_api 將 Promtail 配置為公開 Loki 推送 API 服務(wù)器
# 注意：使用 loki_push_api 配置的每個 job 都將公開此 API鳖昌，并且需要一個單獨的端口
#loki_push_api:
# # 服務(wù)器配置與服務(wù)器相同
# server:
# http_listen_port: 3500
# grpc_listen_port: 3600
# labels:
# pushserver: push1
# use_incoming_timestamp: false
pipeline_stages:
- docker: {}
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels:
- __meta_kubernetes_pod_label_name
target_label: __service__
- source_labels:
- __meta_kubernetes_pod_node_name
target_label: __host__
- action: drop
regex: ''
source_labels:
- __service__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- action: replace
replacement: $1
separator: /
source_labels:
- __meta_kubernetes_namespace
- __service__
target_label: job
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: container
#- replacement: /var/log/pods/*$1/*.log
- replacement: /data0/log-data/*/*/*/$1/msv-access.log
separator: /
source_labels:
- __meta_kubernetes_pod_uid
- __meta_kubernetes_pod_container_name
target_label: __path__
- job_name: kubernetes-pods-app-msv-access
pipeline_stages:
- docker: {}
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: drop
regex: .+
source_labels:
- __meta_kubernetes_pod_label_name
- source_labels:
- __meta_kubernetes_pod_label_app
target_label: __service__
- source_labels:
- __meta_kubernetes_pod_node_name
target_label: __host__
- action: drop
regex: ''
source_labels:
- __service__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- action: replace
replacement: $1
separator: /
source_labels:
- __meta_kubernetes_namespace
- __service__
target_label: job
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: container
#- replacement: /var/log/pods/*$1/*.log
- replacement: /data0/log-data/*/*/*/$1/msv-access.log
separator: /
source_labels:
- __meta_kubernetes_pod_uid
- __meta_kubernetes_pod_container_name
target_label: __path__
- job_name: kubernetes-pods-direct-controllers-msv-access
pipeline_stages:
- docker: {}
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: drop
regex: .+
separator: ''
source_labels:
- __meta_kubernetes_pod_label_name
- __meta_kubernetes_pod_label_app
- action: drop
regex: '[0-9a-z-.]+-[0-9a-f]{8,10}'
source_labels:
- __meta_kubernetes_pod_controller_name
- source_labels:
- __meta_kubernetes_pod_controller_name
target_label: __service__
- source_labels:
- __meta_kubernetes_pod_node_name
target_label: __host__
- action: drop
regex: ''
source_labels:
- __service__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- action: replace
replacement: $1
separator: /
source_labels:
- __meta_kubernetes_namespace
- __service__
target_label: job
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: container
#- replacement: /var/log/pods/*$1/*.log
- replacement: /data0/log-data/*/*/*/$1/msv-access.log
separator: /
source_labels:
- __meta_kubernetes_pod_uid
- __meta_kubernetes_pod_container_name
target_label: __path__
- job_name: kubernetes-pods-indirect-controller-msv-access
pipeline_stages:
- docker: {}
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: drop
regex: .+
separator: ''
source_labels:
- __meta_kubernetes_pod_label_name
- __meta_kubernetes_pod_label_app
- action: keep
regex: '[0-9a-z-.]+-[0-9a-f]{8,10}'
source_labels:
- __meta_kubernetes_pod_controller_name
- action: replace
regex: '([0-9a-z-.]+)-[0-9a-f]{8,10}'
source_labels:
- __meta_kubernetes_pod_controller_name
target_label: __service__
- source_labels:
- __meta_kubernetes_pod_node_name
target_label: __host__
- action: drop
regex: ''
source_labels:
- __service__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- action: replace
replacement: $1
separator: /
source_labels:
- __meta_kubernetes_namespace
- __service__
target_label: job
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: container
#- replacement: /var/log/pods/*$1/*.log
- replacement: /data0/log-data/*/*/*/$1/msv-access.log
separator: /
source_labels:
- __meta_kubernetes_pod_uid
- __meta_kubernetes_pod_container_name
target_label: __path__
- job_name: kubernetes-pods-static-msv-access
pipeline_stages:
- docker: {}
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: drop
regex: ''
source_labels:
- __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror
- action: replace
source_labels:
- __meta_kubernetes_pod_label_component
target_label: __service__
- source_labels:
- __meta_kubernetes_pod_node_name
target_label: __host__
- action: drop
regex: ''
source_labels:
- __service__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- action: replace
replacement: $1
separator: /
source_labels:
- __meta_kubernetes_namespace
- __service__
target_label: job
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: container
#- replacement: /var/log/pods/*$1/*.log
- replacement: /data0/log-data/*/*/*/$1/msv-access.log
separator: /
source_labels:
- __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror
- __meta_kubernetes_pod_container_name
target_label: __path__
tolerations:
- operator: "Exists"
securityContext:
readOnlyRootFilesystem: true
runAsGroup: 0
runAsUser: 0
# Extra volumes to scrape logs from
volumes:
# docker log
- name: docker
hostPath:
path: /var/lib/docker/containers
# pod log
- name: pods
hostPath:
path: /var/log/pods
# Pod 應(yīng)用寫日志文件 log
- name: data0
hostPath:
path: /data0/log-data
volumeMounts:
- name: docker
mountPath: /var/lib/docker/containers
readOnly: true
- name: pods
mountPath: /var/log/pods
readOnly: true
- name: data0
mountPath: /data0/log-data
readOnly: true
config:
# 描述 Promtail 如何連接到 Loki 的多個實例，并向每個實例發(fā)送日志低飒。
# 警告：如果其中一臺遠(yuǎn)程 Loki 服務(wù)器無法響應(yīng)或發(fā)生任何可重試的錯誤许昨，這將影響將日志發(fā)送到任何其他已配置的遠(yuǎn)程 Loki 服務(wù)器。發(fā)送是在單個線程上完成的褥赊！
# 如果要發(fā)送到多個遠(yuǎn)程 Loki 實例车要，通常建議并行運行多個 Promtail 客戶端。
# 注意：官方文檔中的 clinets 字段是錯誤的
client:
url: http://distributor-loki-headless:3100/loki/api/v1/push
# 日志條目在被推送到 Loki 之前會根據(jù)最大批處理持續(xù)時間 batchwait 和大小 batchsize 進(jìn)行批處理崭倘，以先到者為準(zhǔn)
# Maximum wait period before sending batch
batchwait: 5s
# Maximum batch size to accrue before sending, unit is byte
# 每個流日志達(dá)到 400KB 發(fā)送
batchsize: 409600
# Maximum time to wait for server to respond to a request
timeout: 1m
# 注意：Loki 會拒絕接收到的所有順序不正確的日志行
backoff_config:
# Initial backoff time between retries
min_period: 10s
# Maximum backoff time between retries
max_period: 5m
# Maximum number of retries when sending batches, 0 means infinite retries
# 如果所有重試均失敗翼岁，則 promtail 會丟棄該批次的日志條目（這些條目將丟失）并繼續(xù)下一個
max_retries: 10
# The labels to add to any time series or alerts when communicating with loki
external_labels:
env: prod
server:
#http_listen_address:
http_listen_port: 3101
#grpc_listen_address:
grpc_listen_port: 9095
graceful_shutdown_timeout: 60s
http_server_read_timeout: 60s
http_server_write_timeout: 60s
http_server_idle_timeout: 120s
# 設(shè)置 200 MB
grpc_server_max_recv_msg_size: 209715200
grpc_server_max_send_msg_size: 209715200
grpc_server_max_concurrent_streams: 1024
log_level: info
health_check_target: true
# 描述如何將讀取的文件偏移量保存到磁盤
positions:
filename: /run/promtail/positions.yaml
sync_period: 5s
ignore_invalid_yaml: false
target_config:
# Period to resync directories being watched and files being tailed
sync_period: 5s
serviceMonitor:
enabled: true
interval: 30s
additionalLabels: {}
annotations: {}
scrapeTimeout: 25s
EOF
====================================================================================================================================
備注：
docker: {}
docker 階段將匹配并解析此格式的日志行：
`{"log":"level=info ts=2019-04-30T02:12:41.844179Z caller=filetargetmanager.go:180 msg=\"Adding target\"\n","stream":"stderr","time":"2019-04-30T02:12:41.8443515Z"}`
自動將時間提取到日志 timestamp 中类垫，stream 傳輸?shù)綐?biāo)簽中，并將日志字段輸出到 output 中琅坡，因為 docker以這種方式包裝您的應(yīng)用程序日志悉患，這將對其進(jìn)行解包，以便僅對日志內(nèi)容進(jìn)行進(jìn)一步的管道處理 榆俺。
Docker 階段只是如下定義的包裝：
- json:
output: log
stream: stream
timestamp: time
- labels:
stream:
- timestamp:
source: timestamp
format: RFC3339Nano
- output:
source: output
kubernetes_sd_config：
kubernetes_sd_config 發(fā)現(xiàn)規(guī)則跟 prometheus 一樣的包括：
node：
__meta_kubernetes_node_name: The name of the node object.
_meta_kubernetes_node_label<labelname>: Each label from the node object.
_meta_kubernetes_node_labelpresent<labelname>: true for each label from the node object.
_meta_kubernetes_node_annotation<annotationname>: Each annotation from the node object.
_meta_kubernetes_node_annotationpresent<annotationname>: true for each annotation from the node object.
_meta_kubernetes_node_address<address_type>: The first address for each node address type, if it exists.service：
__meta_kubernetes_namespace: The namespace of the service object.
_meta_kubernetes_service_annotation<annotationname>: Each annotation from the service object.
_meta_kubernetes_service_annotationpresent<annotationname>: true for each annotation of the service object.
__meta_kubernetes_service_cluster_ip: The cluster IP address of the service. (Does not apply to services of type ExternalName)
__meta_kubernetes_service_external_name: The DNS name of the service. (Applies to services of type ExternalName)
_meta_kubernetes_service_label<labelname>: Each label from the service object.
_meta_kubernetes_service_labelpresent<labelname>: true for each label of the service object.
__meta_kubernetes_service_name: The name of the service object.
__meta_kubernetes_service_port_name: Name of the service port for the target.
__meta_kubernetes_service_port_protocol: Protocol of the service port for the target.pod：
__meta_kubernetes_namespace: The namespace of the pod object.
__meta_kubernetes_pod_name: The name of the pod object.
__meta_kubernetes_pod_ip: The pod IP of the pod object.
_meta_kubernetes_pod_label<labelname>: Each label from the pod object.
_meta_kubernetes_pod_labelpresent<labelname>: true for each label from the pod object.
_meta_kubernetes_pod_annotation<annotationname>: Each annotation from the pod object.
_meta_kubernetes_pod_annotationpresent<annotationname>: true for each annotation from the pod object.
__meta_kubernetes_pod_container_init: true if the container is an InitContainer
__meta_kubernetes_pod_container_name: Name of the container the target address points to.
__meta_kubernetes_pod_container_port_name: Name of the container port.
__meta_kubernetes_pod_container_port_number: Number of the container port.
__meta_kubernetes_pod_container_port_protocol: Protocol of the container port.
__meta_kubernetes_pod_ready: Set to true or false for the pod’s ready state.
__meta_kubernetes_pod_phase: Set to Pending, Running, Succeeded, Failed or Unknown in the lifecycle.
__meta_kubernetes_pod_node_name: The name of the node the pod is scheduled onto.
__meta_kubernetes_pod_host_ip: The current host IP of the pod object.
__meta_kubernetes_pod_uid: The UID of the pod object.
__meta_kubernetes_pod_controller_kind: Object kind of the pod controller.
__meta_kubernetes_pod_controller_name: Name of the pod controller.endpoints：
__meta_kubernetes_namespace: The namespace of the endpoints object.
__meta_kubernetes_endpoints_name: The names of the endpoints object.
對于直接從端點列表中發(fā)現(xiàn)的所有目標(biāo)（未從基礎(chǔ) pod 中另外推斷出的那些目標(biāo)）售躁，將附加以下標(biāo)簽：
__meta_kubernetes_endpoint_hostname: Hostname of the endpoint.
__meta_kubernetes_endpoint_node_name: Name of the node hosting the endpoint.
__meta_kubernetes_endpoint_ready: Set to true or false for the endpoint’s ready state.
__meta_kubernetes_endpoint_port_name: Name of the endpoint port.
__meta_kubernetes_endpoint_port_protocol: Protocol of the endpoint port.
__meta_kubernetes_endpoint_address_target_kind: Kind of the endpoint address target.
__meta_kubernetes_endpoint_address_target_name: Name of the endpoint address target.
注意：如果端點屬于服務(wù)，則會附加角色：服務(wù)發(fā)現(xiàn)的所有標(biāo)簽茴晋。
注意：對于由 Pod 支持的所有目標(biāo)陪捷，將附加角色的所有標(biāo)簽：Pod 發(fā)現(xiàn)的所有標(biāo)簽。
- ingress：
__meta_kubernetes_namespace: The namespace of the ingress object.
__meta_kubernetes_ingress_name: The name of the ingress object.
_meta_kubernetes_ingress_label<labelname>: Each label from the ingress object.
_meta_kubernetes_ingress_labelpresent<labelname>: true for each label from the ingress object.
_meta_kubernetes_ingress_annotation<annotationname>: Each annotation from the ingress object.
_meta_kubernetes_ingress_annotationpresent<annotationname>: true for each annotation from the ingress object.
__meta_kubernetes_ingress_scheme: Protocol scheme of ingress, https if TLS config is set. Defaults to http.
__meta_kubernetes_ingress_path: Path from ingress spec. Defaults to /.
====================================================================================================================================
安裝 promtail
# helm install promtail promtail-2.0.1.tgz -f promtail-config.yaml -n grafana
# kubectl get all -n grafana
NAME READY STATUS RESTARTS AGE
pod/loki-0 1/1 Running 0 136m
pod/promtail-2b5rt 1/1 Running 0 49s
pod/promtail-58llq 1/1 Running 0 49s
pod/promtail-bvp6q 1/1 Running 0 49s
pod/promtail-cr5hv 1/1 Running 0 49s
pod/promtail-dfq44 1/1 Running 0 49s
pod/promtail-jm9wc 1/1 Running 0 49s
pod/promtail-kl5wr 1/1 Running 0 49s
pod/promtail-mrxjc 1/1 Running 0 49s
pod/promtail-qklwp 1/1 Running 0 49s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/loki ClusterIP 10.96.116.194 <none> 3100/TCP 136m
service/loki-headless ClusterIP None <none> 3100/TCP 136m
service/promtail-headless ClusterIP None <none> 3101/TCP 51s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/promtail 9 9 9 9 9 <none> 51s
NAME READY AGE
statefulset.apps/loki 1/1 136m
修改為 hostNetwork：
# kubectl edit ds -n grafana promtail
...
spec:
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
Promtail 公開了幾個 URL诺擅，可用于了解其服務(wù)發(fā)現(xiàn)的工作方式：
- /service-discovery 服務(wù)發(fā)現(xiàn)頁面市袖，顯示在 relabeling 之前和之后所有發(fā)現(xiàn)的目標(biāo)及其標(biāo)簽，以及刪除目標(biāo)的原因烁涌。
- /targets 目標(biāo)頁面苍碟，僅顯示正在主動抓取的目標(biāo)及其相應(yīng)的標(biāo)簽，文件和位置撮执。
- /ready 健康檢查接口
- /metrics metrics 接口
- /static/ 一些靜態(tài)頁微峰，沒啥用
---
apiVersion: v1
kind: Service
metadata:
labels:
app: promtail
name: promtail
namespace: grafana
spec:
ports:
- name: http-metrics
port: 3101
protocol: TCP
targetPort: http-metrics
selector:
app: promtail
release: promtail
sessionAffinity: None
type: ClusterIP
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: traefik
name: promtail
namespace: grafana
spec:
rules:
- host: promtail.hupu.io
http:
paths:
- backend:
serviceName: promtail
servicePort: 3101
path: /
查看默認(rèn)的任務(wù)：
# kubectl get cm -n grafana promtail -o yaml |grep job_name
- job_name: kubernetes-pods-name
- job_name: kubernetes-pods-app
- job_name: kubernetes-pods-direct-controllers
- job_name: kubernetes-pods-indirect-controller
- job_name: kubernetes-pods-static
查看配置詳情：
# kubectl get cm -n grafana promtail -o yaml
試運行 Promtail：
注意：還可以添加標(biāo)簽
# cat my.log | promtail --stdin --client.url http://127.0.0.1:3100/loki/api/v1/push --dry-run
# cat my.log | promtail --stdin --client.url http://127.0.0.1:3100/loki/api/v1/push -server.grpc-listen-port 9999 -positions.file /tmp/2.yaml --stdin -log.level debug
# cat my.log | promtail --stdin --client.url http://127.0.0.1:3100/loki/api/v1/push --client.external-labels=k1=v1,k2=v2
發(fā)現(xiàn) promtail 性能比 filebeat 好，promtail 還收集了所有的 .log 日志抒钱， filebeat 只收集了一部分：
top - 21:54:40 up 104 days, 1:06, 1 user, load average: 30.98, 31.73, 32.47
Tasks: 472 total, 2 running, 470 sleeping, 0 stopped, 0 zombie
%Cpu(s): 67.4/10.8 78[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
KiB Mem : 65806668 total, 3318964 free, 34830892 used, 27656812 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 30111884 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22705 root 20 0 18.5g 5.8g 25760 S 389.1 9.3 1317:56 java
46827 root 20 0 2557116 371756 17212 S 238.3 0.6 17288:44 filebeat
44872 root 20 0 19.0g 9.2g 18904 S 169.6 14.6 25432:15 java
55659 root 20 0 8576336 102280 22444 S 160.4 0.2 243:31.13 promtail
監(jiān)控邏輯
- 您的應(yīng)用向 Promtail 跟蹤的文件發(fā)出一條日志行
- Promtail 讀取新行并增加其計數(shù)器
- Promtail 將日志行轉(zhuǎn)發(fā)給 distributor蜓肆，在該 distributor 中應(yīng)增加收到的計數(shù)器
- distributor 將日志行轉(zhuǎn)發(fā)到 ingester，請求持續(xù)時間計數(shù)器應(yīng)在此增加
