概要

體系結(jié)構(gòu).jpg

組件簡(jiǎn)介

• spring-actuator
  可以幫助你監(jiān)控和管理Spring Boot應(yīng)用淤年，比如健康檢查瘟栖、審計(jì)葵擎、統(tǒng)計(jì)和HTTP追蹤等
  通過(guò)JMX或者HTTP endpoints來(lái)獲取數(shù)據(jù)

• spring-boot-admin
  對(duì)接spring-actuator，提供展示界面

• micrometer
  提供多種監(jiān)控平臺(tái)可用的JVM應(yīng)用指標(biāo)查詢接口半哟，類似SLF4J

• Prometheus
  可存儲(chǔ)多維時(shí)間序列數(shù)據(jù)的監(jiān)控和報(bào)警工具

• Graphite / InfluxDB / OpenTSDB
  時(shí)間序列數(shù)據(jù)庫(kù)

• Granfana
  可視化指標(biāo)分析平臺(tái)

監(jiān)控搭建

spring-actuator + micrometer + Prometheus + Grafana + <WebHook>

Prometheus

特征

• 多維度數(shù)據(jù)模型-由指標(biāo)鍵值對(duì)標(biāo)識(shí)的時(shí)間序列數(shù)據(jù)組成
• PromQL酬滤，一種靈活的查詢語(yǔ)言
• 不依賴分布式存儲(chǔ); 單個(gè)服務(wù)器節(jié)點(diǎn)是自治的
• 以HTTP方式，通過(guò)pull模型拉取時(shí)間序列數(shù)據(jù)
• 支持通過(guò)中間網(wǎng)關(guān)推送時(shí)間序列數(shù)據(jù)
• 通過(guò)服務(wù)發(fā)現(xiàn)或者靜態(tài)配置寓涨，來(lái)發(fā)現(xiàn)目標(biāo)服務(wù)對(duì)象
• 支持多種多樣的圖表和界面展示

架構(gòu)

prometheus-architecture.jpg

數(shù)據(jù)采集

GET /actuator/prometheus

指標(biāo)查詢接口數(shù)據(jù).jpg

數(shù)據(jù)查詢

prometheus提供了web界面執(zhí)行PromSQL查詢時(shí)序數(shù)據(jù)驾茴，上面的指標(biāo)就可以直接作為查詢條件語(yǔ)句寺旺，且支持多種函數(shù)查詢，參見 https://prometheus.io/docs/prometheus/latest/querying/functions/

promSQL查詢.jpg

預(yù)警規(guī)則

通過(guò)配置文件中的規(guī)則對(duì)每次查詢的時(shí)序數(shù)據(jù)進(jìn)行預(yù)警判定，可以在web界面中展示已加載的規(guī)則項(xiàng)
預(yù)警信息發(fā)送到AlertManager組件中進(jìn)行統(tǒng)一預(yù)警管理

• 規(guī)則配置：

groups:
- name: Instances
  rules:
  - alert: InstanceDown
    expr: up != 1  # 規(guī)則表達(dá)式稍途，支持PromSQL查詢
    for: 1m  # 首次命中規(guī)則1分鐘后發(fā)送報(bào)警，若延遲區(qū)間內(nèi)對(duì)數(shù)據(jù)再次檢查沒(méi)有命中規(guī)則毁兆，就不再報(bào)警
    labels:  # 報(bào)警信息標(biāo)簽
      severity: page # 預(yù)警嚴(yán)重程度想许，后面可以根據(jù)這個(gè)字段抑制某些不需要的告警
      status: High
    annotations:  # 報(bào)警信息描述
      description: "Application: {{ $labels.job }} Instance: {{ $labels.instance }} is Down ! ! !"
      value: '{{ $value }}'
      summary:  "Instance {{ $labels.instance }} down"

• 預(yù)警狀態(tài)
  Inactive： 這里什么都沒(méi)有發(fā)生。
  Pending： 已觸發(fā)閾值沃呢，但未滿足報(bào)警持續(xù)時(shí)間（即rule中的for字段）
  Firing： 已觸發(fā)閾值且滿足告警持續(xù)時(shí)間年栓。警報(bào)發(fā)送到Notification Pipeline，經(jīng)過(guò)處理薄霜，發(fā)送報(bào)警
  Resolved： 預(yù)警已解決某抓，只會(huì)出現(xiàn)報(bào)警的請(qǐng)求報(bào)文中纸兔，在web界面看不到

AlertManager

Prometheus服務(wù)器中的警報(bào)規(guī)則向AlertManager發(fā)送警報(bào)。然后否副，警報(bào)管理器管理這些警報(bào)汉矿，包括沉默、抑制副编、聚合和通過(guò)電子郵件负甸、待命通知系統(tǒng)和聊天平臺(tái)等方法發(fā)送通知
這里我們選擇使用WebHook方式，將報(bào)警信息發(fā)送到指定接口痹届，我們可以針對(duì)報(bào)警數(shù)據(jù)自行選擇通知方式和通知人

• 報(bào)警請(qǐng)求數(shù)據(jù)

{
  "receiver": "web\\.hook",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "InstanceDown",
        "instance": "10.10.10.10:9000",
        "job": "app-1",
        "severity": "page",
        "status": "High"
      },
      "annotations": {
        "description": "Application: bdp-gateway Instance: 10.10.10.10:9000 is Down ! ! !",
        "summary": "Instance 10.10.10.10:9000 down",
        "value": "0"
      },
      "startsAt": "2021-02-20T09:11:43.380766777Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "http://VM-102-32-centos:9090/graph?g0.expr=up+%21%3D+1\u0026g0.tab=1",
      "fingerprint": "8a9aadd8d34d09f7"
    },
    {
      "status": "resolved",
      "labels": {
        "alertname": "InstanceDown",
        "instance": "10.10.10.10:9001",
        "job": "app-2",
        "severity": "page",
        "status": "High"
      },
      "annotations": {
        "description": "Application: app-2 Instance: 10.10.10.10:9001 is Down ! ! !",
        "summary": "Instance 10.10.10.10:9001 down",
        "value": "0"
      },
      "startsAt": "2021-02-20T09:11:28.380766777Z",
      "endsAt": "2021-02-20T09:13:43.380766777Z",
      "generatorURL": "http://VM-102-32-centos:9090/graph?g0.expr=up+%21%3D+1\u0026g0.tab=1",
      "fingerprint": "6070b8cb7389ffc2"
    }
  ],
  "groupLabels": {
    "alertname": "InstanceDown"
  },
  "commonLabels": {
    "alertname": "InstanceDown",
    "severity": "page",
    "status": "High"
  },
  "commonAnnotations": {
    "value": "0"
  },
  "externalURL": "http://VM-102-32-centos:9093",
  "version": "4",
  "groupKey": "{}:{alertname=\"InstanceDown\"}",
  "truncatedAlerts": 0
}

Grafana

用于可視化大型測(cè)量數(shù)據(jù)的開源程序呻待，他提供了強(qiáng)大和優(yōu)雅的方式去創(chuàng)建、共享队腐、瀏覽數(shù)據(jù)蚕捉。dashboard中顯示了你不同metric數(shù)據(jù)源中的數(shù)據(jù)

數(shù)據(jù)源

• 多種時(shí)序數(shù)據(jù)庫(kù)（Prometheus、Graphite柴淘、OpenTSDB迫淹、InfluxDB）

• 文檔數(shù)據(jù)庫(kù)（ElasticSearch）

• 分布式追蹤系統(tǒng)（Jaeger、Zipkin为严、Tempo）

• SQL（MySQL敛熬、PostgreSQL、Sql Server）

• 等等

• 節(jié)點(diǎn)儀表盤
  

  節(jié)點(diǎn)儀表盤.jpg

• 應(yīng)用健康表
  

  應(yīng)用健康表.jpg

部署

應(yīng)用端加入spring-actuator和micrometer組件

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

安裝Prometheus & AlertManager

• 官網(wǎng)下載解壓縮后直接運(yùn)行第股，這里我們只需要用到prometheus和alertmanager
  https://prometheus.io/download/
  prometheus默認(rèn)端口9090应民，alertmanager默認(rèn)端口9093
  
  目錄結(jié)構(gòu).jpg

• 創(chuàng)建prometheus啟動(dòng)腳本start.sh

nohup ./prometheus --web.enable-lifecycle 2>&1 & 

• 創(chuàng)建prometheus熱加載腳本reload.sh

curl -XPOST http://localhost:9090/-/reload 

• Prometheus.yml

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093
      - 127.0.0.1:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"
    - "./rules/*.yml"
    
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'app-1'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    metrics_path: '/actuator/prometheus'
    static_configs:
    - targets: ['10.10.10.10:9000', '10.10.10.11:9000']

  - job_name: 'app-2'
    metrics_path: '/actuator/prometheus'
    static_configs:
    - targets: ['10.10.10.11:9001']

• 創(chuàng)建alertmanager啟動(dòng)腳本start.sh

nohup ./alertmanager 2>&1 & 

• 創(chuàng)建alertmanager熱加載腳本reload.sh

curl -XPOST http://localhost:9093/-/reload 

• alertmanager.yml

route:
  group_by: ['alertname']  # 報(bào)警分組依據(jù)字段
  group_wait: 20s  # 收到新組時(shí)等待時(shí)間，目的是為了等待同組的警報(bào)合并發(fā)送報(bào)警
  group_interval: 5m  # 同組報(bào)警發(fā)送的間隔時(shí)間夕吻，從上次發(fā)送報(bào)警的時(shí)間開始計(jì)算
  repeat_interval: 3m  # 報(bào)警發(fā)送間隔
  receiver: 'web.hook'  # 接收?qǐng)?bào)警的名稱
receivers:
  - name: 'web.hook'
    webhook_configs:
    - send_resolved: false  # 已報(bào)警的指標(biāo)恢復(fù)后是否通知诲锹，默認(rèn)true
      url: 'http://10.10.10.11:8080/xxx/xxx'  #報(bào)警通知接口
inhibit_rules:  # 告警抑制配置，避免當(dāng)某種問(wèn)題告警產(chǎn)生之后用戶接收到大量由此問(wèn)題導(dǎo)致的一系列的其它告警通知
  - source_match:  # 源報(bào)警規(guī)則
      severity: 'critical'
    target_match:  # 抑制的報(bào)警規(guī)則
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']  # 需要都有相同的標(biāo)簽及值涉馅，否則抑制不起作用

安裝Grafana

• 官網(wǎng)下載解壓縮后直接運(yùn)行 ./bin/grafana-server
  https://grafana.com/grafana/download
  默認(rèn)端口3000
  

  grafana目錄結(jié)構(gòu).jpg

• 創(chuàng)建Prometheus數(shù)據(jù)源

  
  
  grafana添加數(shù)據(jù)源.jpg

• 導(dǎo)入儀表盤
  儀表盤可以使用文件導(dǎo)入归园，在官網(wǎng)上可以查詢想要的儀表盤下載后導(dǎo)入到自己的服務(wù)器上
  https://grafana.com/grafana/dashboards/12856
  

  grafana導(dǎo)入儀表盤.jpg