整體架構(gòu)

1. promethueus: 一個開源的服務(wù)監(jiān)控系統(tǒng)和時間序列數(shù)據(jù)庫崭闲。附帶豐富的數(shù)據(jù)采集器寨躁，支持告警處理五嫂。
2. grafana： 一款開源的數(shù)據(jù)可視化工具苞冯，可以非常輕松的將數(shù)據(jù)轉(zhuǎn)成圖表的展現(xiàn)形式來做到數(shù)據(jù)監(jiān)控以及數(shù)據(jù)統(tǒng)計袖牙。支持promethueus、influxDB等多種數(shù)據(jù)源舅锄。

prometheus-architecture.png

基礎(chǔ)安裝

本文采用的主要安裝方式為docker-compose

prometheus

version: '3.8'

services:
  prometheus:
    image: prom/prometheus:v2.37.6
    ports:
      - 9090:9090
    volumes:
      - ./config:/etc/prometheus
      - ./prom-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus/data'
      - '--web.console.libraries=/prometheus/console_libraries'
      - '--web.console.templates=/prometheus/consoles'
    user: '0'

grafana

1. docker-compose啟動

   version: '3.8'
   
   services:
     grafana:
       image: grafana/grafana-oss:9.4.3
       ports:
         - 3000:3000
       volumes:
         - ./gfdata:/var/lib/grafana
         - TZ=Asia/Shanghai
       environment:
         - GF_SERVER_ROOT_URL=http://100.127.9.84:3000/
       user: '0'
   

2. 瀏覽器訪問http://{host}:3000,使用admin/admin登錄鞭达；

3. 根據(jù)提示修改密碼司忱；

4. Config -> Data Sources, 添加Prometheus數(shù)據(jù)源：

image-20230310120537493.png

監(jiān)控配置

服務(wù)器

1. 每臺服務(wù)器分別安裝node-exporter

   wget https://github.com/prometheus/node_exporter/releases/download/v1.5.0/node_exporter-1.5.0.linux-amd64.tar.gz
   tar -xvf node_exporter-1.5.0.linux-amd64.tar.gz & mv node_exporter-1.5.0.linux-amd64 node_exporter
   cd node_exporter
   nohup ./node_exporter >/dev/null 2>&1 &
   

2. prometheus.yml配置job，并重啟prometheus

   scrape_configs:
     - job_name: "node"
       static_configs:
         - targets: ['192.168.0.184:9100','192.168.0.59:9100','192.168.0.182:9100','192.168.1.10:9100','192.168.3.220:9100']
   

3. grafana導(dǎo)入dashboard模板
   node-exporter-full_rev30.json

4. 效果展示

image-20230310132748412.png

mysql

1. 創(chuàng)建用戶并配置權(quán)限

   CREATE USER 'exporter'@'%' IDENTIFIED BY 'xxxxxx' WITH MAX_USER_CONNECTIONS 3;
   GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'exporter'@'%';
   

2. 安裝mysqld-exporter

   version: '3.8'
   
   services:
     mysqld-exporter:
       image: prom/mysqld-exporter:v0.14.0
       ports:
         - "9104:9104"
       environment:
         - DATA_SOURCE_NAME=exporter:xxxxxx@(host:port)/
   

3. prometheus.yml配置job畴蹭，并重啟prometheus

   scrape_configs:
     - job_name: "mysql"
       static_configs:
         - targets:
           # All mysql hostnames to monitor.
           - 192.168.0.184:2206
       relabel_configs:
         - source_labels: [__address__]
           target_label: __param_target
         - source_labels: [__param_target]
           target_label: instance
         - target_label: __address__
           # The mysqld_exporter host:port
           replacement: 192.168.3.39:9104
   

4. grafana導(dǎo)入dashboard模板
   mysql-overview.json

5. 效果展示

image-20230310133226180.png

mongodb

1. 創(chuàng)建用戶并配置權(quán)限

   db.getSiblingDB("admin").createUser({
       user: "exporter",
       pwd: "xxxxxx",
       roles: [
           { role: "clusterMonitor", db: "admin" },
           { role: "read", db: "local" }
       ]
   })
   

2. 下載mongodb-exporter二進(jìn)制文件

   wget https://github.com/percona/mongodb_exporter/releases/download/v0.11.0/mongodb_exporter-0.11.0.linux-amd64.tar.gz
   tar -xvf mongodb_exporter-0.11.0.linux-amd64.tar.gz
   

3. 編寫Dockerfile

   FROM alpine:3.4
   EXPOSE 9216
   
   RUN apk add --update ca-certificates
   COPY mongodb_exporter /bin/mongodb_exporter
   RUN chmod 755 /bin/mongodb_exporter
   
   CMD [ "mongodb_exporter" ]
   

4. 安裝mongodb-exporter

   version: '3.8'
   
   services:
     mongodb-exporter:
       build:
         context: ./
         dockerfile: Dockerfile
       image: percona/mongodb_exporter:0.11.0
       ports:
         - 9216:9216
       environment:
         - MONGODB_URI=mongodb://exporter:xxxxxx@host:port/?authSource=admin
   

5. prometheus.yml配置job坦仍，并重啟prometheus

   scrape_configs:
     - job_name: "mongodb"
       static_configs:
         - targets: ["192.168.3.39:9216"]
   

6. grafana導(dǎo)入dashboard模板
   mongodb-overview_rev1.json

7. 效果展示

image-20230310134222718.png

redis

1. 安裝redis-exporter

   version: '3.8'
   
   services:
     redis-exporter:
       image: oliver006/redis_exporter:v1.48.0
       ports:
         - "9121:9121"
       environment:
         - REDIS_ADDR=redis://host:port # 連接集群中一個節(jié)點
         - REDIS_PASSWORD=xxxxxxx
         - REDIS_EXPORTER_INCL_SYSTEM_METRICS=true
   

2. prometheus.yml配置job，并重啟prometheus

   scrape_configs:
     - job_name: 'redis_exporter_targets'
       static_configs:
         - targets:
           - redis://192.168.0.182:8001
           - redis://192.168.0.182:8002
           - redis://192.168.0.59:8001
           - redis://192.168.0.59:8002
           - redis://192.168.0.184:8001
           - redis://192.168.0.184:8002
       metrics_path: /scrape
       relabel_configs:
         - source_labels: [__address__]
           target_label: __param_target
         - source_labels: [__param_target]
           target_label: instance
         - target_label: __address__
           replacement: 192.168.3.39:9121
   
     ## config for scraping the exporter itself
     - job_name: 'redis_exporter'
       static_configs:
         - targets:
           - 192.168.3.39:9121
   

3. grafana導(dǎo)入dashboard模板
   redis-dashboard.json

4. 效果展示

image-20230310134847117.png

rocketmq

1. 構(gòu)建rocketmq-exporter

   curl -O https://github.com/apache/rocketmq-exporter/archive/refs/heads/master.zip
   unzip master.zip
   cd rocketmq-exporter-master
   mvn package
   cd target
   

   或者下載我構(gòu)建好的制品:

   rocketmq-exporter-0.0.2-SNAPSHOT-exec.jar

2. 編寫Dockerfile

   FROM openjdk:11
   EXPOSE 5557
   
   COPY rocketmq-exporter-0.0.2-SNAPSHOT-exec.jar /usr/local/rocketmq-exporter.jar
   
   CMD [ "java", "-jar", "/usr/local/rocketmq-exporter.jar" ]
   

3. 安裝rocketmq-exporter

   version: '3.8'
   
   services:
     rockermq-exporter:
       build:
         context: ./
         dockerfile: Dockerfile
       image: apache/rocketmq_exporter:0.0.2
       ports:
         - 5557:5557
       environment:
         - ROCKETMQ_CONFIG_NAMESRVADDR=host:port
         - ROCKETMQ_CONFIG_ROCKETMQ_VERSION=V4_4_0
   

4. prometheus.yml配置job叨襟，并重啟prometheus

   scrape_configs:
     - job_name: 'rockermq_exporter'
       static_configs:
         - targets:
           - 192.168.3.39:5557
   

5. grafana導(dǎo)入dashboard模板
   rocketmq-dashboard.json

6. 查看效果

c530acf5ba82fa9196afdab371abc36.png

minIO

Prometheus + Grafana Minio監(jiān)控

docker容器

cAdvisor + Prometheus + Grafana docker容器監(jiān)控

elasticsearch

1. 安裝elasticsearch-exporter

   version: '3.8'
   
   services:
     elasticsearch_exporter:
       image: prometheuscommunity/elasticsearch-exporter:v1.2.1
       command:
         - '--es.uri=http://user1:password1@192.168.0.59:9200' # 連接集群中的一個節(jié)點
         - '--es.all'
         - '--es.indices'
         - '--es.clusterinfo.interval=5m'
       ports:
         - "9114:9114"
   

2. prometheus.yml配置job繁扎，并重啟prometheus

   scrape_configs:
     - job_name: 'elasticsearch_exporter'
       static_configs:
         - targets:
           - 192.168.3.39:9114
   

3. grafana導(dǎo)入dashboard模板
   elasticsearch-dashboard.json

4. 效果展示

image-20230320105112512.png

告警配置

grafana + prometheus告警配置