轉(zhuǎn)：kafka監(jiān)控實(shí)戰(zhàn)(jmxtrans+InfluxDb+Grafana)

kafka監(jiān)控實(shí)戰(zhàn)(jmxtrans+InfluxDb+Grafana)

轉(zhuǎn)自http://navyaijm.blog.51cto.com/4647068/1958376

前言

從上周一直在調(diào)研找一款好用的kafka監(jiān)控，我測試使用過的KafkaOffsetMonitor沛申、Burrow硫眯、kafka-monitor、Kafka-Manager炫乓，他們各有優(yōu)缺點(diǎn)，具體情況我這里就不展開描述了，大家可以到它們的git上去查看陪毡，并且它們基本上都是監(jiān)控topic的寫入和讀取等等队萤，沒有提供對于整體集群的監(jiān)控信息轮锥，比如集群的分片、延時要尔、內(nèi)存使用情況等等舍杜，無意中發(fā)現(xiàn)了jmxtrans新娜，jmxtrans它是一個通過jmx采集java應(yīng)用的數(shù)據(jù)采集器，他的輸出可以是Graphite既绩、StatsD概龄、Ganglia、InfluxDb等等饲握，剛好我們現(xiàn)有的監(jiān)控是通過InfluxDb做數(shù)據(jù)存儲的私杜，通過Grafana做展示，下面就給大家介紹一下jmxtrans+InfluxDb+Grafana監(jiān)控kafka的整體解決方案救欧，并且不需要任何額外的開發(fā)工作衰粹，完全使用原生的。

環(huán)境介紹

角色

10.10.10.10   InfluxDb
10.10.10.100  Grafana
10.10.30.69   jmxtrans
kafka集群
10.10.20.14  node1
10.10.20.15 node2
10.10.20.16 node3
10.10.20.17 node4

軟件版本

influxdb-1.2.4-1.x86_64
grafana-4.1.1-1484211277.x86_64
jmxtrans-266.rpm
kafka_2.10-0.9.0.0.jar.asc

架構(gòu)圖

image

配置規(guī)劃

jmxtrans我們可以分別在每臺kafka節(jié)點(diǎn)上部署颜矿，也可以部署到一臺機(jī)器上寄猩，我這里是選擇了后者，因?yàn)槲业募盒∑锝@樣配置文件可以集中管理田篇，如果集群比較大，可以考慮分散部署箍铭。
關(guān)于jmxtrans的配置文件泊柬，分全局指標(biāo)（每個kafka節(jié)點(diǎn)）和topic指標(biāo)，全局指標(biāo)每個節(jié)點(diǎn)一個配置文件诈火，命名規(guī)則：base_10.10.20.14.json兽赁，topic指標(biāo)是每個topic一個配置文件，命名規(guī)則：falcon_monitor_us_17.json

監(jiān)控指標(biāo)

全局指標(biāo)

每秒輸入的流量:

"obj" : "kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec"
"attr" : [ "Count" ]
"resultAlias":"BytesInPerSec"
"tags"     : {"application" : "BytesInPerSec"}

每秒輸入的流量

"obj" : "kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec"
"attr" : [ "Count" ]
"resultAlias":"BytesOutPerSec"
"tags"     : {"application" : "BytesOutPerSec"}

每秒輸入的流量

"obj" : "kafka.server:type=BrokerTopicMetrics,name=BytesRejectedPerSec"
"attr" : [ "Count" ]
"resultAlias":"BytesRejectedPerSec"
"tags"     : {"application" : "BytesRejectedPerSec"}

每秒的消息寫入總量

"obj" : "kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec"
"attr" : [ "Count" ]
"resultAlias":"MessagesInPerSec"
"tags"     : {"application" : "MessagesInPerSec"}

每秒FetchFollower的請求次數(shù)

"obj" : "kafka.network:type=RequestMetrics,name=RequestsPerSec,request=FetchFollower"
"attr" : [ "Count" ]
"resultAlias":"RequestsPerSec"
"tags"     : {"request" : "FetchFollower"}

每秒FetchConsumer的請求次數(shù)

"obj" : "kafka.network:type=RequestMetrics,name=RequestsPerSec,request=FetchConsumer"
"attr" : [ "Count" ]
"resultAlias":"RequestsPerSec"
"tags"     : {"request" : "FetchConsumer"}

每秒Produce的請求次數(shù)

"obj" : "kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce"
"attr" : [ "Count" ]
"resultAlias":"RequestsPerSec"
"tags"     : {"request" : "Produce"}

內(nèi)存使用的使用情況

"obj" : "java.lang:type=Memory"
"attr" : [ "HeapMemoryUsage", "NonHeapMemoryUsage" ]
"resultAlias":"MemoryUsage"
"tags"     : {"application" : "MemoryUsage"}

GC的耗時和次數(shù)

"obj" : "java.lang:type=GarbageCollector,name=*"
"attr" : [ "CollectionCount","CollectionTime" ]
"resultAlias":"GC"
"tags"     : {"application" : "GC"}

線程的使用情況

"obj" : "java.lang:type=Threading"
"attr" : [ "PeakThreadCount","ThreadCount" ]
"resultAlias":"Thread"
"tags"     : {"application" : "Thread"}

副本落后主分片的最大消息數(shù)量

"obj" : "kafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=Replica"
"attr" : [ "Value" ]
"resultAlias":"ReplicaFetcherManager"
"tags"     : {"application" : "MaxLag"}

該broker上的partition的數(shù)量

"obj" : "kafka.server:type=ReplicaManager,name=PartitionCount"
"attr" : [ "Value" ]
"resultAlias":"ReplicaManager"
"tags"     : {"application" : "PartitionCount"}

正在做復(fù)制的partition的數(shù)量

"obj" : "kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions"
"attr" : [ "Value" ]
"resultAlias":"ReplicaManager"
"tags"     : {"application" : "UnderReplicatedPartitions"}

Leader的replica的數(shù)量

"obj" : "kafka.server:type=ReplicaManager,name=LeaderCount"
"attr" : [ "Value" ]
"resultAlias":"ReplicaManager"
"tags"     : {"application" : "LeaderCount"}

一個請求FetchConsumer耗費(fèi)的所有時間

"obj" : "kafka.network:type=RequestMetrics,name=TotalTimeMs,request=FetchConsumer"
"attr" : [ "Count","Max" ]
"resultAlias":"TotalTimeMs"
"tags"     : {"application" : "FetchConsumer"}

一個請求FetchFollower耗費(fèi)的所有時間

"obj" : "kafka.network:type=RequestMetrics,name=TotalTimeMs,request=FetchFollower"
"attr" : [ "Count","Max" ]
"resultAlias":"TotalTimeMs"
"tags"     : {"application" : "FetchFollower"}

一個請求Produce耗費(fèi)的所有時間

"obj" : "kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce"
"attr" : [ "Count","Max" ]
"resultAlias":"TotalTimeMs"
"tags"     : {"application" : "Produce"}

topic的監(jiān)控指標(biāo)

falcon_monitor_us每秒的寫入流量

"kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=falcon_monitor_us"
"attr" : [ "Count" ]
"resultAlias":"falcon_monitor_us"
"tags"     : {"application" : "BytesInPerSec"}

falcon_monitor_us每秒的輸出流量

"kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec,topic=falcon_monitor_us"
"attr" : [ "Count" ]
"resultAlias":"falcon_monitor_us"
"tags"     : {"application" : "BytesOutPerSec"}

falcon_monitor_us每秒寫入消息的數(shù)量

"obj" : "kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec,topic=falcon_monitor_us"
"attr" : [ "Count" ]
"resultAlias":"falcon_monitor_us"
"tags"     : {"application" : "MessagesInPerSec"}

falcon_monitor_us在每個分區(qū)最后的Offset

"obj" : "kafka.log:type=Log,name=LogEndOffset,topic=falcon_monitor_us,partition=*"
"attr" : [ "Value" ]
"resultAlias":"falcon_monitor_us"
"tags"     : {"application" : "LogEndOffset"}

參數(shù)說明

obj對應(yīng)jmx的ObjectName冷守，就是我們要監(jiān)控的指標(biāo)
attr對應(yīng)ObjectName的屬性刀崖，可以理解為我們要監(jiān)控的指標(biāo)的值
resultAlias對應(yīng)metric 的名稱，在InfluxDb里面就是MEASUREMENTS名
tags對應(yīng)InfluxDb的tag功能拍摇，對與存儲在同一個MEASUREMENTS里面的不同監(jiān)控指標(biāo)可以做區(qū)分亮钦，我們在用Grafana繪圖的時候會用到，建議對每個監(jiān)控指標(biāo)都打上tags

對于全局監(jiān)控充活，每一個監(jiān)控指標(biāo)對應(yīng)一個MEASUREMENTS蜂莉，所有的kafka節(jié)點(diǎn)同一個監(jiān)控指標(biāo)數(shù)據(jù)寫同一個MEASUREMENTS ，對于topc監(jiān)控的監(jiān)控指標(biāo)混卵，同一個topic所有kafka節(jié)點(diǎn)寫到同一個MEASUREMENTS映穗，并且以topic名稱命名

安裝

kafka

這里不詳細(xì)介紹kafka集群的安裝，主要說一下kafka的啟動方式幕随，因?yàn)槲覀冃枰ㄟ^jmx采集kafka的監(jiān)控數(shù)據(jù)蚁滋，所以在kafka的啟動時候需要啟動jmx端口，啟動方式如下：

cd /data/kafka/bin/
JMX_PORT=9999 nohup ./kafka-server-start.sh ../config/server.properties  >/dev/null 2>&1 &

influxDb

yum -y install  influxdb   ##安裝
/etc/init.d/influxdb start ##啟動服務(wù)
[root@ip-10-10-10-10 jmxtrans]# influx
Connected to http://localhost:8086 version 1.3.2
InfluxDB shell version: 1.3.2
> CREATE USER "root" WITH PASSWORD '123456' WITH ALL PRIVILEGES    ##添加一個賬號
>

Grafana

yum -y install  grafana          ##安裝
/etc/init.d/grafana-server start ##啟動服務(wù)

jmxtrans

wget http://central.maven.org/maven2/org/jmxtrans/jmxtrans/266/jmxtrans-266.rpm
rpm -ivh jmxtrans-266.rpm  ##安裝
/etc/init.d/jmxtrans start ##啟動

配置

這里主要介紹jmxtrans采集數(shù)據(jù)的配置文件撰寫和Grafana繪圖的配置注意事項(xiàng)，kafka和InfluxDb的配置這里不做描述枢赔。

jmxtrans

jmxtrans默認(rèn)讀取/var/lib/jmxtrans下的配置文件去采集數(shù)據(jù)的澄阳，所以我們把采集kafka監(jiān)控數(shù)據(jù)的配置文件都在這個目錄下，下面是我的配置文件命名規(guī)范：

[root@ip-10-10-30-69 jmxtrans]# ll
total 96
-rw-r--r-- 1 root root 1657 Aug 18 17:03 article-feedback-10min-json_14.json
-rw-r--r-- 1 root root 1657 Aug 18 17:03 article-feedback-10min-json_15.json
-rw-r--r-- 1 root root 1657 Aug 18 17:04 article-feedback-10min-json_16.json
-rw-r--r-- 1 root root 1657 Aug 18 17:04 article-feedback-10min-json_17.json
-rw-r--r-- 1 root root 8430 Aug 22 08:24 base_10.10.20.14.json
-rw-r--r-- 1 root root 8431 Aug 22 08:24 base_10.10.20.15.json
-rw-r--r-- 1 root root 8431 Aug 22 08:25 base_10.10.20.16.json
-rw-r--r-- 1 root root 8431 Aug 22 08:25 base_10.10.20.17.json
-rw-r--r-- 1 root root 2027 Aug 21 16:19 falcon_monitor_us_14.json
-rw-r--r-- 1 root root 2027 Aug 21 16:20 falcon_monitor_us_15.json
-rw-r--r-- 1 root root 2484 Aug 21 20:58 falcon_monitor_us_16.json
-rw-r--r-- 1 root root 2027 Aug 21 16:20 falcon_monitor_us_17.json
-rw-r--r-- 1 root root 2147 Aug 21 17:43 highgmp-articles-through-primary_14.json
-rw-r--r-- 1 root root 2147 Aug 21 17:46 highgmp-articles-through-primary_15.json
-rw-r--r-- 1 root root 2147 Aug 21 17:46 highgmp-articles-through-primary_16.json
-rw-r--r-- 1 root root 2147 Aug 21 17:47 highgmp-articles-through-primary_17.json
[root@ip-10-10-30-69 jmxtrans]# pwd
/var/lib/jmxtrans

全局監(jiān)控的配置文件踏拜，以10.10.20.14為例：

[root@ip-10-10-30-69 jmxtrans]# cat base_10.10.20.14.json
{
  "servers" : [ {
    "port" : "9999",
    "host" : "10.10.20.14",
    "queries" : [ {
      "obj" : "kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec",
      "attr" : [ "Count","OneMinuteRate" ],
      "resultAlias":"BytesInPerSec",
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
        "url" : "http://10.10.10.10:8086/",
        "username" : "root",
        "password" : "root",
        "database" : "jmxDB",
        "tags"     : {"application" : "BytesInPerSec"}
      } ]
    },
{
      "obj" : "kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec",
      "attr" : [ "Count","OneMinuteRate" ],
      "resultAlias":"BytesOutPerSec",
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
        "url" : "http://10.10.10.10:8086/",
        "username" : "root",
        "password" : "root",
        "database" : "jmxDB",
        "tags"     : {"application" : "BytesOutPerSec"}
      } ]
    },
{
      "obj" : "kafka.server:type=BrokerTopicMetrics,name=BytesRejectedPerSec",
      "attr" : [ "Count","OneMinuteRate" ],
      "resultAlias":"BytesRejectedPerSec",
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
        "url" : "http://10.10.10.10:8086/",
        "username" : "root",
        "password" : "root",
        "database" : "jmxDB",
        "tags"     : {"application" : "BytesRejectedPerSec"}
      } ]
    },
{
      "obj" : "kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec",
      "attr" : [ "Count","OneMinuteRate" ],
      "resultAlias":"MessagesInPerSec",
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
        "url" : "http://10.10.10.10:8086/",
        "username" : "root",
        "password" : "root",
        "database" : "jmxDB",
        "tags"     : {"application" : "MessagesInPerSec"}
      } ]
    },
{
      "obj" : "kafka.network:type=RequestMetrics,name=RequestsPerSec,request=FetchConsumer",
      "attr" : [ "Count" ],
      "resultAlias":"RequestsPerSec",
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
        "url" : "http://10.10.10.10:8086/",
        "username" : "root",
        "password" : "root",
        "database" : "jmxDB",
        "tags"     : {"request" : "FetchConsumer"}
      } ]
    },
{
      "obj" : "kafka.network:type=RequestMetrics,name=RequestsPerSec,request=FetchFollower",
      "attr" : [ "Count" ],
      "resultAlias":"RequestsPerSec",
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
        "url" : "http://10.10.10.10:8086/",
        "username" : "root",
        "password" : "root",
        "database" : "jmxDB",
        "tags"     : {"request" : "FetchFollower"}
      } ]
    },
{
      "obj" : "kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce",
      "attr" : [ "Count" ],
      "resultAlias":"RequestsPerSec",
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
        "url" : "http://10.10.10.10:8086/",
        "username" : "root",
        "password" : "root",
        "database" : "jmxDB",
        "tags"     : {"request" : "Produce"}
      } ]
    },
{
      "obj" : "java.lang:type=Memory",
      "attr" : [ "HeapMemoryUsage", "NonHeapMemoryUsage" ],
      "resultAlias":"MemoryUsage",
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
        "url" : "http://10.10.10.10:8086/",
        "username" : "root",
        "password" : "root",
        "database" : "jmxDB",
        "tags"     : {"application" : "MemoryUsage"}
      } ]
    },
{
      "obj" : "java.lang:type=GarbageCollector,name=*",
      "attr" : [ "CollectionCount","CollectionTime" ],
      "resultAlias":"GC",
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
        "url" : "http://10.10.10.10:8086/",
        "username" : "root",
        "password" : "root",
        "database" : "jmxDB",
        "tags"     : {"application" : "GC"}
      } ]
    },
{
      "obj" : "java.lang:type=Threading",
      "attr" : [ "PeakThreadCount","ThreadCount" ],
      "resultAlias":"Thread",
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
        "url" : "http://10.10.10.10:8086/",
        "username" : "root",
        "password" : "root",
        "database" : "jmxDB",
        "tags"     : {"application" : "Thread"}
      } ]
    },
{
      "obj" : "kafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=Replica",
      "attr" : [ "Value" ],
      "resultAlias":"ReplicaFetcherManager",
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
        "url" : "http://10.10.10.10:8086/",
        "username" : "root",
        "password" : "root",
        "database" : "jmxDB",
        "tags"     : {"application" : "MaxLag"}
      } ]
    },
{
      "obj" : "kafka.server:type=ReplicaManager,name=PartitionCount",
      "attr" : [ "Value" ],
      "resultAlias":"ReplicaManager",
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
        "url" : "http://10.10.10.10:8086/",
        "username" : "root",
        "password" : "root",
        "database" : "jmxDB",
        "tags"     : {"application" : "PartitionCount"}
      } ]
    },
{
      "obj" : "kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions",
      "attr" : [ "Value" ],
      "resultAlias":"ReplicaManager",
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
        "url" : "http://10.10.10.10:8086/",
        "username" : "root",
        "password" : "root",
        "database" : "jmxDB",
        "tags"     : {"application" : "UnderReplicatedPartitions"}
      } ]
    },
{
      "obj" : "kafka.server:type=ReplicaManager,name=LeaderCount",
      "attr" : [ "Value" ],
      "resultAlias":"ReplicaManager",
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
        "url" : "http://10.10.10.10:8086/",
        "username" : "root",
        "password" : "root",
        "database" : "jmxDB",
        "tags"     : {"application" : "LeaderCount"}
      } ]
    },
{
      "obj" : "kafka.network:type=RequestMetrics,name=TotalTimeMs,request=FetchConsumer",
      "attr" : [ "Count","Max" ],
      "resultAlias":"TotalTimeMs",
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
        "url" : "http://10.10.10.10:8086/",
        "username" : "root",
        "password" : "root",
        "database" : "jmxDB",
        "tags"     : {"application" : "FetchConsumer"}
      } ]
    },
{
      "obj" : "kafka.network:type=RequestMetrics,name=TotalTimeMs,request=FetchFollower",
      "attr" : [ "Count","Max" ],
      "resultAlias":"TotalTimeMs",
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
        "url" : "http://10.10.10.10:8086/",
        "username" : "root",
        "password" : "root",
        "database" : "jmxDB",
        "tags"     : {"application" : "FetchConsumer"}
      } ]
    },
{
      "obj" : "kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce",
      "attr" : [ "Count","Max" ],
      "resultAlias":"TotalTimeMs",
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
        "url" : "http://10.10.10.10:8086/",
        "username" : "root",
        "password" : "root",
        "database" : "jmxDB",
        "tags"     : {"application" : "Produce"}
      } ]
    },
{
      "obj" : "kafka.server:type=ReplicaManager,name=IsrShrinksPerSec",
      "attr" : [ "Count" ],
      "resultAlias":"ReplicaManager",
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
        "url" : "http://10.10.10.10:8086/",
        "username" : "root",
        "password" : "root",
        "database" : "jmxDB",
        "tags"     : {"application" : "IsrShrinksPerSec"}
      } ]
    }
]
  } ]
}

topic監(jiān)控的配置文件，以falcon_monitor_us的10.10.20.14節(jié)點(diǎn)為例：

[root@ip-10-10-30-69 jmxtrans]# cat  falcon_monitor_us_14.json
{
  "servers" : [ {
    "port" : "9999",
    "host" : "10.10.20.14",
    "queries" : [ {
      "obj" : "kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=falcon_monitor_us",
      "attr" : [ "Count" ],
      "resultAlias":"falcon_monitor_us",
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
        "url" : "http://10.10.10.10:8086/",
        "username" : "root",
        "password" : "root",
        "database" : "jmxDB",
        "tags"     : {"application" : "BytesInPerSec"}
      } ]
    },
{
      "obj" : "kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec,topic=falcon_monitor_us",
      "attr" : [ "Count" ],
      "resultAlias":"falcon_monitor_us",
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
        "url" : "http://10.10.10.10:8086/",
        "username" : "root",
        "password" : "root",
        "database" : "jmxDB",
        "tags"     : {"application" : "BytesOutPerSec"}
      } ]
    },
{
      "obj" : "kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec,topic=falcon_monitor_us",
      "attr" : [ "Count" ],
      "resultAlias":"falcon_monitor_us",
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
        "url" : "http://10.10.10.10:8086/",
        "username" : "root",
        "password" : "root",
        "database" : "jmxDB",
        "tags"     : {"application" : "MessagesInPerSec"}
      } ]
    },
{
      "obj" : "kafka.log:type=Log,name=LogEndOffset,topic=falcon_monitor_us,partition=*",
      "attr" : [ "Value" ],
      "resultAlias":"falcon_monitor_us",
      "outputWriters" : [ {
        "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
        "url" : "http://10.10.10.10:8086/",
        "username" : "root",
        "password" : "root",
        "database" : "jmxDB",
        "tags"     : {"application" : "LogEndOffset"}
      } ]
    }
]
  } ]
}

Grafana配置

添加數(shù)據(jù)源

image

Url低剔、Database速梗、User、Password需要和jmxtrans采集數(shù)據(jù)配置文件里面的寫一致襟齿，然后點(diǎn)擊Save&Test姻锁，提示成功就正常了

image

創(chuàng)建一個dashboard，然后在這里配置每一個監(jiān)控指標(biāo)的圖

image

要點(diǎn)說明

對于監(jiān)控指標(biāo)為Count的監(jiān)控項(xiàng)猜欺，需要通過Grafana做計算得到我們想要的監(jiān)控位隶，比如BytesInPerSec這個指標(biāo)，它的監(jiān)控值是一個累計值开皿，我們想要取到每秒的流量涧黄，肯定需要計算，(本次采集的值-上次采集的值)/60 ,jmxtrans是一分鐘采集一次數(shù)據(jù)赋荆，具體配置參考下面截圖：

image

因?yàn)槲覀兪且环昼姴杉淮螖?shù)據(jù)笋妥，所以group by 和derivative選1分鐘；因?yàn)槲覀円棵氲牧髁空叮詍ath這里除以60

X軸的單位選擇春宣，比如流量的單位、時間的單位嫉你、每秒消息的個數(shù)無單位等等月帝，下面分布舉一個例子介紹說明
設(shè)置流量的單位，點(diǎn)擊需要設(shè)置的圖幽污，選擇"Edit"進(jìn)入編輯頁面嚷辅，切到Axes這個tab頁，Unit--》data（Metric）--》bytes

image

單位油挥，點(diǎn)擊需要設(shè)置的圖潦蝇，選擇"Edit"進(jìn)入編輯頁面，切到Axes這個tab頁深寥，Unit--》time--》milliseconds(ms)

image

設(shè)置按原始值展示攘乒，無單位，點(diǎn)擊需要設(shè)置的圖惋鹅，選擇"Edit"進(jìn)入編輯頁面则酝，切到Axes這個tab頁，Unit--》none--》none

image

收獲總結(jié)

關(guān)于jmx收集了kafka的那些指標(biāo)，對應(yīng)的值都是那些類型沽讹，對應(yīng)這個問題走了很多彎路般卑，各種谷歌百度拿到了有人整理過的，一個一個試爽雄，發(fā)現(xiàn)很多不能用蝠检，要不就是寫的是錯誤的，要不就是版本不同挚瘟，寫法不一樣叹谁，最后看到了jconsole這個工具，他可以連接到本地或者遠(yuǎn)程的jmx端口乘盖，能看到在收集的所有指標(biāo)焰檩，在windows下裝好jdk，在bin目錄你可以找到這個工具订框。

image

關(guān)于consumer的延時析苫，關(guān)官方介紹有一個type是 type=consumer-fetch-manager-metrics的指標(biāo)，但是我這通過jconsole連進(jìn)來死活沒有找到穿扳，如果親們有使用這套監(jiān)控方案的衩侥，求幫忙解惑我的這個問題，謝了纵揍，官網(wǎng)監(jiān)控指標(biāo)如下：

[http://kafka.apache.org/documentation/#monitoring]

?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請聯(lián)系作者

人面猴
序言：七十年代末顿乒，一起剝皮案震驚了整個濱河市，隨后出現(xiàn)的幾起案子泽谨，更是在濱河造成了極大的恐慌璧榄，老刑警劉巖，帶你破解...
沈念sama閱讀 211,817評論 6贊 492
死咒
序言：濱河連續(xù)發(fā)生了三起死亡事件吧雹，死亡現(xiàn)場離奇詭異骨杂，居然都是意外死亡，警方通過查閱死者的電腦和手機(jī)雄卷，發(fā)現(xiàn)死者居然都...
沈念sama閱讀 90,329評論 3贊 385
救了他兩次的神仙讓他今天三更去死
文/潘曉璐我一進(jìn)店門搓蚪，熙熙樓的掌柜王于貴愁眉苦臉地迎上來，“玉大人丁鹉，你說我怎么就攤上這事妒潭。” “怎么了揣钦？”我有些...
開封第一講書人閱讀 157,354評論 0贊 348
道士緝兇錄：失蹤的賣姜人
文/不壞的土叔我叫張陵雳灾，是天一觀的道長。經(jīng)常有香客問我冯凹，道長谎亩，這世上最難降的妖魔是什么？我笑而不...
開封第一講書人閱讀 56,498評論 1贊 284
?港島之戀（遺憾婚禮）
正文為了忘掉前任，我火速辦了婚禮匈庭，結(jié)果婚禮上夫凸，老公的妹妹穿的比我還像新娘。我一直安慰自己阱持，他們只是感情好夭拌，可當(dāng)我...
茶點(diǎn)故事閱讀 65,600評論 6贊 386
惡毒庶女頂嫁案：這布局不是一般人想出來的
文/花漫我一把揭開白布。她就那樣靜靜地躺著衷咽，像睡著了一般啼止。火紅的嫁衣襯著肌膚如雪。梳的紋絲不亂的頭發(fā)上兵罢，一...
開封第一講書人閱讀 49,829評論 1贊 290
城市分裂傳說
那天，我揣著相機(jī)與錄音滓窍，去河邊找鬼卖词。笑死，一個胖子當(dāng)著我的面吹牛吏夯，可吹牛的內(nèi)容都是我干的此蜈。我是一名探鬼主播，決...
沈念sama閱讀 38,979評論 3贊 408
雙鴛鴦連環(huán)套：你想象不到人心有多黑
文/蒼蘭香墨我猛地睜開眼噪生，長吁一口氣：“原來是場噩夢啊……” “哼裆赵！你這毒婦竟也來了？” 一聲冷哼從身側(cè)響起跺嗽，我...
開封第一講書人閱讀 37,722評論 0贊 266
萬榮殺人案實(shí)錄
序言：老撾萬榮一對情侶失蹤战授，失蹤者是張志新（化名）和其女友劉穎，沒想到半個月后桨嫁，有當(dāng)?shù)厝嗽跇淞掷锇l(fā)現(xiàn)了一具尸體植兰，經(jīng)...
沈念sama閱讀 44,189評論 1贊 303
?護(hù)林員之死
正文獨(dú)居荒郊野嶺守林人離奇死亡，尸身上長有42處帶血的膿包…… 初始之章·張勛以下內(nèi)容為張勛視角年9月15日...
茶點(diǎn)故事閱讀 36,519評論 2贊 327
?白月光啟示錄
正文我和宋清朗相戀三年璃吧，在試婚紗的時候發(fā)現(xiàn)自己被綠了蚓炬。大學(xué)時的朋友給我發(fā)了我未婚夫和他白月光在一起吃飯的照片搁痛。...
茶點(diǎn)故事閱讀 38,654評論 1贊 340
活死人
序言：一個原本活蹦亂跳的男人離奇死亡，死狀恐怖，靈堂內(nèi)的尸體忽然破棺而出沪斟，到底是詐尸還是另有隱情，我是刑警寧澤孤个，帶...
沈念sama閱讀 34,329評論 4贊 330
?日本核電站爆炸內(nèi)幕
正文年R本政府宣布陋率，位于F島的核電站，受9級特大地震影響务冕，放射性物質(zhì)發(fā)生泄漏血当。R本人自食惡果不足惜，卻給世界環(huán)境...
茶點(diǎn)故事閱讀 39,940評論 3贊 313
男人毒藥：我在死后第九天來索命
文/蒙蒙一、第九天我趴在偏房一處隱蔽的房頂上張望臊旭。院中可真熱鬧落恼，春花似錦、人聲如沸离熏。這莊子的主人今日做“春日...
開封第一講書人閱讀 30,762評論 0贊 21
一樁弒父案，背后竟有這般陰謀
文/蒼蘭香墨我抬頭看了看天上的太陽滋戳。三九已至钻蔑，卻和暖如春，著一層夾襖步出監(jiān)牢的瞬間奸鸯，已是汗流浹背咪笑。一陣腳步聲響...
開封第一講書人閱讀 31,993評論 1贊 266
情欲美人皮
我被黑心中介騙來泰國打工，沒想到剛下飛機(jī)就差點(diǎn)兒被人妖公主榨干…… 1. 我叫王不留娄涩，地道東北人窗怒。一個月前我還...
沈念sama閱讀 46,382評論 2贊 360
代替公主和親
正文我出身青樓，卻偏偏與公主長得像蓄拣，于是被迫代替她去往敵國和親扬虚。傳聞我的和親對象是個殘疾皇子，可洞房花燭夜當(dāng)晚...
茶點(diǎn)故事閱讀 43,543評論 2贊 349

轉(zhuǎn)：kafka監(jiān)控實(shí)戰(zhàn)(jmxtrans+InfluxDb+Grafana)

kafka監(jiān)控實(shí)戰(zhàn)(jmxtrans+InfluxDb+Grafana)

前言

環(huán)境介紹

角色

軟件版本

架構(gòu)圖

配置規(guī)劃

監(jiān)控指標(biāo)

全局指標(biāo)

topic的監(jiān)控指標(biāo)

參數(shù)說明

安裝

kafka

influxDb

Grafana

jmxtrans

配置

jmxtrans

Grafana配置

要點(diǎn)說明

收獲總結(jié)

推薦閱讀更多精彩內(nèi)容