Kafka性能測(cè)試之produce

在kafka的bin目錄下芬位，有兩個(gè)腳本kafka-producer-perf-test.sh?和kafka-consumer-perf-test.sh，這兩個(gè)腳本的作用是用來測(cè)試生產(chǎn)者和消費(fèi)者的贸辈。

[root@hostname bin]# ./kafka-producer-perf-test.sh --help

usage: producer-performance [-h] --topic TOPIC --num-records NUM-RECORDS [--payload-delimiter PAYLOAD-DELIMITER] --throughput THROUGHPUT

? ? ? ? ? ? ? ? ? ? ? ? ? ? [--producer-props PROP-NAME=PROP-VALUE [PROP-NAME=PROP-VALUE ...]] [--producer.config CONFIG-FILE] (--record-size RECORD-SIZE |

? ? ? ? ? ? ? ? ? ? ? ? ? ? --payload-file PAYLOAD-FILE)

This tool is used to verify the producer performance.

optional arguments:

? -h, --help? ? ? ? ? ? show this help message and exit

? --topic TOPIC? ? ? ? ? produce messages to this topic

? --num-records NUM-RECORDS

? ? ? ? ? ? ? ? ? ? ? ? number of messages to produce

? --payload-delimiter PAYLOAD-DELIMITER

? ? ? ? ? ? ? ? ? ? ? ? provides delimiter to be used when --payload-file is? provided.? Defaults? to? new? line.? Note? that this parameter will be ignored if --

? ? ? ? ? ? ? ? ? ? ? ? payload-file is not provided. (default: \n)

? --throughput THROUGHPUT

? ? ? ? ? ? ? ? ? ? ? ? throttle maximum message throughput to *approximately* THROUGHPUT messages/sec

? --producer-props PROP-NAME=PROP-VALUE [PROP-NAME=PROP-VALUE ...]

? ? ? ? ? ? ? ? ? ? ? ? kafka producer related configuration properties like bootstrap.servers,client.id etc.? These configs take precedence over those passed via

? ? ? ? ? ? ? ? ? ? ? ? --producer.config.

? --producer.config CONFIG-FILE

? ? ? ? ? ? ? ? ? ? ? ? producer config properties file.

? either --record-size or --payload-file must be specified but not both.

? --record-size RECORD-SIZE

? ? ? ? ? ? ? ? ? ? ? ? message size in bytes. Note that you must provide exactly one of --record-size or --payload-file.

? --payload-file PAYLOAD-FILE

? ? ? ? ? ? ? ? ? ? ? ? file to read the message payloads from. This works only for UTF-8? encoded? text files. Payloads will be read from this file and a payload

? ? ? ? ? ? ? ? ? ? ? ? will be randomly selected when sending messages. Note that you must provide exactly one of --record-size or --payload-file.

根據(jù)help命令可以得知相關(guān)參數(shù)以及參數(shù)的含義。

[root@hadoop-sh1-core1 bin]# ./kafka-topics.sh --zookeeper hadoop-sh1-master2:2181 --topic "test003" --describe? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

Topic:test003? PartitionCount:3? ? ? ? ReplicationFactor:1? ? Configs:

? ? ? ? Topic: test003? Partition: 0? ? Leader: 0? ? ? Replicas: 0? ? Isr: 0

? ? ? ? Topic: test003? Partition: 1? ? Leader: 1? ? ? Replicas: 1? ? Isr: 1

? ? ? ? Topic: test003? Partition: 2? ? Leader: 2? ? ? Replicas: 2? ? Isr: 2

[root@hadoop-sh1-core1 bin]# ./kafka-producer-perf-test.sh --topic test003 --num-records 1000000 --record-size 1024? --throughput -1 --producer.config ../config/producer.properties

[2018-07-02 14:29:03,086] WARN Error while fetching metadata with correlation id 1 : {test003=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

250992 records sent, 50198.4 records/sec (49.02 MB/sec), 425.2 ms avg latency, 1167.0 max latency.

526995 records sent, 103616.8 records/sec (101.19 MB/sec), 266.2 ms avg latency, 1080.0 max latency.

1000000 records sent, 83998.320034 records/sec (82.03 MB/sec), 317.24 ms avg latency, 1314.00 ms max latency, 259 ms 50th, 868 ms 95th, 1212 ms 99th, 1228 ms 99.9th.

不限制吞吐量芒珠，3個(gè)partition的topic琅拌，3個(gè)備份，ack=all的情況下婚度，最高達(dá)到了101.19 MB/sec的速度蘸秘，平均延遲在200多到400多左右，最高延遲是1228 ms蝗茁。

調(diào)節(jié)topic參數(shù)(還是3臺(tái)機(jī)器)醋虏，將topic的partition數(shù)量調(diào)高到10，備份數(shù)降到1哮翘，還是不限制吞吐量

Created topic "test004".

[root@hadoop-sh1-core1 bin]# ./kafka-topics.sh --zookeeper hadoop-sh1-master2:2181 --topic "test004" --describe? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Topic:test004? PartitionCount:10? ? ? ReplicationFactor:1? ? Configs:

? ? ? ? Topic: test004? Partition: 0? ? Leader: 0? ? ? Replicas: 0? ? Isr: 0

? ? ? ? Topic: test004? Partition: 1? ? Leader: 1? ? ? Replicas: 1? ? Isr: 1

? ? ? ? Topic: test004? Partition: 2? ? Leader: 2? ? ? Replicas: 2? ? Isr: 2

? ? ? ? Topic: test004? Partition: 3? ? Leader: 3? ? ? Replicas: 3? ? Isr: 3

? ? ? ? Topic: test004? Partition: 4? ? Leader: 0? ? ? Replicas: 0? ? Isr: 0

? ? ? ? Topic: test004? Partition: 5? ? Leader: 1? ? ? Replicas: 1? ? Isr: 1

? ? ? ? Topic: test004? Partition: 6? ? Leader: 2? ? ? Replicas: 2? ? Isr: 2

? ? ? ? Topic: test004? Partition: 7? ? Leader: 3? ? ? Replicas: 3? ? Isr: 3

? ? ? ? Topic: test004? Partition: 8? ? Leader: 0? ? ? Replicas: 0? ? Isr: 0

? ? ? ? Topic: test004? Partition: 9? ? Leader: 1? ? ? Replicas: 1? ? Isr: 1

[root@hadoop-sh1-core1 bin]# ./kafka-producer-perf-test.sh --topic test004 --num-records 1000000 --record-size 1024? --throughput -1 --producer.config ../config/producer.properties

525946 records sent, 105189.2 records/sec (102.72 MB/sec), 209.1 ms avg latency, 1107.0 max latency.

1000000 records sent, 114155.251142 records/sec (111.48 MB/sec), 226.50 ms avg latency, 1203.00 ms max latency, 152 ms 50th, 1007 ms 95th, 1191 ms 99th, 1196 ms 99.9th.? ? ? ?

從上面可以看見颈嚼，平均延遲和最大延遲以及吞吐量都上升了許多，因?yàn)檫€是只有三臺(tái)機(jī)器忍坷，吞吐量可能只能到100M多每秒粘舟，不知道再增加是否還會(huì)提升熔脂。

還是3個(gè)partiton，3份備份柑肴，將吞吐降到了1w霞揉，再看看測(cè)試情況

[root@hadoop-sh1-core1 bin]# ./kafka-producer-perf-test.sh --topic test003 --num-records 1000000 --record-size 1024? --throughput 10000 --producer.config ../config/producer.properties

49979 records sent, 9995.8 records/sec (9.76 MB/sec), 1.4 ms avg latency, 203.0 max latency.

50043 records sent, 10008.6 records/sec (9.77 MB/sec), 0.6 ms avg latency, 18.0 max latency.

49973 records sent, 9994.6 records/sec (9.76 MB/sec), 0.7 ms avg latency, 41.0 max latency.

50057 records sent, 10009.4 records/sec (9.77 MB/sec), 0.5 ms avg latency, 23.0 max latency.

50002 records sent, 10000.4 records/sec (9.77 MB/sec), 0.6 ms avg latency, 46.0 max latency.

50015 records sent, 10003.0 records/sec (9.77 MB/sec), 0.6 ms avg latency, 24.0 max latency.

49993 records sent, 9996.6 records/sec (9.76 MB/sec), 0.6 ms avg latency, 47.0 max latency.

50010 records sent, 10000.0 records/sec (9.77 MB/sec), 0.6 ms avg latency, 26.0 max latency.

50015 records sent, 10003.0 records/sec (9.77 MB/sec), 0.6 ms avg latency, 26.0 max latency.

50005 records sent, 10001.0 records/sec (9.77 MB/sec), 0.6 ms avg latency, 19.0 max latency.

50010 records sent, 10002.0 records/sec (9.77 MB/sec), 0.6 ms avg latency, 24.0 max latency.

50010 records sent, 10002.0 records/sec (9.77 MB/sec), 0.5 ms avg latency, 18.0 max latency.

50010 records sent, 10002.0 records/sec (9.77 MB/sec), 0.5 ms avg latency, 23.0 max latency.

50000 records sent, 10000.0 records/sec (9.77 MB/sec), 0.6 ms avg latency, 23.0 max latency.

50016 records sent, 10003.2 records/sec (9.77 MB/sec), 0.6 ms avg latency, 31.0 max latency.

49994 records sent, 9998.8 records/sec (9.76 MB/sec), 0.6 ms avg latency, 29.0 max latency.

50010 records sent, 10002.0 records/sec (9.77 MB/sec), 0.6 ms avg latency, 21.0 max latency.

50019 records sent, 10001.8 records/sec (9.77 MB/sec), 0.6 ms avg latency, 32.0 max latency.

1000000 records sent, 9999.200064 records/sec (9.76 MB/sec), 0.61 ms avg latency, 203.00 ms max latency, 1 ms 50th, 1 ms 95th, 2 ms 99th, 19 ms 99.9th.

平均延時(shí)降到了0.6ms，最大延遲在幾十ms以內(nèi)晰骑。

接下來再調(diào)節(jié)record size大小适秩，看是否造成影響。

[root@hadoop-sh1-core1 bin]# ./kafka-producer-perf-test.sh --topic test004 --num-records 1000000 --record-size 258? --throughput -1 --producer.config ../config/producer.properties

1000000 records sent, 323519.896474 records/sec (79.60 MB/sec), 4.77 ms avg latency, 210.00 ms max latency, 3 ms 50th, 18 ms 95th, 39 ms 99th, 46 ms 99.9th.

[root@hadoop-sh1-core1 bin]# ./kafka-producer-perf-test.sh --topic test004 --num-records 1000000 --record-size 1024? --throughput -1 --producer.config ../config/producer.properties

525946 records sent, 105189.2 records/sec (102.72 MB/sec), 209.1 ms avg latency, 1107.0 max latency.

1000000 records sent, 114155.251142 records/sec (111.48 MB/sec), 226.50 ms avg latency, 1203.00 ms max latency, 152 ms 50th, 1007 ms 95th, 1191 ms 99th, 1196 ms 99.9th.

[root@hadoop-sh1-core1 bin]# ./kafka-producer-perf-test.sh --topic test004 --num-records 10000000 --record-size 258? --throughput -1 --producer.config ../config/producer.properties

1949374 records sent, 389874.8 records/sec (95.93 MB/sec), 3.6 ms avg latency, 212.0 max latency.

2392851 records sent, 478570.2 records/sec (117.75 MB/sec), 2.3 ms avg latency, 21.0 max latency.

2372680 records sent, 474536.0 records/sec (116.76 MB/sec), 2.4 ms avg latency, 26.0 max latency.

2362539 records sent, 472507.8 records/sec (116.26 MB/sec), 2.3 ms avg latency, 24.0 max latency.

10000000 records sent, 455435.624175 records/sec (112.06 MB/sec), 2.54 ms avg latency, 212.00 ms max latency, 2 ms 50th, 4 ms 95th, 10 ms 99th, 20 ms 99.9th.

兩者對(duì)比可以發(fā)現(xiàn)硕舆，size從1024降到258秽荞，性能有了顯著提升，延遲就降了不少抚官，第三個(gè)扬跋，再將records數(shù)量上升到1000w條，發(fā)現(xiàn)凌节，延遲并沒有上升對(duì)少钦听，吞吐穩(wěn)定在了110M多，可能三臺(tái)機(jī)器的瓶頸就是這么多了倍奢。

由以上的測(cè)試結(jié)論可以得出朴上，kafka的produce效率，和record size大小卒煞，partition數(shù)量痪宰，備份數(shù)，ack的值等有關(guān)畔裕。（當(dāng)然衣撬，和單臺(tái)機(jī)器的性能也有一些關(guān)系。_）

?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者

人面猴
序言：七十年代末柴钻，一起剝皮案震驚了整個(gè)濱河市淮韭，隨后出現(xiàn)的幾起案子，更是在濱河造成了極大的恐慌贴届，老刑警劉巖靠粪，帶你破解...
沈念sama閱讀 217,509評(píng)論 6贊 504
死咒
序言：濱河連續(xù)發(fā)生了三起死亡事件，死亡現(xiàn)場(chǎng)離奇詭異毫蚓，居然都是意外死亡占键，警方通過查閱死者的電腦和手機(jī)，發(fā)現(xiàn)死者居然都...
沈念sama閱讀 92,806評(píng)論 3贊 394
救了他兩次的神仙讓他今天三更去死
文/潘曉璐我一進(jìn)店門元潘，熙熙樓的掌柜王于貴愁眉苦臉地迎上來畔乙，“玉大人，你說我怎么就攤上這事翩概∩啵” “怎么了返咱？”我有些...
開封第一講書人閱讀 163,875評(píng)論 0贊 354
道士緝兇錄：失蹤的賣姜人
文/不壞的土叔我叫張陵，是天一觀的道長(zhǎng)牍鞠。經(jīng)常有香客問我咖摹，道長(zhǎng)，這世上最難降的妖魔是什么难述？我笑而不...
開封第一講書人閱讀 58,441評(píng)論 1贊 293
?港島之戀（遺憾婚禮）
正文為了忘掉前任萤晴，我火速辦了婚禮，結(jié)果婚禮上胁后，老公的妹妹穿的比我還像新娘店读。我一直安慰自己，他們只是感情好攀芯，可當(dāng)我...
茶點(diǎn)故事閱讀 67,488評(píng)論 6贊 392
惡毒庶女頂嫁案：這布局不是一般人想出來的
文/花漫我一把揭開白布屯断。她就那樣靜靜地躺著，像睡著了一般敲才。火紅的嫁衣襯著肌膚如雪裹纳。梳的紋絲不亂的頭發(fā)上，一...
開封第一講書人閱讀 51,365評(píng)論 1贊 302
城市分裂傳說
那天紧武，我揣著相機(jī)與錄音，去河邊找鬼敏储。笑死阻星，一個(gè)胖子當(dāng)著我的面吹牛，可吹牛的內(nèi)容都是我干的已添。我是一名探鬼主播妥箕，決...
沈念sama閱讀 40,190評(píng)論 3贊 418
雙鴛鴦連環(huán)套：你想象不到人心有多黑
文/蒼蘭香墨我猛地睜開眼，長(zhǎng)吁一口氣：“原來是場(chǎng)噩夢(mèng)啊……” “哼更舞！你這毒婦竟也來了畦幢？” 一聲冷哼從身側(cè)響起，我...
開封第一講書人閱讀 39,062評(píng)論 0贊 276
萬榮殺人案實(shí)錄
序言：老撾萬榮一對(duì)情侶失蹤缆蝉，失蹤者是張志新（化名）和其女友劉穎宇葱，沒想到半個(gè)月后，有當(dāng)?shù)厝嗽跇淞掷锇l(fā)現(xiàn)了一具尸體刊头，經(jīng)...
沈念sama閱讀 45,500評(píng)論 1贊 314
?護(hù)林員之死
正文獨(dú)居荒郊野嶺守林人離奇死亡黍瞧，尸身上長(zhǎng)有42處帶血的膿包…… 初始之章·張勛以下內(nèi)容為張勛視角年9月15日...
茶點(diǎn)故事閱讀 37,706評(píng)論 3贊 335
?白月光啟示錄
正文我和宋清朗相戀三年，在試婚紗的時(shí)候發(fā)現(xiàn)自己被綠了原杂。大學(xué)時(shí)的朋友給我發(fā)了我未婚夫和他白月光在一起吃飯的照片印颤。...
茶點(diǎn)故事閱讀 39,834評(píng)論 1贊 347
活死人
序言：一個(gè)原本活蹦亂跳的男人離奇死亡，死狀恐怖穿肄，靈堂內(nèi)的尸體忽然破棺而出年局，到底是詐尸還是另有隱情际看，我是刑警寧澤，帶...
沈念sama閱讀 35,559評(píng)論 5贊 345
?日本核電站爆炸內(nèi)幕
正文年R本政府宣布矢否，位于F島的核電站仲闽，受9級(jí)特大地震影響，放射性物質(zhì)發(fā)生泄漏兴喂。R本人自食惡果不足惜蔼囊，卻給世界環(huán)境...
茶點(diǎn)故事閱讀 41,167評(píng)論 3贊 328
男人毒藥：我在死后第九天來索命
文/蒙蒙一、第九天我趴在偏房一處隱蔽的房頂上張望衣迷。院中可真熱鬧畏鼓，春花似錦、人聲如沸壶谒。這莊子的主人今日做“春日...
開封第一講書人閱讀 31,779評(píng)論 0贊 22
一樁弒父案，背后竟有這般陰謀
文/蒼蘭香墨我抬頭看了看天上的太陽汗菜。三九已至让禀，卻和暖如春，著一層夾襖步出監(jiān)牢的瞬間陨界，已是汗流浹背巡揍。一陣腳步聲響...
開封第一講書人閱讀 32,912評(píng)論 1贊 269
情欲美人皮
我被黑心中介騙來泰國打工，沒想到剛下飛機(jī)就差點(diǎn)兒被人妖公主榨干…… 1. 我叫王不留菌瘪，地道東北人腮敌。一個(gè)月前我還...
沈念sama閱讀 47,958評(píng)論 2贊 370
代替公主和親
正文我出身青樓，卻偏偏與公主長(zhǎng)得像俏扩，于是被迫代替她去往敵國和親糜工。傳聞我的和親對(duì)象是個(gè)殘疾皇子，可洞房花燭夜當(dāng)晚...
茶點(diǎn)故事閱讀 44,779評(píng)論 2贊 354

Kafka性能測(cè)試之produce

推薦閱讀更多精彩內(nèi)容