我們?cè)谑褂肏Base的時(shí)候缅叠，必須要能夠清楚HBase服務(wù)端的性能逆航，這對(duì)HBase的合理使用以及性能調(diào)優(yōu)都非常重要穴豫，所以一般在使用HBase之前怀大，建議做一些必要的基準(zhǔn)性能測(cè)試纱兑，其中，讀寫P99/P999延時(shí)就是一項(xiàng)衡量HBase性能的關(guān)鍵指標(biāo)化借。本文首先介紹下HBase自帶的性能測(cè)試工具——PerformanceEvaluation的使用潜慎，然后通過它壓測(cè)下HBase讀寫路徑P999延時(shí)情況。

一、HBase PE 參數(shù)介紹

PerformanceEvaluation铐炫，這里簡(jiǎn)稱PE垒手，全名為org.apache.hadoop.hbase.PerformanceEvaluation，是HBase自帶的性能測(cè)試工具倒信，目前主要支持HBase隨機(jī)/順序讀寫延時(shí)的性能測(cè)試科贬。執(zhí)行 bin/hbase pe 可直接使用：

[root@xxx ~]$ hbase pe
Usage: java org.apache.hadoop.hbase.PerformanceEvaluation \
  <OPTIONS> [-D<property=value>]* <command> <nclients>

Options:
 nomapred        Run multiple clients using threads (rather than use mapreduce)
 rows            Rows each client runs. Default: 1048576
 size            Total size in GiB. Mutually exclusive with --rows. Default: 1.0.
 sampleRate      Execute test on a sample of total rows. Only supported by randomRead. Default: 1.0
 traceRate       Enable HTrace spans. Initiate tracing every N rows. Default: 0
 table           Alternate table name. Default: 'TestTable'
 multiGet        If >0, when doing RandomRead, perform multiple gets instead of single gets. Default: 0
 compress        Compression type to use (GZ, LZO, ...). Default: 'NONE'
 flushCommits    Used to determine if the test should flush the table. Default: false
 writeToWAL      Set writeToWAL on puts. Default: True
 autoFlush       Set autoFlush on htable. Default: False
 oneCon          all the threads share the same connection. Default: False
 presplit        Create presplit table. If a table with same name exists, it'll be deleted and recreated (instead of verifying count of its existing regions). Recommended for accurate perf analysis (see guide). Default: disabled
 inmemory        Tries to keep the HFiles of the CF inmemory as far as possible. Not guaranteed that reads are always served from memory.  Default: false
 usetags         Writes tags along with KVs. Use with HFile V3. Default: false
 numoftags       Specify the no of tags that would be needed. This works only if usetags is true. Default: 1
 filterAll       Helps to filter out all the rows on the server side there by not returning any thing back to the client.  Helps to check the server side performance.  Uses FilterAllFilter internally. 
 latency         Set to report operation latencies. Default: False
 bloomFilter      Bloom filter type, one of [NONE, ROW, ROWCOL]
 blockEncoding   Block encoding to use. Value should be one of [NONE, PREFIX, DIFF, FAST_DIFF, PREFIX_TREE]. Default: NONE
 valueSize       Pass value size to use: Default: 1000
 valueRandom     Set if we should vary value size between 0 and 'valueSize'; set on read for stats on size: Default: Not set.
 valueZipf       Set if we should vary value size between 0 and 'valueSize' in zipf form: Default: Not set.
 period          Report every 'period' rows: Default: opts.perClientRunRows / 10 = 104857
 multiGet        Batch gets together into groups of N. Only supported by randomRead. Default: disabled
 addColumns      Adds columns to scans/gets explicitly. Default: true
 replicas        Enable region replica testing. Defaults: 1.
 splitPolicy     Specify a custom RegionSplitPolicy for the table.
 randomSleep     Do a random sleep before each get between 0 and entered value. Defaults: 0
 columns         Columns to write per row. Default: 1
 caching         Scan caching to use. Default: 30

 Note: -D properties will be applied to the conf used. 
  For example: 
   -Dmapreduce.output.fileoutputformat.compress=true
   -Dmapreduce.task.timeout=60000

Command:
 append          Append on each row; clients overlap on keyspace so some concurrent operations
 checkAndDelete  CheckAndDelete on each row; clients overlap on keyspace so some concurrent operations
 checkAndMutate  CheckAndMutate on each row; clients overlap on keyspace so some concurrent operations
 checkAndPut     CheckAndPut on each row; clients overlap on keyspace so some concurrent operations
 filterScan      Run scan test using a filter to find a specific row based on it's value (make sure to use --rows=20)
 increment       Increment on each row; clients overlap on keyspace so some concurrent operations
 randomRead      Run random read test
 randomSeekScan  Run random seek and scan 100 test
 randomWrite     Run random write test
 scan            Run scan test (read every row)
 scanRange10     Run random seek scan with both start and stop row (max 10 rows)
 scanRange100    Run random seek scan with both start and stop row (max 100 rows)
 scanRange1000   Run random seek scan with both start and stop row (max 1000 rows)
 scanRange10000  Run random seek scan with both start and stop row (max 10000 rows)
 sequentialRead  Run sequential read test
 sequentialWrite Run sequential write test

Args:
 nclients        Integer. Required. Total number of clients (and HRegionServers) running. 1 <= value <= 500
Examples:
 To run a single client doing the default 1M sequentialWrites:
 $ bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation sequentialWrite 1
 To run 10 clients doing increments over ten rows:
 $ bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10 --nomapred increment 10

（可左右滑動(dòng)）

不加任何參數(shù)就會(huì)輸出如上usage提示，基本使用就是：

hbase pe <OPTIONS> [-D<property=value>]* <command> <nclients>

這里介紹幾個(gè)常用的重要參數(shù)：

• nomapred：表示采用MapReduce多線程測(cè)試還是本地多線程測(cè)試鳖悠，一般采用本地多線程的方式榜掌，在命令中加上--nomapred即可；

• oneCon：是否所有線程使用一個(gè)Connection連接乘综，默認(rèn)false憎账，表示每個(gè)線程都會(huì)創(chuàng)建一個(gè)HBase Connection，這樣不合理卡辰，建議設(shè)置為true鼠哥，命令中加--oneCon=true即可；

• valueSize：寫入HBase的value的size看政，單位Byte，默認(rèn)值為1000抄罕。需要根據(jù)實(shí)際的業(yè)務(wù)字段值的大小設(shè)置valueSize允蚣，比如--valueSize=100；

• table：測(cè)試表的名稱呆贿，如果不設(shè)置則默認(rèn)為TestTable嚷兔；

• rows：?jiǎn)蝹€(gè)線程測(cè)試的行數(shù)，默認(rèn)值為1048576做入，實(shí)際測(cè)試時(shí)可自行制定冒晰，比如--rows=100000。注意這是單線程的行數(shù)竟块，實(shí)際行數(shù)要乘以線程數(shù)壶运，比如10個(gè)線程寫入時(shí)就會(huì)往HBase中寫100000*10=100w條記錄；

• size：?jiǎn)蝹€(gè)線程測(cè)試的大小浪秘，單位為GB蒋情，默認(rèn)值為1，這個(gè)參數(shù)與rows是互斥的耸携，不能同時(shí)設(shè)置棵癣；

• compress：設(shè)置表的壓縮算法，默認(rèn)None夺衍，表示不壓縮狈谊，可以根據(jù)實(shí)際情況設(shè)置比如--compress=SNAPPY。這個(gè)設(shè)置也可以用來測(cè)試不同壓縮算法對(duì)讀寫性能的影響；

• presplit：表的預(yù)分區(qū)數(shù)量即region個(gè)數(shù)河劝，一般要參考regionserver數(shù)量壁榕，設(shè)置一個(gè)合理值以避免數(shù)據(jù)熱點(diǎn)和影響測(cè)試結(jié)果，比如--presplit=10丧裁；

• autoFlush：寫入操作的autoFlush屬性护桦，默認(rèn)false，這里是BufferedMutator寫入方式煎娇，禁用autoFlush表示會(huì)批量寫入二庵，一般建議設(shè)置為true以獲得單條寫的性能測(cè)試，即--autoFlush=true缓呛；

• caching：Scan讀操作的caching屬性催享，默認(rèn)值為30，一般可以根據(jù)實(shí)際使用設(shè)置哟绊，比如--caching=100因妙；

其他參數(shù)通常可以默認(rèn)或根據(jù)自己的場(chǎng)景調(diào)整票髓，這里不多介紹攀涵。此外，command 是PE支持的讀寫測(cè)試類型洽沟，包括randomRead以故，randomWrite，SequentialRead裆操，SequentialWrite等怒详，具體如上。nclients 就是開啟的線程數(shù)量踪区。

二昆烁、HBase 讀寫壓測(cè) P999

集群環(huán)境

當(dāng)前測(cè)試集群包含2個(gè)HMaster、8個(gè)RS節(jié)點(diǎn)缎岗，服務(wù)器配置：24核CPU静尼；128G內(nèi)存；1T*6 HDD磁盤传泊，HBase堆大小配置為16G茅郎，版本為1.2.0-cdh5.11.0。因此這是一個(gè)HBase 1.x的P999性能壓測(cè)或渤，同樣適用于HBase 2.x系冗。

壓測(cè)案例

這里分別測(cè)試了randomWrite、sequentialWrite薪鹦，randomRead及sequentialRead的延時(shí)情況掌敬，給出當(dāng)前環(huán)境下的P99及P999 latency指標(biāo)供參考惯豆。

在各個(gè)測(cè)試case中，使用PE的本地多線程模式即--nomapred奔害，測(cè)試表包含16個(gè)region楷兽，采用Snappy壓縮，并且value大小為100Byte华临，我們相應(yīng)的開了16個(gè)線程進(jìn)行測(cè)試芯杀，寫入測(cè)試時(shí)均關(guān)閉了autoFlush。PE運(yùn)行完成后會(huì)分別打出每個(gè)線程的延遲狀況雅潭，這里貼出了其中一個(gè)線程的測(cè)試結(jié)果揭厚，具體如下：

1、randomWrite

每個(gè)線程向rw_test_1表中隨機(jī)寫入100w條記錄：

[root@xxx ~]$ hbase pe --nomapred --oneCon=true --table=rw_test_1 --rows=1000000 --valueSize=100 --compress=SNAPPY --presplit=16 --autoFlush=true randomWrite 16

20/02/22 15:06:07 INFO hbase.PerformanceEvaluation: Latency (us) : mean=186.42, min=0.00, max=594880.00, stdDev=6981.60, 50th=1.00, 75th=2.00, 95th=28.00, 99th=1020.00, 99.9th=3941.00, 99.99th=381319.92, 99.999th=503455.66
20/02/22 15:06:07 INFO hbase.PerformanceEvaluation: Num measures (latency) : 1000000
20/02/22 15:06:07 INFO hbase.PerformanceEvaluation: Mean      = 186.42
Min       = 0.00
Max       = 594880.00
StdDev    = 6981.60
50th      = 1.00
75th      = 2.00
95th      = 28.00
99th      = 1020.00
99.9th    = 3941.00
99.99th   = 381319.92
99.999th  = 503455.66

（可左右滑動(dòng)）

測(cè)試結(jié)果：該case中扶供，HBase隨機(jī)寫P999延時(shí)大概在4ms左右筛圆。

2、sequentialWrite

每個(gè)線程向rw_test_2表中順序?qū)懭?G數(shù)據(jù)：

[root@xxx ~]$ hbase pe --nomapred --oneCon=true --table=rw_test_2 --size=1 --valueSize=100 --compress=SNAPPY --presplit=16 --autoFlush=true sequentialWrite 16

20/02/22 16:24:49 INFO hbase.PerformanceEvaluation: Latency (us) : mean=220.51, min=0.00, max=1440185.00, stdDev=10022.38, 50th=1.00, 75th=2.00, 95th=132.00, 99th=396.00, 99.9th=1152.00, 99.99th=515707.37, 99.999th=917447.01
20/02/22 16:24:49 INFO hbase.PerformanceEvaluation: Num measures (latency) : 1048576
20/02/22 16:24:49 INFO hbase.PerformanceEvaluation: Mean      = 220.51
Min       = 0.00
Max       = 1440185.00
StdDev    = 10022.38
50th      = 1.00
75th      = 2.00
95th      = 132.00
99th      = 396.00
99.9th    = 1152.00
99.99th   = 515707.37
99.999th  = 917447.01

（可左右滑動(dòng)）

測(cè)試結(jié)果：該case中椿浓，HBase順序?qū)慞999延時(shí)大概在1.2ms左右太援。

3、randomRead

以rw_test_2表為例扳碍，隨機(jī)讀取數(shù)據(jù)：

[root@xxx ~]$ hbase pe --nomapred --oneCon=true --table=rw_test_2 --size=1 --valueSize=100 randomRead 16

20/02/22 16:53:48 INFO hbase.PerformanceEvaluation: Latency (us) : mean=748.70, min=74.00, max=2161876.00, stdDev=5055.01, 50th=289.00, 75th=364.00, 95th=2665.00, 99th=4579.00, 99.9th=78024.00, 99.99th=100495.98, 99.999th=150378.50
20/02/22 16:53:48 INFO hbase.PerformanceEvaluation: Num measures (latency) : 1048576
20/02/22 16:53:48 INFO hbase.PerformanceEvaluation: Mean      = 748.70
Min       = 74.00
Max       = 2161876.00
StdDev    = 5055.01
50th      = 289.00
75th      = 364.00
95th      = 2665.00
99th      = 4579.00
99.9th    = 78024.00
99.99th   = 100495.98
99.999th  = 150378.50

（可左右滑動(dòng)）

測(cè)試結(jié)果：該case中提岔，HBase隨機(jī)讀P999延時(shí)大概在78ms左右，小于100ms笋敞。

4唧垦、sequentialRead

以rw_test_2表為例，順序讀取數(shù)據(jù)：

[root@xxx ~]$ hbase pe --nomapred --oneCon=true --table=rw_test_2 --size=1 --valueSize=100 sequentialRead 16

20/02/22 17:08:41 INFO hbase.PerformanceEvaluation: Latency (us) : mean=593.44, min=86.00, max=183676.00, stdDev=4299.28, 50th=302.00, 75th=398.00, 95th=633.00, 99th=932.00, 99.9th=75718.98, 99.99th=93035.20, 99.999th=135947.24
20/02/22 17:08:41 INFO hbase.PerformanceEvaluation: Num measures (latency) : 1048576
20/02/22 17:08:42 INFO hbase.PerformanceEvaluation: Mean      = 593.44
Min       = 86.00
Max       = 183676.00
StdDev    = 4299.28
50th      = 302.00
75th      = 398.00
95th      = 633.00
99th      = 932.00
99.9th    = 75718.98
99.99th   = 93035.20
99.999th  = 135947.24

（可左右滑動(dòng)）

測(cè)試結(jié)果：該case中液样，HBase順序讀P999延時(shí)大概在75ms左右。

三巧还、總結(jié)

本文介紹了如何使用HBase自帶的PE工具進(jìn)行讀寫延時(shí)測(cè)試鞭莽，PE主要用于測(cè)試HBase的讀寫延時(shí)指標(biāo)比如P999延時(shí)，但暫時(shí)不支持HBase吞吐量指標(biāo)測(cè)試比如單機(jī)TPS（后面會(huì)介紹YCSB基準(zhǔn)測(cè)試）麸祷。希望通過本文的一些量化指標(biāo)澎怒，能夠讓我們對(duì)HBase讀寫速度有一個(gè)大概認(rèn)識(shí)。

參考：
1阶牍、HBase2.0中的Benchmark工具-PerformanceEvaluation
2喷面、一場(chǎng)HBase2.x的寫入性能優(yōu)化之旅

如果您喜歡這篇文章，點(diǎn)【在看】與轉(zhuǎn)發(fā)都是一種鼓勵(lì)走孽，期待得到您的認(rèn)可 ?(^_-)