前言

先說(shuō)一下測(cè)試的背景伐憾，本次測(cè)試是關(guān)于一個(gè)回調(diào)接口眯漩，沒(méi)有業(yè)務(wù)邏輯菌湃，期望系統(tǒng)的處理能力可以達(dá)到3000TPS。

大概的業(yè)務(wù)調(diào)用關(guān)系是這樣的：

image.png

問(wèn)題

在壓測(cè)開始巷折，沒(méi)覺(jué)得的會(huì)有什么問(wèn)題压鉴，首先腳本很簡(jiǎn)單，一個(gè)post接口锻拘，沒(méi)有進(jìn)行任何的參數(shù)化油吭，只是單純的向服務(wù)器發(fā)送請(qǐng)求，且接口沒(méi)有任何返回值逊拍。

• 接口的壓測(cè)策略上鞠，設(shè)置的梯度加壓，最大并發(fā)用戶就只有4個(gè)

• 這個(gè)是TPS芯丧，從下圖可以看出芍阎，是很有規(guī)律的波動(dòng)

image.png

• 這個(gè)是響應(yīng)時(shí)間，同TPS一樣缨恒，很有規(guī)律的波動(dòng)

image.png

• 下圖為聚合報(bào)告谴咸，由此可以看此轮听，99%的響應(yīng)時(shí)間均小于4ms，最大響應(yīng)時(shí)間為3s多岭佳。

image.png

根據(jù)以上結(jié)果血巍，開始就覺(jué)得，是不是服務(wù)器的問(wèn)題珊随，但是監(jiān)控服務(wù)器述寡，cpu、內(nèi)存叶洞、io等都很正常鲫凶，這時(shí)又把目光轉(zhuǎn)向壓力機(jī)和jmeter本身。

這時(shí)衩辟，就想到是不是壓力機(jī)本身出現(xiàn)了瓶頸螟炫，雖說(shuō)考慮到其配置覺(jué)得可能性比較小（32核64G）艺晴，通過(guò)win任務(wù)管理器查看機(jī)器的性能數(shù)據(jù)如下：

• 壓力機(jī)：

可以看到昼钻，在壓測(cè)時(shí)，機(jī)器用開啟了1130個(gè)線程數(shù)封寞，71個(gè)進(jìn)程數(shù)

image.png

根據(jù)測(cè)試結(jié)果然评，響應(yīng)時(shí)間99%小于4ms，于是就設(shè)置了一個(gè)超時(shí)時(shí)間钥星，設(shè)置為30ms沾瓦，再次執(zhí)行腳本。這時(shí)谦炒，得到的結(jié)果可以看到 大多數(shù)為鏈接失敗贯莺，但有兩種鏈接失敗的情況，于是上網(wǎng)查找以下問(wèn)題可能的原因：

image.png

最后宁改，看到一個(gè)教程說(shuō)缕探，需要設(shè)置以下Advanced-Implementation，選擇“java”还蹲。于是乎爹耗，就嘗試的設(shè)置一下，結(jié)果谜喊，后續(xù)請(qǐng)求一切正常潭兽。

image.png

Abvanced：

java、HttpClint4

jmeter官方文檔給出的說(shuō)明我復(fù)制了下來(lái)：

HTTP Request
This sampler lets you send an HTTP/HTTPS request to a web server.
It also lets you control whether or not JMeter parses HTML files for images 
and other embedded resources and sends HTTP requests to retrieve them. 
The following types of embedded resource are retrieved:
      images
      applets
      stylesheets (CSS) and resources referenced from those files
      external scripts
      frames, iframes
      background images (body, table, TD, TR)
      background sound
      
      
The default parser is org.apache.jmeter.protocol.http.parser.LagartoBasedHtmlParser.
This can be changed by using the property "htmlparser.className" - see jmeter.properties for details.

If you are going to send multiple requests to the same web server, 
consider using an HTTP Request Defaults Configuration Element so you do not have to enter the same information for each HTTP Request.

Or, instead of manually adding HTTP Requests, you may want to use JMeter's HTTP(S) Test Script Recorder to create them.
This can save you time if you have a lot of HTTP requests or requests with many parameters.

There are two different test elements used to define the samplers:

AJP/1.3 Sampler
    uses the Tomcat mod_jk protocol (allows testing of Tomcat in AJP mode without needing Apache httpd) 
  The AJP Sampler does not support multiple file upload; only the first file will be used.
HTTP Request
    this has an implementation drop-down box, which selects the HTTP protocol implementation to be used:
Java
    uses the HTTP implementation provided by the JVM. 
  This has some limitations in comparison with the HttpClient implementations - see below.
HTTPClient4
    uses Apache HttpComponents HttpClient 4.x.
Blank Value
    does not set implementation on HTTP Samplers, 
  so relies on HTTP Request Defaults if present or on jmeter.httpsampler property defined in jmeter.properties

The Java HTTP implementation has some limitations:

There is no control over how connections are re-used. When a connection is released by JMeter, it may or may not be re-used by the same thread.
The API is best suited to single-threaded usage - various settings are defined via system properties, and therefore apply to all connections.
No support of Kerberos authentication
It does not support client based certificate testing with Keystore Config.
Better control of Retry mechanism
It does not support virtual hosts.
It supports only the following methods: GET, POST, HEAD, OPTIONS, PUT, DELETE and TRACE
Better control on DNS Caching with DNS Cache Manager

個(gè)人理解--待完善

關(guān)于此處斗遏，如何去理解山卦？

問(wèn)題2

在壓測(cè)過(guò)程中，當(dāng)線程數(shù)和請(qǐng)求數(shù)比較少的時(shí)候诵次，一切正常账蓉，但是隨著線程數(shù)增加枚碗，請(qǐng)求量上升，就開始一直報(bào)錯(cuò)铸本。
線程數(shù)比較少時(shí)肮雨，正常；
當(dāng)線程說(shuō)多時(shí)箱玷，報(bào)錯(cuò)信息如下：

2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Unexpected end of file from server
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Unexpected end of file from server
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: readResponse: java.net.SocketException: Socket closed
2020-02-18 14:06:01,584 ERROR o.a.j.p.h.s.HTTPJavaImpl: Cause: java.net.SocketException: Socket closed

解決思路

在網(wǎng)上查詢一下報(bào)錯(cuò)信息怨规，是說(shuō)重定向的次數(shù)被限制了，但因?yàn)樽约簤簻y(cè)的接口沒(méi)有重定向锡足，而且前期一直正常椅亚，到后面才出現(xiàn)這個(gè)問(wèn)題，這不科學(xué)舱污。所以就繼續(xù)尋找問(wèn)題。
首先弥虐，因?yàn)槲覊簻y(cè)的鏈路是這樣的

SLB --- nginx--kong---docker swarm --具體應(yīng)用

這時(shí)扩灯，嘗試性的跳過(guò)SLB ，直接到nginx霜瘪，發(fā)現(xiàn)壓測(cè)結(jié)果一切正常珠插，這時(shí)可以初步判斷問(wèn)題在SLB。

經(jīng)過(guò)查詢颖对，發(fā)現(xiàn)是因?yàn)镾LB有一個(gè)防攻擊的配置捻撑，將后面大量的請(qǐng)求，當(dāng)作了攻擊缤底，所以才不行顾患。

至此，問(wèn)題解決个唧。

以下是重定向請(qǐng)求的一些響應(yīng)信息江解，僅作為記錄：

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html>
<head><title>302 Found</title></head>
<body bgcolor="white">
<h1>302 Found</h1>
<p>The requested resource resides temporarily under a different URI.</p>
<hr/>Powered by Tengine</body>
</html>

HTTP/1.1 302 Found
Server: Tengine
Date: Tue, 18 Feb 2020 10:14:20 GMT
Content-Type: text/html
Content-Length: 258
Connection: keep-alive
Set-Cookie: acw_sc__v1=5e4bb8fc37967fa0c859c94a7f9dafd851d8bb95;path=/;Max-Age=1800
cache-control: no-cache, no-store
Pragma: no-cache
Set-Cookie: acw_tc=784c10e715820208607852086e610bc1cb505dd898c94daf023389aca8c4d8;path=/;HttpOnly;Max-Age=2678401
Location: /application-center/api/v1/xxx/xxx/20000001103