現象：隊列在拋出異常后不再進行消費伦泥，但是線程仍然存活
查看線程狀態(tài)
- 進入docker
- jps -l查看pid
- jstack -l pid
"pool-1-thread-1" #37 prio=5 os_prio=0 cpu=100.93ms elapsed=125.98s tid=0x00007fc530ccd800 nid=0x39 waiting on condition [0x00007fc4dc9b9000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.2/Native Method)
- parking to wait for <0x0000000095cc6b80> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.2/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.2/AbstractQueuedSynchronizer.java:2081)
at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:138)
at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:306)
at org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64)
at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:192)
at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:185)
at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:107)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:276)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:263)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at com.XXXXXX.taiji.common.api.ExternalAPIClient.lambda$callWithTracer$0(ExternalAPIClient.java:81)
at com.XXXXXX.taiji.common.api.ExternalAPIClient$$Lambda$374/0x00000001005cd440.run(Unknown Source)
at com.XXXXXX.taiji.common.tracer.TracerSupport.lambda$new$0(TracerSupport.java:6)
at com.XXXXXX.taiji.common.tracer.TracerSupport$$Lambda$310/0x000000010045d840.with(Unknown Source)
at com.XXXXXX.taiji.common.tracer.TracerSupport.lambda$tracer$1(TracerSupport.java:17)
at com.XXXXXX.taiji.common.tracer.TracerSupport$$Lambda$331/0x00000001004d6440.run(Unknown Source)
at com.XXXXXX.taiji.common.cat.CatExternalAPITracer.with(CatExternalAPITracer.java:22)
at com.XXXXXX.taiji.common.cat.CatExternalAPITracer.with(CatExternalAPITracer.java:11)
at com.XXXXXX.taiji.common.tracer.TracerSupport.lambda$tracer$2(TracerSupport.java:16)
at com.XXXXXX.taiji.common.tracer.TracerSupport$$Lambda$312/0x000000010045d040.with(Unknown Source)
at com.XXXXXX.taiji.common.api.ExternalAPIClient.callWithTracer(ExternalAPIClient.java:79)
at com.XXXXXX.taiji.common.api.ExternalAPIClient.executeWithJSON(ExternalAPIClient.java:72)
at com.XXXXXX.taiji.qingniao.service.factory.WeiWangSendService.sendMsg(WeiWangSendService.java:76)
at com.XXXXXX.taiji.qingniao.service.factory.AbstractSendService.sendSMS(AbstractSendService.java:63)
at com.XXXXXX.taiji.qingniao.service.MessageService.deal(MessageService.java:286)
at com.XXXXXX.taiji.qingniao.redismq.SendMessageRedisQueue.lambda$start$0(SendMessageRedisQueue.java:74)
at com.XXXXXX.taiji.qingniao.redismq.SendMessageRedisQueue$$Lambda$315/0x000000010045a440.accept(Unknown Source)
at com.XXXXXX.taiji.common.redismq.RedisQueueWorker.lambda$kickit$1(RedisQueueWorker.java:85)
at com.XXXXXX.taiji.common.redismq.RedisQueueWorker$$Lambda$372/0x00000001005ce440.run(Unknown Source)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.2/Executors.java:515)
at java.util.concurrent.FutureTask.run(java.base@11.0.2/FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.2/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.2/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.2/Thread.java:834)
waiting on condition
通過jstack -l pid發(fā)現pool-1-thread-1這個線程有問題
線程狀態(tài)為“waiting on condition”：
說明它在等待另一個條件的發(fā)生，來把自己喚醒，或者干脆它是調用了 sleep(N)档礁。
此時線程狀態(tài)大致為以下幾種：
java.lang.Thread.State: WAITING (parking)：線程掛起，一直等那個條件發(fā)生脊髓；
java.lang.Thread.State: TIMED_WAITING (parking或sleeping)：定時的恭朗，那個條件不到來，也將定時喚醒自己。
at org.apache.http.pool.PoolEntryFuture.await
由at org.apache.http.pool.PoolEntryFuture.await可知弟翘，是http連接池被拿光了連接數
推測可能是由于http的連接資源沒有正確釋放導致的
配合業(yè)務日志打印的異常“api xxxx respond status code 503”
推測可能是http請求過程中拋出異常導致了沒有正確close
代碼排查
this.tracer().with(traceKey, req, () -> {
try {
var res = this.client.execute(req);
var code = res.getStatusLine().getStatusCode();
if (code < 200 || code >= 300) {
throw GlobalErrors.API_STATUS_ERROR.args(path, code);
}
var output = new ByteArrayOutputStream();
var input = res.getEntity().getContent();
IOUtils.copy(input, output);
input.close();
String content = output.toString();
if (content.isEmpty()) {
throw GlobalErrors.API_ACCESS_ERROR.args(path, "http body is empty");
}
holder.value(content);
} catch (IOException e) {
throw GlobalErrors.API_ACCESS_ERROR.args(path, e.getMessage(), e);
}
});
res、output、input一共三個
OutputStream和InputStream的close方法是一個空方法辟癌，交給jvm的gc來處理黍少，不關閉也沒事
this.client.execute(req)返回的是一個CloseableHttpResponse(接口)厂置，源碼比較復雜，最終找到res的close方法調用了releaseConnection
public void close() throws IOException {
this.releaseConnection(false);
}
那么應該是Response沒有close導致的
AutoCloseable
java的輸入輸出流传于、各種Connection沼溜，都繼承了AutoCloseable接口
看了下AutoCloseable接口的源碼系草，注釋比較長，從注釋可知它的出現是為了更好的管理資源能耻，準確說是資源的釋放晓猛，當一個資源類實現了該接口close方法戒职，在使用try-catch-resources語法創(chuàng)建的資源拋出異常后，JVM會自動調用close 方法進行資源釋放捧韵，當沒有拋出異常正常退出try-block時候也會調用close方法纫版。
try-catch-resources語法
try-catch-resources語法自jdk1.7新增其弊，在try的()內部創(chuàng)建資源梭伐，創(chuàng)建的資源在退出try-block時候會自動調用該資源的close方法
示例：
public class AutoCloseableDemo {
public static void main(String[] args) {
try (AutoCloseableApp app = new AutoCloseableApp(); AutoCloseableApp2 app2 = new AutoCloseableApp2()) {
System.out.println("--執(zhí)行main方法--");
throw new RuntimeException("--exception--");
} catch (Exception e) {
System.out.println(e.getMessage());
} finally {
System.out.println("--finally--");
}
}
public static class AutoCloseableApp implements AutoCloseable {
@Override
public void close() throws Exception {
System.out.println("--close1--");
}
}
public static class AutoCloseableApp2 implements AutoCloseable {
@Override
public void close() throws Exception {
System.out.println("--close2--");
}
}
}
輸出：
--執(zhí)行main方法--
--close2--
--close1--
--exception--
--finally--
- 由帶資源的try語句管理的資源必須是實現了AutoCloseable接口的類的對象
- 在try代碼中聲明的資源被隱式聲明為fianl
- 通過使用分號分隔每個聲明可以管理多個資源
- 執(zhí)行順序：close-catch-finally
- 有多個資源時赂苗，關閉順序為資源聲明順序的反序
修改后的代碼
this.tracer().with(traceKey, req, () -> {
try (var res = this.client.execute(req)) {
var code = res.getStatusLine().getStatusCode();
if (code < 200 || code >= 300) {
throw GlobalErrors.API_STATUS_ERROR.args(path, code);
}
var output = new ByteArrayOutputStream();
var input = res.getEntity().getContent();
IOUtils.copy(input, output);
String content = output.toString();
if (content.isEmpty()) {
throw GlobalErrors.API_ACCESS_ERROR.args(path, "http body is empty");
}
holder.value(content);
} catch (IOException e) {
throw GlobalErrors.API_ACCESS_ERROR.args(path, e.getMessage(), e);
}
});
問題來了拌滋，為什么不拋出異常的時候不close也沒事赌渣？
this.tracer().with(traceKey, req, () -> {
try {
var res = this.client.execute(req);
var code = res.getStatusLine().getStatusCode();
var output = new ByteArrayOutputStream();
var input = res.getEntity().getContent();
IOUtils.copy(input, output);
if (code < 1000) {
throw GlobalErrors.API_STATUS_ERROR.args(path, code);
}
String content = output.toString();
if (content.isEmpty()) {
throw GlobalErrors.API_ACCESS_ERROR.args(path, "http body is empty");
}
holder.value(content);
} catch (IOException e) {
throw GlobalErrors.API_ACCESS_ERROR.args(path, e.getMessage(), e);
}
});
以上這段代碼，一定會拋出異常throw GlobalErrors.API_STATUS_ERROR.args(path, code)鸿竖，res也沒有close，但是并不會出現線程park的情況。
當把IOUtils.copy(input, output)這一行代碼挪到拋出異常下面時桩撮，線程會park
查看了一下IOUtils.copy的源碼，也沒發(fā)現什么特殊之處
還需要繼續(xù)研究
