問題:每日0點(diǎn)系統(tǒng)內(nèi)存占用飆升,大量掛起線程漏健,系統(tǒng)停頓, log4j2版本:2.7
Thread Stack:
AsyncAppender-async
at sun.misc.Unsafe.park(ZJ)V (Native Method)
at java.util.concurrent.locks.LockSupport.park(Ljava/lang/Object;)V (LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()Z (AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(I)V (AbstractQueuedSynchronizer.java:997)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(I)V (AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.Semaphore.acquire()V (Semaphore.java:312)
at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.rollover(Lorg/apache/logging/log4j/core/appender/rolling/RolloverStrategy;)Z (RollingFileManager.java:247)
at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.rollover()V (RollingFileManager.java:192)
at org.apache.logging.log4j.core.appender.rolling.RollingFileManager.checkRollover(Lorg/apache/logging/log4j/core/LogEvent;)V (RollingFileManager.java:175)
at org.apache.logging.log4j.core.appender.RollingFileAppender.append(Lorg/apache/logging/log4j/core/LogEvent;)V (RollingFileAppender.java:280)
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(Lorg/apache/logging/log4j/core/LogEvent;)V (AppenderControl.java:156)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(Lorg/apache/logging/log4j/core/LogEvent;)V (AppenderControl.java:129)
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(Lorg/apache/logging/log4j/core/LogEvent;)V (AppenderControl.java:120)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(Lorg/apache/logging/log4j/core/LogEvent;)V (AppenderControl.java:84)
at org.apache.logging.log4j.core.appender.AsyncAppender$AsyncThread.callAppenders(Lorg/apache/logging/log4j/core/LogEvent;)Z (AsyncAppender.java:451)
at org.apache.logging.log4j.core.appender.AsyncAppender$AsyncThread.run()V (AsyncAppender.java:404)
Leak Report:
One instance of **"java.util.concurrent.ArrayBlockingQueue"** loaded by **"<system class loader>"** occupies **897,341,688 (95.23%)** bytes. The instance is referenced by **org.apache.logging.log4j.core.appender.AsyncAppender$AsyncThread @ 0xc0700150 AsyncAppender-async** , loaded by **"org.springframework.boot.loader.LaunchedURLClassLoader @ 0xc0400000"**. The memory is accumulated in one instance of **"java.lang.Object[]"** loaded by **"<system class loader>"**.
The stacktrace of this Thread is available. [See stacktrace](../pages/24.html).
**Keywords**
java.util.concurrent.ArrayBlockingQueue
java.lang.Object[]
org.springframework.boot.loader.LaunchedURLClassLoader @ 0xc0400000
原因:每日0點(diǎn)觸發(fā)log4j的rollover策略,semaphore設(shè)置的1個橘霎,所有的log輸出都會在未完成切割任務(wù)時掛起等待切割完成蔫浆。程序中多線程任務(wù)很多且打印日志的地方也很多,就早成瞬間積壓超過了承受能力姐叁。
try {
// Block until the asynchronous operation is completed.
semaphore.acquire();
} catch (final InterruptedException e) {
logError("Thread interrupted while attempting to check rollover", e);
return false;
}
解決:
升級log4j2到2.14版本并且啟用Disruptor異步處理瓦盛,這是一種無鎖的線程間通信庫
- 啟用的方式有兩種,一種是在vm參數(shù)中加入
-Dlog4j2.contextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector
一種是在xml文件中不再使用Root標(biāo)簽外潜,而改用:
<AsyncRoot level="info" includeLocation="true">
<appender-ref ref="Console"/>
<appender-ref ref="RollingFileInfo"/>
<appender-ref ref="RollingFileError"/>
</AsyncRoot>
兩種方式以第一種性能更佳原环,差別不大
- 在classpath下新建文件:log4j2.component.properties
增加配置:
log4j2.AsyncQueueFullPolicy=Discard #隊列滿時放棄日志操作
AsyncLoggerConfig.SynchronizeEnqueueWhenQueueFull=true #當(dāng)隊列滿時同步入列,這是默認(rèn)值处窥,實際還是走AsyncQueueFullPolicy策略
AsyncLoggerConfig.RingBufferSize=131072 #隊列長度嘱吗,Disruptor使用的不是真正意義上的隊列,而是類似于計數(shù)器+事件的方式滔驾。