問題
生產環(huán)境springboot工程卡死入愧，檢查日志發(fā)現了以下問題：
Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 60000, active 0, maxActive 20
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
at org.mybatis.spring.transaction.SpringManagedTransaction.openConnection(SpringManagedTransaction.java:84)
at org.mybatis.spring.transaction.SpringManagedTransaction.getConnection(SpringManagedTransaction.java:70)
at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:336)
at org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:84)
at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:62)
at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:324)
at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156)
at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109)
at com.github.pagehelper.PageInterceptor.intercept(PageInterceptor.java:143)
at org.apache.ibatis.plugin.Plugin.invoke(Plugin.java:61)
at com.sun.proxy.$Proxy109.query(Unknown Source)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:148)
... 34 common frames omitted
Caused by: com.alibaba.druid.pool.GetConnectionTimeoutException: wait millis 60000, active 0, maxActive 20
at com.alibaba.druid.pool.DruidDataSource.getConnectionInternal(DruidDataSource.java:1265)
at com.alibaba.druid.pool.DruidDataSource.getConnectionDirect(DruidDataSource.java:1086)
at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:4544)
at com.alibaba.druid.filter.stat.StatFilter.dataSource_getConnection(StatFilter.java:670)
at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:4540)
at com.alibaba.druid.filter.FilterAdapter.dataSource_getConnection(FilterAdapter.java:2723)
at com.alibaba.druid.filter.FilterChainImpl.dataSource_connect(FilterChainImpl.java:4540)
at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1064)
at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:1056)
at com.alibaba.druid.pool.DruidDataSource.getConnection(DruidDataSource.java:104)
at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
... 46 common frames omitted
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at java.util.Timer.<init>(Timer.java:176)
at org.postgresql.util.SharedTimer.getTimer(SharedTimer.java:45)
at org.postgresql.jdbc.PgConnection.getTimer(PgConnection.java:1165)
at org.postgresql.jdbc.PgConnection.addTimerTask(PgConnection.java:1178)
at org.postgresql.jdbc.PgStatement.startTimer(PgStatement.java:889)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:429)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:356)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:303)
at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:289)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:266)
at org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:233)
at com.alibaba.druid.pool.vendor.PGValidConnectionChecker.isValidConnection(PGValidConnectionChecker.java:65)
at com.alibaba.druid.pool.DruidAbstractDataSource.validateConnection(DruidAbstractDataSource.java:1290)
at com.alibaba.druid.pool.DruidAbstractDataSource.createPhysicalConnection(DruidAbstractDataSource.java:1535)
at com.alibaba.druid.pool.DruidDataSource$CreateConnectionThread.run(DruidDataSource.java:2100)
分析原因
可能的原因
OutOfMemoryError: unable to create new native thread出現的情況有兩種：
- 服務器剩余內存不足（非JVM內存）卷中，不能創(chuàng)建新的線程
能創(chuàng)建的線程數的具體計算公式如下：
(MaxProcessMemory - JVMMemory - ReservedOsMemory) / (ThreadStackSize) = Number of threads
MaxProcessMemory 指的是一個進程的最大內存
JVMMemory JVM內存
ReservedOsMemory 保留的操作系統(tǒng)內存
ThreadStackSize 線程棧的大小
- 超出服務器用戶最大進程限制:
通過以下命令可以查看（注意澈魄，不同用戶，最大進程限制配置可能不一樣）：
ulimit -u
image.png
參考資料：
https://blog.csdn.net/thwsir/article/details/86480956
https://www.cnblogs.com/svennee/p/4331549.html
https://blog.csdn.net/qq171563857/article/details/94590992
分析
通過檢查jvm參數窟她，發(fā)現沒有設置-Xss纳账，使用的默認大小兄旬，其次幅聘，使用free -m查看服務器可用內存凡纳，發(fā)現還有很多剩余，懷疑是超出服務器用戶最大進程限制帝蒿，由于需要先修復生產環(huán)境荐糜，沒有過多時間分析定位錯誤，只保留了日志以及設置了用戶的最大進程數葛超，就重啟了工程暴氏。
事后模擬
- 在java工程中定義一個http接口，當調用時绣张，不停的創(chuàng)建和啟動線程答渔。
@GetMapping("/testThread")
public void testThread() {
List<Thread> list = new ArrayList<>();
while(true) {
Thread t = new Thread(() -> {
try {
Thread.sleep(600000L);
} catch (InterruptedException e) {
log.error("###",e);
}
});
t.start();
list.add(t);
log.info(String.valueOf(list.size()));
}
}
-
設置服務器
java用戶的最大進程數量為1000個。
image.png 在服務器啟動java工程侥涵，并調用測試線程接口研儒，成功模擬了生產異常。
image.png
- 通過
pstree查看java工程的線程數量
pstree -p java進程id | wc -l
image.png
修復
在文件/etc/security/limits.d/20-nproc.conf調整用戶的最大進程數
* soft nproc 4096
java soft nproc 1000
root soft nproc unlimited
說明：
root用戶無限制
java用戶1000個
其他用戶4096個
