前幾天公司后端系統(tǒng)出現(xiàn)了故障予借,導(dǎo)致app多個(gè)功能無法使用,查看日志蛔钙,發(fā)現(xiàn)日志出現(xiàn)較多的redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool的異常信息殉挽,顯而易見,jedis/redis出現(xiàn)了問題企垦。因?yàn)槭莄onnection的相關(guān)的問題,所以看了一下jedis和連接數(shù)相關(guān)的配置項(xiàng)晒来,maxIdle和maxTotal都是200钞诡,jedis的封裝也在finally中釋放了connection,所以初步猜測問題發(fā)生在redis服務(wù)端
1.jedis機(jī)器-->ping-->redis機(jī)器,毫秒級的響應(yīng)時(shí)間----網(wǎng)絡(luò)暢通
2.使用netstat -apn |grep redis-server連接數(shù)為20多個(gè)--網(wǎng)絡(luò)連接數(shù)正常
3.free ?-m內(nèi)存使用率60%---(表面上)內(nèi)存夠用
4.df ? ? -h磁盤使用率15%---磁盤空間充足
5.使用redis-cli臭增,執(zhí)行info命令,client部分:
#Clients
connected_clients:18
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
clients數(shù)量也正常
6.使用redis-cli竹习,執(zhí)行ping命令誊抛,異常信息出來了:
(error)MISCONF Redis is configured to save RDB snapshots, but is currently
not able to persist on disk. Commands that may modify the data set
are disabled. Please check Redis logs for details about the error.
然后查看redis日志,出現(xiàn)了
WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
問題已經(jīng)很清晰了整陌,bgsave會fork一個(gè)子進(jìn)程拗窃,因?yàn)関m.overcommit_memory = 0,所以申請的內(nèi)存大小和父進(jìn)程的一樣泌辫,由于redis已經(jīng)使用了60%的內(nèi)存空間随夸,所以fork失敗
解決辦法:
/etc/sysctl.conf 添加 vm.overcommit_memory=1
sysctl ?vm.overcommit_memory=1