故障描述
時間锡溯，監(jiān)控一直報警赶舆，某個接口的響應時間會周期性的變慢；檢查報警日志發(fā)現(xiàn)趾唱，報警機器都來自同一個機房涌乳；奇怪的是，其他機房的同一接口卻并沒有變慢甜癞；
排查過程
首先懷疑訪問的資源是不是有問題夕晓，檢查之后發(fā)現(xiàn)并沒有任何問題；向運維申請權限之后悠咱，查看機器的GC日志蒸辆，一看，嚇了一條析既，機器在不停的進行CMS GC,日志如下：
2017-07-10T00:27:38.561+0800: 222046.226: [GC (CMS Initial Mark) [1 CMS-initial-mark: 761762K(1048576K)] 866632K(1992320K), 0.0188394 secs] [Times: user=0.11 sys=0.00, real=0.02 secs]
2017-07-10T00:27:38.581+0800: 222046.245: [CMS-concurrent-mark-start]
2017-07-10T00:27:38.992+0800: 222046.656: [CMS-concurrent-mark: 0.411/0.411 secs] [Times: user=0.85 sys=0.01, real=0.41 secs]
2017-07-10T00:27:38.992+0800: 222046.656: [CMS-concurrent-preclean-start]
2017-07-10T00:27:38.996+0800: 222046.660: [CMS-concurrent-preclean: 0.004/0.004 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2017-07-10T00:27:38.996+0800: 222046.660: [CMS-concurrent-abortable-preclean-start]
CMS: abort preclean due to time 2017-07-10T00:27:44.134+0800: 222051.799: [CMS-concurrent-abortable-preclean: 5.137/5.138 secs] [Times: user=5.32 sys=0.06, real=5.14 secs]
2017-07-10T00:27:44.135+0800: 222051.800: [GC (CMS Final Remark) [YG occupancy: 140252 K (943744 K)]2017-07-10T00:27:44.136+0800: 222051.800: [Rescan (parallel) , 0.0261302 secs]2017-07-10T00:27:44.162+0800: 222051.826: [weak refs processing, 0.0001257 secs]2017-07-10T00:27:44.162+0800: 222051.826: [class unloading, 0.0235377 secs]2017-07-10T00:27:44.185+0800: 222051.850: [scrub symbol table, 0.0064071 secs]2017-07-10T00:27:44.192+0800: 222051.856: [scrub string table, 0.0012990 secs][1 CMS-remark: 761762K(1048576K)] 902015K(1992320K), 0.0577149 secs] [Times: user=0.24 sys=0.00, real=0.06 secs]
2017-07-10T00:27:44.193+0800: 222051.858: [CMS-concurrent-sweep-start]
2017-07-10T00:27:44.963+0800: 222052.627: [CMS-concurrent-sweep: 0.769/0.769 secs] [Times: user=0.80 sys=0.02, real=0.77 secs]
2017-07-10T00:27:44.963+0800: 222052.627: [CMS-concurrent-reset-start]
2017-07-10T00:27:44.965+0800: 222052.629: [CMS-concurrent-reset: 0.002/0.002 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2017-07-10T00:28:36.805+0800: 222104.470: [GC (Allocation Failure) 2017-07-10T00:28:36.805+0800: 222104.470: [ParNew
Desired survivor size 80510976 bytes, new threshold 6 (max 6)
- age 1: 1670280 bytes, 1670280 total
: 943744K->104832K(943744K), 0.0949423 secs] 1323361K->589707K(1992320K), 0.0951679 secs] [Times: user=0.53 sys=0.01, real=0.10 secs]
2017-07-10T00:28:37.823+0800: 222105.488: [GC (Allocation Failure) 2017-07-10T00:28:37.824+0800: 222105.488: [ParNew
Desired survivor size 80510976 bytes, new threshold 1 (max 6)
- age 1: 106303472 bytes, 106303472 total
- age 2: 1043512 bytes, 107346984 total
: 943744K->104832K(943744K), 0.4966488 secs] 1428619K->717627K(1992320K), 0.4968997 secs] [Times: user=3.66 sys=0.01, real=0.50 secs]
2017-07-10T00:29:20.274+0800: 222147.938: [GC (Allocation Failure) 2017-07-10T00:29:20.274+0800: 222147.939: [ParNew
Desired survivor size 80510976 bytes, new threshold 1 (max 6)
- age 1: 107347632 bytes, 107347632 total
: 943744K->104832K(943744K), 0.3041730 secs] 1556539K->867316K(1992320K), 0.3044353 secs] [Times: user=2.12 sys=0.01, real=0.30 secs]
2017-07-10T00:29:20.580+0800: 222148.244: [GC (CMS Initial Mark) [1 CMS-initial-mark: 762484K(1048576K)] 867363K(1992320K), 0.0184973 secs] [Times: user=0.10 sys=0.01, real=0.02 secs]
2017-07-10T00:29:20.599+0800: 222148.263: [CMS-concurrent-mark-start]
2017-07-10T00:29:21.034+0800: 222148.698: [CMS-concurrent-mark: 0.435/0.435 secs] [Times: user=0.91 sys=0.01, real=0.44 secs]
2017-07-10T00:29:21.034+0800: 222148.698: [CMS-concurrent-preclean-start]
2017-07-10T00:29:21.040+0800: 222148.704: [CMS-concurrent-preclean: 0.006/0.006 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2017-07-10T00:29:21.040+0800: 222148.704: [CMS-concurrent-abortable-preclean-start]
CMS: abort preclean due to time 2017-07-10T00:29:26.197+0800: 222153.862: [CMS-concurrent-abortable-preclean: 5.157/5.157 secs] [Times: user=5.38 sys=0.07, real=5.16 secs]
2017-07-10T00:29:26.199+0800: 222153.863: [GC (CMS Final Remark) [YG occupancy: 152740 K (943744 K)]2017-07-10T00:29:26.199+0800: 222153.863: [Rescan (parallel) , 0.0263032 secs]2017-07-10T00:29:26.225+0800: 222153.889: [weak refs processing, 0.0001888 secs]2017-07-10T00:29:26.225+0800: 222153.890: [class unloading, 0.0286006 secs]2017-07-10T00:29:26.254+0800: 222153.918: [scrub symbol table, 0.0113296 secs]2017-07-10T00:29:26.265+0800: 222153.930: [scrub string table, 0.0016177 secs][1 CMS-remark: 762484K(1048576K)] 915225K(1992320K), 0.0682597 secs] [Times: user=0.24 sys=0.00, real=0.07 secs]
2017-07-10T00:29:26.267+0800: 222153.932: [CMS-concurrent-sweep-start]
2017-07-10T00:29:26.985+0800: 222154.650: [CMS-concurrent-sweep: 0.718/0.718 secs] [Times: user=0.77 sys=0.01, real=0.72 secs]
2017-07-10T00:29:26.986+0800: 222154.650: [CMS-concurrent-reset-start]
2017-07-10T00:29:26.988+0800: 222154.652: [CMS-concurrent-reset: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2017-07-10T00:29:40.748+0800: 222168.412: [GC (Allocation Failure) 2017-07-10T00:29:40.748+0800: 222168.412: [ParNew
Desired survivor size 80510976 bytes, new threshold 6 (max 6)
- age 1: 40404536 bytes, 40404536 total
: 943744K->100515K(943744K), 0.1270597 secs] 1324617K->585666K(1992320K), 0.1273832 secs] [Times: user=0.73 sys=0.02, real=0.12 secs]
2017-07-10T00:29:41.544+0800: 222169.208: [GC (Allocation Failure) 2017-07-10T00:29:41.544+0800: 222169.208: [ParNew
Desired survivor size 80510976 bytes, new threshold 2 (max 6)
- age 1: 67918336 bytes, 67918336 total
- age 2: 39428848 bytes, 107347184 total
: 939427K->104832K(943744K), 0.4560649 secs] 1424578K->782170K(1992320K), 0.4564709 secs] [Times: user=3.26 sys=0.01, real=0.46 secs]
2017-07-10T00:29:42.002+0800: 222169.667: [GC (CMS Initial Mark) [1 CMS-initial-mark: 677338K(1048576K)] 796205K(1992320K), 0.0171894 secs] [Times: user=0.11 sys=0.00, real=0.02 secs]
可以看到每隔一分鐘左右躬贡，由于Allocation Failure導致ParNew GC，而由于存活對象太大眼坏，導致又進行了CMS GC;
通過jmap發(fā)現(xiàn)果然每隔一段時間拂玻，內存中BaseEntry類的對象實例數(shù)目急劇增大，檢查業(yè)務代碼發(fā)現(xiàn)宰译，業(yè)務代碼中有段邏輯會周期性的獲取最新的IP列表檐蚜；而判斷列表是否有更新，是通過對文件內容md5判斷的沿侈，正常情況下闯第，文件變化頻率會很小，那為什么現(xiàn)在會頻繁更新呢缀拭？
調用接口發(fā)現(xiàn)咳短，IP服務返回的md5并沒有變化填帽，但本地文件的md5和服務端不一樣；打開文件發(fā)現(xiàn)咙好，本地文件存在亂碼篡腌；
問題的原因基本找到，應該是由于文件亂碼導致的敷扫，讓開發(fā)人員修改代碼哀蘑，在保存文件時指定編碼，問題得到解決葵第；
故障原因
線上問題是得到了解決绘迁，但是奇怪的是，同樣的代碼為什么其他機房不存在問題呢卒密？檢查服務器編碼配置：
image.png
服務器的Locale配置都是相同的缀台，但是通過jinfo查看java應用的file.encoding,返回的卻不一樣；
詢問運維人員哮奇，不同機房的差異膛腐，運維人員提到，由于發(fā)布系統(tǒng)故障鼎俘，部分機房的應用是通過手工運行發(fā)布腳本發(fā)布的哲身；這樣問題也就清楚了，應該是ssh會將本地的locale帶到服務器環(huán)境導致贸伐；具體的可以參考這篇文章
