docker中如何做jvm的gc、內(nèi)存等問題排查

背景：docker內(nèi)運行的Java服務(wù)不像宿主機直接運行那么方便查詢問題侥祭，今天記錄下，首先鏡像內(nèi)有些命令是不支持的，比如tomcat鏡像沒有l(wèi)s矮冬、top等命令（jdk鏡像默認(rèn)有這些命令）谈宛，不方便查找問題，tomcat的openjdk版本是有的胎署，如tomcat:8.5.49-jdk8-openjdk

參考
碰到系統(tǒng)CPU飆高和頻繁GC吆录，你會怎么排查？
Linux使用jstat命令查看jvm的GC情況

小知識

1 根據(jù)容器名查看對應(yīng)宿主機的進程id

docker inspect -f '{{.State.Pid}}' competent_lichterman

2 查看gc情況琼牧，9是進程id 恢筝，每5s自動刷新一次。

jstat -gc 9 5000

full gc的次數(shù)太多說明程序有問題巨坊，fgc 期間會導(dǎo)致服務(wù)不可用撬槽，后面有fgc的時間

圖片.png

3 查看jvm線程的堆棧信息。查看進程id為1的線程信息趾撵，包含iot代碼字樣的附近30行內(nèi)容侄柔，一般程序出問題，這里會定位到有問題的代碼部分

jstack 1 | grep "iot" -A 30

導(dǎo)出線程信息到文本
jstack -l <pid> >jvm.txt

4 查看整個JVM內(nèi)存狀態(tài)
jmap -heap [pid]
查看JVM堆中對象詳細(xì)占用情況
jmap -histo [pid]
導(dǎo)出整個JVM 中內(nèi)存信息
jmap -dump:format=b,file=文件名 [pid]
jmap -dump:live,format=b,file=d:/20170417.dump PID

[root@01384fe8ffb3 /]# jmap -heap 1
Attaching to process ID 1, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.131-b11

using thread-local object allocation.
Parallel GC with 2 thread(s)

Heap Configuration:
   MinHeapFreeRatio         = 0
   MaxHeapFreeRatio         = 100
   MaxHeapSize              = 2092957696 (1996.0MB)
   NewSize                  = 44040192 (42.0MB)
   MaxNewSize               = 697303040 (665.0MB)
   OldSize                  = 88080384 (84.0MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB
   G1HeapRegionSize         = 0 (0.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 490733568 (468.0MB)
   used     = 24698304 (23.55413818359375MB)
   free     = 466035264 (444.44586181640625MB)
   5.032935509314904% used
From Space:
   capacity = 6815744 (6.5MB)
   used     = 6531296 (6.228729248046875MB)
   free     = 284448 (0.271270751953125MB)
   95.82660381610577% used
To Space:
   capacity = 14155776 (13.5MB)
   used     = 0 (0.0MB)
   free     = 14155776 (13.5MB)
   0.0% used
PS Old Generation
   capacity = 150470656 (143.5MB)
   used     = 50911384 (48.552879333496094MB)
   free     = 99559272 (94.9471206665039MB)
   33.83475911741888% used

35304 interned Strings occupying 4184288 bytes.
[root@01384fe8ffb3 /]#

問題排查思路

1通過docker stats 查看服務(wù)cpu占调、內(nèi)存占用情況暂题，定位有問題的服務(wù)

圖片.png

2 進入容器top命令查看cpu占用最多的進程id，一般jdk鏡像java進程pid都是1

圖片.png

3 查看進程中各線程占用情況
top -Hp 1

圖片.png

4 jstack -F 線程id ：即可查看某線程正在運行情況(不加F會報錯＞可骸８颐摇！)

[root@826d95fa3a92 /]# jstack -F 167
Attaching to process ID 167, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.131-b11
Deadlock Detection:

No deadlocks found.

Thread 436: (state = BLOCKED)
Error occurred during stack walking:
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
        at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:163)
        at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet(LinuxDebuggerLocal.java:482)
        at sun.jvm.hotspot.debugger.linux.LinuxThread.getContext(LinuxThread.java:65)
        at sun.jvm.hotspot.runtime.linux_amd64.LinuxAMD64JavaThreadPDAccess.getCurrentFrameGuess(LinuxAMD64JavaThreadPDAccess.java:93)
        at sun.jvm.hotspot.runtime.JavaThread.getCurrentFrameGuess(JavaThread.java:256)
        at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:218)
        at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:86)
        at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45)
        at sun.jvm.hotspot.tools.JStack.run(JStack.java:66)
        at sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260)
        at sun.jvm.hotspot.tools.Tool.start(Tool.java:223)
        at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
        at sun.jvm.hotspot.tools.JStack.main(JStack.java:92)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at sun.tools.jstack.JStack.runJStackTool(JStack.java:140)
        at sun.tools.jstack.JStack.main(JStack.java:106)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
        at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
        at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLo

5導(dǎo)出堆內(nèi)存數(shù)據(jù)
jmap -dump:format=b,file=myjvm.dump 1