問(wèn)題描述
配置好Hadoop集群(包括hdfs和yarn)富雅，配置好spark-on-yarn，提交任務(wù)后發(fā)現(xiàn)container異常退出肛搬，有core dump產(chǎn)生没佑；修改yarn的資源配置，依然crash温赔。
container日志一旦任務(wù)結(jié)束就被系統(tǒng)自動(dòng)清除蛤奢，只能通過(guò)當(dāng)前控制臺(tái)分析日志；日志中有core dump字樣和java error report的具體地址 /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/application_1523510046494_0001/container_1523510046494_0001_01_000002/hs_err_pid4883.log陶贼，但去查看卻看不到這個(gè)文件啤贩。
# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc:??SuppressErrorAt=/memnode.cpp:2307#
# A fatal error has been detected by the Java Runtime Environment:#
#??Internal Error (/builddir/build/BUILD/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/openjdk/hotspot/src/share/vm/opto/memnode.cpp:2307), pid=4883, tid=0x00007f930d7f6700
#??assert(Opcode() == mem->Opcode() || phase->C->get_alias_index(adr_type()) == Compile::AliasIdxRaw) failed: no mismatched stores, except on raw memory
#
# JRE version: OpenJDK Runtime Environment (8.0_161-b14) (build 1.8.0_161-debug-b14)
# Java VM: OpenJDK 64-Bit Server VM (25.161-b14-debug mixed mode linux-amd64 compressed oops)
#Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
#/tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/application_1523510046494_0001/container_1523510046494_0001_01_000002/hs_err_pid4883.log
#
# Compiler replay data is saved as:
# /tmp/hadoop-hadoop/nm-local-dir/usercache/hadoop/appcache/application_1523510046494_0001/container_1523510046494_0001_01_000002/replay_pid4883.log
#
# If you would like to submit a bug report, please visit:
#http://bugreport.java.com/bugreport/crash.jsp#
Current thread is 140269563373312
Dumping core ...
解決方式
首先修改‘yarn.nodemanager.delete.debug-delay-sec’配置，保留日志文件
yarn-site.xml????????????yarn.nodemanager.delete.debug-delay-sec????????????360000
Error report里看到了錯(cuò)誤信息拜秧， 執(zhí)行的JVM的源代碼
nosuchmethoderror openjdk/hotspot/src/share/vm/prims/jni.cpp
開(kāi)始懷疑是JDK的版本瓜晤，安裝的是debug版本。java-1.8.0-openjdk有如下版本可以安裝腹纳，重新安裝為java-1.8.0-openjdk-devel.x86_64痢掠，問(wèn)題解決驱犹。程序能夠穩(wěn)定執(zhí)行了，速度也變快了足画。
java-1.8.0-openjdk.x86_64 : OpenJDK Runtime Environment
java-1.8.0-openjdk-devel.x86_64 : OpenJDK Development Environment
java-1.8.0-openjdk-debug.x86_64 : OpenJDK Runtime Environment with full debug on
java-1.8.0-openjdk-devel-debug.x86_64 : OpenJDK Development Environment with full debug on
