HDP3.1.5 默認(rèn)spark版本2.4 昌妹，希望將spark升級到3
1篙耗、去spark官網(wǎng)上下載spark3版本尊剔，選擇spark3-hadoop3.2 崖媚，雖然hadoop版本高于hdp3.15沒關(guān)系，不然spark-sql不能用塑顺。
2汤求、把下載好的spark3目錄放在 /usr/hdp/3.1.5-xx/spark3 （放啥地方都行）
3、拷貝/etc/spark2/conf/spark-defaults.conf 茬暇、/etc/spark2/conf/spark-env.sh首昔、hive-site.xml 到 spark3/conf 下
4寡喝、在spark-defaults.conf 添加 下面參數(shù)
spark.driver.extraJavaOptions -Dhdp.version=3.1.5.0-152
spark.yarn.am.extraJavaOptions -Dhdp.version=3.1.5.0-152
spark.yarn.archive hdfs:///hdp/apps/spark3/spark3.zip
5糙俗、到/usr/hdp/3.1.5-xx/spark3/jars目錄下
zip spark3.zip ./*
hdfs dfs -put ./spark3.zip hdfs:///hdp/apps/spark3/
6、在spark3/conf目錄下的spark-env.sh 添加
export SPARK_DIST_CLASSPATH=$(hadoop classpath)
下面是已經(jīng)存儲需要修改成正確的
export SPARK_CONF_DIR=${SPARK_CONF_DIR:-/usr/hdp/3.1.5.0-152/spark3/conf/}
7预鬓、測試
/usr/hdp/3.1.5-xx/spark3/bin/spark-sql
/usr/hdp/3.1.5-xx/spark3/bin/spark-submit
Note: 這種多版本共存的版本巧骚，需要在gateway的機(jī)器上操作赊颠。
進(jìn)入到spark3目錄，執(zhí)行bin/spark-submit 就是spark3
進(jìn)入/usr/hdp/3.1.5-xx/spark2 目錄劈彪，執(zhí)行bin/spark-submit 就是 spark2
互相不干擾竣蹦，工作節(jié)點(diǎn)不需要調(diào)整。
升級完以后spark3 cluster模式提交任務(wù)報(bào)錯(cuò)沧奴。
15/09/01 21:54:05 INFO yarn.Client: Application report for application_1441066518301_0013 (state: ACCEPTED)
15/09/01 21:54:05 INFO yarn.Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1441144443866
final status: UNDEFINED
tracking URL: http://yarnmaster-8245.lvs01.dev.ebayc3.com:8088/proxy/application_1441066518301_0013/
user: stack
15/09/01 21:54:06 INFO yarn.Client: Application report for application_1441066518301_0013 (state: ACCEPTED)
15/09/01 21:54:10 INFO yarn.Client: Application report for application_1441066518301_0013 (state: FAILED)
15/09/01 21:54:10 INFO yarn.Client:
client token: N/A
diagnostics: Application application_1441066518301_0013 failed 2 times due to AM Container for appattempt_1441066518301_0013_000002 exited with exitCode: 1
For more detailed output, check application tracking page:http://yarnmaster-8245.lvs01.dev.ebayc3.com:8088/cluster/app/application_1441066518301_0013Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_e03_1441066518301_0013_02_000001
Exit code: 1
Exception message: /mnt/yarn/nm/local/usercache/stack/appcache/
application_1441066518301_0013/container_e03_1441066518301_0013_02_000001/
launch_container.sh: line 24: $PWD:$PWD/__hadoop_conf__:$PWD/__spark__.jar:$HADOOP_CONF_DIR:
/usr/hdp/current/hadoop-client/*::$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:
/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-.6.0.${hdp.version}.jar:
/etc/hadoop/conf/secure: bad substitution
Stack trace: ExitCodeException exitCode=1: /mnt/yarn/nm/local/usercache/stack/appcache/application_1441066518301_0013/container_e03_1441066518301_0013_02_000001/launch_container.sh: line 24: $PWD:$PWD/__hadoop_conf__:$PWD/__spark__.jar:$HADOOP_CONF_DIR:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure: bad substitution
at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
需要再spark3/conf目錄下創(chuàng)建一個(gè)文件痘括，叫java-opts內(nèi)容-Dhdp.version=xxx
版本可以通過hdp-select status hadoop-client 查看
關(guān)于報(bào)錯(cuò)的參考 https://try2explore.com/questions/11286719
