spark 2.2.1 client
1. JAVA_HOME not set
vim spark-env.sh
export JAVA_HOME=/usr/local/jdk
export HADOOP_CONF_DIR=/etc/hadoop/conf
export HADOOP_HOME=/usr/hdp/current/hadoop-client
2. Exception in thread "main" java.lang.NoClassDefFoundError: com/sun/jersey/api/client/config/ClientConfig
cp /usr/hdp/current/hadoop-client/client/jersey-core-1.9.jar /usr/local/spark-2.2.1/jars/
cp /usr/hdp/current/hadoop-client/client/jersey-client-1.9.jar /usr/local/spark-2.2.1/jars/
rm -rf /usr/local/spark-2.2.1/jars/jersey-client-2.22.2.jar
3. java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider not found (Centos7出現(xiàn)的問(wèn)題)
從ambari上修改yarn-site配置
搜索關(guān)鍵字: yarn.client.failover-proxy-provider
修改為: org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider
4. bad substitution
/hadoop/yarn/local/usercache/wormhole/appcache/application_1527591700659_0001/container_e05_1527591700659_0001_01_000001/launch_container.sh: line 30: PWD:PWD/spark_conf:PWD/__spark_libs__/*:/usr/hdp/2.6.4.0-91/hadoop/conf:/usr/hdp/2.6.4.0-91/hadoop/*:/usr/hdp/2.6.4.0-91/hadoop/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:/usr/hdp/current/ext/hadoop/*:PWD/mr-framework/hadoop/share/hadoop/mapreduce/:PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:PWD/mr-framework/hadoop/share/hadoop/common/:PWD/mr-framework/hadoop/share/hadoop/common/lib/*:PWD/mr-framework/hadoop/share/hadoop/yarn/:PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:PWD/mr-framework/hadoop/share/hadoop/hdfs/:PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:PWD/mr-framework/hadoop/share/hadoop/tools/lib/:/usr/hdp/{hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.{hdp.version}.jar:/etc/hadoop/conf/secure:/usr/hdp/current/ext/hadoop/: bad substitution
step 1:
vim java-opts
-Dhdp.version=2.6.4.0-91
step 2:
1. Go to 'Ambari -> YARN -> configs' and go to 'Advanced' tab.
2. scroll down the page to till end, there will find an option to add custom property for yarn-site
3. click on 'add property' and enter 'hdp.version' and the version value.
hdp.version=2.6.4.0-91
4. save the changes and restart the required services. It will deploy the hdp.verion property in yarn-site.xml
注意：之前幫客戶解決問(wèn)題的時(shí)候懈玻，hdp.version=2.5.0.0-1245版本時(shí)按照上述方式設(shè)置不能解決問(wèn)題锥累，貌似HDP 2.6以下版本需要按照下面的方式解決栗恩。
step 1:
vim java-opts
-Dhdp.version=2.6.4.0-91
step 2:
vim spark-defaults.conf
spark.driver.extraJavaOptions -Dhdp.version=2.5.0.0-1245
spark.yarn.am.extraJavaOptions -Dhdp.version=2.5.0.0-1245
注意：用戶提交spark作業(yè)時(shí)，若重新設(shè)置這兩個(gè)參數(shù)的值，會(huì)覆蓋默認(rèn)配置诱告，所以用戶在配置這兩個(gè)變量時(shí)需要加上hdp.version的配置扮匠。
5. spark applicaiton executor ui顯示問(wèn)題
vi spark-default.conf
spark.hadoop.yarn.timeline-service.enabled false
