spark下載地址:?https://archive.apache.org/dist/spark/
一、配置Spark環(huán)境變量及將jars中 以下jar包拷貝至hive lib目錄中
spark-core_2.12-3.0.0.jar
spark-kvstore_2.12-3.0.0.jar?
spark-launcher_2.12-3.0.0.jar?
spark-network-common_2.12-3.0.0.jar
spark-network-shuffle_2.12-3.0.0.jar?
spark-tags_2.12-3.0.0.jar?
spark-unsafe_2.12-3.0.0.jar
二、在Hive中創(chuàng)建spark配置文件?
1、vim?/opt/module/hive/conf/spark-defaults.conf??
spark.master yarn
spark.eventLog.enabled? ? ? ? ? true
spark.eventLog.dir? ? ? ? ? ? ? hsfs://hadoop102:8020/spark-history spark.executor.memory? ? ? ? ? 1g
spark.driver.memory? ? ? ? ? ? 1g?
2蹬屹、hadoop fs -mkdir?/spark-history
三误趴、上傳純凈版spark jar包至hdfs?
1、hadoop fs -mkdir /spark-jars?
2点额、hadoop fs -put /opt/module/spark-3.0.0-bin-without-hadoop/jars/*? /spark-jars
四钾挟、修改Hive中hive-site.xml??
<!--spark依賴包(注:端口號必須和hadoop中namenode端口號一致) -->
<property>
<name>spark.yarn.jars</name>
<value>hdfs://hadoop102:8020/spark-jars/*</value>
</property>
<!--hive執(zhí)行引擎 -->
<property>
<name>hive.execution.engine</name>
<value>spark</value>
</property>
<!--hive和spark連接超時(shí)時(shí)間 -->
<property>
<name>hive.spark.client.connect.timeout</name>
<value>10000ms</value>
</property>
