說明：相關(guān)文章內(nèi)容為本人學(xué)習(xí)記錄义黎，參考網(wǎng)絡(luò)分享诉瓦，如有侵權(quán)聯(lián)系刪除！僅供技術(shù)分享非商用漱挎！

Hadoop官方下載地址

Hive官方下載地址

概述

工具：VMware 14

目的：創(chuàng)建三個(gè)虛擬機(jī)系枪，網(wǎng)絡(luò)以橋接模式，三臺(tái)虛擬機(jī)在同一網(wǎng)段磕谅，保證三臺(tái)機(jī)器能夠相互ping通私爷。

流程步驟：
① 下載 CentOS7 ISO鏡像使用VM創(chuàng)建第一個(gè)虛擬機(jī)；
② 通過 VM 克隆創(chuàng)建剩下兩個(gè)虛擬機(jī)膊夹；
③ 設(shè)置三個(gè)系統(tǒng)的主機(jī)名以及網(wǎng)絡(luò)衬浑，并相互設(shè)置ssh免密登錄；
④ 安裝JDK放刨；
⑤ 安裝 Hadoop3.2 嚎卫；
⑥ 安裝 Hive3.1 ；

配置Linux主機(jī)信息

• 修改主機(jī)名稱(CentOS7)

 # 使用這個(gè)命令會(huì)立即生效且重啟也生效
[root@smallsuperman ~]# hostnamectl set-hostname outman00
[root@smallsuperman ~]# hostname
outman00

 # 編輯下hosts文件， 給127.0.0.1添加hostname
[root@smallsuperman ~]# vi /etc/hosts  
[root@smallsuperman ~]# cat /etc/hosts 
127.0.0.1   localhost smallsuperman.centos localhost4 localhost4.localdomain4 outman00
::1         localhost smallsuperman.centos localhost6 localhost6.localdomain6

• 內(nèi)網(wǎng)映射

# 主機(jī)名代替ip訪問
[root@smallsuperman ~]# sed -i '$a\192.168.233.132 outman00' /etc/hosts
[root@smallsuperman ~]# sed -i '$a\192.168.233.130 outman01' /etc/hosts
[root@smallsuperman ~]# sed -i '$a\192.168.233.131 outman02' /etc/hosts
[root@smallsuperman ~]# ping outman00 # 測(cè)試通否
PING localhost (127.0.0.1) 56(84) bytes of data

• 關(guān)閉防火墻

# 檢查防火墻狀態(tài)
# 看到綠色字樣標(biāo)注的“active（running）”拓诸，說明防火墻是開啟狀態(tài)
# disavtive（dead）的字樣，說明防火墻已經(jīng)關(guān)閉
[root@smallsuperman ~]# systemctl status firewalld.service

# 關(guān)閉運(yùn)行的防火墻
[root@smallsuperman ~]# systemctl stop firewalld.service

# 禁止防火墻服務(wù)器麻昼，系統(tǒng)重啟不會(huì)開啟防火墻
[root@smallsuperman ~]# systemctl disable firewalld.service

• 創(chuàng)建Hadoop用戶奠支，并賦權(quán)

# 增加一個(gè)用戶
[root@smallsuperman ~]# adduser hadoop

# 賦權(quán)
以root用戶身份為hadoop用戶賦權(quán)，在 root 賬號(hào)下抚芦，命令終端輸入：vi /etc/sudoers
找到
root ALL=(ALL) ALL
添加一行內(nèi)容
hadoop ALL=(ALL) ALL

• 設(shè)置SSH免密登錄

# 每個(gè)主機(jī)都生成密鑰（一直回車） 
[hadoop@outman02 ~]$ ssh-keygen

Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): 
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:rKWn0xjk+J3ZShFMO3pYx6sBxCpl1YkUojNsG4HQ1iE hadoop@outman02
The key's randomart image is:
+---[RSA 2048]----+
|ooE.o==+..       |
|..o++ooooo       |
| .Bo .. * o      |
| ..=. .* + .     |
|  .. +o S .      |
|    . o= +       |
|     .o=++       |
|      ++= .      |
|      ....       |
+----[SHA256]-----+

# 與其他主機(jī)建立免密連接倍谜，將自己的公鑰拷貝至其他主機(jī)的authorized_keys文件中。
[hadoop@outman02 ~]$ ssh-copy-id outman01
[hadoop@outman02 ~]$ ssh-copy-id outman00
[hadoop@outman01 ~]$ ssh-copy-id outman00
[hadoop@outman01 ~]$ ssh-copy-id outman02
[hadoop@outman00 ~]$ ssh-copy-id outman01
[hadoop@outman00 ~]$ ssh-copy-id outman02

# 測(cè)試一下免密登錄
[hadoop@outman00 ~]$ ssh hadoop@outman01
Last failed login: Mon Jun  3 02:16:04 CST 2019 from outman00 on ssh:notty
There were 3 failed login attempts since the last successful login.
Last login: Mon Jun  3 02:13:15 2019
[hadoop@outman01 ~]$ pwd
/home/hadoop

安裝JDK

1叉抡、去官網(wǎng)下載軟件的安裝壓縮包 尔崔；
2、上傳一份到主機(jī)臨時(shí)目錄下（我的是在/tmp/tar_gz）褥民，然后通過 scp 到另外兩個(gè)服務(wù)器相同位置季春；

• 解壓到指定目錄下

[root@outman02 tar_gz]# tar -zxvf jdk-8u211-linux-x64.tar.gz -C /usr/local/my_app

• 配置環(huán)境變量

[root@outman00 jdk1.8.0_211]# sed  -i '$a\\nexport JAVA_HOME=/usr/local/my_app/jdk1.8.0_211\nexport PATH=$PATH:$JAVA_HOME/bin ' /etc/profile
# 更新加載環(huán)境變量 
[root@outman00 jdk1.8.0_211]# source /etc/profile
# 檢查是否安裝成功
[root@outman00 jdk1.8.0_211]# java -version
java version "1.8.0_211"
Java(TM) SE Runtime Environment (build 1.8.0_211-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode)

安裝Hadoop

主要目錄
（1）bin目錄：存放對(duì)Hadoop相關(guān)服務(wù)（HDFS,YARN）進(jìn)行操作的腳本
（2）etc目錄：Hadoop的配置文件目錄，存放Hadoop的配置文件
（3）lib目錄：存放Hadoop的本地庫（對(duì)數(shù)據(jù)進(jìn)行壓縮解壓縮功能）
（4）sbin目錄：存放啟動(dòng)或停止Hadoop相關(guān)服務(wù)的腳本
（5）share目錄：存放Hadoop的依賴jar包消返、文檔载弄、和官方案例

• 解壓到指定目錄下

[root@outman00 tar_gz]# tar -zxvf hadoop-3.2.0.tar.gz  -C /usr/local/my_app/hadoop

修改配置文件

• 進(jìn)入解壓路徑

[root@outman00 hadoop-3.2.0]# cd /usr/local/my_app/hadoop/hadoop-3.2.0/etc/hadoop/

• 修改配置文件中JDK路徑

[root@outman00 hadoop]# vi /usr/local/my_app/hadoop/hadoop-3.2.0/etc/hadoop/hadoop-env.sh

# 修改內(nèi)容，添加JDK的路徑信息
    52 # The java implementation to use. By default, this environment
    53 # variable is REQUIRED on ALL platforms except OS X!
    54 export JAVA_HOME=/usr/local/my_app/jdk1.8.0_211

• 修改 core-site.xml 文件

fs.defaultFS HDFS中NameNode的地址 端口撵颊。
hadoop.tmp.dir Hadoop運(yùn)行時(shí)產(chǎn)生文件的存儲(chǔ)目錄宇攻。

<configuration>
<property>
 <name>fs.defaultFS</name>
 <value>hdfs://outman00:9000</value>
</property>
<property>
 <name>hadoop.tmp.dir</name>
 <value>/usr/local/my_app/hadoop/hadoop_data</value>
</property>
</configuration>

• 修改 hdfs-site.xml

<configuration>
<property>
  <name>dfs.namenode.name.dir</name>
  <value>/usr/local/my_app/hadoop/hadoop_data/namenode_data</value>
  <description>元數(shù)據(jù)存儲(chǔ)目錄，安全起見可配置到其他目錄</description>
</property>

<property>
  <name>dfs.datanode.data.dir</name>
  <value>/usr/local/my_app/hadoop/hadoop_data/datanode_data</value>
  <description>datanode 的數(shù)據(jù)存儲(chǔ)目錄</description>
</property>

<property>
   <name>dfs.replication</name>
   <value>2</value>
   <description>HDFS 的數(shù)據(jù)塊的副本個(gè)數(shù)</description>
</property>

<property>
   <name>dfs.secondary.http.address</name>
   <value>outman01:50090</value>
   <description>secondarynamenode 節(jié)點(diǎn)信息倡勇，最好是和namenode 設(shè)置為不同節(jié)點(diǎn)</description>
   </property>
</configuration>

• 修改 yarn-site.xml

yarn.nodemanager.aux-services YARN 集群為 MapReduce 程序提供的 shuffle 服務(wù)
yarn.resourcemanager.hostname ResourceManager的信息

<configuration>

<!-- Reducer獲取數(shù)據(jù)的方式 -->
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

<!-- 指定YARN的ResourceManager的地址 -->
<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>outman00</value>
</property>

</configuration>

• 修改 mapred-site.xml

配置采用yarn作為資源調(diào)度框架（指定MR運(yùn)行在YARN上）

<configuration>
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>
</configuration>

• 修改workers （3.0之前是slaves文件 ）

outman00
outman01
outman02

• 把安裝的Hadoop目錄文件分發(fā)給另外兩個(gè)節(jié)點(diǎn)（scp過去）

[hadoop@outman00 hadoop]$ scp -r hadoop-3.2.0/ hadoop@outman02:/usr/local/my_app/hadoop
[hadoop@outman00 hadoop]$ scp -r hadoop-3.2.0/ hadoop@outman01:/usr/local/my_app/hadoop

• 配置環(huán)境變量（三個(gè)節(jié)點(diǎn) /etc/profile）

[root@outman00 hadoop]# sed -i '$a\export HADOOP_HOME=/usr/local/my_app/hadoop/hadoop-3.2.0\nexport PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin' /etc/profile

# 更新環(huán)境變量配置
[root@outman00 hadoop]# source /etc/profile

# 驗(yàn)證
[root@outman02 hadoop]# hadoop --help

• 初始化namenode

在HDFS 主節(jié)點(diǎn)(core-site.xml中配置的fs.defaultFS),執(zhí)行初始化命令逞刷，成功后會(huì)根據(jù)配置的信息創(chuàng)建對(duì)應(yīng)的data目錄如果需要重新初始化，刪除后重新執(zhí)行即可妻熊！

[root@outman00 hadoop]# hadoop namenode -format

# 判斷成功關(guān)鍵信息
2019-06-05 01:58:12,198 INFO common.Storage: Storage directory /usr/local/my_app/hadoop/hadoop_data/namenode_data has been successfully formatted.

• 啟動(dòng)hdfs（執(zhí)行啟動(dòng)腳本任意節(jié)點(diǎn)）

[hadoop@outman00 ~]$ start-dfs.sh

[hadoop@outman00 ~]$ jps
8949 DataNode
8840 NameNode
9229 Jps
[hadoop@outman01 hadoop_data]$ jps
8071 SecondaryNameNode
8137 Jps
7997 DataNode
[hadoop@outman02 hadoop_data]$ jps
7973 Jps
7817 DataNode

• 出現(xiàn)問題

  • NameDode 主節(jié)點(diǎn)可以訪問hdfs夸浅，但是另外兩個(gè)節(jié)點(diǎn)無法訪問
• 報(bào)錯(cuò)信息

# 主節(jié)點(diǎn)正常訪問
[hadoop@outman00 ~]$ hadoop fs -ls /
Found 1 items
drwxr-xr-x   - hadoop supergroup          0 2019-06-06 00:34 /dyp

# 次節(jié)點(diǎn)無法訪問
[hadoop@outman01 ~]$ hadoop fs -ls /
ls: Call From outman01/192.168.233.130 to outman00:9000 failed on connection exception: java.net.ConnectException: 拒絕連接; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused

# 次節(jié)點(diǎn)無法訪問
[hadoop@outman02 ~]$ hadoop fs -ls /
ls: Call From localhost/127.0.0.1 to outman00:9000 failed on connection exception: java.net.ConnectException: 拒絕連接; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused

• 根據(jù)配置信息 core-site.xml 檢查是否可以訪問主節(jié)點(diǎn)的9000端口

# 主節(jié)點(diǎn)可以
[root@outman00 datanode_data]# telnet  outman00 9000
Trying 127.0.0.1...
Connected to outman00.
Escape character is '^]'.

# 次節(jié)點(diǎn)不可以
[root@outman01 ~]# telnet  outman00 9000
Trying 192.168.233.132...
telnet: connect to address 192.168.233.132: Connection refused
[root@outman02 xinetd.d]# telnet  outman00 9000
Trying 192.168.233.132...
telnet: connect to address 192.168.233.132: Connection refused

• 檢查主節(jié)點(diǎn)9000端口占用情況

發(fā)現(xiàn)9000端口被 127.0.0.1:本地占用，也就是只有本地才能訪問 （HDFS監(jiān)聽的9000端口默認(rèn)綁定127.0.0.1地址）

[root@outman00 datanode_data]# lsof -i:9000
COMMAND  PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
java    7339 hadoop  269u  IPv4  42236      0t0  TCP localhost:cslistener (LISTEN)
java    7339 hadoop  279u  IPv4  44037      0t0  TCP localhost:cslistener->localhost:51560 (ESTABLISHED)
java    7413 hadoop  328u  IPv4  44036      0t0  TCP localhost:51560->localhost:cslistener (ESTABLISHED)
[root@outman00 datanode_data]# netstat -tunlp |grep 9000
tcp        0      0 127.0.0.1:9000          0.0.0.0:*               LISTEN      7339/java   

• 嘗試修改host配置

#127.0.0.1   localhost smallsuperman.centos localhost4 localhost4.localdomain4 outman00
#::1         localhost smallsuperman.centos localhost6 localhost6.localdomain6
192.168.233.132 outman00
192.168.233.130 outman01
192.168.233.131 outman02

• 重啟hadoop

[hadoop@outman00 ~]$ stop-all.sh
[hadoop@outman00 ~]$ start-all.sh

• 檢查主節(jié)點(diǎn)NameNode上9000占用（占用的ip變成了主節(jié)點(diǎn)其他節(jié)點(diǎn)可以訪問了）

[root@outman00 datanode_data]# netstat -tunlp | grep 9000
tcp        0      0 192.168.233.132:9000    0.0.0.0:*               LISTEN      10843/java 

• 其他節(jié)點(diǎn)訪問hdfs

[hadoop@outman01 ~]$ hadoop fs -ls /
Found 1 items
drwxr-xr-x   - hadoop supergroup          0 2019-06-06 00:34 /dyp
[hadoop@outman02 ~]$ hadoop fs -ls /
Found 1 items
drwxr-xr-x   - hadoop supergroup          0 2019-06-06 00:34 /dyp

• 訪問 HDFS 前臺(tái)頁面

http://192.168.233.132:9870

特別注意

hadoop3.0之前web訪問端口是50070
hadoop3.0之后web訪問端口為9870

• 啟動(dòng)yarn（要在yarn的主節(jié)點(diǎn)啟動(dòng)）

[hadoop@outman00 hadoop_data]$ start-yarn.sh
Starting resourcemanager
Starting nodemanagers

# 查看進(jìn)程（主節(jié)點(diǎn)增加 ResourceManager 固耘、NodeManager 其他節(jié)點(diǎn)增加 NodeManager）
[hadoop@outman00 hadoop_data]$ jps
9811 Jps
8949 DataNode
9463 NodeManager
8840 NameNode
9353 ResourceManager
[hadoop@outman01 hadoop_data]$ jps
8227 NodeManager
8071 SecondaryNameNode
8327 Jps
7997 DataNode

• 測(cè)試yarn

[hadoop@outman00 mapreduce]$ hadoop jar hadoop-mapreduce-examples-3.2.0.jar wordcount  /dyp/test/test  /dyp/test/test_out

# 報(bào)錯(cuò)
[2019-06-06 02:01:09.415]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
錯(cuò)誤: 找不到或無法加載主類 org.apache.hadoop.mapreduce.v2.app.MRAppMaster
# 解決方法  ：  在配置文件 mapred-site.xml 文件中添加 mapreduce 程序所用到的 classpath 如下
# /usr/local/my_app/hadoop/hadoop-3.2.0/ 就是hadoop安裝路徑
<configuration>
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>

<property>
    <name>mapreduce.application.classpath</name>
    <value>/usr/local/my_app/hadoop/hadoop-3.2.0/share/hadoop/mapreduce/*, /usr/local/my_app/hadoop/hadoop-3.2.0/share/hadoop/mapreduce/lib/*</value>
</property>
</configuration>

• 再次測(cè)試

[hadoop@outman00 mapreduce]$ hadoop jar hadoop-mapreduce-examples-3.2.0.jar wordcount  /dyp/test/test  /dyp/test/test_out

[hadoop@outman00 mapreduce]$ hadoop fs -ls  /dyp/test/test_out
Found 2 items
-rw-r--r--   2 hadoop supergroup          0 2019-06-06 02:16 /dyp/test/test_out/_SUCCESS
-rw-r--r--   2 hadoop supergroup         29 2019-06-06 02:16 /dyp/test/test_out/part-r-00000
[hadoop@outman00 mapreduce]$ hadoop fs -cat  /dyp/test/test_out/part-r-00000
1|2|3   1
A|B|C   1
A|B|C1|2|3  1

安裝 MySQL

我這里使用騰訊云服務(wù)器上安裝在 Docker 中的 MySQL题篷，所以在虛擬上只需要安裝MySQL的客戶端就可以了，只用于訪問

• 安裝mysql-client

[root@outman00 ~]# yum  install mysql

# 連接騰訊云MySQL
[root@outman00 ~]# mysql -h 騰訊云MySQL的IP -u root -p

安裝Hive3

• 解壓到指定目錄下

[root@outman00 tar_gz]# tar -zxvf apache-hive-3.1.1-bin.tar.gz -C /usr/local/my_app/hive

• 下載 MySQL 的 Java 驅(qū)動(dòng)并上傳到 Hive 的lib路徑下
  官網(wǎng)下載地址

-rw-r--r--. 1 root root  2293144 6月   7 02:07 mysql-connector-java-8.0.16.jar
[root@outman00 lib]# pwd
/usr/local/my_app/hive/hive-3.1.1/lib

• 配置Hive環(huán)境變量

[root@outman00 lib]# sed -i '$a\export HIVE_HOME=/usr/local/my_app/hive/hive-3.1.1\nexport PATH=$PATH:$HIVE_HOME/bin' /etc/profile

# 更新生效
[root@outman00 lib]# source /etc/profile

• 修改配置文件

[root@outman00 conf]# cd /usr/local/my_app/hive/hive-3.1.1/conf
[root@outman00 conf]# cp hive-env.sh.template hive-env.sh
[root@outman00 conf]# cp hive-default.xml.template hive-site.xml

• hive-env.sh 添加以下內(nèi)容

export JAVA_HOME=/usr/local/my_app/jdk1.8.0_211

export HADOOP_HOME=/usr/local/my_app/hadoop/hadoop-3.2.0

export HIVE_HOME=/usr/local/my_app/hive/hive-3.1.1

• 更新生效

[root@outman00 conf]# source hive-env.sh

• 修改 hive-site.xml

• 先創(chuàng)建目錄

[root@outman00 conf]# mkdir -p /usr/local/my_app/hive/hive_data/warehouse
[root@outman00 conf]# mkdir -p /usr/local/my_app/hive/hive_data/tmp
[root@outman00 conf]# mkdir -p /usr/local/my_app/hive/hive_data/log

• 修改 hive-site.xml

<property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <description>Driver class name for a JDBC metastore</description>
</property>
<property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>用戶名</value>
    <description>username to use against metastore database</description>
 </property>
<property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>密碼</value>
    <description>password to use against metastore database</description>
</property>
<property>
    <name>hive.metastore.warehouse.dir</name>
    <value>/usr/local/my_app/hive/hive_datawarehouse</value>
    <description>location of default database for the warehouse</description>
</property>
<property>
    <name>hive.exec.scratchdir</name>
    <value>/usr/local/my_app/hive/hive_data/tmp</value>
    <description>HDFS root scratch dir for Hive jobs which gets created with write all (733) permission. For each connecting user, an HDFS scratch dir: ${hive.exec.scratchdir}/&lt;username&gt; is created, with ${hive.scratch.dir.permission}.</description>
</property>
<property>
    <name>hive.querylog.location</name>
    <value>/usr/local/my_app/hive/hive_data/log</value>
    <description>Location of Hive run time structured log file</description>
</property>

# 我們把變量  $system:java.io.tmpdir 替換成我們的臨時(shí)數(shù)據(jù)存放目錄 /usr/local/my_app/hive/hive_data/tmp

• 修改 hive-log4j.proprties 文件

• hive啟動(dòng)報(bào)錯(cuò)如下

Caused by: com.ctc.wstx.exc.WstxParsingException: Illegal character entity: expansion character (code 0x8
at [row,col,system-id]: [3186,96,"file:/usr/local/my_app/hive/hive-3.1.1/conf/hive-site.xml"

配置文件 hive-site.xml 3186行96個(gè)字符不合法

# 詳細(xì)報(bào)錯(cuò)
Exception in thread "main" java.lang.RuntimeException: com.ctc.wstx.exc.WstxParsingException: Illegal character entity: expansion character (code 0x8
 at [row,col,system-id]: [3186,96,"file:/usr/local/my_app/hive/hive-3.1.1/conf/hive-site.xml"]
    at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2981)
    at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2930)
    at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2805)
    at org.apache.hadoop.conf.Configuration.get(Configuration.java:1459)
    at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:4990)
    at org.apache.hadoop.hive.conf.HiveConf.getVar(HiveConf.java:5063)
    at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5150)
    at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5093)
    at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:97)
    at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:81)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:699)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
Caused by: com.ctc.wstx.exc.WstxParsingException: Illegal character entity: expansion character (code 0x8
 at [row,col,system-id]: [3186,96,"file:/usr/local/my_app/hive/hive-3.1.1/conf/hive-site.xml"]
    at com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:621)
    at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:491)
    at com.ctc.wstx.sr.StreamScanner.reportIllegalChar(StreamScanner.java:2456)
    at com.ctc.wstx.sr.StreamScanner.validateChar(StreamScanner.java:2403)
    at com.ctc.wstx.sr.StreamScanner.resolveCharEnt(StreamScanner.java:2369)
    at com.ctc.wstx.sr.StreamScanner.fullyResolveEntity(StreamScanner.java:1515)
    at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2828)
    at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1123)
    at org.apache.hadoop.conf.Configuration$Parser.parseNext(Configuration.java:3277)
    at org.apache.hadoop.conf.Configuration$Parser.parse(Configuration.java:3071)
    at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2964)
    ... 17 more

• 啟動(dòng) metastore 報(bào)錯(cuò)

報(bào)錯(cuò)信息

hadoop@outman00 hive-3.1.1]$  hive --service metastore 

2019-06-07 23:25:27: Starting Hive Metastore Server
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/my_app/hive/hive-3.1.1/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/my_app/hadoop/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
MetaException(message:Version information not found in metastore.)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:84)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:93)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8661)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8656)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:8926)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:8843)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:236)

分析報(bào)錯(cuò)

• jar包沖突厅目，關(guān)鍵信息

[jar:file:/usr/local/my_app/hive/hive-3.1.1/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/my_app/hadoop/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class

• 刪除 HVIE_HOME/bin 下的jar
• 不要?jiǎng)h除HADOOP_HOME/bin下的jar包否則start-all.sh遠(yuǎn)程啟動(dòng)hadoop時(shí)會(huì)報(bào)找不到log4j包的錯(cuò)誤番枚。

rm -rf /usr/local/my_app/hive/hive-3.1.1/lib/log4j-slf4j-impl-2.10.0.jar

• 啟動(dòng)metastore仍然報(bào)錯(cuò)內(nèi)容如下

2019-06-07 23:41:43: Starting Hive Metastore Server
MetaException(message:Version information not found in metastore.)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:84)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:93)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8661)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8656)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:8926)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:8843)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:236)

• 嘗試解決 在hvie-site.xml（繼續(xù)報(bào)錯(cuò)）

# 關(guān)閉元數(shù)據(jù)驗(yàn)證
<property>     
    <name>datanucleus.metadata.validate</name>    
    <value>false</value>    
</property> 
# 關(guān)閉元數(shù)據(jù)存儲(chǔ)模式驗(yàn)證
<property>  
    <name>hive.metastore.schema.verification</name> 
    <value>false</value> 
</property>
<property>
    <name>datanucleus.schema.autoCreateAll</name>
    <value>ture</value>
</property>
# 其中hive.metastore.schema.verification防止架構(gòu)版本不兼容時(shí)的 Metastore 操作∷鸱螅考慮將此設(shè)置為“True”葫笼，以減少 Metastore 操作期間發(fā)生架構(gòu)損壞的可能性

• 如果是第一次需要執(zhí)行初始化命令：schematool -dbType mysql -initSchema

[hadoop@outman00 hive-3.1.1]$ schematool -dbType mysql -initSchema

# 發(fā)現(xiàn)MySQL中創(chuàng)建了hive庫
MySQL [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| dyp                |
| hive               |
| information_schema |
| mysql              |
| performance_schema |
| sys                |
+--------------------+

正常啟動(dòng) metastore 后 進(jìn)入 hive 交互，報(bào)錯(cuò)如下

hive> show databases;
OK
Failed with exception java.io.IOException:java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:user.name%7D

• 修改配置

  <property>
    <name>hive.exec.local.scratchdir</name>
    <value>/usr/local/my_app/hive/hive_data/tmp/${user.name}</value>
    <description>Local scratch space for Hive jobs</description>
  </property>

啟動(dòng)后節(jié)點(diǎn)信息

啟動(dòng)后節(jié)點(diǎn)信息

概念說明

• NameNode

負(fù)責(zé)管理整個(gè) HDFS 文件系統(tǒng)的元數(shù)據(jù):配置副本策略拗馒、管理我們存儲(chǔ)數(shù)據(jù)塊(Block)映射信息路星、管理HDFS名稱空間、處理客戶端讀寫請(qǐng)求

• SecondaryNameNode

是 NameNode 的輔助分擔(dān)任務(wù)，定期合并fsimage和edits文件洋丐，可以輔助恢復(fù)NameNode呈昔；

• DataNode

負(fù)責(zé)管理用戶的文件數(shù)據(jù)塊:根據(jù)NameNode下發(fā)的任務(wù)命令，DataNode去執(zhí)行對(duì)應(yīng)的操作友绝。（存儲(chǔ)實(shí)際數(shù)據(jù)塊堤尾、執(zhí)行數(shù)據(jù)塊的讀寫操作）
文件會(huì)按照固定大小（blocksize）來切分成塊后分布式存儲(chǔ)在若干臺(tái)DataNode上
每一個(gè)文件快可以有多個(gè)副本迁客，并存放在不同的 DataNode 上 DataNode 會(huì)定期向 NameNode 匯報(bào)自身所保存的文件block信息郭宝，而 NameNode 則會(huì)負(fù)責(zé)保持文件的副本數(shù)量（當(dāng)減少DataNode的時(shí)候，NameNode才知道當(dāng)前副本狀態(tài)掷漱，從而進(jìn)行副本維持）

• YARN 負(fù)責(zé)將系統(tǒng)資源分配給在 Hadoop 集群中運(yùn)行的各種應(yīng)用程序粘室，并調(diào)度要在不同集群節(jié)點(diǎn)上執(zhí)行的任務(wù)
• YARN 的組件 ResourceManager

ResourceManager 是在系統(tǒng)中的所有應(yīng)用程序之間仲裁資源的最終權(quán)限。

• YARN 的組件 ResourceManager

NodeManager 是每臺(tái)機(jī)器框架代理卜范，負(fù)責(zé) Containers衔统，監(jiān)視其資源使用情況（CPU，內(nèi)存先朦，磁盤缰冤，網(wǎng)絡(luò)）并將其報(bào)告給 ResourceManager。