Hive2.3.4安裝[hadoop完全分布式中]

一动猬、安裝Hadoop

Hive運行在Hadoop環(huán)境之上，因此需要hadoop環(huán)境表箭，本次在安裝在hadoop完全分布式模式的namennode節(jié)點上

請參考：hadoop搭建[完全分布式]

二赁咙、安裝Hive

下載

[hadoop@s101 /home/hadoop]$cd /app/
[hadoop@s101 /app]$wget http://archive.apache.org/dist/hive/stable-2/apache-hive-2.3.4-bin.tar.gz

解壓安裝

[hadoop@s101 /app]$cd /app
[hadoop@s101 /app]$ln -s apache-hive-2.3.4-bin hive
[hadoop@s101 /app]$tar zxvf apache-hive-2.3.4-bin.tar.gz

配置hadoop用戶的hive環(huán)境變量

echo -e '##################HIVE環(huán)境變量配置#############\nexport HIVE_HOME=/app/hive\nexport PATH=$HIVE_HOME/bin:$PATH' >> ~/.bash_profile&& source ~/.bash_profile&&tail -3 ~/.bash_profile

三、配置hive

3.1 拷貝出hive-site.xml和hive-env.sh配置文件

[hadoop@s101 /app/hive/conf]$cd /app/hive/conf/
[hadoop@s101 /app/hive/conf]$cp hive-default.xml.template hive-site.xml
[hadoop@s101 /app]$cp hive-env.sh.template hive-env.sh

3.2 修改hive-site.xm中系統(tǒng)無法識別的變量`${system:java.io.tmpdir}`和`${system:user.name}`

測試

[hadoop@s101 /app/hive/conf]$sed -n 's#${system:java.io.tmpdir}#/app/hive.java.io.tmpdir#pg' hive-site.xml 
    <value>/app/hive.java.io.tmpdir/${system:user.name}</value>
    <value>/app/hive.java.io.tmpdir/${hive.session.id}_resources</value>
    <value>/app/hive.java.io.tmpdir/${system:user.name}</value>
    <value>/app/hive.java.io.tmpdir/${system:user.name}/operation_logs</value>
[hadoop@s101 /app/hive/conf]$sed -n 's#${system:user.name}#hadoop#pg' hive-site.xml 
    <value>/app/hive.java.io.tmpdir/hadoop</value>
    <value>/app/hive.java.io.tmpdir/hadoop</value>
    <value>/app/hive.java.io.tmpdir/hadoop/operation_logs</value>

替換

[hadoop@s101 /app/hive/conf]$sed -i 's#${system:java.io.tmpdir}#/app/hive.java.io.tmpdir#g' hive-site.xml 
[hadoop@s101 /app/hive/conf]$sed -i 's#${system:user.name}#hadoop#g' hive-site.xml

檢查

[hadoop@s101 /app/hive/conf]$grep 'hive.java.io.tmpdir' hive-site.xml 
    <value>/app/hive.java.io.tmpdir/${system:user.name}</value>
    <value>/app/hive.java.io.tmpdir/${hive.session.id}_resources</value>
    <value>/app/hive.java.io.tmpdir/${system:user.name}</value>
    <value>/app/hive.java.io.tmpdir/${system:user.name}/operation_logs</value>
[hadoop@s101 /app/hive/conf]$grep 'hive.java.io.tmpdir/hadoop' hive-site.xml 
    <value>/app/hive.java.io.tmpdir/hadoop</value>
    <value>/app/hive.java.io.tmpdir/hadoop</value>
    <value>/app/hive.java.io.tmpdir/hadoop/operation_logs</value>

創(chuàng)建出文件夾

[hadoop@s101 /app]$mkdir -p /app/hive.java.io.tmpdir/hadoop

3.3 初始化默認derby數(shù)據(jù)庫

[hadoop@s101 /home/hadoop]$schematool -initSchema -dbType derby

.......
Metastore connection URL: jdbc:derby:;databaseName=metastore_db;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 2.3.0
Initialization script hive-schema-2.3.0.derby.sql
Initialization script completed
schemaTool completed

初始化完成后，會在當前目錄產(chǎn)生一個metastore_db文件夾彼水，這就是數(shù)據(jù)庫的名稱崔拥。
HDFS文件系統(tǒng)中也會產(chǎn)生相應的目錄

[hadoop@s101 /home/hadoop]$ll /home/hadoop/metastore_db/
[hadoop@s101 /home/hadoop]$hdfs dfs -ls  /tmp
Found 2 items
drwx-wx-wx   - hadoop supergroup          0 2018-11-26 19:12 /tmp/hive

四、進入hive命令行

檢查hadoop集群是否啟動凤覆，如果沒有啟動需要啟動起來

[hadoop@s101 /home/hadoop]start-dfs.sh
[hadoop@s101 /home/hadoop]start-yarn.sh

進入hive命令行

[hadoop@s101 /home/hadoop]$hive
...
hive>

五链瓦、配置MySQL數(shù)據(jù)庫存儲元數(shù)據(jù)

5.1 修改URL、drivername盯桦、username慈俯、password等配置信息

[hadoop@s101 /home/hadoop]$vim /app/hive/conf/hive-site.xml

<property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://192.168.200.9:3306/hive</value>
</property>
<property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
</property>
<property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>hive</value>
</property>
<property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>123456</value>
</property>

請參考：mysql安裝

5.2 mysql的相關設置

查看mysql配置文件/etc/my.cnf，看屬性binlog_format是否設置為ROW

binlog_format=ROW

否則在后續(xù)初始化時會出現(xiàn)一下問題

圖片.png

登錄 mysql -uroot -p

create database hive;
GRANT ALL PRIVILEGES ON hive.* TO 'hive'@'192.168.%.%' IDENTIFIED BY '123456' WITH GRANT OPTION;
flush privileges;

上傳mysql驅(qū)動mysql-connector-java-5.1.38.jar到/app/hive/lib下拥峦。

[hadoop@s101 /home/hadoop]$ls /app/hive/lib/mysql-connector-java-5.1.38.jar 
/app/hive/lib/mysql-connector-java-5.1.38.jar

初始化數(shù)據(jù)庫

[hadoop@s101 /home/hadoop]$schematool -initSchema -dbType mysql
....
Metastore connection URL:    jdbc:mysql://192.168.200.9:3306/hive?createDatabaseIfNotExist=true
Metastore Connection Driver :    com.mysql.jdbc.Driver
Metastore connection User:   hive
Starting metastore schema initialization to 2.3.0
Initialization script hive-schema-2.3.0.mysql.sql
Initialization script completed
schemaTool completed

查看生成的表

mysql> use hive;
Database changed
mysql> show tables;