Ambari2.6.2 和HDP2.6.5搭建Hadoop集群-04_Ambari2.6.2 安裝HDP2.6.5

本節(jié)我們開始正式構建HDP的軟件環(huán)境

一浩村、服務器環(huán)境配置

1.服務器列表
2.配置SSH免密登陸
3.更改hosts
4.配置防火墻
5.配置阿里yum源(可選)
6.NTP時鐘同步
7.關閉SELinux牲览、配置umask 值

通過前面的三章以上內容均已完成（即基本的軟件安裝環(huán)境已經(jīng)具備）

二屿聋、軟件環(huán)境

1.安裝mysql數(shù)據(jù)庫

在安裝Ambari時谈跛，Ambari默認的數(shù)據(jù)庫是ProstgreSQL，這里選擇使用MySql乘盖。但Centos 7 默認支持的是MariaDB數(shù)據(jù)庫. MariaDB是MySQL的一個分支呐赡，主要由開源社區(qū)在維護。安裝過程中先將Centos7默認安裝的MariaDB數(shù)據(jù)庫刪除焚虱，再重新安裝MySql

安裝步驟：

軟件下載（mysql的rpm包有很多文件购裙，只需要安裝Server 和 Client）

由于操作系統(tǒng)是用CentOS7.5 根據(jù)官網(wǎng)查詢匹配的Mysql 5.7版本為 5.7.26

https://dev.mysql.com/downloads/file/?id=485654

image.png

數(shù)據(jù)庫安裝
刪除Centos默認安裝的MariaDB

[root@node1 ambari]# rpm -qa|grep mariadb
mariadb-libs-5.5.56-2.el7.x86_64
[root@node1 ambari]# rpm -e --nodeps mariadb-libs-5.5.56-2.el7.x86_64

解壓mysql-5.7.26-1.el7.x86_64.rpm-bundle.tar

[root@node1 mysql]# tar -xvf mysql-5.7.26-1.el7.x86_64.rpm-bundle.tar
mysql-community-embedded-devel-5.7.26-1.el7.x86_64.rpm
mysql-community-libs-5.7.26-1.el7.x86_64.rpm
mysql-community-embedded-5.7.26-1.el7.x86_64.rpm
mysql-community-test-5.7.26-1.el7.x86_64.rpm
mysql-community-embedded-compat-5.7.26-1.el7.x86_64.rpm
mysql-community-common-5.7.26-1.el7.x86_64.rpm
mysql-community-devel-5.7.26-1.el7.x86_64.rpm
mysql-community-client-5.7.26-1.el7.x86_64.rpm
mysql-community-server-5.7.26-1.el7.x86_64.rpm
mysql-community-libs-compat-5.7.26-1.el7.x86_64.rpm

安裝Mysql【注意：執(zhí)行有依賴順序】

[root@node1 mysql]# rpm -ivh mysql-community-common-5.7.26-1.el7.x86_64.rpm
warning: mysql-community-common-5.7.26-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:mysql-community-common-5.7.26-1.e################################# [100%]
[root@node1 mysql]# rpm -ivh mysql-community-libs-5.7.26-1.el7.x86_64.rpm
warning: mysql-community-libs-5.7.26-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:mysql-community-libs-5.7.26-1.el7################################# [100%]
[root@node1 mysql]# rpm -ivh mysql-community-client-5.7.26-1.el7.x86_64.rpm
warning: mysql-community-client-5.7.26-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:mysql-community-client-5.7.26-1.e################################# [100%]
[root@node1 mysql]# rpm -ivh mysql-community-server-5.7.26-1.el7.x86_64.rpm
warning: mysql-community-server-5.7.26-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
error: Failed dependencies:
    libaio.so.1()(64bit) is needed by mysql-community-server-5.7.26-1.el7.x86_64
    libaio.so.1(LIBAIO_0.1)(64bit) is needed by mysql-community-server-5.7.26-1.el7.x86_64
    libaio.so.1(LIBAIO_0.4)(64bit) is needed by mysql-community-server-5.7.26-1.el7.x86_64

【此處報錯：】這個報錯的意思是需要安裝libaio包 libaio-0.3.109-13.el7.x86_64.rpm

[root@node1 tmp]# yum -y install wget
[root@node1 tmp]# wget http://mirror.centos.org/centos/7/os/x86_64/Packages/libaio-0.3.109-13.el7.x86_64.rpm
[root@node1 tmp]# rpm -ivh libaio-0.3.109-13.el7.x86_64.rpm

再次執(zhí)行安裝server(執(zhí)行成功)

[root@node1 mysql]# rpm -ivh mysql-community-server-5.7.26-1.el7.x86_64.rpm
warning: mysql-community-server-5.7.26-1.el7.x86_64.rpm: Header V3 DSA/SHA1 Signature, key ID 5072e1f5: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:mysql-community-server-5.7.26-1.e################################# [100%]

數(shù)據(jù)庫初始化
為了保證數(shù)據(jù)庫目錄為與文件的所有者為 mysql 登陸用戶，如果你是以 root 身份運行 mysql 服務鹃栽，需要執(zhí)行下面的命令初始化

mysqld --initialize --user=mysql

如果是以 mysql 身份運行躏率，則可以去掉 --user 選項。
另外 --initialize 選項默認以“安全”模式來初始化民鼓，則會為 root 用戶生成一個密碼并將該密碼標記為過期薇芝，登陸后你需要設置一個新的密碼，而使用 --initialize-insecure 命令則不使用安全模式摹察，則不會為 root 用戶生成一個密碼恩掷。
這里演示使用的 --initialize 初始化的，會生成一個 root 賬戶密碼供嚎，密碼在log文件里黄娘，紅色區(qū)域的就是自動生成的密碼

[root@node1 mysql]# cat /var/log/mysqld.log
2019-06-12T07:58:54.568405Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2019-06-12T07:58:55.838148Z 0 [Warning] InnoDB: New log files created, LSN=45790
2019-06-12T07:58:56.155262Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
2019-06-12T07:58:56.317281Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: ed9a8884-8ce7-11e9-aafe-1c1b0df90270.
2019-06-12T07:58:56.356702Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2019-06-12T07:58:56.358045Z 1 [Note] A temporary password is generated for root@localhost: n?kifcur4sUe

啟動mysql

[root@node1 mysql]# systemctl start mysqld.service

使用臨時密碼登陸系統(tǒng)【A temporary password is generated for root@localhost: n?kifcur4sUe】

[root@node1 mysql]# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.7.26

Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

修改root用戶密碼

mysql> show databases;
ERROR 1820 (HY000): You must reset your password using ALTER USER statement before executing this statement.

以前的 password()函數(shù)將會被拋棄逼争，官方建議使用下面的命令來修改密碼（避免使用簡單密碼否則也會報錯）

ALTER USER 'root'@'localhost' IDENTIFIED BY 'nihao2019@mysql'; 此處自行設置

安裝目錄及功能
mysql安裝完成后劝赔，文件分布在以下幾個目錄:
/var/lib/mysql 數(shù)據(jù)庫文件,目錄下的 ibdata1 用于存放數(shù)據(jù)，索引等杂伟，是MYSQL的最主要的數(shù)據(jù)仍翰。ib_filelog是數(shù)據(jù)庫日志文件予借，采用二進制存儲频蛔。
/usr/share/mysql 命令及配置文件目錄
/usr/bin mysqladmin秦叛、mysqldump 等命令
在linux命令行下挣跋，可使用 whereis mysql 命令查看其安裝目錄

[root@node1 mysql]# whereis mysql
mysql: /usr/bin/mysql /usr/lib64/mysql /usr/share/mysql /usr/share/man/man1/mysql.1.gz

開機自啟動

[root@node1 mysql]# vi /etc/rc.local
#添加service mysqld start

2.安裝mysql-java驅動

命令：yum install -y mysql-connector-java （安裝mysql驅動)
檢查：cd /usr/share/java (安裝完成后，切換至java目錄嫌术，如下圖牌借，則ok)

image.png

cp /usr/share/java/mysql-connector-java.jar /var/lib/ambari-server/resources/mysql-jdbc-driver.jar
vi /etc/ambari-server/conf/ambari.properties
添加server.jdbc.driver.path=/usr/share/java/mysql-connector-java.jar
[非常重要]

三、修改yum源磷籍，實現(xiàn)離線安裝

1现柠、安裝httpd服務（主服務器）

[root@node1 ~]# yum install -y httpd

[root@node1 ~]# systemctl start httpd

[root@node1 ~]# chkconfig httpd on
Note: Forwarding request to 'systemctl enable httpd.service'.
Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service.

2、將上面下載的三個包放到/var/www/html目錄下（主服務器）

[root@node1 tmp]# cd /var/www/html/
[root@node1 html]# mkdir ambari
拷貝文件到ambari下面
[root@node1 html]# cd ambari/
[root@node1 ambari]# ls
ambari-2.6.2.2-centos7.tar.gz  HDP-2.6.5.0-centos7-rpm.tar.gz  HDP-UTILS-1.1.0.22-centos7.tar.gz
[root@node1 ambari]# tar -zxvf ambari-2.6.2.2-centos7.tar.gz
[root@node1 ambari]# tar -zxvf HDP-2.6.5.0-centos7-rpm.tar.gz 
[root@node1 ambari]# mkdir HDP-UTILS
[root@node1 ambari]# tar -zxvf HDP-UTILS-1.1.0.22-centos7.tar.gz
[root@node1 ambari]# rm -rf ambari-2.6.2.2-centos7.tar.gz  HDP-2.6.5.0-centos7-rpm.tar.gz  HDP-UTILS-1.1.0.22-centos7.tar.gz
[root@node1 ambari]# ls
ambari  HDP  HDP-UTILS

現(xiàn)在可以通過訪問http://192.168.12.191/ambari/查看是否能成功訪問

image.png

3比然、制作本地源

（1）安裝本地源制作相關工具（主服務器）

[root@node1 ambari]# yum install yum-utils createrepo yum-plugin-priorities -y
[root@node1 ambari]# createrepo  ./
Spawning worker 0 with 67 pkgs
Spawning worker 1 with 67 pkgs
Spawning worker 2 with 66 pkgs
Spawning worker 3 with 66 pkgs
Workers Finished
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete

（2）修改文件里面的源地址（主服務器）

[root@node1 ambari]# cp ambari/centos7/2.6.2.2-1/ambari.repo ambari/centos7/2.6.2.2-1/ambari.repo.bak
[root@node1 ambari]# vi ambari/centos7/2.6.2.2-1/ambari.repo
#VERSION_NUMBER=2.6.2.2-1
[ambari-2.6.2.2]
name=ambari Version - ambari-2.6.2.2
baseurl=http://192.168.12.191/ambari/ambari/centos7/2.6.2.2-1
gpgcheck=1
gpgkey=http://192.168.12.191/ambari/ambari/centos7/2.6.2.2-1/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
[root@node1 ambari]# cp ambari/centos7/2.6.2.2-1/ambari.repo /etc/yum.repos.d/


[root@node1 2.6.5.0-292]# cp hdp.repo hdp.repo.bak
[root@node1 2.6.5.0-292]# vi hdp.repo
#VERSION_NUMBER=2.6.5.0-292
[HDP-2.6.5.0]
name=HDP Version - HDP-2.6.5.0
baseurl=http://192.168.12.191/ambari/HDP/centos7/2.6.5.0-292
gpgcheck=1
gpgkey=http://192.168.12.191/ambari/HDP/centos7/2.6.5.0-292/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://192.168.12.191/ambari/HDP-UTILS/centos7/1.1.0.22
gpgcheck=1
gpgkey=http://192.168.12.191/ambari/HDP-UTILS/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[root@node1 2.6.5.0-292]# cp hdp.repo /etc/yum.repos.d/

上面就創(chuàng)建好了主機上的文件周循，然后可以通過下面命令清楚一下yum的緩存就可以了

[root@node1 ambari]# yum clean all
[root@node1 ambari]# yum makecache
[root@node1 ambari]# yum repolist

(3) 將文件分發(fā)到其他機器

[root@node1 yum.repos.d]# cd /etc/yum.repos.d
[root@node1 yum.repos.d]# scp ambari.repo root@node2:/etc/yum.repos.d/
ambari.repo                                                                                 100%  268   285.9KB/s   00:00
[root@node1 yum.repos.d]# scp ambari.repo root@node3:/etc/yum.repos.d/
ambari.repo                                                                                 100%  268   308.5KB/s   00:00
[root@node1 yum.repos.d]# scp ambari.repo root@node4:/etc/yum.repos.d/
ambari.repo                                                                                 100%  268   273.3KB/s   00:00
[root@node1 yum.repos.d]# scp ambari.repo root@node5:/etc/yum.repos.d/
ambari.repo                                                                                 100%  268   309.9KB/s   00:00
[root@node1 yum.repos.d]# scp hdp.repo root@node2:/etc/yum.repos.d/
hdp.repo                                                                                    100%  482   697.2KB/s   00:00
[root@node1 yum.repos.d]# scp hdp.repo root@node3:/etc/yum.repos.d/
hdp.repo                                                                                    100%  482   557.2KB/s   00:00
[root@node1 yum.repos.d]# scp hdp.repo root@node4:/etc/yum.repos.d/
hdp.repo                                                                                    100%  482   499.9KB/s   00:00
[root@node1 yum.repos.d]# scp hdp.repo root@node5:/etc/yum.repos.d/
hdp.repo                                                                                    100%  482   527.9KB/s   00:00
[root@node1 yum.repos.d]#

測試http://192.168.12.191/ambari/ambari/centos7/2.6.2.2-1/ 是否可以訪問湾笛，如果可以則沒有問題

四、安裝ambari-server

ambari-server有兩種數(shù)據(jù)庫安裝模式

4.1默認postgresql數(shù)據(jù)庫的安裝方式

這種不推薦生產(chǎn)環(huán)境使用
（1）蓖墅、安裝和配置ambari-server

[root@master yum.repos.d]# yum -y install ambari-server
[root@master yum.repos.d]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? n
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==============================================================================
Enter choice (1): 3
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/java/jdk1.8.0_161
Validating JDK on Ambari Server...done.
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? n
Configuring database...
Default properties detected. Using built-in database.
Configuring ambari database...
Checking PostgreSQL...
Running initdb: This may take up to a minute.
Initializing database ... OK


About to start PostgreSQL
Configuring local database...
Configuring PostgreSQL...
Restarting PostgreSQL
Creating schema and user...
done.
Creating tables...
done.
Extracting system views...
ambari-admin-2.6.0.0.267.jar
...........
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

（2）啟動ambari

[root@master ~]# ambari-server start

成功啟動后在瀏覽器輸入Ambari地址：
http://192.168.12.191:8080

4.2第三方數(shù)據(jù)庫的安裝方式

生產(chǎn)使用Mysql
（1）安裝mysql
（2）登錄mysql论矾，執(zhí)行下面的語句
[root@node1 share]# mysql -uroot -p

CREATE DATABASE ambari;  
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'localhost' IDENTIFIED BY 'bigdata'; 
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%' IDENTIFIED BY 'bigdata';
FLUSH PRIVILEGES;  
use ambari;
source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql; 
show tables;  
quit; 

CREATE DATABASE hive;  
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'localhost' IDENTIFIED BY 'hive'; 
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%' IDENTIFIED BY 'hive';
FLUSH PRIVILEGES;

（3）mysql與ambari-server的連接
ambari-server 安裝

[root@node1 java]# yum install ambari-server

ambari-server 初始化

ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? n
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==============================================================================
Enter choice (1): 3
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/java/jdk1.8.0_121
Validating JDK on Ambari Server...done.
Checking GPL software agreement...
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? n
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3
Hostname (localhost):
Port (3306):
Database name (ambari):
Username (ambari):
Enter Database Password (bigdata):
Invalid characters in password. Use only alphanumeric or _ or - characters
Enter Database Password (bigdata):
Configuring ambari database...
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)?
Extracting system views...
....ambari-admin-2.6.2.2.1.jar
.......
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

（4）啟動Ambari

[root@node1 java]# ambari-server start

登陸http://192.168.12.191:8080 admin/admin[初始]

image.png

五靶橱、安裝配置部署HDP集群

1、登錄
登錄界面传黄，默認管理員賬戶登錄队寇，賬戶：admin 密碼：admin
2、安裝向導

（1）配置集群的名字為hadoop
（2）選擇版本并修改為本地源地址

image.png

3）安裝配置
選擇要安裝的服務器和上傳上面教程所拷貝出來的秘鑰文件id_rsa

image.png

（4）安裝ambari的agent识埋，同時檢查系統(tǒng)問題

image.png

如果這里出了問題窒舟，請檢查上面所有的步驟有沒有遺漏和未設置的參數(shù)。同時在重新修改了配置以后诵盼，最好是重置ambari-server來重新進行安裝,確保每一步無警告再執(zhí)行下一步惠豺。

[root@master ~]# # ambari-server stop    
[root@master ~]# # ambari-server reset   #重置命令
[root@master ~]# # ambari-server setup   #重新設置
[root@master ~]# # ambari-server start

（5）選擇要安裝的服務

image.png

（6）選擇分配服務

image.png

最后安裝完成

image.png

我的上面有一些警告和關閉的服務风宁，不用擔心洁墙。大體上就是這樣的安裝流程戒财。
對于HDP的安裝個人感覺比CDH使用CM的安裝方式要繁瑣很多热监。過程中出現(xiàn)錯誤需要重新安裝時需要刪除干凈相關文件和依賴饮寞，在重新安裝時Ambari會對已存在的包孝扛、文件、用戶進行檢測骂际、可根據(jù)提示完善刪除操作疗琉。