Cloudera Manager 安裝 CDH5.x 心得

廢話不多說(shuō)合冀，先展示下這幾天搗鼓的成果

• Cloudera Manager 管理配置界面

這里寫圖片描述

• Hbase 管理界面及Hbase Web UI

[圖片上傳失敗...(image-1c6abb-1513138023093)]

這里寫圖片描述

• HDFS 管理界面 及 NameNode Web UI

[圖片上傳失敗...(image-82c66f-1513138023093)]

這里寫圖片描述

• Hive 管理界面 及 HiveServer2 Web UI

這里寫圖片描述

這里寫圖片描述

• Hue 管理界面 及 Web UI

這里寫圖片描述

這里寫圖片描述

首次登錄會(huì)提示讓創(chuàng)建管理員賬號(hào)和密碼豆拨，筆者 admin admin

這里寫圖片描述

• Oozie 管理界面 及 Oozie Web UI

這里寫圖片描述

# Oozie  Web UI 報(bào)錯(cuò)展运，后期修復(fù)

Oozie web console is disabled.
To enable Oozie web console install the Ext JS library.

參考：http://cdh01.aniu.so:11000/oozie/docs/DG_QuickStart.html

• YARN (MR2 Included) 管理界面 及 Web UI

這里寫圖片描述

• ResourceManager Web UI

  
  
  這里寫圖片描述

• HistoryServer Web UI
  [圖片上傳失敗...(image-47c126-1513138023093)]

• Zookeeper 管理界面

這里寫圖片描述

筆者這里zookeeper安裝的時(shí)候選擇的默認(rèn)剥险，因此只安裝了一個(gè)zookeeper纺涤，但個(gè)人感覺(jué)后期應(yīng)該需要增加zookeeper的界面數(shù)量

下面開(kāi)始說(shuō)安裝的注事事項(xiàng)

• 1五垮、配置環(huán)境要符合要求，要純凈的系統(tǒng)環(huán)境

# 筆者環(huán)境
# CM env
192.168.1.137 cdh01.aniu.so CentOS6.9 16G Memory 100G LVM卷 （Manger 節(jié)點(diǎn)）
192.168.1.148 cdh02.aniu.so CentOS6.9 4G Memory 70G LVM卷
192.168.1.149 cdh03.aniu.so CentOS6.9 4G Memory 70G LVM卷
192.168.1.150 cdh04.aniu.so CentOS6.9 4G Memory 70G LVM卷

建議小白參考筆者的環(huán)境配置琳疏，主機(jī)名可以自定義

#對(duì)四個(gè)節(jié)點(diǎn)的系統(tǒng)進(jìn)行更新有决，安裝開(kāi)發(fā)工具包
yum update -y && yum -y groupinstall "Development Tools"

• 2、關(guān)閉防火墻空盼、禁用Selinux

# 關(guān)閉防火墻
/etc/init.d/iptables stop && /etc/init.d/ip6tables stop
chkconfig iptables off && chkconfig ip6tables off

# 建議采用修改內(nèi)核參數(shù)的方式關(guān)閉ip6tables 
vim /etc/modprobe.d/dist.conf # 編輯此文件，在最后加入：

# Disable ipv6
alias net-pf-10 off
alias ipv6 off

# 禁用selinux
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
setenforce 0  # 不重啟臨時(shí)生效

• 3新荤、內(nèi)核參數(shù)調(diào)整

# 內(nèi)存參數(shù)調(diào)整
sysctl -w vm.swappiness=10 或者 編輯vim /etc/sysctl.conf揽趾，在最后加入：

vm.swappiness = 10

編輯啟動(dòng)項(xiàng)vim /etc/rc.local,最后加入：
echo never > /sys/kernel/mm/transparent_hugepage/defrag 
echo never > /sys/kernel/mm/transparent_hugepage/enabled

注：上面所有操作在所有節(jié)點(diǎn)都需要執(zhí)行

• 4、所有節(jié)點(diǎn)間配置免密認(rèn)證

# CM節(jié)點(diǎn)執(zhí)行
ssh-keygen -t rsa -b 2048 # 有確認(rèn)提示苛骨，一直按回車即可
cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
# 筆者 hosts.conf
# CM node
192.168.1.137 cdh01.aniu.so
192.168.1.148 cdh02.aniu.so
192.168.1.149 cdh03.aniu.so
192.168.1.150 cdh04.aniu.so

# 同步密鑰
for ip in $(awk '{print $1}' hosts.conf );do scp ~/.ssh/authorized_keys root@$ip:/root/.ssh ;done
或者使用
ssh-copy-id root@cdh01.aniu.so
ssh-copy-id root@cdh02.aniu.so
ssh-copy-id root@cdh03.aniu.so
ssh-copy-id root@cdh04.aniu.so
# 上面操作也需要在所有節(jié)點(diǎn)執(zhí)行

• 5篱瞎、使用cloudera-manger repo安裝CM

# 在CM節(jié)點(diǎn)執(zhí)行
wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/cloudera-manager.repo -P /etc/yum.repos.d
wget https://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/cloudera-cdh5.repo -P /etc/yum.repos.d
# 
yum clean all && yum makecache # 建議執(zhí)行不強(qiáng)制
yum install oracle-j2sdk1.7 -y
yum install cloudera-manager-daemons cloudera-manager-server -y

# 在其他節(jié)點(diǎn)執(zhí)行
wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/cloudera-manager.repo -P /etc/yum.repos.d
yum install oracle-j2sdk1.7 -y

# 配置JAVA_HOME
編輯vim /etc/profile
export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
保存退出執(zhí)行：
source /etc/profile 使更改的環(huán)境變量生效

# 在所有節(jié)點(diǎn)執(zhí)行配置JAVA_HOME的操作

• 6苟呐、CM節(jié)點(diǎn)安裝數(shù)據(jù)庫(kù)，或使用已有的數(shù)據(jù)

# 筆者使用mysql57-community.repo,安裝的mysql
[mysql57-community]
name=MySQL 5.7 Community Server
baseurl=http://repo.mysql.com/yum/mysql-5.7-community/el/6/$basearch/
enabled=1
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql
#
yum install mysql-community-embedded  mysql-community-server  mysql-community-devel mysql-community-client -y

# 筆者my.cnf
[root@cdh01 yum.repos.d]# cat /etc/my.cnf
[client]
port        = 3306
socket      = /var/lib/mysql/mysql.sock

[mysqld]
datadir     = /opt/mysql
socket      = /var/lib/mysql/mysql.sock
#skip-grant-tables 
skip-ssl
disable-partition-engine-check
port        = 3306
skip-external-locking
key_buffer_size = 16M
max_allowed_packet = 1M
table_open_cache = 64
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M
thread_cache_size = 8
query_cache_size = 8M
tmp_table_size = 16M
performance_schema_max_table_instances = 500

explicit_defaults_for_timestamp = true
max_connections = 500
max_connect_errors = 100
open_files_limit = 8192

log-bin=mysql-bin
binlog_format=mixed
server-id   = 1
expire_logs_days = 10
early-plugin-load = ""

default_storage_engine          = InnoDB
innodb_file_per_table           = 1
innodb_data_home_dir            = /opt/mysql
innodb_data_file_path           = ibdata1:1024M;ibdata2:10M:autoextend
innodb_log_group_home_dir       = /opt/mysql
innodb_buffer_pool_size         = 16M
innodb_log_file_size            = 5M
innodb_log_buffer_size          = 8M
innodb_flush_log_at_trx_commit  = 1
innodb_lock_wait_timeout        = 50
innodb_log_files_in_group       = 3 
innodb_buffer_pool_size         = 12G
innodb_log_file_size            = 512M
innodb_log_buffer_size          = 256M
innodb_flush_log_at_trx_commit  = 2
innodb_lock_wait_timeout        = 150
innodb_open_files               = 600
innodb_max_dirty_pages_pct      = 50
innodb_file_per_table           = 1

[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash

[myisamchk]
key_buffer_size = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M

[mysqlhotcopy]
xinteractive-timeout

symbolic-links=0
slow_query_log
long_query_time = 5
slow_query_log_file = /var/log/mysql-slow.log
log-error = /var/log/mysqld.log
pid-file = /var/run/mysqld/mysqld.pid

# 初始化mysql俐筋，并設(shè)置啟動(dòng)數(shù)據(jù)庫(kù)設(shè)置root密碼
/usr/sbin/mysqld --initialize --user=mysql --socket=/var/lib/mysql/mysql.sock # 先執(zhí)行
mysql_secure_installation # 再執(zhí)行

# 創(chuàng)建CM啟動(dòng)用到的數(shù)據(jù)庫(kù)
mysql -u root -pAniuops123. -e "create database cmf DEFAULT CHARACTER SET utf8;"
mysql -u root -pAniuops123. -e "GRANT ALL PRIVILEGES ON `cmf`.* TO 'cmf'@'localhost' IDENTIFIED BY 'Aniunas123.'";"

• 啟動(dòng)cloudera-scm-server牵素，并配置parcel

# 生成db配置文件
/usr/share/cmf/schema/scm_prepare_database.sh mysql cmf cmf Aniucmf123.

# 啟動(dòng)cloudera-scm-server
/etc/init.d/cloudera-scm-server start # 查看啟動(dòng)日志

# 配置parcel離線
cd /opt/cloudera/parcel-repo/ # 然后下載

wget http://archive.cloudera.com/cdh5/parcels/latest/CDH-5.13.1-1.cdh5.13.1.p0.2-el6.parcel
wget http://archive.cloudera.com/cdh5/parcels/latest/CDH-5.13.1-1.cdh5.13.1.p0.2-el6.parcel.sha1
wget http://archive.cloudera.com/cdh5/parcels/latest/manifest.json

# 注：讀者根據(jù)cloudera當(dāng)前CDH最新版本更改下載用到的URL
mv CDH-5.13.1-1.cdh5.13.1.p0.2-el6.parcel.sha1 CDH-5.13.1-1.cdh5.13.1.p0.2-el6.parcel.sha # 強(qiáng)制執(zhí)行、默認(rèn)使用本地的parcels包澄者，不更改sha1,cloudera-scm-server啟動(dòng)安裝時(shí)會(huì)去cloudera官網(wǎng)找匹配的parcel安裝包

重啟cloudera-scm-server笆呆，查看實(shí)時(shí)日志
/etc/init.d/cloudera-scm-server restart
tailf /var/log/cloudera-scm-server/cloudera-scm-server.log 

• 通過(guò)CM管理界面安裝CDH，注意事項(xiàng)

# CM server啟動(dòng)成功即可通過(guò)http://192.168.1.137:7180訪問(wèn)粱挡，默認(rèn)賬戶密碼：admin admin

# **重點(diǎn)內(nèi)容** 下面的話很重要：


不要勾選：?jiǎn)斡脩裟Ｊ?赠幕，筆者在此模式下安裝多次都沒(méi)成功，有心人可以測(cè)試

• 能一次性安裝成功的最好询筏，安裝不成功建議多試幾次榕堰，對(duì)初始化完成的虛擬機(jī)進(jìn)行快照操作，便于恢復(fù)