一柴底、基礎(chǔ)環(huán)境設(shè)置
注意：基礎(chǔ)環(huán)境需在每臺集群服務器設(shè)置顽素。
1.1、軟件環(huán)境
| 編號 | 軟件名稱 | 軟件版本 |
|---|---|---|
| 1 | 操作系統(tǒng) | CentOS Linux release 7.5.1804 (Core) |
| 2 | JDK | jdk-8u144-linux-x64 |
| 3 | Scala | scala-2.11.8 |
1.2烛占、服務器規(guī)劃
| 編號 | 服務器名稱 | 內(nèi)網(wǎng)IP | 內(nèi)存 | 用途 |
|---|---|---|---|---|
| 1 | cdhserver1 | 192.168.26.21 | 20G | CM-server胎挎、CM-agent、MySQL忆家、HTTP-server |
| 2 | cdhserver2 | 192.168.26.22 | 9G | CM-agent |
| 3 | cdhserver3 | 192.168.26.23 | 9G | CM-agent |
1.3犹菇、配置主機名IP映射
注意：CDH中謹慎使用FQDN,與主機名保持一致
vi /etc/hosts
#添加如下地址映射
192.168.26.21 cdhserver1
192.168.26.22 cdhserver2
192.168.26.23 cdhserver3
1.4、SSH免密登錄配置
ssh-keygen
ssh-copy-id -i ~/.ssh/id_rsa.pub root@cdhserver1
ssh-copy-id -i ~/.ssh/id_rsa.pub root@cdhserver2
ssh-copy-id -i ~/.ssh/id_rsa.pub root@cdhserver3
1.5芽卿、防火墻配置
為了安裝時簡單揭芍，可以執(zhí)行以下命令暫時關(guān)閉所有防火墻。
systemctl stop firewalld.service systemctl disable firewalld.service firewall-cmd --state
安裝完成開啟防火墻卸例，設(shè)置防火墻策略称杨。
#開啟防火墻
systemctl start firewalld.service
#設(shè)置可以訪問的端口
firewall-cmd –permanent –add-rich-rule="rule family='ipv4' source address='192.168.26.21' port protocol='tcp' port='0-65535' accept"
firewall-cmd –permanent –add-rich-rule="rule family='ipv4' source address='192.168.26.22' port protocol='tcp' port='0-65535' accept"
firewall-cmd –permanent –add-rich-rule="rule family='ipv4' source address='192.168.26.23' port protocol='tcp' port='0-65535' accept"
firewall-cmd –permanent –add-port=0-65535/tcp firewall-cmd –reload
1.6、關(guān)閉SELinux
vi /etc/sysconfig/selinux
#修改SELINUX為disabled
SELINUX=disabled
1.7币厕、開啟NTP服務實現(xiàn)時間同步
yum install -y ntp systemctl enable ntpd systemctl start ntpd
1.8列另、安裝JDK環(huán)境
話說JDK8最新更新的本版不開源了，下載歷史本版：jdk-8u181-linux-x64.rpm
https://www.oracle.com/technetwork/java/javase/downloads/java-archive-javase8-2177648.html
若系統(tǒng)自帶openJDK旦装，首先卸載
java -version rpm -qa | grep jdk
yum -y remove java-xxx-openjdk-xxx.el7_0.x86_64
yum -y remove java-xxx-openjdk-headless-xxx.el7_0.x86_64
安裝JDK
rpm -ivh jdk-8u181-linux-x64.rpm
配置JDK環(huán)境變量
vi /etc/profile source /etc/bashrc
export JAVA_HOME=/usr/java/jdk1.8.0_181-amd64 export JRE_HOME=$JAVA_HOME/jre export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib export PATH=$PATH:$JAVA_HOME/bin
1.9 安裝SCALA環(huán)境
下載scala-2.11.8 :
https://downloads.lightbend.com/scala/2.11.8/scala-2.11.8.rpm 或
https://www.scala-lang.org/files/archive/scala-2.11.8.rpm
rpm -ivh scala-2.11.8.rpm
1.10 修改Linux內(nèi)核參數(shù)
Cloudera 建議將 swappiness 設(shè)置為 10（控制換出運行時內(nèi)存權(quán)重）
#查看swappiness
cat /proc/sys/vm/swappiness
#永久性修改页衙，執(zhí)行下面兩條命令
sysctl -w vm.swappiness=10
echo vm.swappiness = 10 >> /etc/sysctl.conf
修改文件句柄數(shù)
vi /etc/security/limits.conf
在limits.conf文件末尾添加如下配置：
# End of file
* soft nofile 65536
* hard nofile 65536
* soft nproc 131072
* hard nproc 131072
vi /etc/security/limits.d/20-nproc.conf
修改 ‘* soft nproc 4096’ 成更大的值或注釋掉，因為該文件的配置會覆蓋limits.conf的配置阴绢。
* soft nproc 131072
透明大頁面壓縮
臨時關(guān)閉：
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
永久關(guān)閉：
vi /etc/rc.d/rc.local chmod +x /etc/rc.d/rc.local
在文件后添加下面內(nèi)容: 店乐，每次添加完記得執(zhí)行chmod +x /etc/rc.d/rc.local
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag
查看狀態(tài)
cat /sys/kernel/mm/transparent_hugepage/defrag
cat /sys/kernel/mm/transparent_hugepage/enabled
1.11 重啟集群服務器
init 6
二、MYSQL安裝
2.1 卸載centos7自帶的mariadb
rpm -qa | grep mariadb rpm -e --nodeps mariadb-libs-5.5.56-2.el7.x86_64
2.2 安裝依賴
yum install -y net-tools perl wget
2.3 下載MySQL安裝包并解壓
wget https://cdn.mysql.com//archives/mysql-5.7/mysql-5.7.18-1.el7.x86_64.rpm-bundle.tar
tar -xvf mysql-5.7.18-1.el7.x86_64.rpm-bundle.tar
mysql-community-server-5.7.18-1.el7.x86_64.rpm
mysql-community-embedded-devel-5.7.18-1.el7.x86_64.rpm
mysql-community-devel-5.7.18-1.el7.x86_64.rpm
mysql-community-client-5.7.18-1.el7.x86_64.rpm
mysql-community-common-5.7.18-1.el7.x86_64.rpm
mysql-community-embedded-5.7.18-1.el7.x86_64.rpm
mysql-community-embedded-compat-5.7.18-1.el7.x86_64.rpm
mysql-community-libs-5.7.18-1.el7.x86_64.rpm
mysql-community-server-minimal-5.7.18-1.el7.x86_64.rpm
mysql-community-test-5.7.18-1.el7.x86_64.rpm
mysql-community-minimal-debuginfo-5.7.18-1.el7.x86_64.rpm
mysql-community-libs-compat-5.7.18-1.el7.x86_64.rpm
2.4 依次安裝mysql rpm包
按common–>libs–>client–>server->devel->libs-compat的順序安裝
rpm -ivh mysql-community-common-5.7.18-1.el7.x86_64.rpm
rpm -ivh mysql-community-libs-5.7.18-1.el7.x86_64.rpm
rpm -ivh mysql-community-client-5.7.18-1.el7.x86_64.rpm
rpm -ivh mysql-community-server-5.7.18-1.el7.x86_64.rpm
rpm -ivh mysql-community-devel-5.7.18-1.el7.x86_64.rpm
#安裝下面包呻袭，否則Hue安裝報連接錯誤
rpm -ivh mysql-community-libs-compat-5.7.18-1.el7.x86_64.rpm
2.5 修改MySQL端口和數(shù)據(jù)存放路徑
mkdir -p /data/mysql/data
vi /etc/my.cnf
2.6 啟動mysql服務及查看狀態(tài)
systemctl start mysqld.service systemctl status mysqld.service
2.7 初始化數(shù)據(jù)庫密碼
#查看初始密碼
grep "password" /var/log/mysqld.log
mysql -uroot -p
#默認密碼檢查策略要求密碼必須包含：大小寫字母眨八、數(shù)字和特殊符號，并且長度不能少于8位 ALTER USER 'root'@'localhost' IDENTIFIED BY 'Root@2019';
#遠程連接數(shù)據(jù)庫的時候需要輸入用戶名和密碼
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'Root@2019' WITH GRANT OPTION;
FLUSH PRIVILEGES;
#查看端口
show global variables like 'port';
方式二：
修改配置文件/etc/my.cnf令MySQL跳過登錄時的權(quán)限檢驗左电，在[mysqld]下加入一行：
skip-grant-tables
使用 mysql登陸
mysql> use mysql;
mysql> UPDATE user SET authentication_string = password('root') WHERE host = 'localhost' AND user = 'root';
mysql> select host,user, authentication_string, password_expired from user;
mysql> update user set password_expired='N' where password_expired='Y' //密碼不過期
mysql> update user set host='%' where user='root' and host='localhost'; //遠程可訪問
mysql> flush privileges; //刷新
mysql> exit;//退出
刪除skip-grant-tables
2.8 設(shè)置開機啟動
systemctl enable mysqld systemctl daemon-reload
