一慕的、服務(wù)器配置說明
1.四臺(tái)服務(wù)器
2.每臺(tái)服務(wù)器內(nèi)存4G
3./boot空間3072M
4.【可選擇配置】/swap交互虛擬內(nèi)存,4096
5./根目錄空間57G
二该窗、服務(wù)器網(wǎng)絡(luò)選擇配置
1.選擇NAT鏈接網(wǎng)絡(luò)/橋接
2.編輯~/.bashrc文件
vim ~/.bashrc
export all_proxy=socks5://192.168.197.2:21881
export http_proxy=http://192.168.197.2:21882
export https_proxy=http://192.168.197.2:21882
三危虱、關(guān)閉防火墻
1.查看防火墻的狀態(tài)
systemctl status firewalld.service
2.關(guān)閉防火墻
systemctl stop firewalld.service
3.永久關(guān)閉防火墻
systemctl disable firewalld.service
4.關(guān)閉每臺(tái)虛擬機(jī)的SELinux
修改/etc/selinux/config 文件
將SELINUX=enforcing改為SELINUX=disabled
5.系統(tǒng)內(nèi)核調(diào)優(yōu)參考sysctl.conf
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.ipv4.tcp_max_tw_buckets = 60000
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.tcp_wmem = 4096 16384 4194304
net.core.wmem_default = 8388608
net.core.rmem_default = 8388608
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.netdev_max_backlog = 262144
net.core.somaxconn = 262144
net.ipv4.tcp_max_orphans = 3276800
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_synack_retries = 1
net.ipv4.tcp_syn_retries = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_mem = 94500000 915000000 927000000
net.ipv4.tcp_fin_timeout = 1
net.ipv4.tcp_keepalive_time = 1200
net.ipv4.tcp_max_syn_backlog = 65536
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 2
net.ipv4.tcp_tw_recycle = 1
#net.ipv4.tcp_tw_len = 1
net.ipv4.tcp_tw_reuse = 1
#net.ipv4.tcp_fin_timeout = 30
#net.ipv4.tcp_keepalive_time = 120
net.ipv4.ip_local_port_range = 1024? 65535
四、建立用戶
1.創(chuàng)建組和用戶:bdadmin
# groupadd bdadmin
# useradd bdadmin-r -m -g bdadmin
設(shè)置密碼:
# passwd bdadmin
New password: <changeme>
Retype new password: <changeme>
2.切換root與用戶之間免密配置 /etc/sudoers
bdadmin ALL=(ALL) NOPASSWD: ALL
【注:為方便執(zhí)行g(shù)p的一些命令翼馆,在切換用戶時(shí)可以免密執(zhí)行】
3.授予bdadmin數(shù)控權(quán)限
? 確保取消注釋包含 NOPASSWD 關(guān)鍵字的行。使用此命令將 bdadmin 用戶添加到 wheel 組中金度。
# usermod -aG wheel bdadmin
五应媚、配置SSH密鑰以及服務(wù)器之間免密配置
1.開啟公鑰配置。?root用戶下猜极,修改配置文件 /etc/ssh/sshd_config
PubkeyAuthentication yes
PasswordAuthentication yes
2.在主節(jié)點(diǎn)配置私鑰與公鑰中姜,切換到對(duì)應(yīng)的用戶上
ssh-keygen -t ed25519 -b 4096
cat id_dsa.pub >> authorized_keys
3.給服務(wù)器添加公鑰
將主節(jié)點(diǎn)上的密鑰分發(fā)給其他機(jī)器
ssh-copy-id -i /home/bdadmin/.ssh/authorized_keys bdadmin@segment1-server
ssh-copy-id -i /home/bdadmin/.ssh/authorized_keys bdadmin@segment2-server
ssh-copy-id -i /home/bdadmin/.ssh/authorized_keys bdadmin@segment3-server
4.驗(yàn)證
ssh bdadmin@segment1-server
六、安裝JDK
可參考JAVA中的跟伏,Centos7 部署jdk
七丢胚、部署Scala
下載https://downloads.lightbend.com/scala/2.11.8/scala-2.11.8.tgz
1.解壓scala-2.11.8.tgz
2.配置環(huán)境變量 /etc/profile
export SCALA_HOME=/usr/local/src/scala-2.11.8
export PATH=$PATH:${SCALA_HOME}/bin
八翩瓜、在master節(jié)點(diǎn)上安裝mysql
1.檢查centos 自帶的mariadb,并卸載
rpm -qa|grep mariadb? (檢查是否存在)
rpm -e --nodeps mariadb-libs-5.5.68-1.el7.x86_64 (卸載相應(yīng)的內(nèi)容)
2.下載
wget https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-community-libs-8.0.18-1.el7.x86_64.rpm
wget https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-community-common-8.0.18-1.el7.x86_64.rpm
wget https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-community-client-8.0.18-1.el7.x86_64.rpm
wget https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-community-server-8.0.18-1.el7.x86_64.rpm
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-8.0.18-1.el7.noarch.rpm
3.安裝
rpm -ivh mysql-community-common-8.0.18-1.el7.x86_64.rpm
rpm -ivh mysql-community-libs-8.0.18-1.el7.x86_64.rpm
rpm -ivh mysql-community-client-8.0.18-1.el7.x86_64.rpm
rpm -ivh mysql-community-server-8.0.18-1.el7.x86_64.rpm
rpm -ivh mysql-connector-java-8.0.18-1.el7.noarch.rpm --force --nodeps
啟動(dòng)
systemctl start mysqld
查看啟動(dòng)狀態(tài)
systemctl status mysqld
開機(jī)啟動(dòng)
systemctl enable mysqld
查看初始root密碼
cat /var/log/mysqld.log | grep password
把密碼記錄下來嗜桌,接下來登陸需要輸入
登陸MySQL
mysql -u root -p
輸入上面記錄的密碼
改變r(jià)oot的登陸密碼
use mysql;
ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'Root@123';
允許非本機(jī)登陸
update user set host='%' where user='root';
grant all privileges on *.* to 'root' with grant option;
flush privileges;
添加hivedb用戶
create user hivedb identified by 'Hivedb@123';
grant all privileges on *.* to 'hivedb' with grant option;
flush privileges;
查看用戶:
select user,host from user;
配置文件 /etc/my.cnf
character_set_server=utf8mb4
collation-server=utf8mb4_general_ci
default-storage-engine=INNODB
九奥溺、zookeeper部署
下載zookeeper3.6.3版本,解壓文件到/app路徑下
下載地址:https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.6.3/apache-zookeeper-3.6.3-bin.tar.gz
1.建立對(duì)應(yīng)的數(shù)據(jù)存放和日志目錄
mkdir -p /data/zookeeper/zdata/data
mkdir -p /data/zookeeper/zdata/logs
2.修改配置
cd /app
ln -s apache-zookeeper-3.6.3-bin zookeeper
cd zookeeper/conf
cp zoo_sample.cfg zoo.cfg
# 修改zoo.cfg,以下為配置信息
dataDir=/data/zookeeper/zdata/data
dataLogsDir=/data/zookeeper/zdata/logs
# 副本數(shù)
autopurge.snapRetainCount=3
server.0=master-server:2888:3888
server.1=segment1-server:2888:3888
server.2=segment2-server:2888:3888
server.3=segment3-server:2888:3888
4lw.commands.whitelist=*
3.再dataDir的目錄下創(chuàng)建myid文件骨宠,并在對(duì)應(yīng)服務(wù)器添加服務(wù)編號(hào)
[master-server] # vim /data/zookeeper/zdata/data/myid
myid文件中添加對(duì)應(yīng)服務(wù)器的編號(hào):
matser-server 編輯myid文件 寫入數(shù)子0;
segment1-server?編輯myid文件 寫入數(shù)子1;
segment2-server?編輯myid文件 寫入數(shù)子2;
segment3-server?編輯myid文件 寫入數(shù)子3;
4.配置環(huán)境變量
# 在/etc/profile.d/env.sh中添加
export ZOOKEEPER_HOME=/app/apache-zookeeper-3.6.3
5.編寫集群啟動(dòng)管理腳本 vim?execZookeeper.sh
#!/bin/bash
if [ $# -lt 1 ]
then
? ? echo "No Args Input..."
? ? exit ;
fi
case $1 in
"start")
? ? echo " =================== 啟動(dòng) zookeeper 集群 ======================"
? ? for host in hadoop101 hadoop102 hadoop103
? ? do
? ? ? echo "----------------- 啟動(dòng) $host -------------------"
? ? ? ssh $host "/app/apache-zookeeper-3.6.3/bin/zkServer.sh start"
? ? done
;;
"stop")
? ? echo " =================== 關(guān)閉 zookeeper 集群 ======================"
? ? for host in hadoop101 hadoop102 hadoop103
? ? do
? ? ? echo "----------------- 關(guān)閉 $host -------------------"
? ? ? ssh $host "/app/apache-zookeeper-3.6.3/bin/zkServer.sh stop"
? ? done
;;
"status")
? ? echo " =================== 檢查 zookeeper 集群 ======================"
? ? for host in hadoop101 hadoop102 hadoop103
? ? do
? ? ? echo "----------------- 檢查 $host -------------------"
? ? ? ssh $host "/app/apache-zookeeper-3.6.3/bin/zkServer.sh status"
? ? done
esac
6.啟動(dòng):sh execZookeeper.sh start;停止:sh?execZookeeper.sh stop;狀態(tài):sh?execZookeeper.sh status
7.啟動(dòng)zookeeper之后需要格式化一下:
hdfs zkfc -formatZK
十浮定、hadoop部署
1.下載、解壓
https://archive.apache.org/dist/hadoop/common/hadoop-3.3.1
tar -zxvf hadoop-3.3.1.tar.gz?
2.配置環(huán)境變量
在/etc/profie.d/ 的文件夾下面新建一個(gè)文件 my_env.sh存儲(chǔ)系統(tǒng)環(huán)境變量
#HADOOP_HOME
export HADOOP_HOME=/usr/local/src/hadoop-3.3.1
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
3.生效環(huán)境變量层亿、并驗(yàn)證hadoop
source /etc/profile
hadoop
4.配置.sh文件
4.1.hadoop-3.3.1/etc/hadoop/hadoop_env.sh
配置用戶
export HDFS_NAMENODE_USER="bdadmin"
export HDFS_DATANODE_USER="bdadmin"
export HDFS_SECONDARYNAMENODE_USER="bdadmin"
export YARN_RESOURCEMANAGER_USER="bdadmin"
export YARN_NODEMANAGER_USER="bdadmin"
配置路徑
export JAVA_HOME=/usr/local/src/jdk1.8.0_321
export HADOOP_HOME=/usr/local/src/hadoop-3.3.1
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}
配置網(wǎng)絡(luò)接口:優(yōu)先先擇IPv4
export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true"
配置操作系統(tǒng)類型
export HADOOP_OS_TYPE=${HADOOP_OS_TYPE:-$(uname -s)}
4.2.hadoop-3.3.1/etc/hadoop/mapred_env.sh
export JAVA_HOME=/usr/local/src/jdk1.8.0_321
4.3.hadoop-3.3.1/etc/hadoop/yarn_env.sh
export JAVA_HOME=/usr/local/src/jdk1.8.0_321
5.配置自定以site.xml文件
5.1hadoop-3.3.1/etc/hadoop/core-site.xml
<configuration>
? ? ? ? <!--設(shè)置默認(rèn)的訪問端-->
? ? <property>
? ? ? ? <name>fs.defaultFS</name>
? ? ? ? <value>hdfs://master-server:8020</value>
? ? ? ? <description>Where HDFS NameNode can be found on the network</description>
? ? </property>
? ? ? ? <!--hadoop存儲(chǔ)零時(shí)文件-->
? ? <property>
? ? ? ? <name>hadoop.tmp.dir</name>
? ? ? ? <value>/home/bdadmin/hadoop/core-site/data/hdfs/tmp</value>
? ? ? ? <description>Where Hadoop will place all of its working files</description>
? ? </property>
? ? <!-- 用戶角色配置桦卒,不配置此項(xiàng)會(huì)導(dǎo)致web頁面報(bào)錯(cuò)(不能操作數(shù)據(jù)) -->
? ? <property>
? ? ? ? <name>hadoop.http.staticuser.user</name>
? ? ? ? <value>hadoop</value>
? ? </property>
? ? <!-- 權(quán)限配置 hadoop.proxyuser.{填寫自己的用戶名}.hosts-->
? ? <property>
? ? ? ? <name>hadoop.proxyuser.hadoop.hosts</name>
? ? ? ? <value>*</value>
? ? </property>
? ? <property>
? ? ? ? <name>hadoop.proxyuser.hadoop.groups</name>
? ? ? ? <value>*</value>
? ? </property>
</configuration>
5.2hadoop-3.3.1/etc/hadoop/hdfs-site.xml
<configuration>
? ? <!-- 副本數(shù)配置 -->
? ? <property>
? ? ? <name>dfs.replication</name>
? ? ? <value>2</value>
? ? </property>
? ? <property>
? ? <!--namenode 節(jié)點(diǎn)數(shù)據(jù)(即元數(shù)據(jù))的存放位置,可以指定多個(gè)目錄實(shí)現(xiàn)容錯(cuò)匿又,多個(gè)目錄用逗號(hào)分隔-->
? ? ? <name>dfs.namenode.name.dir</name>
? ? ? <value>/home/bdadmin/hadoop/hdfs-site/dfs/name</value>
? ? </property>
? ? <property>
? ? <!--datanode 節(jié)點(diǎn)數(shù)據(jù)(即數(shù)據(jù)塊)的存放位置-->
? ? ? <name>dfs.datanode.data.dir</name>
? ? ? <value>/home/bdadmin/hadoop/hdfs-site/dfs/data</value>
? ? </property>
? ? <!-- 集群名稱方灾,此值在接下來的配置中將多次出現(xiàn)務(wù)必注意同步修改 -->
? ? <property>
? ? ? <name>dfs.nameservices</name>
? ? ? <value>mycluster</value>
? ? </property>
? ? <!-- 所有的namenode列表,此處也只是邏輯名稱碌更,非namenode所在的主機(jī)名稱 -->
? ? <property>
? ? ? <name>dfs.ha.namenodes.mycluster</name>
? ? ? <value>nn1,nn2</value>
? ? </property>
? ? <!-- namenode之間用于RPC通信的地址裕偿,value填寫namenode所在的主機(jī)地址 -->
? ? <!-- 默認(rèn)端口8020,注意mycluster與nn1要和上文的配置一致 -->
? ? <property>
? ? ? <name>dfs.namenode.rpc-address.mycluster.nn1</name>
? ? ? <value>master-server:8020</value>
? ? </property>
? ? <property>
? ? ? <name>dfs.namenode.rpc-address.mycluster.nn2</name>
? ? ? <value>segment1-server:8020</value>
? ? </property>
? ? <!-- namenode的web訪問地址痛单,默認(rèn)端口9870 -->
? ? <property>
? ? ? <name>dfs.namenode.http-address.mycluster.nn1</name>
? ? ? <value>master-server:9870</value>
? ? </property>
? ? <property>
? ? ? <name>dfs.namenode.http-address.mycluster.nn2</name>
? ? ? <value>segment1-server:9870</value>
? ? </property>
? ? <!-- journalnode主機(jī)地址嘿棘,最少三臺(tái),默認(rèn)端口8485 -->
? ? <!-- 格式為 qjournal://jn1:port;jn2:port;jn3:port/${nameservices} -->
? ? <!-- a shared edits dir must not be specified if HA is not enabled -->
? ? <!-- 偽分布式時(shí),取消該配置 -->
? ? <property>
? ? ? <name>dfs.namenode.shared.edits.dir</name>
? ? ? <value>qjournal://master-server:8485;segment1-server:8485;segment2-server:8485/mycluster</value>
? ? </property>
? ? <!-- namenode日志文件輸出路徑旭绒,即journalnode讀取變更的位置 -->
? ? <property>
? ? ? <name>dfs.journalnode.edits.dir</name>
? ? ? <value>/home/bdadmin/hadoop/log/journalnode-namenode</value>
? </property>
</configuration>
5.3hadoop-3.3.1/etc/hadoop/yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
? ? <!-- Reducer獲取數(shù)據(jù)的方式 -->
? ? <property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
? ? </property>
? ? <!-- 指定YARN的ResourceManager的地址 -->
? ? <property>
<name>yarn.resourcemanager.hostname</name>
<value>master-server</value>
? ? </property>
</configuration>
5.4hadoop-3.3.1/etc/hadoop/mapred-site.xml
<configuration>
? ? <!-- 指定MR運(yùn)行在Yarn上 -->
? ? <property>
? ? ? <name>mapreduce.framework.name</name>
? ? ? <value>yarn</value>
? ? </property>
</configuration>
6.初始化namenode
hdfs namenode -format
7.當(dāng)無法啟動(dòng)namenode的standby節(jié)點(diǎn)時(shí)鸟妙,把namenode的active節(jié)點(diǎn)的current文件下的所有文件copy到standby下的current文件夾下面
出現(xiàn)如下情況時(shí):
2022-03-24 14:49:52,315 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: segment1-server/192.168.197.124:8020. Already tried 42 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=100, sleepTime=10000 MILLISECONDS)
執(zhí)行
scp -r /home/bdadmin/hadoop/hdfs-site/dfs/name/current/* bdadmin@segment1-server:/home/bdadmin/hadoop/hdfs-site/dfs/name/current/
【注意】
1.第一次執(zhí)行hadoop啟動(dòng),配置了resourcemanager后挥吵,初始化時(shí)考慮權(quán)限問題重父,如果resourcemanager不能夠正常啟動(dòng)起來,停止服務(wù)resourcemanager忽匈,刪除服務(wù)器中零時(shí)的文件內(nèi)容:
rm -rf /tmp/*
再重新啟動(dòng)即可房午,若還有問題,則查看日志丹允。
2.再重新格式化namenode后歪沃,需要操作清空如下路勁中的文件
①、再hadoop中的core-site文件找到配置的零時(shí)文件路勁嫌松,刪除此路徑下所有的內(nèi)容
<!--hadoop存儲(chǔ)零時(shí)文件-->
? ? <property>
? ? ? ? <name>hadoop.tmp.dir</name>
? ? ? ? <value>/home/bdadmin/hadoop/core-site/data/hdfs/tmp</value>
? ? ? ? <description>Where Hadoop will place all of its working files</description>
? ? </property>
$ rm -rf?/home/bdadmin/hadoop/core-site/data/hdfs/tmp/*
②、再hadoop中的hdfs-site文件找到配置的數(shù)據(jù)節(jié)點(diǎn)存放位置奕污,刪除此路徑下的所有內(nèi)容
<property>
? ? <!--namenode 節(jié)點(diǎn)數(shù)據(jù)(即元數(shù)據(jù))的存放位置萎羔,可以指定多個(gè)目錄實(shí)現(xiàn)容錯(cuò),多個(gè)目錄用逗號(hào)分隔-->
? ? <name>dfs.namenode.name.dir</name>
? ? ? <value>file:///home/bdadmin/hadoop/hdfs-site/dfs/name</value>
? ? </property>
? ? <property>
? ? <!--datanode 節(jié)點(diǎn)數(shù)據(jù)(即數(shù)據(jù)塊)的存放位置-->
? ? ? <name>dfs.datanode.data.dir</name>
? ? ? <value>file:///home/bdadmin/hadoop/hdfs-site/dfs/data</value>
? ? </property>
$ rm -rf?/home/bdadmin/hadoop/hdfs-site/dfs/name
$ rm -rf?/home/bdadmin/hadoop/hdfs-site/dfs/data
③碳默、再hadoop中的hdfs-site文件找到配置的journalnode存放位置贾陷,刪除此路徑下的所有內(nèi)容
<!-- namenode日志文件輸出路徑缘眶,即journalnode讀取變更的位置 -->
? ? <property>
? ? ? <name>dfs.journalnode.edits.dir</name>
? ? ? <value>/home/bdadmin/hadoop/log/journalnode-namenode</value>
? ? </property>
$ rm -rf?/home/bdadmin/hadoop/log/journalnode-namenode
④、重啟hadoop服務(wù)髓废,并對(duì)namenode初始化(格式化)
這時(shí)候會(huì)報(bào)錯(cuò)/home/bdadmin/hadoop/log/journalnode-namenode下存在文件巷懈,不能初始化。
再次直接重復(fù)①-②-③步驟
然后停掉hadoop服務(wù)慌洪,在重啟就可以了顶燕。
⑤、最后把a(bǔ)ctive中的namenode下的current下的所有文件復(fù)制拷貝一份到standby中的namenode
scp -r /home/bdadmin/hadoop/hdfs-site/dfs/name/current/* bdadmin@segment1-server:/home/bdadmin/hadoop/hdfs-site/dfs/name/current/
⑥冈爹、再次停掉hadoop服務(wù)涌攻,重新啟動(dòng)。