一布朦、網(wǎng)絡(luò)設(shè)置以及集群部署規(guī)劃
image
二查描、Docker容器
- 拉取鏡像
docker pull daocloud.io/library/centos:latest
-
創(chuàng)建容器
- 按照集群的架構(gòu)扣蜻，創(chuàng)建容器時需要設(shè)置固定IP走孽，所以先要在docker使用如下命令創(chuàng)建固定IP的子網(wǎng)
docker network create --subnet=172.18.0.0/16 netgroup
- docker的子網(wǎng)創(chuàng)建完成之后就可以創(chuàng)建固定IP的容器了
docker run -d --privileged -ti -v /sys/fs/cgroup:/sys/fs/cgroup --name Cluster-master -h Cluster-master -p 50070:50070 -p 50075:50075 -p 50080:50080 --net netgroup --ip 172.18.0.2 daocloud.io/library/centos /usr/sbin/init
docker run -d --privileged -ti -v /sys/fs/cgroup:/sys/fs/cgroup --name cluster-slave1 -h cluster-slave1 -p 8088:8088 -p 8042:8042 -p 8044:8044 --net netgroup --ip 172.18.0.3 daocloud.io/library/centos /usr/sbin/init
docker run -d --privileged -ti -v /sys/fs/cgroup:/sys/fs/cgroup --name cluster-slave2 -h cluster-slave2 -p 18000:18000 -p 18001:18001 -p 18002:18002 --net netgroup --ip 172.18.0.4 daocloud.io/library/centos /usr/sbin/init
- 啟動控制臺并進(jìn)入docker容器中
docker exec -it master /bin/bash
三钝的、軟件安裝
3.1 安裝OpenSSH免密登錄
[root@cluster-master /]# yum -y install openssh openssh-server openssh-clients
[root@cluster-master /]# systemctl restart sshd
由于 cluster-master 和cluster-slave1 需要訪問其他三個集群翁垂，需要修改ssh
[root@cluster-master /]# vi /etc/ssh/ssh_config
將原來的StrictHostKeyChecking ask 設(shè)置StrictHostKeyChecking為no 保存
master公鑰分發(fā)
[root@cluster-master /]# ssh-keygen -t rsa
[root@cluster-master .ssh]# cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys
文件生成之后用scp將公鑰文件分發(fā)到集群slave主機
docker的root密碼需要重新設(shè)置铆遭，先檢查passwd命令，沒有則安裝
[root@cluster-master .ssh]# yum -y install passwd
[root@cluster-master .ssh]# passwd
[root@cluster-master .ssh]# ssh root@slave1 'mkdir ~/.ssh'
[root@cluster-master .ssh]# scp ~/.ssh/authorized_keys root@slave1:~/.ssh [root@cluster-master .ssh]# ssh root@slave2 'mkdir ~/.ssh’
[root@cluster-master .ssh]# scp ~/.ssh/authorized_keys root@slave2:~/.ssh
分發(fā)完成之后測試(ssh root@slave1)是否已經(jīng)可以免輸入密碼登錄
cluster-slave1 也需要生成公鑰
[root@cluster-slave1 .ssh]# ssh-keygen -t rsa
[root@cluster-slave1 .ssh]# ssh-copy-id master
[root@cluster-slave1 .ssh]# ssh-copy-id slave2
[root@cluster-slave1 .ssh]# ssh-copy-id slave1
3.2 Ansible安裝
[root@cluster-master ~]# yum -y install epel-release
[root@cluster-master ~]# yum -y install ansible
[root@cluster-master ~]# vi /etc/ansible/hosts
[cluster]
cluster-master
cluster-slave1
cluster-slave2
[master]
cluster-master
[slaves]
cluster-slave1
cluster-slave2
配置docker容器hosts
由于/etc/hosts文件在容器啟動時被重寫沿猜，直接修改內(nèi)容在容器重啟后不能保留枚荣，為了讓容器在重啟之后獲取集群hosts，使用了一種啟動容器后重寫hosts的方法啼肩。需要在~/.bashrc中追加以下指令
[root@cluster-master ~]# vi .bashrc
:>/etc/hosts
cat >>/etc/hosts<<EOF
127.0.0.1 localhost
172.18.0.2 cluster-master
172.18.0.3 cluster-slave1
172.18.0.4 cluster-slave2
EOF
[root@cluster-master ~]# source ~/.bashrc
使配置文件生效,可以看到/etc/hosts文件已經(jīng)被改為需要的內(nèi)容
[root@cluster-master ~]# cat /etc/hosts
127.0.0.1 localhost
172.18.0.2 cluster-master
172.18.0.3 cluster-slave1
172.18.0.4 cluster-slave2
用ansible分發(fā).bashrc至集群slave下
[root@cluster-master ~]# ansible cluster -m copy -a "src=~/.bashrc dest=~/"
3.3 JDK && Hadoop
下載hadoop3 到/opt目錄下橄妆，解壓安裝包，并創(chuàng)建鏈接文件
[root@cluster-master opt]# yum install java-1.8.0-openjdk-devel.x86_64
[root@cluster-master opt]# wget https://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.9.2/hadoop-2.9.2.tar.gz
[root@cluster-master opt]# tar -xzvf hadoop-2.9.2.tar.gz
[root@cluster-master opt]# ln -s hadoop-2.9.2 hadoop
各個機器需要有which命令 如果沒有祈坠，hadoop 會報錯, 需要安裝
[root@cluster-master opt]# yum -y install which
配置java和hadoop環(huán)境變量 編輯 ~/.bashrc文件
#hadoop
export HADOOP_HOME=/opt/hadoop-2.9.2
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
#java
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el8_1.x86_64
export PATH=$JAVA_HOME/bin:$PATH
[root@cluster-master opt]# source .bashrc
配置hadoop運行所需配置文件
- 修改core-site.xml
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/tmp</value>
<description>A base for other temporary directories.</description>
</property>
<!-- file system properties -->
<property>
<name>fs.default.name</name>
<value>hdfs://cluster-master:9000</value>
</property>
<property>
<name>fs.trash.interval</name>
<value>4320</value>
</property>
</configuration>
- hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>/home/hadoop/tmp/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/hadoop/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.permissions.superusergroup</name>
<value>staff</value>
</property>
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>cluster-slave2:50090</value>
</property>
</configuration>
- mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
- yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>cluster-slave1</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
打包hadoop 向slaves分發(fā)
[root@master ~]# tar -cvf hadoop-dis.tar hadoop hadoop-2.9.2
使用ansible-playbook分發(fā).bashrc和hadoop-dis.tar至slave主機
---
- hosts: cluster
tasks:
- name: copy .bashrc to slaves
copy: src=~/.bashrc dest=~/
notify:
- exec source
- name: copy hadoop-dis.tar to slaves
unarchive: src=/opt/hadoop-dis.tar dest=/opt
handlers:
- name: exec source
shell: source ~/.bashrc
將以上yaml保存為hadoop-dis.yaml害碾，并執(zhí)行
[root@cluster-master ~]# ansible-playbook hadoop-dis.yaml
四、Hadoop 啟動
4.1 格式化namenode
[root@cluster-master ~]# hadoop namenode -format
4.2 HDFS 在cluster-master 上執(zhí)行
[root@cluster-master ~]# ./start-dfs.sh
4.3 YARN 在cluster-slave1上執(zhí)行
[root@cluster-master ~]# ./start-yarn.sh
4.4 驗證服務(wù)
http://宿主機IP:50070/ HDFS
http://宿主機IP:8088/ Yarn
