基礎(chǔ)環(huán)境
MBP, Palallels Desktop, Centos7
關(guān)鍵詞
docker, ansible, hadoop
集群架構(gòu)
集群包含4臺“虛擬主機(jī)”，采用Docker創(chuàng)建容器的方式創(chuàng)建噪伊，無需創(chuàng)建多臺虛擬機(jī)竹握，簡單方便薪缆。
| OS | hostname | IP |
|---|---|---|
| Centos7 | cluster-master | 172.18.0.2 |
| Centos7 | cluster-slave1 | 172.18.0.3 |
| Centos7 | cluster-slave1 | 172.18.0.4 |
| Centos7 | cluster-slave1 | 172.18.0.5 |
Docker
docker 安裝
登錄到Centos 7寄主機(jī)中安裝docker
[root@centos-linux ~]# yum -y install docker.x86_64
啟動docker服務(wù)
[root@centos-linux ~]# systemctl start docker
Centos鏡像拉取
國內(nèi)使用docker.io拉取鏡像的時候非常慢烛缔，所以找了鏡像倉庫進(jìn)行拉取怒竿。
本次用到的鏡像倉庫是http://hub.daocloud.io, 在搜索欄中輸入centos并搜索
搜索centos
搜索結(jié)果
Centos鏡像描述
在檢索到的鏡像詳情右側(cè)給出了pull的命令荠雕，直接運(yùn)行即可稳其。
[root@centos-linux ~]# docker pull daocloud.io/library/centos:latest
拉取完成之后可以看到
[root@centos-linux ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
daocloud.io/library/centos latest 328edcd84f1b 3 weeks ago 192.5 MB
創(chuàng)建容器
按照集群的架構(gòu)驶赏，創(chuàng)建容器時需要設(shè)置固定IP，所以先要在docker下創(chuàng)建固定IP的子網(wǎng)
[root@centos-linux ~]# docker network create --subnet=172.18.0.0/16 netgroup
docker的子網(wǎng)創(chuàng)建完成之后就可以創(chuàng)建固定IP的容器了
[root@centos-linux ~]# docker run -d --privileged -ti -v /sys/fs/cgroup:/sys/fs/cgroup --name cluster-master -h cluster-master --net netgroup --ip 172.18.0.2 daocloud.io/library/centos /usr/sbin/init
[root@centos-linux ~]# docker run -d --privileged -ti -v /sys/fs/cgroup:/sys/fs/cgroup --name cluster-slave1 -h cluster-slave1 --net netgroup --ip 172.18.0.3 daocloud.io/library/centos /usr/sbin/init
[root@centos-linux ~]# docker run -d --privileged -ti -v /sys/fs/cgroup:/sys/fs/cgroup --name cluster-slave2 -h cluster-slave2 --net netgroup --ip 172.18.0.4 daocloud.io/library/centos /usr/sbin/init
[root@centos-linux ~]# docker run -d --privileged -ti -v /sys/fs/cgroup:/sys/fs/cgroup --name cluster-slave3 -h cluster-slave3 --net netgroup --ip 172.18.0.5 daocloud.io/library/centos /usr/sbin/init
在Centos7下使用簡單方式創(chuàng)建容器后遇到sshd啟動失敗的問題既鞠，所以需要添加參數(shù)--privileged和-v /sys/fs/cgroup:/sys/fs/cgroup煤傍，并在啟動的時候運(yùn)行/usr/sbin/init。
openssh
創(chuàng)建的centos容器中并沒有預(yù)裝ssh服務(wù)嘱蛋，需要登錄到所有容器中手動安裝蚯姆。
登錄容器
[root@centos-linux ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
328913df0d52 daocloud.io/library/centos "/usr/sbin/init" 1 hours ago Up 1 hours cluster-slave3
83361d4bf079 daocloud.io/library/centos "/usr/sbin/init" 1 hours ago Up 1 hours cluster-slave2
35d1e5732340 daocloud.io/library/centos "/usr/sbin/init" 1 hours ago Up 1 hours cluster-slave1
37fbcc24b0e3 daocloud.io/library/centos "/usr/sbin/init" 1 hours ago Up 1 hours cluster-master
[root@centos-linux ~]# docker exec -it 37fbcc24b0e3 /bin/bash
安裝openssh
[root@cluster-master /]# yum -y install openssh openssh-server openssh-clients
啟動ssh服務(wù)
[root@cluster-master /]# systemctl start sshd
ssh自動接受新的公鑰
master設(shè)置ssh登錄自動添加kown_hosts
vi編輯/etc/ssh/ssh_config配置文件
設(shè)置StrictHostKeyChecking為no
cluster-master公鑰分發(fā)
在master機(jī)上執(zhí)行ssh-keygen -t rsa并一路回車，完成之后會生成~/.ssh目錄洒敏，目錄下有id_rsa（私鑰文件）和id_rsa.pub（公鑰文件）龄恋，再將id_rsa.pub重定向到文件authorized_keys
[root@cluster-master /]# cd ~/.ssh
[root@cluster-master /]# cat id_rsa.pub>authorized_keys
文件生成之后用scp將公鑰文件分發(fā)到集群slave主機(jī)
[root@cluster-master /]# ssh root@cluster-slave1 'mkdir ~/.ssh'
[root@cluster-master /]# scp ~/.ssh/authorized_keys root@cluster-slave1:~/.ssh
[root@cluster-master /]# ssh root@cluster-slave2 'mkdir ~/.ssh'
[root@cluster-master /]# scp ~/.ssh/authorized_keys root@cluster-slave2:~/.ssh
[root@cluster-master /]# ssh root@cluster-slave3 'mkdir ~/.ssh'
[root@cluster-master /]# scp ~/.ssh/authorized_keys root@cluster-slave3:~/.ssh
分發(fā)完成之后測試是否已經(jīng)可以免輸入密碼登錄。另外本次實驗使用到了root用戶凶伙，如果在其他用戶下使用免密碼登錄郭毕，需要確保用戶對~/.ssh/authorized_keys文件有可操作權(quán)限。
Ansible
源碼安裝ansible
使用git將文件安裝在cluster-master的/opt目錄下镊靴，因為無需agent铣卡，所以安裝在一臺主控機(jī)上即可管理集群。容器中的centos沒有預(yù)裝git偏竟，在安裝之前需要事先安裝git:
[root@cluster-master /]# yum -y install git
在/opt下執(zhí)行
[root@cluster-master opt]# git clone git://github.com/ansible/ansible.git --recursive
在執(zhí)行ansible的env-setup之前需要安裝一些容器中沒有預(yù)裝的軟件包
[root@cluster-master opt]# yum -y install python-setuptools
[root@cluster-master opt]# easy_install pip
[root@cluster-master opt]# pip install paramiko PyYAML Jinja2 httplib2 six
進(jìn)入ansible目錄煮落，執(zhí)行安裝腳本
[root@cluster-master opt]# cd ./ansible
[root@cluster-master ansible]# source ./hacking/env-setup
配置ansible hosts
創(chuàng)建/opt/ansible/hosts文件，并將cluster內(nèi)的主機(jī)按組的方式寫入該文件
[cluster]
cluster-master
cluster-slave1
cluster-slave2
cluster-slave3
[master]
cluster-master
[slaves]
cluster-slave1
cluster-slave2
cluster-slave3
配置docker容器hosts
/etc/hosts文件在容器啟動時被重寫踊谋，直接修改內(nèi)容在容器重啟后不能保留蝉仇，為了讓容器在重啟之后獲取集群hosts，使用了一種啟動容器后重寫hosts的方法殖蚕。
需要在~/.bashrc中追加以下指令
:>/etc/hosts
cat >>/etc/hosts<<EOF
127.0.0.1 localhost
172.18.0.2 cluster-master
172.18.0.3 cluster-slave1
172.18.0.4 cluster-slave2
172.18.0.5 cluster-slave3
EOF
運(yùn)行source ~/.bashrc使之生效
這時可以看到/etc/hosts文件已經(jīng)被改為需要的內(nèi)容
[root@cluster-master ansible]# cat /etc/hosts
127.0.0.1 localhost
172.18.0.2 cluster-master
172.18.0.3 cluster-slave1
172.18.0.4 cluster-slave2
172.18.0.5 cluster-slave3
用ansible分發(fā).bashrc至集群slave下
ansible cluster -m copy -a "src=~/.bashrc dest=~/"
Hadoop
在集群中安裝openjdk
使用ansible在在集群中安裝openjdk
[root@cluster-master ansible]# ansible cluster -m yum -a "name=java-1.8.0-openjdk,java-1.8.0-openjdk-devel state=latest"
在cluster-master上安裝hadoop
將hadoop安裝包下載至/opt目錄下
[root@cluster-master opt]# wget http://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-2.7.4/hadoop-2.7.4.tar.gz
下載完成之后解壓安裝包轿衔，并創(chuàng)建鏈接文件
[root@cluster-master opt]# tar -xzvf hadoop-2.7.4.tar.gz
[root@cluster-master opt]# ln -s hadoop-2.7.4 hadoop
設(shè)置java和hadoop環(huán)境變量(.bashrc)
# java
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-1.b16.el7_3.x86_64
export PATH=$HADOOP_HOME/bin:$PATH
# hadoop
export HADOOP_HOME=/opt/hadoop
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
修改hadoop運(yùn)行所需配置文件
core-site.xml
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/tmp</value>
<description>A base for other temporary directories.</description>
</property>
<!-- file system properties -->
<property>
<name>fs.default.name</name>
<value>hdfs://cluster-master:9000</value>
</property>
</configuration>
hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
</configuration>
mapred-site.xml
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>http://cluster-master:9001</value>
</property>
</configuration>
yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:18040</value>
</property>
</configuration>
打包hadoop文件
將hadoop鏈接文件和hadoop-2.7.4打包成一個文件方便ansible分發(fā)到slave主機(jī)
[root@cluster-master opt]# tar -cvf hadoop-dis.tar hadoop hadoop-2.7.4
使用ansible-playbook分發(fā).bashrc和hadoop-dis.tar至slave主機(jī)
---
- hosts: cluster
tasks:
- name: copy .bashrc to slaves
copy: src=~/.bashrc dest=~/
notify:
- exec source
- name: copy hadoop-dis.tar to slaves
unarchive: src=/opt/hadoop-dis.tar dest=/opt
handlers:
- name: exec source
shell: source ~/.bashrc
將以上yaml保存為hadoop-dis.yaml，并執(zhí)行
[root@cluster-master opt]# ansible-playbook hadoop-dis.yaml
hadoop-dis.tar會自動解壓到slave主機(jī)的/opt目錄下睦疫。
格式化namenode
[root@cluster-master opt]# hadoop namenode -format
啟動hadoop集群
到這一步已經(jīng)可以開始hadoop之旅了害驹，啟動比較簡單，在$HADOOP_HOME/sbin下有幾個啟動和停止的腳本如下：
[root@cluster-master opt]# cd $HADOOP_HOME/sbin
[root@cluster-master sbin]# ls -l
total 120
-rwxr-xr-x. 1 20415 101 2752 Aug 1 00:35 distribute-exclude.sh
-rwxr-xr-x. 1 20415 101 6452 Aug 1 00:35 hadoop-daemon.sh
-rwxr-xr-x. 1 20415 101 1360 Aug 1 00:35 hadoop-daemons.sh
-rwxr-xr-x. 1 20415 101 1640 Aug 1 00:35 hdfs-config.cmd
-rwxr-xr-x. 1 20415 101 1427 Aug 1 00:35 hdfs-config.sh
-rwxr-xr-x. 1 20415 101 2291 Aug 1 00:35 httpfs.sh
-rwxr-xr-x. 1 20415 101 3128 Aug 1 00:35 kms.sh
-rwxr-xr-x. 1 20415 101 4080 Aug 1 00:35 mr-jobhistory-daemon.sh
-rwxr-xr-x. 1 20415 101 1648 Aug 1 00:35 refresh-namenodes.sh
-rwxr-xr-x. 1 20415 101 2145 Aug 1 00:35 slaves.sh
-rwxr-xr-x. 1 20415 101 1779 Aug 1 00:35 start-all.cmd
-rwxr-xr-x. 1 20415 101 1471 Aug 1 00:35 start-all.sh
-rwxr-xr-x. 1 20415 101 1128 Aug 1 00:35 start-balancer.sh
-rwxr-xr-x. 1 20415 101 1401 Aug 1 00:35 start-dfs.cmd
-rwxr-xr-x. 1 20415 101 3734 Aug 1 00:35 start-dfs.sh
-rwxr-xr-x. 1 20415 101 1357 Aug 1 00:35 start-secure-dns.sh
-rwxr-xr-x. 1 20415 101 1571 Aug 1 00:35 start-yarn.cmd
-rwxr-xr-x. 1 20415 101 1347 Aug 1 00:35 start-yarn.sh
-rwxr-xr-x. 1 20415 101 1770 Aug 1 00:35 stop-all.cmd
-rwxr-xr-x. 1 20415 101 1462 Aug 1 00:35 stop-all.sh
-rwxr-xr-x. 1 20415 101 1179 Aug 1 00:35 stop-balancer.sh
-rwxr-xr-x. 1 20415 101 1455 Aug 1 00:35 stop-dfs.cmd
-rwxr-xr-x. 1 20415 101 3206 Aug 1 00:35 stop-dfs.sh
-rwxr-xr-x. 1 20415 101 1340 Aug 1 00:35 stop-secure-dns.sh
-rwxr-xr-x. 1 20415 101 1642 Aug 1 00:35 stop-yarn.cmd
-rwxr-xr-x. 1 20415 101 1340 Aug 1 00:35 stop-yarn.sh
-rwxr-xr-x. 1 20415 101 4295 Aug 1 00:35 yarn-daemon.sh
-rwxr-xr-x. 1 20415 101 1353 Aug 1 00:35 yarn-daemons.sh
主要使用HDFS蛤育，啟動start-dfs.sh即可
[root@cluster-master sbin]# ./start-dfs.sh
Starting namenodes on [cluster-master]
cluster-master: starting namenode, logging to /opt/hadoop-2.7.4/logs/hadoop-root-namenode-cluster-master.out
cluster-slave1: starting datanode, logging to /opt/hadoop-2.7.4/logs/hadoop-root-datanode-cluster-slave1.out
cluster-slave3: starting datanode, logging to /opt/hadoop-2.7.4/logs/hadoop-root-datanode-cluster-slave3.out
cluster-slave2: starting datanode, logging to /opt/hadoop-2.7.4/logs/hadoop-root-datanode-cluster-slave2.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /opt/hadoop-2.7.4/logs/hadoop-root-secondarynamenode-cluster-master.out
啟動完成之后會在master和slave看到以下進(jìn)程
cluster-master
[root@cluster-master sbin]# jps
2484 SecondaryNameNode
2648 Jps
2315 NameNode
cluster-slave
[root@cluster-slave1 logs]# jps
27502 DataNode
27583 Jps
體驗hadoop
在master端上傳文件試試
[root@cluster-master sbin]# hadoop dfs -put start-dfs.sh /
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
[root@cluster-master sbin]# hadoop dfs -ls /
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
Found 1 items
-rw-r--r-- 3 root supergroup 3734 2017-08-26 10:57 /start-dfs.sh
總結(jié)
docker下部署hadoop確實比較奇葩宛官，但是對于個人學(xué)習(xí)來說確是十分簡便，不用在虛擬機(jī)里復(fù)制多套環(huán)境瓦糕，也簡化了網(wǎng)絡(luò)配置的步驟底洗，加上ansible的使用使得集群的管理更加高效。
