一：環(huán)境準(zhǔn)備：
- 首先要有一個Centos7操作系統(tǒng)查蓉，可以在虛擬機(jī)中安裝晒屎。
- 在centos7中安裝docker甫恩，docker的版本為1.8.2
安裝步驟如下：
<1>安裝制定版本的dockeryum install -y docker-1.8.2-10.el7.centos
<2>安裝的時候可能會報錯钾菊，需要安裝這個依賴
rpm -e lvm2-7:2.02.105-14.el7.x86_64
啟動docker：
root@gavin:/zhangkun/docker# service docker start
- 關(guān)閉防火墻并重啟系統(tǒng)：
root@gavin:/zhangkun/docker# systemctl stop firewalld
root@gavin:/zhangkun/docker# systemctl disable firewalld
注意：執(zhí)行完上面的命令之后需要重啟系統(tǒng)
reboot -h(重啟系統(tǒng))
- 關(guān)閉selinux
1：setenforce 0(立刻生效羞反，不需要重啟操作系統(tǒng))
2：修改/etc/selinux/config文件中的SELINUX=disabled ，然后重啟系統(tǒng)生效
建議兩個步驟都執(zhí)行腿倚，這樣可以確保系統(tǒng)重啟之后selinux也是關(guān)閉狀態(tài)
二：構(gòu)建一個hadoop的基礎(chǔ)鏡像纯出，使用dockerfile文件方式進(jìn)行構(gòu)建
- 先構(gòu)建一個具備ssh功能的鏡像，方便后期使用
- 創(chuàng)建目錄和Dockerfile
root@gavin:/zhangkun/docker# mkdir centos-ssh-root
root@gavin:/zhangkun/docker# cd centos-ssh-root
root@gavin:/zhangkun/docker/centos-ssh-root# vim Dockerfile
Dockerfile內(nèi)容如下：
# 選擇一個已有的os鏡像作為基礎(chǔ)
FROM centos
# 鏡像的作者
MAINTAINER gavin
# 安裝openssh-server和sudo軟件包敷燎，并且將sshd的UsePAM參數(shù)設(shè)置成no
RUN yum install -y openssh-server sudo
RUN sed -i 's/UsePAM yes/UsePAM no/g' /etc/ssh/sshd_config
#安裝openssh-clients
RUN yum install -y openssh-clients
# 添加測試用戶root暂筝，密碼wjht2014，并且將此用戶添加到sudoers里
RUN echo "root:wjht2014" | chpasswd
RUN echo "root ALL=(ALL) ALL" >> /etc/sudoers
# 下面這兩句比較特殊硬贯，在centos6上必須要有焕襟，否則創(chuàng)建出來的容器sshd不能登錄
RUN ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key
RUN ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key
# 啟動sshd服務(wù)并且暴露22端口
RUN mkdir /var/run/sshd
EXPOSE 22
CMD ["/usr/sbin/sshd", "-D"]
- 執(zhí)行docker build 命令
root@gavin:/zhangkun/docker/centos-ssh-root# docker build -t gavin/centos-ssh-root:1.0 .
結(jié)果如下：
Sending build context to Docker daemon 2.56 kB
Step 1 : FROM centos
---> a8493f5f50ff
Step 2 : MAINTAINER gavin
---> Running in 61eb94a2295e
---> 699e5edbbe6c
Removing intermediate container 61eb94a2295e
Step 3 : RUN yum install -y openssh-server sudo
---> Running in c0b03b62bc6a
Loaded plugins: fastestmirror, ovl
Determining fastest mirrors
* base: mirrors.aliyun.com
* extras: mirrors.aliyun.com
* updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package openssh-server.x86_64 0:7.4p1-13.el7_4 will be installed
--> Processing Dependency: openssh = 7.4p1-13.el7_4 for package: openssh-server-7.4p1-13.el7_4.x86_64
--> Processing Dependency: fipscheck-lib(x86-64) >= 1.3.0 for package: openssh-server-7.4p1-13.el7_4.x86_64
--> Processing Dependency: libcrypto.so.10(OPENSSL_1.0.2)(64bit) for package: openssh-server-7.4p1-13.el7_4.x86_64
--> Processing Dependency: libwrap.so.0()(64bit) for package: openssh-server-7.4p1-13.el7_4.x86_64
--> Processing Dependency: libfipscheck.so.1()(64bit) for package: openssh-server-7.4p1-13.el7_4.x86_64
---> Package sudo.x86_64 0:1.8.19p2-11.el7_4 will be installed
--> Running transaction check
---> Package fipscheck-lib.x86_64 0:1.4.1-6.el7 will be installed
--> Processing Dependency: /usr/bin/fipscheck for package: fipscheck-lib-1.4.1-6.el7.x86_64
---> Package openssh.x86_64 0:7.4p1-13.el7_4 will be installed
---> Package openssl-libs.x86_64 1:1.0.1e-60.el7_3.1 will be updated
---> Package openssl-libs.x86_64 1:1.0.2k-8.el7 will be an update
---> Package tcp_wrappers-libs.x86_64 0:7.6-77.el7 will be installed
--> Running transaction check
---> Package fipscheck.x86_64 0:1.4.1-6.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
openssh-server x86_64 7.4p1-13.el7_4 updates 458 k
sudo x86_64 1.8.19p2-11.el7_4 updates 1.1 M
Installing for dependencies:
fipscheck x86_64 1.4.1-6.el7 base 21 k
fipscheck-lib x86_64 1.4.1-6.el7 base 11 k
openssh x86_64 7.4p1-13.el7_4 updates 509 k
tcp_wrappers-libs x86_64 7.6-77.el7 base 66 k
Updating for dependencies:
openssl-libs x86_64 1:1.0.2k-8.el7 base 1.2 M
Transaction Summary
================================================================================
Install 2 Packages (+4 Dependent packages)
Upgrade ( 1 Dependent package)
Total download size: 3.3 M
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
warning: /var/cache/yum/x86_64/7/base/packages/fipscheck-lib-1.4.1-6.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Public key for fipscheck-lib-1.4.1-6.el7.x86_64.rpm is not installed
Public key for openssh-server-7.4p1-13.el7_4.x86_64.rpm is not installed
--------------------------------------------------------------------------------
Total 1.6 MB/s | 3.3 MB 00:02
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Importing GPG key 0xF4A80EB5:
Userid : "CentOS-7 Key (CentOS 7 Official Signing Key) <security@centos.org>"
Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
Package : centos-release-7-3.1611.el7.centos.x86_64 (@CentOS)
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Updating : 1:openssl-libs-1.0.2k-8.el7.x86_64 1/8
Installing : fipscheck-1.4.1-6.el7.x86_64 2/8
Installing : fipscheck-lib-1.4.1-6.el7.x86_64 3/8
Installing : openssh-7.4p1-13.el7_4.x86_64 4/8
Installing : tcp_wrappers-libs-7.6-77.el7.x86_64 5/8
Installing : openssh-server-7.4p1-13.el7_4.x86_64 6/8
Installing : sudo-1.8.19p2-11.el7_4.x86_64 7/8
Cleanup : 1:openssl-libs-1.0.1e-60.el7_3.1.x86_64 8/8
Verifying : fipscheck-lib-1.4.1-6.el7.x86_64 1/8
Verifying : tcp_wrappers-libs-7.6-77.el7.x86_64 2/8
Verifying : fipscheck-1.4.1-6.el7.x86_64 3/8
Verifying : openssh-7.4p1-13.el7_4.x86_64 4/8
Verifying : openssh-server-7.4p1-13.el7_4.x86_64 5/8
Verifying : 1:openssl-libs-1.0.2k-8.el7.x86_64 6/8
Verifying : sudo-1.8.19p2-11.el7_4.x86_64 7/8
Verifying : 1:openssl-libs-1.0.1e-60.el7_3.1.x86_64 8/8
Installed:
openssh-server.x86_64 0:7.4p1-13.el7_4 sudo.x86_64 0:1.8.19p2-11.el7_4
Dependency Installed:
fipscheck.x86_64 0:1.4.1-6.el7 fipscheck-lib.x86_64 0:1.4.1-6.el7
openssh.x86_64 0:7.4p1-13.el7_4 tcp_wrappers-libs.x86_64 0:7.6-77.el7
Dependency Updated:
openssl-libs.x86_64 1:1.0.2k-8.el7
Complete!
---> c996aae83415
Removing intermediate container c0b03b62bc6a
Step 4 : RUN sed -i 's/UsePAM yes/UsePAM no/g' /etc/ssh/sshd_config
---> Running in 1dfac6d4e6c4
---> 2659da5a35a8
Removing intermediate container 1dfac6d4e6c4
Step 5 : RUN yum install -y openssh-clients
---> Running in aad53bc05a7b
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
* base: mirrors.aliyun.com
* extras: mirrors.aliyun.com
* updates: mirrors.aliyun.com
Resolving Dependencies
--> Running transaction check
---> Package openssh-clients.x86_64 0:7.4p1-13.el7_4 will be installed
--> Processing Dependency: libedit.so.0()(64bit) for package: openssh-clients-7.4p1-13.el7_4.x86_64
--> Running transaction check
---> Package libedit.x86_64 0:3.0-12.20121213cvs.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
openssh-clients x86_64 7.4p1-13.el7_4 updates 654 k
Installing for dependencies:
libedit x86_64 3.0-12.20121213cvs.el7 base 92 k
Transaction Summary
================================================================================
Install 1 Package (+1 Dependent package)
Total download size: 746 k
Installed size: 2.8 M
Downloading packages:
--------------------------------------------------------------------------------
Total 2.0 MB/s | 746 kB 00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : libedit-3.0-12.20121213cvs.el7.x86_64 1/2
Installing : openssh-clients-7.4p1-13.el7_4.x86_64 2/2
Verifying : libedit-3.0-12.20121213cvs.el7.x86_64 1/2
Verifying : openssh-clients-7.4p1-13.el7_4.x86_64 2/2
Installed:
openssh-clients.x86_64 0:7.4p1-13.el7_4
Dependency Installed:
libedit.x86_64 0:3.0-12.20121213cvs.el7
Complete!
---> 1fd66ba4a305
Removing intermediate container aad53bc05a7b
Step 6 : RUN echo "root:wjht2014" | chpasswd
---> Running in f1386389be9d
---> 11efd1c6468e
Removing intermediate container f1386389be9d
Step 7 : RUN echo "root ALL=(ALL) ALL" >> /etc/sudoers
---> Running in f2fa0bf8f0ff
---> 66f1dbf23fcb
Removing intermediate container f2fa0bf8f0ff
Step 8 : RUN ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key
---> Running in f200ca8c6f74
Enter passphrase (empty for no passphrase): Enter same passphrase again: Generating public/private dsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_dsa_key.
Your public key has been saved in /etc/ssh/ssh_host_dsa_key.pub.
The key fingerprint is:
SHA256:JXnM8q/diLBuHm4imfhwLXuI1LijSKSkPfAXWwf5V+E root@f85c553c1496
The key's randomart image is:
+---[DSA 1024]----+
| . |
| . + . . |
| o + = E |
| o * . |
|.o o. . S o |
|=+o .= . . . |
|oo=+=+. o . |
|o ===oo.o+ + o |
|o. ooo *= o o . |
+----[SHA256]-----+
---> e2e0212a4a62
Removing intermediate container f200ca8c6f74
Step 9 : RUN ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key
---> Running in 13e6e8697fae
Enter passphrase (empty for no passphrase): Enter same passphrase again: Generating public/private rsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_rsa_key.
Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub.
The key fingerprint is:
SHA256:CP3IFXtxF21OCa3DQ8pqXCT/P0t4laSMnFhlylIBbMY root@f85c553c1496
The key's randomart image is:
+---[RSA 2048]----+
| +.ooo+=o.|
| . Eo++o o+|
| . . =.*++ .= |
| o = .==+=o o|
| + S.o+.oo..|
| + .. .|
| . ..o |
| oo |
| .o|
+----[SHA256]-----+
---> 62506c433eda
Removing intermediate container 13e6e8697fae
Step 10 : RUN mkdir /var/run/sshd
---> Running in 3db5261c4757
---> cec11f2ba2d8
Removing intermediate container 3db5261c4757
Step 11 : EXPOSE 22
---> Running in bd793050ef89
---> d8afedd8edc3
Removing intermediate container bd793050ef89
Step 12 : CMD /usr/sbin/sshd -D
---> Running in c44d33719c82
---> 7fe06a0a3cf2
Removing intermediate container c44d33719c82
Successfully built 7fe06a0a3cf2
看到Successfully built 即構(gòu)建成功
- 基于這個鏡像再構(gòu)建一個帶有jdk的鏡像
- 創(chuàng)建目錄和Dockerfile
root@gavin:/zhangkun/docker# mkdir centos-ssh-root-jdk
root@gavin:/zhangkun/docker# cd centos-ssh-root-jdk
root@gavin:/zhangkun/docker/centos-ssh-root-jdk#vim Dockerfile
Dockerfile內(nèi)容如下：
FROM gavin/centos-ssh-root:v1.0
ADD jdk-8u161-linux-x64.tar.gz /usr/local/
RUN mv /usr/local/jdk1.8.0_161 /usr/local/jdk1.8
ENV JAVA_HOME /usr/local/jdk1.8
ENV PATH $JAVA_HOME/bin:$PATH
下載jdk-8u161-linux-x64.tar.gz
鏈接: https://pan.baidu.com/s/1kWHkmIN 密碼: dt56將下載的jdk放到centos-ssh-root-jdk目錄下
root@gavin:/zhangkun/docker/centos-ssh-root-jdk#sudo cp /home/gavin/下載/jdk-8u161-linux-x64.tar.gz ./
- 構(gòu)建鏡像：
root@gavin:/zhangkun/docker/centos-ssh-root-jdk# docker build -t gavin/centos-ssh-root-jdk:v1.0 .
結(jié)果如下：
Sending build context to Docker daemon 189.8 MB
Step 1 : FROM gavin/centos-ssh-root:v1.0
---> 7fe06a0a3cf2
Step 2 : ADD jdk-8u161-linux-x64.tar.gz /usr/local/
---> dbd69ff9138e
Removing intermediate container 349dc08eeb98
Step 3 : RUN mv /usr/local/jdk1.8.0_161 /usr/local/jdk1.8
---> Running in 749f2cd8d1eb
---> 1b3da34e1112
Removing intermediate container 749f2cd8d1eb
Step 4 : ENV JAVA_HOME /usr/local/jdk1.8
---> Running in 3ca9941d839e
---> 570fe947048e
Removing intermediate container 3ca9941d839e
Step 5 : ENV PATH $JAVA_HOME/bin:$PATH
---> Running in dac0d7608397
---> bfa7c2cb375e
Removing intermediate container dac0d7608397
Successfully built bfa7c2cb375e
- 基于這個jdk鏡像再構(gòu)建一個帶有hadoop的鏡像
- 創(chuàng)建目錄和Dockerfile
root@gavin:/zhangkun/docker# mkdir centos-ssh-root-jdk-hadoop
root@gavin:/zhangkun/docker# cd centos-ssh-root-jdk-hadoop/
root@gavin:/zhangkun/docker/centos-ssh-root-jdk-hadoop# vim Dockerfile
Dockerfile內(nèi)容如下：
FROM gavin/centos-ssh-root-jdk:v1.0
ADD hadoop-2.4.1.tar.gz /usr/local
RUN mv /usr/local/hadoop-2.4.1 /usr/local/hadoop
ENV HADOOP_HOME /usr/local/hadoop
ENV PATH $HADOOP_HOME/bin:$PATH
下載hadoop-2.4.1.tar.gz
鏈接: https://pan.baidu.com/s/1o9fkxRG 密碼: t8hb構(gòu)建鏡像
root@gavin:/zhangkun/docker/centos-ssh-root-jdk-hadoop# docker build -t gavin/gavin/centos-ssh-root-jdk-hadoop:v1.0 .
結(jié)果如下：
Sending build context to Docker daemon 138.7 MB
Step 1 : FROM gavin/centos-ssh-root-jdk:v1.0
---> bfa7c2cb375e
Step 2 : ADD hadoop-2.4.1.tar.gz /usr/local
---> d7615a5d6599
Removing intermediate container 85de43d7f162
Step 3 : RUN mv /usr/local/hadoop-2.4.1 /usr/local/hadoop
---> Running in 889d669375bd
---> ef463ffd5e70
Removing intermediate container 889d669375bd
Step 4 : ENV HADOOP_HOME /usr/local/hadoop
---> Running in 34aeb57db70d
---> f24fdd8f7e2d
Removing intermediate container 34aeb57db70d
Step 5 : ENV PATH $HADOOP_HOME/bin:$PATH
---> Running in 5bee04e663b4
---> 556d46804532
Removing intermediate container 5bee04e663b4
Successfully built 556d46804532
構(gòu)建的三個鏡像如下：
root@gavin:/zhangkun/docker/centos-ssh-root-jdk-hadoop# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
gavin/gavin/centos-ssh-root-jdk-hadoop v1.0 556d46804532 3 hours ago 1.631 GB
gavin/centos-ssh-root-jdk v1.0 bfa7c2cb375e 4 hours ago 1.162 GB
gavin/centos-ssh-root v1.0 7fe06a0a3cf2 4 hours ago 393 MB
三：搭建hadoop分布式集群
- 集群規(guī)劃
準(zhǔn)備搭建一個具有三個節(jié)點(diǎn)的集群，一主兩從
主節(jié)點(diǎn)：hadoop0 ip：192.168.10.3(5022)
從節(jié)點(diǎn)1：hadoop1 ip：192.168.10.4 (3022)
從節(jié)點(diǎn)2：hadoop2 ip：192.168.10.5(4022)
但是由于docker容器重新啟動之后ip會發(fā)生變化饭豹，所以需要我們給docker設(shè)置固定ip鸵赖。使用pipework給 docker容器設(shè)置固定ip
- 啟動三個容器，分別作為hadoop0 hadoop1 hadoop2
在宿主機(jī)上執(zhí)行下面命令拄衰，給容器設(shè)置主機(jī)名和容器的名稱它褪，并且在hadoop0中對外開放端口50070 和8088
docker run --name hadoop0 --hostname hadoop0 -d -P -p 50070:50070 -p 8088:8088 crxy/centos-ssh-root-jdk-hadoop
docker run --name hadoop1 --hostname hadoop1 -d -P crxy/centos-ssh-root-jdk-hadoop
docker run --name hadoop2 --hostname hadoop2 -d -P crxy/centos-ssh-root-jdk-hadoop
- 給這三臺容器設(shè)置固定IP
- 下載PipeWoke
root@gavin:/zhangkun/docker#git clone https://github.com/jpetazzo/pipework
root@gavin:/zhangkun/docker#cp pipework/pipework /usr/local/bin/
- 安裝網(wǎng)橋工具
root@gavin:/zhangkun/docker#yum -y install bridge-utils
- 創(chuàng)建網(wǎng)絡(luò)
root@gavin:/zhangkun/docker#brctl addbr br0
root@gavin:/zhangkun/docker#ip link set dev br0 up
root@gavin:/zhangkun/docker#ip addr add 192.168.10.1/24 dev br0
- 給容器設(shè)置固定ip
root@gavin:/zhangkun/docker#pipework br0 hadoop0 192.168.10.3/24
root@gavin:/zhangkun/docker#pipework br0 hadoop1 192.168.10.4/24
root@gavin:/zhangkun/docker#pipework br0 hadoop2 192.168.10.5/24
- 配置hadoop集群
- 先連接到hadoop0上，
root@gavin:/zhangkun/docker#docker exec -it hadoop0 /bin/bash
下面的步驟就是hadoop集群的配置過程
- 設(shè)置主機(jī)名與ip的映射翘悉，修改三臺容器：vi /etc/hosts
添加下面配置
192.168.2.10 hadoop0
192.168.2.11 hadoop1
192.168.2.12 hadoop2
- 設(shè)置ssh免密碼登錄
在hadoop0上執(zhí)行下面操作
cd ~
mkdir .ssh
cd .ssh
ssh-keygen -t rsa(一直按回車即可)
ssh-copy-id -i localhost
ssh-copy-id -i hadoop0
ssh-copy-id -i hadoop1
ssh-copy-id -i hadoop2
在hadoop1上執(zhí)行下面操作
cd ~
cd .ssh
ssh-keygen -t rsa(一直按回車即可)
ssh-copy-id -i localhost
ssh-copy-id -i hadoop1
在hadoop2上執(zhí)行下面操作
cd ~
cd .ssh
ssh-keygen -t rsa(一直按回車即可)
ssh-copy-id -i localhost
ssh-copy-id -i hadoop2
- 在hadoop0上修改hadoop的配置文件
進(jìn)入到/usr/local/hadoop/etc/hadoop目錄
修改目錄下的配置文件core-site.xml茫打、hdfs-site.xml、yarn-site.xml妖混、mapred-site.xml
(1) hadoop-env.sh
export JAVA_HOME=/usr/local/jdk1.8
(2)core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop0:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop/tmp</value>
</property>
<property>
<name>fs.trash.interval</name>
<value>1440</value>
</property>
</configuration>
(3)hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
</configuration>
(4)yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
</configuration>
(5)修改文件名：mv mapred-site.xml.template mapred-site.xml
vi mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
(6)格式化
進(jìn)入到/usr/local/hadoop目錄下
- 執(zhí)行格式化命令
bin/hdfs namenode -format
注意：在執(zhí)行的時候會報錯包吝，是因為缺少which命令，安裝即可
執(zhí)行下面命令安裝
yum install -y which
看到下面命令說明格式化成功源葫。
這里寫圖片描述
格式化操作不能重復(fù)執(zhí)行诗越。如果一定要重復(fù)格式化，帶參數(shù)-force即可息堂。
(7)啟動偽分布hadoop
命令：sbin/start-all.sh
第一次啟動的過程中需要輸入yes確認(rèn)一下嚷狞。
這里寫圖片描述
使用jps块促，檢查進(jìn)程是否正常啟動？能看到下面幾個進(jìn)程表示偽分布啟動成功
[root@hadoop0 hadoop]# jps
818 ResourceManager
1239 Jps
522 DataNode
411 NameNode
669 SecondaryNameNode
909 NodeManager
(8)停止偽分布hadoop
命令：sbin/stop-all.sh
(9)指定nodemanager的地址床未，修改文件yarn-site.xml
<property>
<description>The hostname of the RM.</description>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop0</value>
</property>
(10)修改hadoop0中hadoop的一個配置文件etc/hadoop/slaves
刪除原來的所有內(nèi)容竭翠，修改為如下
hadoop1
hadoop2
(11)在hadoop0中執(zhí)行命令
scp -rq /usr/local/hadoop hadoop1:/usr/local
scp -rq /usr/local/hadoop hadoop2:/usr/local
(12)啟動hadoop分布式集群服務(wù)
執(zhí)行sbin/start-all.sh
注意：在執(zhí)行的時候會報錯，是因為兩個從節(jié)點(diǎn)缺少which命令薇搁，安裝即可
分別在兩個從節(jié)點(diǎn)執(zhí)行下面命令安裝
yum install -y which
1
再啟動集群(如果集群已啟動斋扰，需要先停止)
sbin/start-all.sh
(13)驗證集群是否正常
首先查看進(jìn)程：
Hadoop0上需要有這幾個進(jìn)程
[root@hadoop0 hadoop]# jps
4643 Jps
4073 NameNode
4216 SecondaryNameNode
4381 ResourceManager
Hadoop1上需要有這幾個進(jìn)程
[root@hadoop1 hadoop]# jps
715 NodeManager
849 Jps
645 DataNode
Hadoop2上需要有這幾個進(jìn)程
[root@hadoop2 hadoop]# jps
456 NodeManager
589 Jps
388 DataNode
使用程序驗證集群服務(wù)
創(chuàng)建一個本地文件
vi a.txt
hello you
hello me
上傳a.txt到hdfs上
hdfs dfs -put a.txt /
執(zhí)行wordcount程序
cd /usr/local/hadoop/share/hadoop/mapreduce
hadoop jar hadoop-mapreduce-examples-2.4.1.jar wordcount /a.txt /out
查看程序執(zhí)行結(jié)果
這里寫圖片描述
這樣就說明集群正常了。
在這可以直接通過宿主機(jī)訪問容器中hadoop集群的服務(wù)
宿主機(jī)的ip為：192.168.1.27
http://192.168.1.27:50070/
http://192.168.1.27:8088/
四：集群節(jié)點(diǎn)重啟
停止三個容器啃洋，在宿主機(jī)上執(zhí)行下面命令
docker stop hadoop0
docker stop hadoop1
docker stop hadoop2
容器停止之后传货，之前設(shè)置的固定ip也會消失，重新再使用這幾個容器的時候還需要重新設(shè)置固定ip
先把之前停止的三個容器起來
docker start hadoop0
docker start hadoop1
docker start hadoop2
在宿主機(jī)上執(zhí)行下面命令重新給容器設(shè)置固定ip
pipework br0 hadoop0 192.168.2.10/24
pipework br0 hadoop1 192.168.2.11/24
pipework br0 hadoop2 192.168.2.12/24
還需要重新在容器中配置主機(jī)名和ip的映射關(guān)系宏娄，每次都手工寫比較麻煩
寫一個腳本问裕，runhosts.sh
#!/bin/bash
echo 192.168.2.10 hadoop0 >> /etc/hosts
echo 192.168.2.11 hadoop1 >> /etc/hosts
echo 192.168.2.12 hadoop2 >> /etc/hosts
添加執(zhí)行權(quán)限，chmod +x runhosts.sh
把這個腳本拷貝到所有節(jié)點(diǎn)孵坚，并且分別執(zhí)行這個腳本
scp runhosts.sh hadoop1:~
scp runhosts.sh hadoop2:~
執(zhí)行腳本的命令 ./runhosts.sh
查看/etc/hosts文件中是否添加成功
這里寫圖片描述
注意：有一些docker版本中不會在hosts文件中自動生成下面這些映射粮宛，所以我們才在這里手工給容器設(shè)置固定ip，并設(shè)置主機(jī)名和ip的映射關(guān)系卖宠。
172.17.0.25 hadoop0
172.17.0.25 hadoop0.bridge
172.17.0.26 hadoop1
172.17.0.26 hadoop1.bridge
172.17.0.27 hadoop2
172.17.0.27 hadoop2.bridge
啟動hadoop集群
sbin/start-all.sh
