11、用Docker構(gòu)建Hadoop分布式集群

后文用到的配置文件可以從https://github.com/yinbodotcc/-Docker-Hadoop-.git下載

一溜歪、環(huán)境說(shuō)明

1、虛擬4臺(tái)主機(jī)许蓖，IP地址和主機(jī)名如下（以下是dockerfile中表達(dá)方式）
RUN echo "172.18.12.4 hadoop-slave3" >> /etc/hosts
RUN echo "172.18.12.3 hadoop-slave2" >> /etc/hosts
RUN echo "172.18.12.2 hadoop-slave1" >> /etc/hosts
RUN echo "172.18.12.1 hadoop-master" >> /etc/hosts
2蝴猪、不需要單獨(dú)部署Zookeeper

圖片.png

圖片.png

二、Ubuntu基礎(chǔ)鏡像構(gòu)建

2.1膊爪、ubuntubase.dockerfile

############################################
# version : yayubuntubase/withssh:v1
# desc : ubuntu14.04 上安裝的ssh
############################################
# 設(shè)置繼承自u(píng)buntu14.04官方鏡像
FROM ubuntu:14.04 

# 下面是一些創(chuàng)建者的基本信息
MAINTAINER yayubuntubase


RUN echo "root:root" | chpasswd


#安裝ssh-server
RUN rm -rvf /var/lib/apt/lists/*
RUN apt-get update 
RUN apt-get install -y openssh-server openssh-client vim wget curl sudo



#配置ssh
RUN mkdir /var/run/sshd
RUN sed -i 's/PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config
RUN sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd
ENV NOTVISIBLE "in users profile"
RUN echo "export VISIBLE=now" >> /etc/profile

EXPOSE 22

2.2自阱、制作鏡像

sudo docker image build --file ubuntubase.dockerfile --tag ubuntuforhadoop/withssh:v1 .

三、加入hadoop的ubuntu基礎(chǔ)鏡像構(gòu)建

3.1米酬、hadoop.dockerfile

############################################ 
# desc : 安裝JAVA  HADOOP 
############################################
 
FROM ubuntuforhadoop/withssh:v1
 
 
MAINTAINER yay 
  
#為hadoop集群提供dns服務(wù)
USER root
RUN sudo apt-get -y install dnsmasq

#安裝和配置java環(huán)境
#安裝和配置java 
ADD jdk-8u191-linux-x64.tar.gz /usr/local/
ENV JAVA_HOME /usr/local/jdk1.8.0_191
ENV CLASSPATH ${JAVA_HOME}/lib/dt.jar:$JAVA_HOME/lib/tools.jar
ENV PATH $PATH:${JAVA_HOME}/bin


#安裝HADOOP
ADD hadoop-3.2.1.tar.gz /usr/local/
RUN cd /usr/local && ln -s ./hadoop-3.2.1 hadoop

ENV HADOOP_PREFIX /usr/local/hadoop
ENV HADOOP_HOME /usr/local/hadoop
ENV HADOOP_COMMON_HOME /usr/local/hadoop
ENV HADOOP_HDFS_HOME /usr/local/hadoop
ENV HADOOP_MAPRED_HOME /usr/local/hadoop
ENV HADOOP_YARN_HOME /usr/local/hadoop
ENV HADOOP_CONF_DIR /usr/local/hadoop/etc/hadoop
ENV PATH ${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin:$PATH

RUN mkdir -p /home/hadooptmp


RUN echo "hadoop ALL= NOPASSWD: ALL" >> /etc/sudoers



#在每臺(tái)主機(jī)上輸入 ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa 創(chuàng)建一個(gè)無(wú)密碼的公鑰沛豌，-t是類(lèi)型的意思，dsa是生成的密鑰類(lèi)型，-P是密碼，’’表示無(wú)密碼枯跑，-f后是秘鑰生成后保存的位置
RUN ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
RUN cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
CMD ["/usr/sbin/sshd","-D"] 



ADD hadoop-conf/.       /usr/local/hadoop/etc/hadoop/
#ADD start/.             /usr/local/hadoop/sbin/

#在制定行插入
RUN sed -i "1i HADOOP_SECURE_DN_USER=hdfs"  /usr/local/hadoop/sbin/start-dfs.sh
RUN sed -i "1i HDFS_NAMENODE_USER=root"  /usr/local/hadoop/sbin/start-dfs.sh
RUN sed -i "1i HDFS_SECONDARYNAMENODE_USER=root"  /usr/local/hadoop/sbin/start-dfs.sh
RUN sed -i "1i HDFS_DATANODE_USER=root"  /usr/local/hadoop/sbin/start-dfs.sh


RUN sed -i "1i HADOOP_SECURE_DN_USER=hdfs"  /usr/local/hadoop/sbin/stop-dfs.sh
RUN sed -i "1i HDFS_NAMENODE_USER=root"  /usr/local/hadoop/sbin/stop-dfs.sh
RUN sed -i "1i HDFS_SECONDARYNAMENODE_USER=root"  /usr/local/hadoop/sbin/stop-dfs.sh
RUN sed -i "1i HDFS_DATANODE_USER=root"  /usr/local/hadoop/sbin/stop-dfs.sh

RUN sed -i "1i HADOOP_SECURE_DN_USER=yarn"  /usr/local/hadoop/sbin/start-yarn.sh
RUN sed -i "1i YARN_NODEMANAGER_USER=root"  /usr/local/hadoop/sbin/start-yarn.sh
RUN sed -i "1i YARN_RESOURCEMANAGER_USER=root"  /usr/local/hadoop/sbin/start-yarn.sh

RUN sed -i "1i HADOOP_SECURE_DN_USER=yarn"  /usr/local/hadoop/sbin/stop-yarn.sh
RUN sed -i "1i YARN_NODEMANAGER_USER=root"  /usr/local/hadoop/sbin/stop-yarn.sh
RUN sed -i "1i YARN_RESOURCEMANAGER_USER=root"  /usr/local/hadoop/sbin/stop-yarn.sh

3.2龄毡、幾個(gè)配置文件

hadoop-conf/core-site.xml

    <property>
        <name>fs.default.name</name>
        <value>hdfs://hadoop-master:9000</value>
        </property>
    <property>
        <name>io.file.buffer.size</name>
        <value>131072</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/home/hadooptmp</value>
        <description>Abase for other temporary directories.</description>
    </property>
</configuration>

注意，如果你在配置非容器環(huán)境推姻，可能需要如下配置
<property>
<name>hadoop.tmp.dir</name>
<value>/home/${user.name}/hadoop_tmp</value>
<description>A base for other temporary directories.</description>
</property>
即使用${user.name}
否則會(huì)出現(xiàn)錯(cuò)誤：java.io.IOException: Cannot create directory /home/hadoop/dfs/name/current

hadoop-conf/hadoop-env.sh

JAVA_HOME=/usr/local/jdk1.8.0_191

hadoop-conf/hdfs-site.xml

<configuration>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>hadoop-master:9001</value>
        <description># 通過(guò)web界面來(lái)查看HDFS狀態(tài) </description>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/home/hadoop/dfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>/home/hadoop/dfs/data</value>
    </property>
    <property>
        <name>dfs.replication</name>
        <value>3</value>
        <description># 每個(gè)Block有3個(gè)備份</description>
    </property>
    <property>
        <name>dfs.webhdfs.enabled</name>
        <value>true</value>
    </property>
</configuration>

注意，如果你在配置非容器環(huán)境，可能需要如下配置
<property>
<name>dfs.namenode.name.dir</name>
<value>/home/${user.name}/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/${user.name}/dfs/data</value>
</property>
<property>
即使用${user.name}
否則會(huì)出現(xiàn)錯(cuò)誤：java.io.IOException: Cannot create directory /home/hadoop/dfs/name/current

hadoop-conf/mapred-site.xml

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>hadoop-master:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>hadoop-master:19888</value>
    </property>
</configuration>

hadoop-conf/yarn-env.sh

JAVA_HOME=/usr/local/jdk1.8.0_191

hadoop-conf/yarn-site.xml

<configuration>
    <!-- Site specific YARN configuration properties -->
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>hadoop-master:8032</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>hadoop-master:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>hadoop-master:8031</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>hadoop-master:8033</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>hadoop-master:8088</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>1024</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.cpu-vcores</name>
        <value>1</value>
    </property>
</configuration>

hadoop-conf/workers

hadoop-master
hadoop-slave1
hadoop-slave2
hadoop-slave3

務(wù)必注意醉旦，Hadoop3之前叫做slaves，Hadoop3之后才改為workers文件的

start/start-dfs.sh

圖片.png

#!/usr/bin/env bash

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

HDFS_DATANODE_USER=root
HADOOP_SECURE_DN_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root



# Start hadoop dfs daemons.
# Optinally upgrade or rollback dfs state.
# Run this on master node.

## startup matrix:
#
# if $EUID != 0, then exec
# if $EUID =0 then
#    if hdfs_subcmd_user is defined, su to that user, exec
#    if hdfs_subcmd_user is not defined, error
#
# For secure daemons, this means both the secure and insecure env vars need to be
# defined.  e.g., HDFS_DATANODE_USER=root HDFS_DATANODE_SECURE_USER=hdfs
#

## @description  usage info
## @audience     private
## @stability    evolving
## @replaceable  no
function hadoop_usage
{
  echo "Usage: start-dfs.sh [-upgrade|-rollback] [-clusterId]"
}

this="${BASH_SOURCE-$0}"
bin=$(cd -P -- "$(dirname -- "${this}")" >/dev/null && pwd -P)

# let's locate libexec...
if [[ -n "${HADOOP_HOME}" ]]; then
  HADOOP_DEFAULT_LIBEXEC_DIR="${HADOOP_HOME}/libexec"
else
  HADOOP_DEFAULT_LIBEXEC_DIR="${bin}/../libexec"
fi

HADOOP_LIBEXEC_DIR="${HADOOP_LIBEXEC_DIR:-$HADOOP_DEFAULT_LIBEXEC_DIR}"
# shellcheck disable=SC2034
HADOOP_NEW_CONFIG=true
if [[ -f "${HADOOP_LIBEXEC_DIR}/hdfs-config.sh" ]]; then
  . "${HADOOP_LIBEXEC_DIR}/hdfs-config.sh"
else
  echo "ERROR: Cannot execute ${HADOOP_LIBEXEC_DIR}/hdfs-config.sh." 2>&1
  exit 1
fi

# get arguments
if [[ $# -ge 1 ]]; then
  startOpt="$1"
  shift
  case "$startOpt" in
    -upgrade)
      nameStartOpt="$startOpt"
    ;;
    -rollback)
      dataStartOpt="$startOpt"
    ;;
    *)
      hadoop_exit_with_usage 1
    ;;
  esac
fi


#Add other possible options
nameStartOpt="$nameStartOpt $*"

#---------------------------------------------------------
# namenodes

NAMENODES=$("${HADOOP_HDFS_HOME}/bin/hdfs" getconf -namenodes 2>/dev/null)

if [[ -z "${NAMENODES}" ]]; then
  NAMENODES=$(hostname)
fi

echo "Starting namenodes on [${NAMENODES}]"
hadoop_uservar_su hdfs namenode "${HADOOP_HDFS_HOME}/bin/hdfs" \
    --workers \
    --config "${HADOOP_CONF_DIR}" \
    --hostnames "${NAMENODES}" \
    --daemon start \
    namenode ${nameStartOpt}

HADOOP_JUMBO_RETCOUNTER=$?

#---------------------------------------------------------
# datanodes (using default workers file)

echo "Starting datanodes"
hadoop_uservar_su hdfs datanode "${HADOOP_HDFS_HOME}/bin/hdfs" \
    --workers \
    --config "${HADOOP_CONF_DIR}" \
    --daemon start \
    datanode ${dataStartOpt}
(( HADOOP_JUMBO_RETCOUNTER=HADOOP_JUMBO_RETCOUNTER + $? ))

#---------------------------------------------------------
# secondary namenodes (if any)

SECONDARY_NAMENODES=$("${HADOOP_HDFS_HOME}/bin/hdfs" getconf -secondarynamenodes 2>/dev/null)

if [[ -n "${SECONDARY_NAMENODES}" ]]; then

  if [[ "${NAMENODES}" =~ , ]]; then

    hadoop_error "WARNING: Highly available NameNode is configured."
    hadoop_error "WARNING: Skipping SecondaryNameNode."

  else

    if [[ "${SECONDARY_NAMENODES}" == "0.0.0.0" ]]; then
      SECONDARY_NAMENODES=$(hostname)
    fi

    echo "Starting secondary namenodes [${SECONDARY_NAMENODES}]"

    hadoop_uservar_su hdfs secondarynamenode "${HADOOP_HDFS_HOME}/bin/hdfs" \
      --workers \
      --config "${HADOOP_CONF_DIR}" \
      --hostnames "${SECONDARY_NAMENODES}" \
      --daemon start \
      secondarynamenode
    (( HADOOP_JUMBO_RETCOUNTER=HADOOP_JUMBO_RETCOUNTER + $? ))
  fi
fi

#---------------------------------------------------------
# quorumjournal nodes (if any)

JOURNAL_NODES=$("${HADOOP_HDFS_HOME}/bin/hdfs" getconf -journalNodes 2>&-)

if [[ "${#JOURNAL_NODES}" != 0 ]]; then
  echo "Starting journal nodes [${JOURNAL_NODES}]"

  hadoop_uservar_su hdfs journalnode "${HADOOP_HDFS_HOME}/bin/hdfs" \
    --workers \
    --config "${HADOOP_CONF_DIR}" \
    --hostnames "${JOURNAL_NODES}" \
    --daemon start \
    journalnode
   (( HADOOP_JUMBO_RETCOUNTER=HADOOP_JUMBO_RETCOUNTER + $? ))
fi

#---------------------------------------------------------
# ZK Failover controllers, if auto-HA is enabled
AUTOHA_ENABLED=$("${HADOOP_HDFS_HOME}/bin/hdfs" getconf -confKey dfs.ha.automatic-failover.enabled | tr '[:upper:]' '[:lower:]')
if [[ "${AUTOHA_ENABLED}" = "true" ]]; then
  echo "Starting ZK Failover Controllers on NN hosts [${NAMENODES}]"

  hadoop_uservar_su hdfs zkfc "${HADOOP_HDFS_HOME}/bin/hdfs" \
    --workers \
    --config "${HADOOP_CONF_DIR}" \
    --hostnames "${NAMENODES}" \
    --daemon start \
    zkfc
  (( HADOOP_JUMBO_RETCOUNTER=HADOOP_JUMBO_RETCOUNTER + $? ))
fi

exit ${HADOOP_JUMBO_RETCOUNTER}

# eof

start/start-yarn.sh

圖片.png

#!/usr/bin/env bash

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root

## @description  usage info
## @audience     private
## @stability    evolving
## @replaceable  no
function hadoop_usage
{
  hadoop_generate_usage "${MYNAME}" false
}

MYNAME="${BASH_SOURCE-$0}"

bin=$(cd -P -- "$(dirname -- "${MYNAME}")" >/dev/null && pwd -P)

# let's locate libexec...
if [[ -n "${HADOOP_HOME}" ]]; then
  HADOOP_DEFAULT_LIBEXEC_DIR="${HADOOP_HOME}/libexec"
else
  HADOOP_DEFAULT_LIBEXEC_DIR="${bin}/../libexec"
fi

HADOOP_LIBEXEC_DIR="${HADOOP_LIBEXEC_DIR:-$HADOOP_DEFAULT_LIBEXEC_DIR}"
# shellcheck disable=SC2034
HADOOP_NEW_CONFIG=true
if [[ -f "${HADOOP_LIBEXEC_DIR}/yarn-config.sh" ]]; then
  . "${HADOOP_LIBEXEC_DIR}/yarn-config.sh"
else
  echo "ERROR: Cannot execute ${HADOOP_LIBEXEC_DIR}/yarn-config.sh." 2>&1
  exit 1
fi

HADOOP_JUMBO_RETCOUNTER=0

# start resourceManager
HARM=$("${HADOOP_HDFS_HOME}/bin/hdfs" getconf -confKey yarn.resourcemanager.ha.enabled 2>&-)
if [[ ${HARM} = "false" ]]; then
  echo "Starting resourcemanager"
  hadoop_uservar_su yarn resourcemanager "${HADOOP_YARN_HOME}/bin/yarn" \
      --config "${HADOOP_CONF_DIR}" \
      --daemon start \
      resourcemanager
  (( HADOOP_JUMBO_RETCOUNTER=HADOOP_JUMBO_RETCOUNTER + $? ))
else
  logicals=$("${HADOOP_HDFS_HOME}/bin/hdfs" getconf -confKey yarn.resourcemanager.ha.rm-ids 2>&-)
  logicals=${logicals//,/ }
  for id in ${logicals}
  do
      rmhost=$("${HADOOP_HDFS_HOME}/bin/hdfs" getconf -confKey "yarn.resourcemanager.hostname.${id}" 2>&-)
      RMHOSTS="${RMHOSTS} ${rmhost}"
  done
  echo "Starting resourcemanagers on [${RMHOSTS}]"
  hadoop_uservar_su yarn resourcemanager "${HADOOP_YARN_HOME}/bin/yarn" \
      --config "${HADOOP_CONF_DIR}" \
      --daemon start \
      --workers \
      --hostnames "${RMHOSTS}" \
      resourcemanager
  (( HADOOP_JUMBO_RETCOUNTER=HADOOP_JUMBO_RETCOUNTER + $? ))
fi

# start nodemanager
echo "Starting nodemanagers"
hadoop_uservar_su yarn nodemanager "${HADOOP_YARN_HOME}/bin/yarn" \
    --config "${HADOOP_CONF_DIR}" \
    --workers \
    --daemon start \
    nodemanager
(( HADOOP_JUMBO_RETCOUNTER=HADOOP_JUMBO_RETCOUNTER + $? ))


# start proxyserver
PROXYSERVER=$("${HADOOP_HDFS_HOME}/bin/hdfs" getconf -confKey  yarn.web-proxy.address 2>&- | cut -f1 -d:)
if [[ -n ${PROXYSERVER} ]]; then
 hadoop_uservar_su yarn proxyserver "${HADOOP_YARN_HOME}/bin/yarn" \
      --config "${HADOOP_CONF_DIR}" \
      --workers \
      --hostnames "${PROXYSERVER}" \
      --daemon start \
      proxyserver
 (( HADOOP_JUMBO_RETCOUNTER=HADOOP_JUMBO_RETCOUNTER + $? ))
fi

exit ${HADOOP_JUMBO_RETCOUNTER}

start/stop-dfs.sh

圖片.png

#!/usr/bin/env bash

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.


HDFS_DATANODE_USER=root
HADOOP_SECURE_DN_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root


# Stop hadoop dfs daemons.
# Run this on master node.

## @description  usage info
## @audience     private
## @stability    evolving
## @replaceable  no
function hadoop_usage
{
  echo "Usage: stop-dfs.sh"
}

this="${BASH_SOURCE-$0}"
bin=$(cd -P -- "$(dirname -- "${this}")" >/dev/null && pwd -P)

# let's locate libexec...
if [[ -n "${HADOOP_HOME}" ]]; then
  HADOOP_DEFAULT_LIBEXEC_DIR="${HADOOP_HOME}/libexec"
else
  HADOOP_DEFAULT_LIBEXEC_DIR="${bin}/../libexec"
fi

HADOOP_LIBEXEC_DIR="${HADOOP_LIBEXEC_DIR:-$HADOOP_DEFAULT_LIBEXEC_DIR}"
# shellcheck disable=SC2034
HADOOP_NEW_CONFIG=true
if [[ -f "${HADOOP_LIBEXEC_DIR}/hdfs-config.sh" ]]; then
  . "${HADOOP_LIBEXEC_DIR}/hdfs-config.sh"
else
  echo "ERROR: Cannot execute ${HADOOP_LIBEXEC_DIR}/hdfs-config.sh." 2>&1
  exit 1
fi

#---------------------------------------------------------
# namenodes

NAMENODES=$("${HADOOP_HDFS_HOME}/bin/hdfs" getconf -namenodes 2>/dev/null)

if [[ -z "${NAMENODES}" ]]; then
  NAMENODES=$(hostname)
fi

echo "Stopping namenodes on [${NAMENODES}]"

  hadoop_uservar_su hdfs namenode "${HADOOP_HDFS_HOME}/bin/hdfs" \
    --workers \
    --config "${HADOOP_CONF_DIR}" \
    --hostnames "${NAMENODES}" \
    --daemon stop \
    namenode

#---------------------------------------------------------
# datanodes (using default workers file)

echo "Stopping datanodes"

hadoop_uservar_su hdfs datanode "${HADOOP_HDFS_HOME}/bin/hdfs" \
  --workers \
  --config "${HADOOP_CONF_DIR}" \
  --daemon stop \
  datanode

#---------------------------------------------------------
# secondary namenodes (if any)

SECONDARY_NAMENODES=$("${HADOOP_HDFS_HOME}/bin/hdfs" getconf -secondarynamenodes 2>/dev/null)

if [[ "${SECONDARY_NAMENODES}" == "0.0.0.0" ]]; then
  SECONDARY_NAMENODES=$(hostname)
fi

if [[ -n "${SECONDARY_NAMENODES}" ]]; then
  echo "Stopping secondary namenodes [${SECONDARY_NAMENODES}]"

  hadoop_uservar_su hdfs secondarynamenode "${HADOOP_HDFS_HOME}/bin/hdfs" \
    --workers \
    --config "${HADOOP_CONF_DIR}" \
    --hostnames "${SECONDARY_NAMENODES}" \
    --daemon stop \
    secondarynamenode
fi

#---------------------------------------------------------
# quorumjournal nodes (if any)

JOURNAL_NODES=$("${HADOOP_HDFS_HOME}/bin/hdfs" getconf -journalNodes 2>&-)

if [[ "${#JOURNAL_NODES}" != 0 ]]; then
  echo "Stopping journal nodes [${JOURNAL_NODES}]"

  hadoop_uservar_su hdfs journalnode "${HADOOP_HDFS_HOME}/bin/hdfs" \
    --workers \
    --config "${HADOOP_CONF_DIR}" \
    --hostnames "${JOURNAL_NODES}" \
    --daemon stop \
    journalnode
fi

#---------------------------------------------------------
# ZK Failover controllers, if auto-HA is enabled
AUTOHA_ENABLED=$("${HADOOP_HDFS_HOME}/bin/hdfs" getconf -confKey dfs.ha.automatic-failover.enabled | tr '[:upper:]' '[:lower:]')
if [[ "${AUTOHA_ENABLED}" = "true" ]]; then
  echo "Stopping ZK Failover Controllers on NN hosts [${NAMENODES}]"

  hadoop_uservar_su hdfs zkfc "${HADOOP_HDFS_HOME}/bin/hdfs" \
    --workers \
    --config "${HADOOP_CONF_DIR}" \
    --hostnames "${NAMENODES}" \
    --daemon stop \
    zkfc
fi



# eof

start/stop-yarn.sh

圖片.png

#!/usr/bin/env bash

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root

## @description  usage info
## @audience     private
## @stability    evolving
## @replaceable  no
function hadoop_usage
{
  hadoop_generate_usage "${MYNAME}" false
}

MYNAME="${BASH_SOURCE-$0}"

bin=$(cd -P -- "$(dirname -- "${MYNAME}")" >/dev/null && pwd -P)

# let's locate libexec...
if [[ -n "${HADOOP_HOME}" ]]; then
  HADOOP_DEFAULT_LIBEXEC_DIR="${HADOOP_HOME}/libexec"
else
  HADOOP_DEFAULT_LIBEXEC_DIR="${bin}/../libexec"
fi

HADOOP_LIBEXEC_DIR="${HADOOP_LIBEXEC_DIR:-$HADOOP_DEFAULT_LIBEXEC_DIR}"
# shellcheck disable=SC2034
HADOOP_NEW_CONFIG=true
if [[ -f "${HADOOP_LIBEXEC_DIR}/yarn-config.sh" ]]; then
  . "${HADOOP_LIBEXEC_DIR}/yarn-config.sh"
else
  echo "ERROR: Cannot execute ${HADOOP_LIBEXEC_DIR}/yarn-config.sh." 2>&1
  exit 1
fi

# stop nodemanager
echo "Stopping nodemanagers"
hadoop_uservar_su yarn nodemanager "${HADOOP_YARN_HOME}/bin/yarn" \
    --config "${HADOOP_CONF_DIR}" \
    --workers \
    --daemon stop \
    nodemanager

# stop resourceManager
HARM=$("${HADOOP_HDFS_HOME}/bin/hdfs" getconf -confKey yarn.resourcemanager.ha.enabled 2>&-)
if [[ ${HARM} = "false" ]]; then
  echo "Stopping resourcemanager"
  hadoop_uservar_su yarn resourcemanager "${HADOOP_YARN_HOME}/bin/yarn" \
      --config "${HADOOP_CONF_DIR}" \
      --daemon stop \
      resourcemanager
else
  logicals=$("${HADOOP_HDFS_HOME}/bin/hdfs" getconf -confKey yarn.resourcemanager.ha.rm-ids 2>&-)
  logicals=${logicals//,/ }
  for id in ${logicals}
  do
      rmhost=$("${HADOOP_HDFS_HOME}/bin/hdfs" getconf -confKey "yarn.resourcemanager.hostname.${id}" 2>&-)
      RMHOSTS="${RMHOSTS} ${rmhost}"
  done
  echo "Stopping resourcemanagers on [${RMHOSTS}]"
  hadoop_uservar_su yarn resourcemanager "${HADOOP_YARN_HOME}/bin/yarn" \
      --config "${HADOOP_CONF_DIR}" \
      --daemon stop \
      --workers \
      --hostnames "${RMHOSTS}" \
      resourcemanager
fi

# stop proxyserver
PROXYSERVER=$("${HADOOP_HDFS_HOME}/bin/hdfs" getconf -confKey  yarn.web-proxy.address 2>&- | cut -f1 -d:)
if [[ -n ${PROXYSERVER} ]]; then
  echo "Stopping proxy server [${PROXYSERVER}]"
  hadoop_uservar_su yarn proxyserver "${HADOOP_YARN_HOME}/bin/yarn" \
      --config "${HADOOP_CONF_DIR}" \
      --workers \
      --hostnames "${PROXYSERVER}" \
      --daemon stop \
      proxyserver
fi
YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root

3.3桨啃、制作鏡像

sudo docker image build --file hadoop.dockerfile --tag hadoop/nozookeeper:v1 .

四车胡、啟動(dòng)容器

4.1、設(shè)置一個(gè)network

設(shè)置一個(gè)network的目的是為了在啟動(dòng)容器的時(shí)候就能指定容器的IP地址

yay@yay-ThinkPad-T470:~/software/dockerfileForHBase$ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
c02cc96fb47c        bridge              bridge              local
827bcf1293c0        host                host                local
2d8cd675265a        none                null                local
yay@yay-ThinkPad-T470:~/software/dockerfileForHBase$ docker network create --driver bridge --subnet=172.18.12.0/16 --gateway=172.18.1.1 mynet
...
yay@yay-ThinkPad-T470:~/software/dockerfileForHBase$ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
c02cc96fb47c        bridge              bridge              local
827bcf1293c0        host                host                local
8fccbdc49d54        mynet               bridge              local
2d8cd675265a        none                null                local

4.2照瘾、啟動(dòng)hadoop-mast和hadoop-slave1匈棘、hadoop-slave2、hadoop-slave3

注意啟動(dòng)的時(shí)候 指定容器名析命、主機(jī)名主卫、網(wǎng)絡(luò)地址逃默、配置/etc/hosts

docker run -it --name master -h hadoop-master --network=mynet --ip 172.18.12.1 --add-host hadoop-slave1:172.18.12.2 --add-host hadoop-slave2:172.18.12.3 --add-host hadoop-slave3:172.18.12.4 -d -P -p 2020:22 -p 9870:9870 -p 8088:8088 hadoop/nozookeeper:v1

docker run -it --name slave1 -h hadoop-slave1 --network=mynet --ip 172.18.12.2  --add-host hadoop-master:172.18.12.1 --add-host hadoop-slave2:172.18.12.3 --add-host hadoop-slave3:172.18.12.4 -d -P -p 2021:22 hadoop/nozookeeper:v1

docker run -it --name slave2 -h hadoop-slave2 --network=mynet --ip 172.18.12.3  --add-host hadoop-master:172.18.12.1 --add-host hadoop-slave1:172.18.12.2 --add-host hadoop-slave3:172.18.12.4 -d -P -p 2022:22 hadoop/nozookeeper:v1

docker run -it --name slave3 -h hadoop-slave3 --network=mynet --ip 172.18.12.4  --add-host hadoop-master:172.18.12.1 --add-host hadoop-slave1:172.18.12.2 --add-host hadoop-slave2:172.18.12.3  -d -P -p 2023:22 hadoop/nozookeeper:v1

也可用&&把它們拼接在一起：

docker run -it --name master -h hadoop-master --network=mynet --ip 172.18.12.1 --add-host hadoop-slave1:172.18.12.2 --add-host hadoop-slave2:172.18.12.3 --add-host hadoop-slave3:172.18.12.4 -d -P -p 2020:22 -p 9870:9870 -p 8088:8088 hadoop/nozookeeper:v1 && docker run -it --name slave1 -h hadoop-slave1 --network=mynet --ip 172.18.12.2  --add-host hadoop-master:172.18.12.1 --add-host hadoop-slave2:172.18.12.3 --add-host hadoop-slave3:172.18.12.4 -d -P -p 2021:22 hadoop/nozookeeper:v1 && docker run -it --name slave2 -h hadoop-slave2 --network=mynet --ip 172.18.12.3  --add-host hadoop-master:172.18.12.1 --add-host hadoop-slave1:172.18.12.2 --add-host hadoop-slave3:172.18.12.4 -d -P -p 2022:22 hadoop/nozookeeper:v1 && docker run -it --name slave3 -h hadoop-slave3 --network=mynet --ip 172.18.12.4  --add-host hadoop-master:172.18.12.1 --add-host hadoop-slave1:172.18.12.2 --add-host hadoop-slave2:172.18.12.3  -d -P -p 2023:22 hadoop/nozookeeper:v1

4.3、master容器上執(zhí)行hdfs namenode -format

yay@yay-ThinkPad-T470:~/software/dockerfileForHadoop$ docker exec -it master /bin/bash
root@hadoop-master:/# ./usr/local/hadoop/bin/hdfs namenode -format

4.4簇搅、master容器上執(zhí)行start-all.sh

root@hadoop-master:/# start-all.sh
WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of HADOOP_PREFIX.
WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of HADOOP_PREFIX.
WARNING: HADOOP_SECURE_DN_USER has been replaced by HDFS_DATANODE_SECURE_USER. Using value of HADOOP_SECURE_DN_USER.
Starting namenodes on [hadoop-master]
hadoop-master: Warning: Permanently added 'hadoop-master,172.18.12.1' (ECDSA) to the list of known hosts.
Starting datanodes
hadoop-slave1: Warning: Permanently added 'hadoop-slave1,172.18.12.2' (ECDSA) to the list of known hosts.
hadoop-slave3: Warning: Permanently added 'hadoop-slave3,172.18.12.4' (ECDSA) to the list of known hosts.
hadoop-slave2: Warning: Permanently added 'hadoop-slave2,172.18.12.3' (ECDSA) to the list of known hosts.
hadoop-slave3: WARNING: /usr/local/hadoop-3.2.1/logs does not exist. Creating.
hadoop-slave2: WARNING: /usr/local/hadoop-3.2.1/logs does not exist. Creating.
hadoop-slave1: WARNING: /usr/local/hadoop-3.2.1/logs does not exist. Creating.
Starting secondary namenodes [hadoop-master]
WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of HADOOP_PREFIX.
WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of HADOOP_PREFIX.
Starting resourcemanager
Starting nodemanagers
root@hadoop-master:/#

使用jps查看主從節(jié)點(diǎn)的狀態(tài)

root@hadoop-master:/# jps
1392 Jps
900 ResourceManager
1047 NodeManager
407 DataNode
632 SecondaryNameNode
270 NameNode
root@hadoop-master:/# exit
exit
yay@yay-ThinkPad-T470:~/software/dockerfileForHadoop$ docker exec -it slave1 /bin/bash
root@hadoop-slave1:/# jps
178 NodeManager
295 Jps
60 DataNode
root@hadoop-slave1:/# exit
exit
yay@yay-ThinkPad-T470:~/software/dockerfileForHadoop$ docker exec -it slave2 /bin/bash
root@hadoop-slave2:/# jps
292 Jps
57 DataNode
175 NodeManager
root@hadoop-slave2:/# exit
exit
yay@yay-ThinkPad-T470:~/software/dockerfileForHadoop$ docker exec -it slave3 /bin/bash
root@hadoop-slave3:/# jps
292 Jps
57 DataNode
175 NodeManager
root@hadoop-slave3:/#

說(shuō)明：至于SSH完域，我只在dockerfile里面進(jìn)行了配置，啟動(dòng)容器后沒(méi)有作別的配置

4.5瘩将、master容器上執(zhí)行stop-all.sh

yay@yay-ThinkPad-T470:~/software/dockerfileForHadoop$ docker exec -it master /bin/bash
root@hadoop-master:/# stop-all.sh
WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of HADOOP_PREFIX.
WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of HADOOP_PREFIX.
WARNING: HADOOP_SECURE_DN_USER has been replaced by HDFS_DATANODE_SECURE_USER. Using value of HADOOP_SECURE_DN_USER.
Stopping namenodes on [hadoop-master]
Stopping datanodes
Stopping secondary namenodes [hadoop-master]
WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of HADOOP_PREFIX.
WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of HADOOP_PREFIX.
Stopping nodemanagers
Stopping resourcemanager
root@hadoop-master:/# jps
2313 Jps
root@hadoop-master:/# exit
exit
yay@yay-ThinkPad-T470:~/software/dockerfileForHadoop$ docker exec -it slave3 /bin/bash
root@hadoop-slave3:/# jps
459 Jps
root@hadoop-slave3:/#

圖片.png

MapReduce的管理界面如下:
圖片.png

五吟税、停止所有容器（本身和本主題無(wú)關(guān)）

列出所有容器ID

yay@yay-ThinkPad-T470:~/software/dockerfileForHadoop$ docker ps -aq
34b959936f6a
32da063264ca
43d836e58198
40d552711425

停止所有容器

yay@yay-ThinkPad-T470:~/software/dockerfileForHadoop$ docker stop $(docker ps -aq)
34b959936f6a
32da063264ca
43d836e58198
40d552711425
yay@yay-ThinkPad-T470:~/software/dockerfileForHadoop$

刪除所有容器

docker rm $(docker ps -aq)
最后編輯于 ：
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
  • 人面猴
    序言：七十年代末，一起剝皮案震驚了整個(gè)濱河市姿现，隨后出現(xiàn)的幾起案子肠仪，更是在濱河造成了極大的恐慌，老刑警劉巖备典，帶你破解...
    沈念sama閱讀 211,290評(píng)論 6贊 491
  • 死咒
    序言：濱河連續(xù)發(fā)生了三起死亡事件异旧，死亡現(xiàn)場(chǎng)離奇詭異，居然都是意外死亡提佣，警方通過(guò)查閱死者的電腦和手機(jī)吮蛹，發(fā)現(xiàn)死者居然都...
    沈念sama閱讀 90,107評(píng)論 2贊 385
  • 救了他兩次的神仙讓他今天三更去死
    文/潘曉璐 我一進(jìn)店門(mén)，熙熙樓的掌柜王于貴愁眉苦臉地迎上來(lái)镐依，“玉大人匹涮，你說(shuō)我怎么就攤上這事』笨牵” “怎么了然低？”我有些...
    開(kāi)封第一講書(shū)人閱讀 156,872評(píng)論 0贊 347
  • 道士緝兇錄：失蹤的賣(mài)姜人
    文/不壞的土叔 我叫張陵，是天一觀的道長(zhǎng)务唐。 經(jīng)常有香客問(wèn)我雳攘，道長(zhǎng)，這世上最難降的妖魔是什么枫笛？ 我笑而不...
    開(kāi)封第一講書(shū)人閱讀 56,415評(píng)論 1贊 283
  • ?港島之戀（遺憾婚禮）
    正文 為了忘掉前任吨灭，我火速辦了婚禮刑巧，結(jié)果婚禮上喧兄，老公的妹妹穿的比我還像新娘。我一直安慰自己啊楚，他們只是感情好吠冤，可當(dāng)我...
    茶點(diǎn)故事閱讀 65,453評(píng)論 6贊 385
  • 惡毒庶女頂嫁案：這布局不是一般人想出來(lái)的
    文/花漫 我一把揭開(kāi)白布。 她就那樣靜靜地躺著恭理，像睡著了一般拯辙。 火紅的嫁衣襯著肌膚如雪。 梳的紋絲不亂的頭發(fā)上颜价，一...
    開(kāi)封第一講書(shū)人閱讀 49,784評(píng)論 1贊 290
  • 城市分裂傳說(shuō)
    那天涯保，我揣著相機(jī)與錄音诉濒，去河邊找鬼。 笑死夕春，一個(gè)胖子當(dāng)著我的面吹牛未荒，可吹牛的內(nèi)容都是我干的。 我是一名探鬼主播撇他，決...
    沈念sama閱讀 38,927評(píng)論 3贊 406
  • 雙鴛鴦連環(huán)套：你想象不到人心有多黑
    文/蒼蘭香墨 我猛地睜開(kāi)眼茄猫，長(zhǎng)吁一口氣：“原來(lái)是場(chǎng)噩夢(mèng)啊……” “哼！你這毒婦竟也來(lái)了困肩？” 一聲冷哼從身側(cè)響起，我...
    開(kāi)封第一講書(shū)人閱讀 37,691評(píng)論 0贊 266
  • 萬(wàn)榮殺人案實(shí)錄
    序言：老撾萬(wàn)榮一對(duì)情侶失蹤脆侮，失蹤者是張志新（化名）和其女友劉穎锌畸，沒(méi)想到半個(gè)月后，有當(dāng)?shù)厝嗽跇?shù)林里發(fā)現(xiàn)了一具尸體靖避，經(jīng)...
    沈念sama閱讀 44,137評(píng)論 1贊 303
  • ?護(hù)林員之死
    正文 獨(dú)居荒郊野嶺守林人離奇死亡潭枣，尸身上長(zhǎng)有42處帶血的膿包…… 初始之章·張勛 以下內(nèi)容為張勛視角 年9月15日...
    茶點(diǎn)故事閱讀 36,472評(píng)論 2贊 326
  • ?白月光啟示錄
    正文 我和宋清朗相戀三年，在試婚紗的時(shí)候發(fā)現(xiàn)自己被綠了幻捏。 大學(xué)時(shí)的朋友給我發(fā)了我未婚夫和他白月光在一起吃飯的照片盆犁。...
    茶點(diǎn)故事閱讀 38,622評(píng)論 1贊 340
  • 活死人
    序言：一個(gè)原本活蹦亂跳的男人離奇死亡，死狀恐怖篡九，靈堂內(nèi)的尸體忽然破棺而出谐岁，到底是詐尸還是另有隱情，我是刑警寧澤榛臼，帶...
    沈念sama閱讀 34,289評(píng)論 4贊 329
  • ?日本核電站爆炸內(nèi)幕
    正文 年R本政府宣布伊佃，位于F島的核電站，受9級(jí)特大地震影響沛善，放射性物質(zhì)發(fā)生泄漏航揉。R本人自食惡果不足惜，卻給世界環(huán)境...
    茶點(diǎn)故事閱讀 39,887評(píng)論 3贊 312
  • 男人毒藥：我在死后第九天來(lái)索命
    文/蒙蒙 一金刁、第九天 我趴在偏房一處隱蔽的房頂上張望帅涂。 院中可真熱鬧，春花似錦尤蛮、人聲如沸媳友。這莊子的主人今日做“春日...
    開(kāi)封第一講書(shū)人閱讀 30,741評(píng)論 0贊 21
  • 一樁弒父案抵屿，背后竟有這般陰謀
    文/蒼蘭香墨 我抬頭看了看天上的太陽(yáng)庆锦。三九已至，卻和暖如春轧葛，著一層夾襖步出監(jiān)牢的瞬間搂抒，已是汗流浹背艇搀。 一陣腳步聲響...
    開(kāi)封第一講書(shū)人閱讀 31,977評(píng)論 1贊 265
  • 情欲美人皮
    我被黑心中介騙來(lái)泰國(guó)打工， 沒(méi)想到剛下飛機(jī)就差點(diǎn)兒被人妖公主榨干…… 1. 我叫王不留求晶，地道東北人焰雕。 一個(gè)月前我還...
    沈念sama閱讀 46,316評(píng)論 2贊 360
  • 代替公主和親
    正文 我出身青樓，卻偏偏與公主長(zhǎng)得像芳杏，于是被迫代替她去往敵國(guó)和親矩屁。 傳聞我的和親對(duì)象是個(gè)殘疾皇子，可洞房花燭夜當(dāng)晚...
    茶點(diǎn)故事閱讀 43,490評(píng)論 2贊 348

推薦閱讀更多精彩內(nèi)容