【Hadoop大數(shù)據(jù)基礎(chǔ)】##

第二節(jié) Hadoop安裝與使用

2.1 Hadoop簡介
2.2 MR實現(xiàn)第一節(jié)作業(yè)
2.3 HDFS分布式存儲系統(tǒng)
2.4 單機(jī)安裝 
2.5 分布式集群搭建
2.6 課后作業(yè)

2.1 Hadoop簡介

     在文章的時候已經(jīng)講解了Hadoop的簡介以及生態(tài)圈,有什么不懂的可以"出門右轉(zhuǎn)"

http://dwz.cn/4rdSdU

2.2 MR實現(xiàn)第一節(jié)作業(yè)

首先運(yùn)行計算就要啟動mapreduce,我們剛剛才啟動了start-dfs.sh (HDFS文件系統(tǒng)),所以不能計算那么我們把mapreduce的程序也啟動起來貌虾。

啟動mapreduce命令
[root@localhost hadoop]# start-yarn.sh 
starting yarn daemons
starting resourcemanager, logging to /opt/hadoop/hadoop-2.6.0-cdh5.8.2/logs/yarn-root-resourcemanager-localhost.localdomain.out
localhost: starting nodemanager, logging to /opt/hadoop/hadoop-2.6.0-cdh5.8.2/logs/yarn-root-nodemanager-localhost.localdomain.out

啟動完成以后我們調(diào)用"jps"命令看是否真的啟動成功了
28113 NodeManager
28011 ResourceManager
28442 Jps
27137 NameNode
27401 SecondaryNameNode
27246 DataNode

可以很明顯的看到多出來兩個程序裙犹。

創(chuàng)建測試文件

創(chuàng)建一個測試的數(shù)據(jù):
vi /opt/test/test.txt
麒麟
小張
張張
果哥
澤安
跨越
天天順利
澤安
祖淵
張張

2.2-01.jpg

將測試文件上傳到HDFS
首先我們要在HDFS上再創(chuàng)建兩個文件,一個 input(輸入) / ouput(輸出)的文件夾袄膏。

[root@localhost ~]# hdfs dfs -mkdir /input /ouput
16/10/26 04:30:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

將測試數(shù)據(jù)上傳到"input"文件夾
[root@localhost ~]# hdfs dfs -put /opt/test/test.txt /input
16/10/26 04:33:03 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

查看是上傳成功
[root@localhost ~]# hdfs dfs -cat /input/test.txt
16/10/26 04:34:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
麒麟
小張
張張
果哥
澤安
跨越
天天順利
澤安
祖淵
張張

調(diào)用Hadoop自帶的WordCount方法

[root@localhost ~]# hadoop jar /opt/hadoop/hadoop-2.6.0-cdh5.8.2/share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.8.2.jar wordcount /input /ouput/test
16/10/26 04:49:37 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/10/26 04:49:38 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032
16/10/26 04:49:42 INFO input.FileInputFormat: Total input paths to process : 1
16/10/26 04:49:43 INFO mapreduce.JobSubmitter: number of splits:1
16/10/26 04:49:44 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1477471653063_0001
16/10/26 04:49:46 INFO impl.YarnClientImpl: Submitted application application_1477471653063_0001
16/10/26 04:49:47 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1477471653063_0001/
16/10/26 04:49:47 INFO mapreduce.Job: Running job: job_1477471653063_0001
16/10/26 04:50:21 INFO mapreduce.Job: Job job_1477471653063_0001 running in uber mode : false
16/10/26 04:50:21 INFO mapreduce.Job:  map 0% reduce 0%
16/10/26 04:50:44 INFO mapreduce.Job:  map 100% reduce 0%
16/10/26 04:51:04 INFO mapreduce.Job:  map 100% reduce 100%
16/10/26 04:51:06 INFO mapreduce.Job: Job job_1477471653063_0001 completed successfully
16/10/26 04:51:06 INFO mapreduce.Job: Counters: 49
    File System Counters
            FILE: Number of bytes read=116
            FILE: Number of bytes written=232107
            FILE: Number of read operations=0
            FILE: Number of large read operations=0
            FILE: Number of write operations=0
            HDFS: Number of bytes read=177
            HDFS: Number of bytes written=78
            HDFS: Number of read operations=6
            HDFS: Number of large read operations=0
            HDFS: Number of write operations=2
    Job Counters 
            Launched map tasks=1
            Launched reduce tasks=1
            Data-local map tasks=1
            Total time spent by all maps in occupied slots (ms)=18128
            Total time spent by all reduces in occupied slots (ms)=17756
            Total time spent by all map tasks (ms)=18128
            Total time spent by all reduce tasks (ms)=17756
            Total vcore-seconds taken by all map tasks=18128
            Total vcore-seconds taken by all reduce tasks=17756
            Total megabyte-seconds taken by all map tasks=18563072
            Total megabyte-seconds taken by all reduce tasks=18182144
    Map-Reduce Framework
            Map input records=10
            Map output records=10
            Map output bytes=116
            Map output materialized bytes=116
            Input split bytes=101
            Combine input records=10
            Combine output records=8
            Reduce input groups=8
            Reduce shuffle bytes=116
            Reduce input records=8
            Reduce output records=8
            Spilled Records=16
            Shuffled Maps =1
            Failed Shuffles=0
            Merged Map outputs=1
            GC time elapsed (ms)=454
            CPU time spent (ms)=3450
            Physical memory (bytes) snapshot=306806784
            Virtual memory (bytes) snapshot=3017633792
            Total committed heap usage (bytes)=163450880
    Shuffle Errors
            BAD_ID=0
            CONNECTION=0
            IO_ERROR=0
            WRONG_LENGTH=0
            WRONG_MAP=0
            WRONG_REDUCE=0
    File Input Format Counters 
            Bytes Read=76
    File Output Format Counters 
            Bytes Written=78

運(yùn)行完成我們看看計算出來的結(jié)果:
[root@localhost ~]# hdfs dfs -ls /ouput/test
16/10/26 04:53:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r--   1 root supergroup          0 2016-10-26 04:51 /ouput/test/_SUCCESS
-rw-r--r--   1 root supergroup         78 2016-10-26 04:51 /ouput/test/part-r-00000

[root@localhost ~]# hdfs dfs -cat /ouput/test/part-r-00000
16/10/26 04:53:41 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
天天順利        1
小張    1
張張    2
果哥    1
澤安    2
祖淵    1
跨越    1
麒麟    1

2.3 HDFS分布式存儲系統(tǒng)

HDFS優(yōu)點

1. 高容錯性
   1. 數(shù)據(jù)自動保存多個副本
   2. 副本都時候會自動恢復(fù)
2. 適合批量處理
   1. 移動計算而非數(shù)據(jù)
   2. 數(shù)據(jù)位置暴露給計算框架
3. 適合大數(shù)據(jù)處理
   1. GB/TB/甚至PB級數(shù)據(jù)
   2. 百萬規(guī)模以上文件數(shù)量
   3. 10k+
4. 可構(gòu)建廉價的機(jī)器上
   1. 不管機(jī)器人有多么的垃圾,只要有空間內(nèi)存就可以搭建Hadoop

****HDFS缺點****

1. 低延遲數(shù)據(jù)訪問
   1. 比如毫秒級
   2. 低延遲與高吞吐率
2. 小文件存取
   1. 占用NameNode大量內(nèi)存
   2. 尋道時間超過讀取時間
3. 并發(fā)寫入/文件隨機(jī)修改
   1. 一個文件只能有一個寫者
   2. 只支持append

HDFS架構(gòu)

2.3-01.jpg

如圖所示,HDFS是按照Master和Slave的結(jié)構(gòu)

NameNode:類似項目經(jīng)理職位 下達(dá)需求,監(jiān)控團(tuán)隊進(jìn)度,工作分配蒸健。
管理數(shù)據(jù)塊映射；處理客戶端的讀寫請求；配置副本策略渡嚣；管理HDFS的名稱空間敷硅；

NameNode的主要功能:接受客戶端的讀寫服務(wù)章办。
    上傳文件 讀取文件都要聯(lián)系NameNode
Namenode保存了metadate信息,而metadate保存著datanode的元數(shù)據(jù) 包含了block的位置信息(路徑),以及datanode的詳細(xì)信息辜羊。
Namenode的metadate信息啟動后會加載到內(nèi)存,metadate存儲到磁盤的文件名;"fsimage",但是其中不包含Block的位置信息,而block的位置信息是在每一次Datanode啟動的時候?qū)lock的位置信息發(fā)送給NameNode 內(nèi)存中腋颠。
**edits**記錄對metadate的操作日志。

SecondaryNameNode:類似經(jīng)理助理的職位 分擔(dān)領(lǐng)導(dǎo)(Namenode)的工作量茵肃；

SecondaryNameNode他不是主節(jié)點也不是從節(jié)點,它是用來輔助NameNode來完成editslog操作日志的跟继。
合并edits的操作可以配置:
    設(shè)置fs.checkpoint.period 默認(rèn)3600秒
    設(shè)置edits log大小fs.checkpoint.size規(guī)定edits文件的最大默認(rèn)64MB

NN合并流程

DataNode: 程序員干活的,上頭(NameNode)下達(dá)一個需求功能需要你(DataNode)實現(xiàn)。存儲client發(fā)來的數(shù)據(jù)塊block；執(zhí)行數(shù)據(jù)塊的讀寫操作杉辙。

dataNode(block)它是數(shù)據(jù)存儲單元,就是數(shù)據(jù)真正存儲的地方。每一個block都會被固定大小的數(shù)
據(jù)塊,默認(rèn)數(shù)據(jù)塊的大小是64MB**可配置**赶舆。

問:我這個文件大小沒有達(dá)到64MB怎么辦呢?
答:如果該文件還不到64MB的時候,則單獨(dú)成一個block

問:那如果大于64MB怎么辦?
答:如果大了就在切分一個block出來,就像我們切肉/切菜一樣,想把他們切的均衡(64MB),假如肉切的1厘米,到最后還剩下3厘米怎么辦?那就繼續(xù)再切分哑姚。

datanode設(shè)計思想

如圖所示

假如我上傳文件大小為"50G"-->切分為多個block。

每一個Block都會有一個3個副本(可配置)

每一個block的3個副本都存在不同的節(jié)點(機(jī)器)上,如圖"block1"芜茵。

HDFS寫流程

簡單易懂

1.HDFS Client-->NameNode:我要上次新文件
2.NameNode檢查DataNode空間,檢查出: xxx機(jī)器上的節(jié)點可以存儲
3.NameNode上傳
4.記錄log日志

下面是專業(yè)術(shù)語

客戶端調(diào)用create()來創(chuàng)建文件

DistributedFileSystem用RPC調(diào)用元數(shù)據(jù)節(jié)點叙量，在文件系統(tǒng)的命名空間中創(chuàng)建一個新的文件。

元數(shù)據(jù)節(jié)點首先確定文件原來不存在九串，并且客戶端有創(chuàng)建文件的權(quán)限绞佩，然后創(chuàng)建新文件。

DistributedFileSystem返回DFSOutputStream猪钮，客戶端用于寫數(shù)據(jù)品山。

客戶端開始寫入數(shù)據(jù)，DFSOutputStream將數(shù)據(jù)分成塊烤低，寫入data queue谆奥。

Data queue由Data Streamer讀取，并通知元數(shù)據(jù)節(jié)點分配數(shù)據(jù)節(jié)點拂玻，用來存儲數(shù)據(jù)塊(每塊默認(rèn)復(fù)制3塊)酸些。分配的數(shù)據(jù)節(jié)點放在一個pipeline里。

Data Streamer將數(shù)據(jù)塊寫入pipeline中的第一個數(shù)據(jù)節(jié)點檐蚜。第一個數(shù)據(jù)節(jié)點將數(shù)據(jù)塊發(fā)送給第二個數(shù)據(jù)節(jié)點魄懂。第二個數(shù)據(jù)節(jié)點將數(shù)據(jù)發(fā)送給第三個數(shù)據(jù)節(jié)點。

DFSOutputStream為發(fā)出去的數(shù)據(jù)塊保存了ack queue闯第，等待pipeline中的數(shù)據(jù)節(jié)點告知數(shù)據(jù)已經(jīng)寫入成功市栗。

如果數(shù)據(jù)節(jié)點在寫入的過程中失敗：

關(guān)閉pipeline，將ack queue中的數(shù)據(jù)塊放入data queue的開始填帽。

當(dāng)前的數(shù)據(jù)塊在已經(jīng)寫入的數(shù)據(jù)節(jié)點中被元數(shù)據(jù)節(jié)點賦予新的標(biāo)示蛛淋，則錯誤節(jié)點重啟后能夠察覺其數(shù)據(jù)塊是過時的，會被刪除篡腌。

失敗的數(shù)據(jù)節(jié)點從pipeline中移除褐荷，另外的數(shù)據(jù)塊則寫入pipeline中的另外兩個數(shù)據(jù)節(jié)點。

元數(shù)據(jù)節(jié)點則被通知此數(shù)據(jù)塊是復(fù)制塊數(shù)不足嘹悼，將來會再創(chuàng)建第三份備份叛甫。

當(dāng)客戶端結(jié)束寫入數(shù)據(jù)，則調(diào)用stream的close函數(shù)杨伙。此操作將所有的數(shù)據(jù)塊寫入pipeline中的數(shù)據(jù)節(jié)點其监，并等待ack queue返回成功。最后通知元數(shù)據(jù)節(jié)點寫入完畢限匣。

寫原理

讀文件

客戶端(client)用FileSystem的open()函數(shù)打開文件

DistributedFileSystem用RPC調(diào)用元數(shù)據(jù)節(jié)點抖苦，得到文件的數(shù)據(jù)塊信息。

對于每一個數(shù)據(jù)塊米死，元數(shù)據(jù)節(jié)點返回保存數(shù)據(jù)塊的數(shù)據(jù)節(jié)點的地址锌历。

DistributedFileSystem返回FSDataInputStream給客戶端，用來讀取數(shù)據(jù)哲身。

客戶端調(diào)用stream的read()函數(shù)開始讀取數(shù)據(jù)辩涝。

DFSInputStream連接保存此文件第一個數(shù)據(jù)塊的最近的數(shù)據(jù)節(jié)點。

Data從數(shù)據(jù)節(jié)點讀到客戶端(client)

當(dāng)此數(shù)據(jù)塊讀取完畢時勘天，DFSInputStream關(guān)閉和此數(shù)據(jù)節(jié)點的連接怔揩，然后連接此文件下一個數(shù)據(jù)塊的最近的數(shù)據(jù)節(jié)點。

當(dāng)客戶端讀取完畢數(shù)據(jù)的時候脯丝，調(diào)用FSDataInputStream的close函數(shù)商膊。

在讀取數(shù)據(jù)的過程中，如果客戶端在與數(shù)據(jù)節(jié)點通信出現(xiàn)錯誤宠进，則嘗試連接包含此數(shù)據(jù)塊的下一個數(shù)據(jù)節(jié)點晕拆。

失敗的數(shù)據(jù)節(jié)點將被記錄，以后不再連接材蹬。

讀原理

安全模式

Name啟動的時候,首先將映像文件(fsimage)載入內(nèi)存,并執(zhí)行編輯日志(edits)中的各項操作,一旦在內(nèi)存中成功建立文件系統(tǒng)元數(shù)據(jù)映射,則創(chuàng)建一個新的fsimage文件(SecondaryNameNode)和一個空的(edits)編輯日志实幕。
此刻N(yùn)amenode運(yùn)行"安全模式",即Namenode的文件系統(tǒng)對于客戶端來說是只讀的。(顯示目錄,顯示文件內(nèi)容等堤器。但是 增刪改都會失敗)昆庇。

2.4 單機(jī)安裝

系統(tǒng)環(huán)境

jdk安裝

下載地址百度即可,我使用的JDK版本是 1.7 ;

2.4-01.jpg

1.在linux創(chuàng)建一個文件夾放jdk
    mkdir /usr/jvm
2.將jdk移到剛剛創(chuàng)建的JVM文件夾下,也可以解壓到j(luò)vm文件夾下
    我這里使用解壓
    tar -zxvf /opt/hadoop/tar/jdk-7u79-linux-x64.tar.gz -C /usr/jvm/
3.解壓完成以后配置JDK環(huán)境變量 /etc/profile
    
    使用命令:
    vi /etc/profile

2.4-02.jpg

    將↓↓↓↓加在profile中    
    export JAVA_HOME=/usr/jvm/jdk1.7.0_79
    export JAVA_BIN=/usr/jvm/jdk1.7.0_79/bin
    export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
    #PATH
    PATH=$PATH:$JAVA_HOME/bin

2.4-03.jpg

==在linux配置中有hadoop在這里先不要管==

4.到這里jdk配置已經(jīng)完成了,我們需要刷新一下剛剛修改的profile。否則我們改改新配置的參數(shù)是無法生效的闸溃。
    
    [root@zero230 tar]# source /etc/profile
    
5.調(diào)用命令查看是否配置成功并且查看版本是否與我們配置的一致

2.4-04.jpg

    [root@zero230 tar]# java -version
    java version "1.7.0_79"
    Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
    Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)
    
到這里JDK已經(jīng)安裝完成!

SSH免驗證

$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

配置Hadoop

一.Hadoop-env.sh

Hadoop也是需要java的,所以我們需要配置(hadoop-env.sh)它調(diào)用的java整吆。
vi hadoop-env.sh
它默認(rèn)的配置是:export JAVA_HOME=${JAVA_HOME}拱撵。這樣的話必須定義變量才能啟動JAVA,所以我們需要將(hadoop-env.sh)里面的JAVA_HOME變量指定的內(nèi)容改成我們的JAVA安裝的絕對路徑,我JDK安裝的路徑在"/usr/bin/java"目錄下。
export  JAVA_HOME=/usr/bin/java

二.core-site.xml

我們需要配置兩個參數(shù) defaultFS(HDFS默認(rèn)名稱)
//配置HDFS名稱
    hdfs:協(xié)議頭,如:http https 等
    localhost:主機(jī)名
    9000:端口
<property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
</property>

三.hdfs-site.xml

replication(副本個數(shù) 默認(rèn)為3個),因為我這里搭建的是單機(jī)所以只要一個表蝙。
    replication:設(shè)置的副本個數(shù)
<property>
    <name>dfs.replication</name>
    <value>1</value>
</property>

四.mapred-site.xml

==注意:mapred.site.xml開始并不存在,而是mapred-site.xml.template拴测。我們需要使用cp命令一份命名為mapred.site.xml==

<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>

五.yarn-site.xml

 <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>
<property>
　　<name>yarn.resourcemanager.address</name>
　　<value>localhost:8032</value>
</property>  

格式化HDFS

1. $ bin/hdfs namenode -format 
    格式化NameNode,不需要每次啟動都需要格式化,機(jī)器第一次啟動的時候需要格式化一次

16/10/26 03:30:03 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
16/10/26 03:30:03 INFO namenode.NameNode: createNameNode [-format]
16/10/26 03:30:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Formatting using clusterid: CID-2eba6061-bea3-4e86-85cc-ebb1b51cf730
16/10/26 03:30:05 INFO namenode.FSNamesystem: No KeyProvider found.
16/10/26 03:30:05 INFO namenode.FSNamesystem: fsLock is fair:true
16/10/26 03:30:06 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
16/10/26 03:30:06 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
16/10/26 03:30:06 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
16/10/26 03:30:06 INFO blockmanagement.BlockManager: The block deletion will start around 2016 Oct 26 03:30:06
16/10/26 03:30:06 INFO util.GSet: Computing capacity for map BlocksMap
16/10/26 03:30:06 INFO util.GSet: VM type       = 64-bit
16/10/26 03:30:06 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB
16/10/26 03:30:06 INFO util.GSet: capacity      = 2^21 = 2097152 entries
16/10/26 03:30:06 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
16/10/26 03:30:06 INFO blockmanagement.BlockManager: defaultReplication         = 1
16/10/26 03:30:06 INFO blockmanagement.BlockManager: maxReplication             = 512
16/10/26 03:30:06 INFO blockmanagement.BlockManager: minReplication             = 1
16/10/26 03:30:06 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
16/10/26 03:30:06 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
16/10/26 03:30:06 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
16/10/26 03:30:06 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
16/10/26 03:30:06 INFO namenode.FSNamesystem: fsOwner             = root (auth:SIMPLE)
16/10/26 03:30:06 INFO namenode.FSNamesystem: supergroup          = supergroup
16/10/26 03:30:06 INFO namenode.FSNamesystem: isPermissionEnabled = true
16/10/26 03:30:06 INFO namenode.FSNamesystem: HA Enabled: false
16/10/26 03:30:06 INFO namenode.FSNamesystem: Append Enabled: true
16/10/26 03:30:06 INFO util.GSet: Computing capacity for map INodeMap
16/10/26 03:30:06 INFO util.GSet: VM type       = 64-bit
16/10/26 03:30:06 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB
16/10/26 03:30:06 INFO util.GSet: capacity      = 2^20 = 1048576 entries
16/10/26 03:30:06 INFO namenode.NameNode: Caching file names occuring more than 10 times
16/10/26 03:30:06 INFO util.GSet: Computing capacity for map cachedBlocks
16/10/26 03:30:06 INFO util.GSet: VM type       = 64-bit
16/10/26 03:30:06 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB
16/10/26 03:30:06 INFO util.GSet: capacity      = 2^18 = 262144 entries
16/10/26 03:30:06 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
16/10/26 03:30:06 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
16/10/26 03:30:06 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000
16/10/26 03:30:06 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
16/10/26 03:30:06 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
16/10/26 03:30:06 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
16/10/26 03:30:06 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
16/10/26 03:30:06 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
16/10/26 03:30:06 INFO util.GSet: Computing capacity for map NameNodeRetryCache
16/10/26 03:30:06 INFO util.GSet: VM type       = 64-bit
16/10/26 03:30:06 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
16/10/26 03:30:06 INFO util.GSet: capacity      = 2^15 = 32768 entries
16/10/26 03:30:06 INFO namenode.NNConf: ACLs enabled? false
16/10/26 03:30:06 INFO namenode.NNConf: XAttrs enabled? true
16/10/26 03:30:06 INFO namenode.NNConf: Maximum size of an xattr: 16384
16/10/26 03:30:06 INFO namenode.FSImage: Allocated new BlockPoolId: BP-857467713-127.0.0.1-1477467006754
16/10/26 03:30:06 INFO common.Storage: Storage directory /tmp/hadoop-root/dfs/name has been successfully formatted.
16/10/26 03:30:07 INFO namenode.FSImageFormatProtobuf: Saving image file /tmp/hadoop-root/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
16/10/26 03:30:07 INFO namenode.FSImageFormatProtobuf: Image file /tmp/hadoop-root/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 351 bytes saved in 0 seconds.
16/10/26 03:30:07 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
16/10/26 03:30:07 INFO util.ExitUtil: Exiting with status 0
16/10/26 03:30:07 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localhost.localdomain/127.0.0.1
************************************************************/
[root@localhost hadoop-2.6.0-cdh5.8.2]#         

格式化

截取后半部分,這樣就表示格式化完成

2. sbin/start-dfs.sh
啟動HDFS(NameNode,secondarynamenode,DataNode)
    
[root@localhost hadoop]# start-dfs.sh 
16/10/26 03:57:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: starting namenode, logging to /opt/hadoop/hadoop-2.6.0-cdh5.8.2/logs/hadoop-root-namenode-localhost.localdomain.out
localhost: starting datanode, logging to /opt/hadoop/hadoop-2.6.0-cdh5.8.2/logs/hadoop-root-datanode-localhost.localdomain.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /opt/hadoop/hadoop-2.6.0-cdh5.8.2/logs/hadoop-root-secondarynamenode-localhost.localdomain.out
16/10/26 03:58:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

2.4-06.jpg

啟動完成以后我們調(diào)用"jps"命令看是否真的啟動成功了
[root@localhost hadoop]# jps
28442 Jps
27137 NameNode
27401 SecondaryNameNode
27246 DataNode

Web瀏覽HDFS文件

到這一步我們可以通過ip地址訪問HDFS文件系統(tǒng)。

[root@localhost hadoop]# ifconfig 
    inet **172.19.4.239**  netmask 255.255.255.0  broadcast 172.19.4.255
    inet6 fe80::250:56ff:fe8d:1144  prefixlen 64  scopeid 0x20<link>
    ether 00:50:56:8d:11:44  txqueuelen 1000  (Ethernet)
    RX packets 1638661  bytes 777256881 (741.2 MiB)
    RX errors 0  dropped 8125  overruns 0  frame 0
    TX packets 333206  bytes 24964212 (23.8 MiB)
    TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

我這里的地址為 172.19.4.239 那么外部訪問的地址是:http://172.19.4.239:50070(默認(rèn)的端口號)/就可以訪問府蛇。

2.4-07.jpg

命令操作HDFS文件

我們講到了配置-->啟動,現(xiàn)在我們來講講對HDFS文件系統(tǒng)的操作,操作HDFS 在linux中的命令:
[root@localhost hadoop]# hdfs dfs
Usage: hadoop fs [generic options]
    [-appendToFile <localsrc> ... <dst>]
    [-cat [-ignoreCrc] <src> ...]
    [-checksum <src> ...]
    [-chgrp [-R] GROUP PATH...]
    [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
    [-chown [-R] [OWNER][:[GROUP]] PATH...]
    [-copyFromLocal [-f] [-p] [-l] <localsrc> ... <dst>]
    [-copyToLocal [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
    [-count [-q] [-h] [-v] <path> ...]
    [-cp [-f] [-p | -p[topax]] <src> ... <dst>]
    [-createSnapshot <snapshotDir> [<snapshotName>]]
    [-deleteSnapshot <snapshotDir> <snapshotName>]
    [-df [-h] [<path> ...]]
    [-du [-s] [-h] <path> ...]
    [-expunge]
    [-find <path> ... <expression> ...]
    [-get [-p] [-ignoreCrc] [-crc] <src> ... <localdst>]
    [-getfacl [-R] <path>]
    [-getfattr [-R] {-n name | -d} [-e en] <path>]
    [-getmerge [-nl] <src> <localdst>]
    [-help [cmd ...]]
    [-ls [-C] [-d] [-h] [-q] [-R] [-t] [-S] [-r] [-u] [<path> ...]]
    [-mkdir [-p] <path> ...]
    [-moveFromLocal <localsrc> ... <dst>]
    [-moveToLocal <src> <localdst>]
    [-mv <src> ... <dst>]
    [-put [-f] [-p] [-l] <localsrc> ... <dst>]
    [-renameSnapshot <snapshotDir> <oldName> <newName>]
    [-rm [-f] [-r|-R] [-skipTrash] <src> ...]
    [-rmdir [--ignore-fail-on-non-empty] <dir> ...]
    [-setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>]]
    [-setfattr {-n name [-v value] | -x name} <path>]
    [-setrep [-R] [-w] <rep> <path> ...]
    [-stat [format] <path> ...]
    [-tail [-f] <file>]
    [-test -[defsz] <path>]
    [-text [-ignoreCrc] <src> ...]
    [-touchz <path> ...]
    [-usage [cmd ...]]

Generic options supported are
-conf <configuration file>     specify an application configuration file
-D <property=value>            use value for given property
-fs <local|namenode:port>      specify a namenode
-jt <local|resourcemanager:port>    specify a ResourceManager
-files <comma separated list of files>    specify comma separated files to be copied to the map reduce cluster
-libjars <comma separated list of jars>    specify comma separated jar files to include in the classpath.
-archives <comma separated list of archives>    specify comma separated archives to be unarchived on the compute machines.

The general command line syntax is
bin/hadoop command [genericOptions] [commandOptions]

**建立一個文件夾**
[root@localhost hadoop]# hdfs dfs -mkdir /user
16/10/26 04:10:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

**查看是否創(chuàng)建成功**
[root@localhost hadoop]# hdfs dfs -ls /
16/10/26 04:12:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
drwxr-xr-x   - root supergroup          0 2016-10-26 04:10 /user

簡單命令就講到這里以后會更單獨(dú)講解HDFS

2.5 分布式集群搭建

系統(tǒng)環(huán)境

安裝JDK

下載地址百度即可,我使用的JDK版本是 1.7 ;

2.5-01.jpg

1.在linux創(chuàng)建一個文件夾放jdk
    mkdir /usr/jvm
2.將jdk移到剛剛創(chuàng)建的JVM文件夾下,也可以解壓到j(luò)vm文件夾下
    我這里使用解壓
    tar -zxvf /opt/hadoop/tar/jdk-7u79-linux-x64.tar.gz -C /usr/jvm/
3.解壓完成以后配置JDK環(huán)境變量 /etc/profile
    
    使用命令:
    vi /etc/profile

2.5-02.jpg

    將↓↓↓↓加在profile中    
    export JAVA_HOME=/usr/jvm/jdk1.7.0_79
    export JAVA_BIN=/usr/jvm/jdk1.7.0_79/bin
    export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
    #PATH
    PATH=$PATH:$JAVA_HOME/bin

2.5-03.jpg

==在linux配置中有hadoop在這里先不要管==

4.到這里jdk配置已經(jīng)完成了,我們需要刷新一下剛剛修改的profile集索。否則我們改改新配置的參數(shù)是無法生效的。
    
    [root@zero230 tar]# source /etc/profile
    
5.調(diào)用命令查看是否配置成功并且查看版本是否與我們配置的一致

2.5-04.jpg

    [root@zero230 tar]# java -version
    java version "1.7.0_79"
    Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
    Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)
    
到這里JDK已經(jīng)安裝完成!

SSH互免驗證

1.在 zero230 zero231 zero239 機(jī)器上分別執(zhí)行一下命令
    ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
    執(zhí)行完以后會獲得 id_dsa(私鑰) 和 id_dsa.pub(公鑰) 兩個文件
2.在zero239 機(jī)器上執(zhí)行該命令
    cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
    意思就是將 公鑰 的內(nèi)容追加到 authorized_keys 文件中欲诺。
3.將 zero231/zero230 的 id_dsa.pub(公鑰)復(fù)制到zero239機(jī)器上
    [root@zero230 ~]# scp -r ~/.ssh/id_dsa.pub root@zero239:~/.ssh/230.pub
    [root@zero231 ~]# scp -r ~/.ssh/id_dsa.pub root@zero239:~/.ssh/231.pub
    **注意:**在這里復(fù)制到zero239機(jī)器上必須重新命令否則會將 zero239 的給覆蓋抄谐。
4.將 230 231 機(jī)器上的 id_dsa.pub 復(fù)制到 239機(jī)器上了將他們追加到 "authorized_keys" 文件中渺鹦。
    [root@zero239 ~]# cat ~/.ssh/23*.pub >> ~/.ssh/authorized_keys

2.5-05.jpg

5.將239機(jī)器上已經(jīng)添加了 230/231機(jī)器的公鑰"authorized_keys"文件復(fù)制到230和231機(jī)器上
    scp -r ~/.ssh/authorized_keys root@zero230:~/.ssh/
    scp -r ~/.ssh/authorized_keys root@zero231:~/.ssh/
    中有提示(yes/no),選擇yes

6.測試SSH是否互通
    ssh zero230 
    回車即可,如果需要輸入密碼請檢查是否配置正確
    [root@zero239 ~]# ssh zero230
    Last login: Thu Oct 27 14:56:54 2016 from a-bg-519.ztequantum.com
    [root@zero230 ~]# exit

配置hadoop

1.將hadoop安裝包復(fù)制到230 231 239 三臺機(jī)器上
   使用命令Scp 
2.解壓到/opt/hadoop(解壓到那個目錄下看自己,但一定要記得在哪個目錄下)文件夾下,建議*三臺機(jī)器上的hadoop存放的位置相同*
    tar zxvf /opt/hadoop/tar/hadoop-2.6.0-cdh5.8.2.tar.gz -C /opt/hadoop/
3.修改配置文件
    首先我們要做分布式的Hadoop,那么配置一定需要將他們連串起來,否則怎么可以相互傳遞數(shù)據(jù)呢?

一.core-site.xml

二.hdfs-site.xml

三.mapred-site.xml

四.yarn-site.xml

五.slaves

將 zero239 zero230 zero231 主機(jī)名添加進(jìn)去

2.5-06.jpg

4.配置修改完成以后將它復(fù)制分配到各個節(jié)點上\
    scp -r /opt/hadoop/hadoop-2.6.0-cdh5.8.2/etc/hadoop/* root@zero230:/opt/hadoop/hadoop-2.6.0-cdh5.8.2/etc/hadoop/
    
    scp -r /opt/hadoop/hadoop-2.6.0-cdh5.8.2/etc/hadoop/* root@zero231:/opt/hadoop/hadoop-2.6.0-cdh5.8.2/etc/hadoop/
    
5.格式化hdfs
    hdfs namenode -format

2.5-07.jpg

6.啟動hadoop
    方法一: start-dfs.sh 然后 yarn-start.sh
    方法二:start-all.sh

2.5-08.jpg

7.查看進(jìn)程是否全部啟動使用命令:JPS
    239機(jī)器:
    [root@zero239 hadoop-2.6.0-cdh5.8.2]# jps
    7518 NodeManager 
    6813 NameNode
    7874 Jps
    5292 DataNode
    7113 SecondaryNameNode
    7404 ResourceManager
    
    230機(jī)器:
    [root@zero230 ~]# jps
    5690 Jps
    5545 NodeManager
    5692 DataNode
    
    231機(jī)器
    [root@zero231 ~]# jps
    3855 NodeManager
    3631 DataNode
    4082 Jps
    
到這一步說明分布式集群搭建完成。

錯誤記錄

[root@localhost ~]# hadoop jar /opt/hadoop/hadoop-2.6.0-cdh5.8.2/share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.8.2.jar wordcount /input /ouput
16/10/26 04:40:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/10/26 04:40:33 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
16/10/26 04:40:34 WARN security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to localhost:9001 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to localhost:9001 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:731)
    at org.apache.hadoop.ipc.Client.call(Client.java:1475)
    at org.apache.hadoop.ipc.Client.call(Client.java:1408)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
    at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:762)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
    at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2121)
    at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1215)
    at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1412)
    at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:145)
    at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:270)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:143)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1325)
    at org.apache.hadoop.examples.WordCount.main(WordCount.java:87)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
    at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
    at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
    at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)
    at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614)
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:713)
    at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1524)
    at org.apache.hadoop.ipc.Client.call(Client.java:1447)
    ... 41 more
[root@localhost ~]# 

問題:沒有關(guān)閉防火墻
    service iptables stop 關(guān)閉
    chkconfig iptables off 永久關(guān)閉

2.6 課后作業(yè)###

1. 搭建一個3個節(jié)點的Hadoop分布式環(huán)境
2. 用MR的方式實現(xiàn)第一課的作業(yè)吸耿，對文本進(jìn)行統(tǒng)計祠锣，輸出統(tǒng)計結(jié)果到HDFS

更多課程可以關(guān)注

【微信公眾號】小張網(wǎng)校

公眾號內(nèi)有微信群，掃描加入后更多大牛交流學(xué)習(xí)