環(huán)境說(shuō)明：
CentOS-6.4-x86_64-bin-DVD1.iso
hadoop-2.4.1.tar.gz
hbase-0.98.3-hadoop2-bin.tar.gz
jdk-7u79-linux-x64.tar.gz
scala-2.10.4.tgz
spark-1.2.0-bin-hadoop2.4.tgz.tar
zookeeper-3.4.6.tar.gz
附上下載地址：
hadoop2.4.1:http://archive.apache.org/dist/hadoop/common/hadoop-2.4.1/hadoop-2.4.1.tar.gz
hbase0.98.3:http://archive.apache.org/dist/hbase/hbase-0.98.3/hbase-0.98.3-hadoop2-bin.tar.gz
spark1.2.0:http://archive.apache.org/dist/spark/spark-1.2.0/spark-1.2.0-bin-hadoop2.4.tgz
zookeeper3.4.6:http://apache.fayea.com/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
scala2.10.4:http://www.scala-lang.org/files/archive/scala-2.10.4.tgz
1、hadoop與hbase依賴(lài)關(guān)系【沒(méi)選好渡处，后面就等著推倒重來(lái)吧】
見(jiàn)：Apache HBase Reference Guide
https://hbase.apache.org/book.html#configuration
----4.1.Hadoop章節(jié)
2嗜憔、namenode無(wú)法啟動(dòng)
日志報(bào)錯(cuò)“ulimit -a for user root”
解決：
重新格式化namenode锻离，然后啟動(dòng)hadoop酒繁，jps存在namenode皮钠。
3识樱、然也有可能datanode無(wú)法啟動(dòng)
日志報(bào)錯(cuò)“ulimit -a for user root”
原因：datanamenode運(yùn)行時(shí)打開(kāi)文件數(shù)艳汽，達(dá)到系統(tǒng)最大限制
當(dāng)前最大限制
[root@centos-FI hadoop-2.4.1]# ulimit -n
1024
解決：
調(diào)整最大打開(kāi)文件數(shù)
[root@centos-FI hadoop-2.4.1]# ulimit -n 65536
[root@centos-FI hadoop-2.4.1]# ulimit -n
65536
再次啟動(dòng)hadoop
[root@centos-FI hadoop-2.4.1]# jps
6330 WebAppProxyServer
6097 NameNode
6214 ResourceManager
6148 DataNode
6441 Jps
6271 NodeManager
6390 JobHistoryServer
ps：ulimit命令只是臨時(shí)修改猴贰，重啟又恢復(fù)默認(rèn)，可在/etc/security/limits.conf 里修改 nofile 的限制河狐。
參考：http://labs.chinamobile.com/mblog/225_17546
4米绕、hadoop fs -ls報(bào)“WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable”
還是中招了64位操作系統(tǒng)，32位的jdk馋艺，不過(guò)還好該warn無(wú)影響可忽視栅干。
5、hadoop fs -ls報(bào)“l(fā)s: `.': No such file or directory”
hadoop2.x不同于hadoop1.x的地方捐祠，
1.x是可以執(zhí)行的碱鳞，
而2.x的執(zhí)行命令為：hadoop fs -ls /
【坑啊】
6、啟動(dòng)hbase時(shí)雏赦，hmaster啟動(dòng)后又消失了劫笙，且hin/base shell后芙扎，list執(zhí)行報(bào)錯(cuò)"ERROR:can't get master address from ZooKeeper; znode data == null"
[root@centos-FI hbase-0.98.3-hadoop2]# jps
28406 NameNode
28576 ResourceManager
32196 HRegionServer
32079 HMaster
28464 DataNode
32253 Jps
28748 JobHistoryServer
28635 NodeManager
24789 QuorumPeerMain
[root@centos-FI hbase-0.98.3-hadoop2]# jps
28406 NameNode
28576 ResourceManager
32196 HRegionServer
28464 DataNode
32293 Jps
28748 JobHistoryServer
28635 NodeManager
24789 QuorumPeerMain
[root@centos-FI hbase-0.98.3-hadoop2]#
日志提示：
2015-07-13 13:25:53,904 DEBUG [main-EventThread] master.ActiveMasterManager: A master is now available
2015-07-13 13:25:53,912 INFO [master:centos-FI:60000] master.ActiveMasterManager: Registered Active Master=centos-FI,60000,1436819147113
2015-07-13 13:25:53,921 INFO [master:centos-FI:60000] Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS
2015-07-13 13:25:54,250 FATAL [master:centos-FI:60000] master.HMaster: Unhandled exception. Starting shutdown.
java.net.ConnectException: Call From centos-FI/127.0.0.1 to centos-FI:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
解決方法：
根據(jù) http://wiki.apache.org/hadoop/ConnectionRefused 的這條提示信息"Check that there isn't an entry for your hostname mapped to 127.0.0.1 or 127.0.1.1 in /etc/hosts (Ubuntu is notorious for this)"檢查：
[root@centos-FI ~]# hostname -i
127.0.0.1 192.168.128.120
[root@centos-FI hbase-0.98.3-hadoop2]# cat /etc/hosts
127.0.0.1 centos-FI localhost #localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 centos-FI #localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.128.120 master
192.168.128.120 slave
192.168.128.120 centos-FI
改為：
[root@centos-FI hbase-0.98.3-hadoop2]# cat /etc/hosts
127.0.0.1 localhost #localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 centos-FI #localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.128.120 master
192.168.128.120 slave
192.168.128.120 centos-FI
即確保
[root@centos-FI hbase-0.98.3-hadoop2]# hostname -i
192.168.128.120
hmaster進(jìn)程正常后，hin/base shell再次執(zhí)行l(wèi)ist也就正常了
當(dāng)然hin/base shell后填大，list執(zhí)行報(bào)錯(cuò)"ERROR:can't get master address from ZooKeeper; znode data == null"還有可能是其他原因戒洼，詳見(jiàn)：
7、啟動(dòng)hbase時(shí)報(bào)錯(cuò)“l(fā)ocalhost: ssh: Could not resolve hostname localhost: Temporary failure in name resolution”
[root@centos-FI hbase-0.98.3-hadoop2]# bin/start-hbase.sh
starting master, logging to /opt/program/hbase-0.98.3-hadoop2/bin/../logs/hbase-root-master-centos-FI.out
localhost: ssh: Could not resolve hostname localhost: Temporary failure in name resolution
以上報(bào)錯(cuò)允华，/etc/hosts中必須存在127.0.0.1 localhost
8圈浇、CRT進(jìn)入hbase shell后無(wú)法退格
在secureCRT中，點(diǎn)擊【選項(xiàng)】【會(huì)話選項(xiàng)】【終端】【仿真】靴寂，右邊的終端選擇linux磷蜀，在hbase shell中如輸入出錯(cuò)，按住Ctrl+刪除鍵(backspace) 即可刪除百炬！
9褐隆、執(zhí)行spark-example：HBaseTest.scala 如下報(bào)錯(cuò)：
WARN TableInputFormatBase: Cannot resolve the host name for master/192.168.128.120 because of javax.naming.CommunicationException: DNS error [Root exception is java.net.PortUnreachableException: ICMP Port Unreachable]; remaining name '120.128.168.192.in-addr.arpa'
這是由于DNS服務(wù)器沒(méi)有當(dāng)前節(jié)點(diǎn)的記錄（都沒(méi)使用到DNS服務(wù)器），手動(dòng)在/etc/hosts中添加一條記錄"master 192.168.128.120"
10剖踊、使用flume攔截器時(shí)庶弃，報(bào)錯(cuò)"java.lang.NullPointerException: Expected timestamp in the Flume event headers, but it was null"
sink是hdfs，然后使用目錄自動(dòng)生成功能德澈。出現(xiàn)如題的錯(cuò)誤歇攻，看官網(wǎng)文檔說(shuō)的是需要在每個(gè)文件記錄行的開(kāi)頭需要有時(shí)間戳，但是時(shí)間戳的格式可能比較難調(diào)節(jié)梆造，所以亦可設(shè)置hdfs.useLocalTimeStamp這個(gè)參數(shù)缴守，比如以每個(gè)小時(shí)作為一個(gè)文件夾，那么配置應(yīng)該是這樣：
a1.sinks.k1.hdfs.path = hdfs://ubuntu:9000/flume/events/%y-%m-%d/%H
a1.sinks.k1.hdfs.filePrefix = events-
a1.sinks.k1.hdfs.round = true
a1.sinks.k1.hdfs.roundValue = 1
a1.sinks.k1.hdfs.roundUnit = hour
a1.sinks.k1.hdfs.useLocalTimeStamp = true
修改之后再次執(zhí)行flume確實(shí)是自動(dòng)生成了hdfs目錄镇辉，
flume日志:
2015-07-23 15:16:28,405 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:261)] Creating hdfs://master:9000/flume_test/15-07-23/events-.1437689788240.tmp
備忘
查看RM：http://ip:8088/
查看hdfs：http://ip:50070/
查看jobtrack屡穗；http://ip:50030/
hbase master web ui:http://ip:60010/
hbase region web ui:http://ip:60030/
- spark監(jiān)控之web UI:
每個(gè)SparkContext啟動(dòng)一個(gè)UI，默認(rèn)端口4040摊聋，多個(gè)SparkContext端口4040以此累加鸡捐，顯示應(yīng)用信息：
高度器stage和task列表
RDD大小和內(nèi)存占用概況
環(huán)境信息
正在運(yùn)行的executors信息
此信息之在應(yīng)用運(yùn)行時(shí)才能顯示栈暇，若想運(yùn)行后查看web UI麻裁，須在啟動(dòng)應(yīng)用之前將spark-default.conf的spark.eventLog.enabled設(shè)置為true：spark將UI中顯示的信息記錄為spark事件，并記錄到持久化存儲(chǔ)中
spark.eventLog.enabled true
spark.eventLog.dir hdfs://centos-FI:9000/spark_eventLog
