安裝環(huán)境
| 主機(jī) | ip | 操作系統(tǒng) |
|---|---|---|
| master | 10.30.45.239 | centos7.4 |
| slave1 | 10.31.155.33 | centos7.4 |
| slave2 | 10.81.233.67 | centos7.4 |
阿里云3臺(tái)ecs搭建一個(gè)小型的hbase分布式數(shù)據(jù)存儲(chǔ)（這里的ip都是內(nèi)網(wǎng)）。一臺(tái)作為主master兩臺(tái)作為從slave莺葫。
安裝步驟
- 安裝vim編輯器
- 安裝ssh
- 配置ssh免密碼登陸
- 安裝jdk
- 安裝hadoop
- 安裝zookeeper
- 安裝時(shí)間同步服務(wù) NTP
- 安裝hbase
準(zhǔn)備工作
版本
centos7.4 內(nèi)核為3.10
Hadoop 2.8.3
Hbase 1.2.6
Zookeeper 3.4.10
Sqoop 1.4.7
Phoenix 4.13.1
修改hostname
# vim /etc/hostname
輸入自定義主機(jī)名 ,三臺(tái)都要改
修改hosts文件
分別登陸三臺(tái)主機(jī)，修改hosts
vim /etc/hosts
--添加以下信息（三臺(tái)都要改）
10.30.45.239 master
10.31.155.33 slave1
10.81.233.67 slave2
登陸master莲绰，下載所有安裝包
--下載jdk8u31
wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-x64.tar.gz
-- 下載 hadoop-2.8.3
wget http://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-2.8.3/hadoop-2.8.3.tar.gz
--下載hbase 1.2.6
wget http://mirrors.hust.edu.cn/apache/hbase/1.2.6/hbase-1.2.6-bin.tar.gz
--下載zookeeper 3.4.10
wget http://apache.claz.org/zookeeper/zookeeper-3.4.10/zookeeper-3.4.10.tar.gz
--下載phoenix4.13.1
wget http://mirror.bit.edu.cn/apache/phoenix/apache-phoenix-4.13.1-HBase-1.2/bin/apache-phoenix-4.13.1-HBase-1.2-bin.tar.gz
--下載sqoop1.4.7
wget http://mirror.cogentco.com/pub/apache/sqoop/1.4.7/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz
寫這篇文章的時(shí)候是2018年4月示罗。適當(dāng)選擇穩(wěn)定的版本安裝妖啥。下載官網(wǎng)如下：
- jdk ： http://www.oracle.com/technetwork/java/javase/downloads/index.html
- hadoop：http://hadoop.apache.org/releases.html
- hbase: http://www.apache.org/dyn/closer.lua/hbase/
- zookeeper: http://zookeeper.apache.org/releases.html
- sqoop: http://sqoop.apache.org
- phoenix: https://phoenix.apache.org/
設(shè)置3臺(tái)ecs之間免密登陸
方便后期之間文件復(fù)制。
# # slave1
# ssh-keygen -t rsa
-- 不用輸入密碼查牌，一路回車就行
# cp ~/.ssh/id_rsa.pub ~/.ssh/slave1_id_rsa.pub
# scp ~/.ssh/slave1_id_rsa.pub master:~/.ssh/
# # slave2
# ssh-keygen -t rsa
# cp ~/.ssh/id_rsa.pub ~/.ssh/slave2_id_rsa.pub
# scp ~/.ssh/slave2_id_rsa.pub master:~/.ssh/
# # master
ssh-keygen -t rsa
# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
# cat ~/.ssh/slave1_id_rsa.pub >> ~/.ssh/authorized_keys
# cat ~/.ssh/slave2_id_rsa.pub >> ~/.ssh/authorized_kyes
-- 拷貝文件至slave1及slave2
# scp ~/.ssh/authorized_keys slave1:~/.ssh
# scp ~/.ssh/authorized_keys slave2:~/.ssh
關(guān)閉防火墻及SELINUX（master事期、slave1、slave2均需修改）
-- 關(guān)閉防火墻
# systemctl stop firewalld.service
# systemctl disable firewalld.service
-- 關(guān)閉SELINUX
# vim /etc/selinux/config
-- 注釋掉
#SELINUX=enforcing
#SELINUXTYPE=targeted
-- 添加
SELINUX=disabled
設(shè)置時(shí)間同步
這里省略纸颜，第一我懶得配兽泣，第二阿里云有自動(dòng)同步
安裝jdk
登陸master
-- 解壓jdk安裝包
# mkdir /usr/java
# tar -zxvf jdk-8u131-linux-x64.tar.gz -C /usr/java
-- 拷貝jdk至slave1及slave2中
# scp -r /usr/java slave1:/usr
# scp -r /usr/java slave2:/usr
-- 設(shè)置jdk環(huán)境變量（master、slave1胁孙、slave2均需修改）
# vim /etc/environment
JAVA_HOME=/usr/java/jdk1.8.0_131
JRE_HOME=/usr/java/jdk1.8.0_131/jre
# vim /etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_131
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin
搭建Hadoop環(huán)境
解壓安裝
登陸master
# tar -zxvf hadoop-2.8.3-x64.tar.gz -C /usr
# cd /usr/hadoop-2.8.3
# mkdir tmp logs hdf hdf/data hdf/name
修改Hadoop配置文件
-- 修改 slaves 文件
# vim /usr/hadoop-2.8.3/etc/hadoop/slaves
-- 刪除 localhost唠倦，添加
slave1
slave2
-- 修改 core-site.xml 文件
# vim /usr/hadoop-2.8.3/etc/hadoop/core-site.xml
-- 在 configuration 節(jié)點(diǎn)中添加以下內(nèi)容
<property>
<name>fs.default.name</name>
<value>hdfs://master:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/usr/hadoop-2.8.3/tmp</value>
</property>
-- 修改 hdfs-site.xml 文件
# vim /usr/hadoop-2.8.3/etc/hadoop/hdfs-site.xml
-- 在 configuration 節(jié)點(diǎn)添加以下內(nèi)容
<property>
<name>dfs.datanode.data.dir</name>
<value>/usr/hadoop-2.8.3/hdf/data</value>
<final>true</final>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/usr/hadoop-2.8.3/hdf/name</value>
<final>true</final>
</property>
-- 修改 mapred-site.xml 文件
# cp /usr/hadoop-2.8.3/etc/hadoop/mapred-site.xml.template /usr/hadoop-2.8.3/etc/hadoop/mapred-site.xml
# vim /usr/hadoop-2.8.3/etc/hadoop/mapred-site.xml
-- 在 configuration 節(jié)點(diǎn)添加以下內(nèi)容
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
-- 修改 yarn-site.xml 文件
# vim /usr/hadoop-2.8.3/etc/hadoop/yarn-site.xml
-- 在 configuration 節(jié)點(diǎn)添加以下內(nèi)容
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:8088</value>
</property>
復(fù)制hadoop到slave節(jié)點(diǎn)
# scp -r /usr/hadoop-2.8.3 slave1:/usr
# scp -r /usr/hadoop-2.8.3 slave2:/usr
配置 master 和 slave 的 hadoop 環(huán)境變量
# vim /etc/profile
-- 添加如下內(nèi)容
export HADOOP_HOME=/usr/hadoop-2.8.3
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
export HADOOP_LOG_DIR=/usr/hadoop-2.8.3/logs
export YARN_LOG_DIR=$HADOOP_LOG_DIR
-- 保存后執(zhí)行
# source /etc/profile
# vim ~/.bashrc
-- 添加如下內(nèi)容
export HADOOP_PREFIX=/usr/hadoop-2.8.3/
-- 保存后執(zhí)行
# source ~/.bashrc
格式化 namenode
# /usr/hadoop-2.8.3/sbin/hdfs namenode -format
啟動(dòng) hadoop（僅在master節(jié)點(diǎn)執(zhí)行）
# ssh master
# /usr/hadoop-2.8.3/sbin/start-all.sh
到這一步已經(jīng)成功完成了hadoop環(huán)境的搭建
Zookeeper環(huán)境搭建
解壓縮 zookeeper 安裝包到master，并建立基本目錄
# tar -zxvf zookeeper-3.4.10.tar.gz -C /usr
# mkdir /usr/zookeeper-3.4.10/data
修改master配置文件
-- 復(fù)制配置文件模板
# cp /usr/zookeeper-3.4.10/conf/zoo-sample.cfg /usr/zookeeper-3.4.10/conf/zoo.cfg
-- 修改配置文件
# vim /usr/zookeeper-3.4.10/conf/zoo.cfg
-- 添加如下內(nèi)容
dataDir=/usr/zookeeper-3.4.10/data
server.1=master:2888:3888
server.2=slave1:2888:3888
server.3=slave2:2888:3888
復(fù)制到各個(gè)子節(jié)點(diǎn)
# scp -r /usr/zookeeper-3.4.10 slave1:/usr
# scp -r /usr/zookeeper-3.4.10 slave2:/usr
創(chuàng)建myid文件
-- master節(jié)點(diǎn)添加myid文件
# ssh master
# touch /usr/zookeeper-3.4.10/data/myid
# echo 1 > /usr/zookeeper-3.4.10/data/myid
-- slave1節(jié)點(diǎn)添加myid文件
# ssh slave1
# touch /usr/zookeeper-3.4.10/data/myid
# echo 2 > /usr/zookeeper-3.4.10/data/myid
-- slave2節(jié)點(diǎn)添加myid文件
# ssh slave2
# touch /usr/zookeeper-3.4.10/data/myid
# echo 3 > /usr/zookeeper-3.4.10/data/myid
啟動(dòng)zookeeper（master涮较、slave1稠鼻、slave2均需執(zhí)行）
-- 啟動(dòng)master
# ssh master
# cd /usr/zookeeper-3.4.10/bin
# ./zkServer.sh start
-- 啟動(dòng)slave1
# ssh slave1
# cd /usr/zookeeper-3.4.10/bin
# ./zkServer.sh start
-- 啟動(dòng)slave2
# ssh slave2
# cd /usr/zookeeper-3.4.10/bin
# ./zkServer.sh start
到這一步完成了zookeeper環(huán)境的搭建
HBase環(huán)境搭建
解壓縮hbase安裝包
# tar -zxvf hbase-1.2.6-bin.star.gz -C /usr
# mkdir /usr/hbase-1.2.6-bin/logs
修改Hbase啟動(dòng)時(shí)要使用的環(huán)境變量（hbase-env.sh）
-- 打開環(huán)境變量配置文件
# vim /usr/hbase-1.2.6/conf/hbase-env.sh
-- 添加如下內(nèi)容
-- 1、設(shè)置java安裝路徑
export JAVA_HOME=/usr/java/jdk1.8.0_131
-- 2狂票、設(shè)置hbase的日志地址
export HBASE_LOG_DIR=${HBASE_HOME}/logs
-- 3候齿、設(shè)置是否使用hbase管理zookeeper（因使用zookeeper管理的方式，故此參數(shù)設(shè)置為false）
export HBASE_MANAGES_ZK=false
-- 4闺属、設(shè)置hbase的pid文件存放路徑
export HBASE_PID_DIR=/var/hadoop/pids
添加所有的region服務(wù)器到regionservers文件中
– 打開regionservers配置文件
# vim /usr/hbase-1.2.5/conf/regionservers
-- 刪除localhost慌盯，新增如下內(nèi)容
master
slave1
slave2
注：hbase在啟動(dòng)或關(guān)閉時(shí)會(huì)依次迭代訪問每一行來啟動(dòng)或關(guān)閉所有的region服務(wù)器進(jìn)程
修改Hbase集群的基本配置信息（hbase-site.xml）当编，該配置將會(huì)覆蓋Hbase的默認(rèn)配置
-- 打開配置文件
# vim /usr/hbase-1.2.6/conf/hbase-site.xml
-- 在configuration節(jié)點(diǎn)下添加如下內(nèi)容
<property>
<name>hbase.rootdir</name>
<value>hdfs://master:9000/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>master,slave1,slave2</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/usr/zookeeper-3.4.10/data</value>
</property>
<property>
<name>hbase.master</name>
<value>hdfs://master:60000</value>
</property>
復(fù)制hbase到slave中
# scp -r /usr/hbase-1.2.6 slave1:/usr
# scp -r /usr/hbase-1.2.6 slave2:/usr
啟動(dòng)hbase（僅在master節(jié)點(diǎn)上執(zhí)行即可）
# ssh master
# /usr/hbase-1.2.5/bin/start-hbase.sh
到這一步hbase環(huán)境搭建完成
本地測(cè)試
使用Habse shell連接Hbase
cd /usr/hbase-1.2.6/bin/
./hbase shell
創(chuàng)建user1表
對(duì)user1簡(jiǎn)單CURD
插入
put 'user1','roow1','personalinfo:name','zhangsan'
查詢
get 'user1','row1','personalinfo:name'
其他命令 help查詢就好
hbase(main):001:0> help
HBase Shell, version 1.2.6, rUnknown, Mon May 29 02:25:32 CDT 2017
Type 'help "COMMAND"', (e.g. 'help "get"' -- the quotes are necessary) for help on a specific command.
Commands are grouped. Type 'help "COMMAND_GROUP"', (e.g. 'help "general"') for help on a command group.
COMMAND GROUPS:
Group name: general
Commands: status, table_help, version, whoami
Group name: ddl
Commands: alter, alter_async, alter_status, create, describe, disable, disable_all, drop, drop_all, enable, enable_all, exists, get_table, is_disabled, is_enabled, list, locate_region, show_filters
Group name: namespace
Commands: alter_namespace, create_namespace, describe_namespace, drop_namespace, list_namespace, list_namespace_tables
Group name: dml
Commands: append, count, delete, deleteall, get, get_counter, get_splits, incr, put, scan, truncate, truncate_preserve
Group name: tools
Commands: assign, balance_switch, balancer, balancer_enabled, catalogjanitor_enabled, catalogjanitor_run, catalogjanitor_switch, close_region, compact, compact_rs, flush, major_compact, merge_region, move, normalize, normalizer_enabled, normalizer_switch, split, trace, unassign, wal_roll, zk_dump
Group name: replication
Commands: add_peer, append_peer_tableCFs, disable_peer, disable_table_replication, enable_peer, enable_table_replication, list_peers, list_replicated_tables, remove_peer, remove_peer_tableCFs, set_peer_tableCFs, show_peer_tableCFs
Group name: snapshots
Commands: clone_snapshot, delete_all_snapshot, delete_snapshot, list_snapshots, restore_snapshot, snapshot
Group name: configuration
Commands: update_all_config, update_config
Group name: quotas
Commands: list_quotas, set_quota
Group name: security
Commands: grant, list_security_capabilities, revoke, user_permission
Group name: procedures
Commands: abort_procedure, list_procedures
Group name: visibility labels
Commands: add_labels, clear_auths, get_auths, list_labels, set_auths, set_visibility
SHELL USAGE:
Quote all names in HBase Shell such as table and column names. Commas delimit
command parameters. Type <RETURN> after entering a command to run it.
Dictionaries of configuration used in the creation and alteration of tables are
Ruby Hashes. They look like this:
{'key1' => 'value1', 'key2' => 'value2', ...}
and are opened and closed with curley-braces. Key/values are delimited by the
'=>' character combination. Usually keys are predefined constants such as
NAME, VERSIONS, COMPRESSION, etc. Constants do not need to be quoted. Type
'Object.constants' to see a (messy) list of all constants in the environment.
If you are using binary keys or values and need to enter them in the shell, use
double-quote'd hexadecimal representation. For example:
hbase> get 't1', "key\x03\x3f\xcd"
hbase> get 't1', "key\003\023\011"
hbase> put 't1', "test\xef\xff", 'f1:', "\x01\x33\x40"
The HBase shell is the (J)Ruby IRB with the above HBase-specific commands added.
For more on the HBase Shell, see http://hbase.apache.org/book.html
