1 安裝sqoop,并且把Mysql中的表數(shù)據(jù)導(dǎo)出到HDFS下的文本文件里吓坚,整個(gè)過程抓圖
2 安裝flume或chukwa,并作簡單測試
1小染、
(1)下載sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz享钞,并解壓
(2)修改sqoop配置文件
Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/usr/hadoop-2.7.2
Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=
set the path to where bin/hbase is available
export HBASE_HOME=/usr/hbase_123
Set the path to where bin/hive is available
export HIVE_HOME=/usr/hive
Set the path for where zookeper config dir is
export ZOOCFGDIR=/usr/zookeeper_349/conf
(3)運(yùn)行sqoop
./bin/sqoop list-databases --connect jdbc:mysql://192.168.31.247:3306/ --username hive --password zaq1XSW@
(4)導(dǎo)入hdsf
./bin/sqoop import --connect jdbc:mysql://192.168.31.247:3306/sqoop --username hive --password zaq1XSW@ --table test -m 1
2尤蒿、
(1)下載apache-flume-1.7.0-bin.tar.gz假消,并解壓
(2)修改配置文件
flume-env.sh
Enviroment variables can be set here.
export JAVA_HOME=/usr/jdk1.8.0_101
Give Flume more memory and pre-allocate, enable remote monitoring via JMX
export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote"
flume-conf
a1.sources = r1
a1.channels = c1
a1.sinks = k1
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444
a1.sources.r1.channels = c1
a1.sinks.k1.type = logger
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
a1.sinks.k1.channel = c1
(3)啟動flume
./bin/flume-ng agent --conf ./conf/ --conf-file ./conf/flume-conf --name a1 -Dflume.root.logger=INFO,console
(4)輸出消息