今天接到客戶是咨詢诈火,要對現(xiàn)有的數(shù)據(jù)進(jìn)行分流沿后,雖然感覺flume實(shí)現(xiàn)沒問題柜去,但是心里總不踏實(shí)塘砸,要是出什么幺蛾子就打臉了足淆,隨后做了一個(gè)小測試,測試的結(jié)果也驗(yàn)證了我的想法丧裁,廢話不多說护桦,下面直接上配置:
#配置文件名為:flume-test.conf
# list the sources, sinks and channels in the agent
a1.sources= source_1
a1.sinks= fs_sink_local_file_1 fs_sink_local_file_2
a1.channels= fs_channel_memory_1 fs_channel_memory_2
# source from spooldir
a1.sources.source_1.type = spooldir
a1.sources.source_1.spoolDir = /home/users/flume/data
a1.sources.source_1.fileHeader = true
#channel for source configure
a1.sources.source_1.channels= fs_channel_memory_1 fs_channel_memory_2
##sinks
#sink1
a1.sinks.fs_sink_local_file_1.channel = fs_channel_memory_1
a1.sinks.fs_sink_local_file_1.type = file_roll
a1.sinks.fs_sink_local_file_1.sink.directory = /home/users/flume/dataA
#sink2
a1.sinks.fs_sink_local_file_2.channel = fs_channel_memory_2
a1.sinks.fs_sink_local_file_2.type = file_roll
a1.sinks.fs_sink_local_file_2.sink.directory = /home/users/flume/dataB
###channels
#channels1
a1.channels.fs_channel_memory_1.type = memory
a1.channels.fs_channel_memory_1.capacity = 5000000
a1.channels.fs_channel_memory_1.transactionCapacity = 5000000
a1.channels.fs_channel_memory_1.byteCapacityBufferPercentage = 40
a1.channels.fs_channel_memory_1.byteCapacity = 8000000000
a1.channels.fs_channel_memory_1.keep-alive = 30
#channels2
a1.channels.fs_channel_memory_2.type = memory
a1.channels.fs_channel_memory_2.capacity = 5000000
a1.channels.fs_channel_memory_2.transactionCapacity = 5000000
a1.channels.fs_channel_memory_2.byteCapacityBufferPercentage = 40
a1.channels.fs_channel_memory_2.byteCapacity = 8000000000
a1.channels.fs_channel_memory_2.keep-alive = 30
可以直接拿去貼就可以跑的配置,啟動(dòng)命令如下:
flume-ng agent -n a1 -f /path/to/conf/flume-test.conf -Xmx100M
以上flume分流復(fù)制實(shí)現(xiàn)的功能為:把 /home/users/flume/data目錄下的文件分別復(fù)制到/home/users/flume/dataA和/home/users/flume/dataB煎娇,既然這個(gè)功能能這樣實(shí)現(xiàn)嘶炭,那么擴(kuò)展到其他source,sink和channel都是沒問題的逊桦,
ok,今天就到這抑进,
小小菜鳥强经,不喜勿噴