1 Star 0 Fork 0

木哥儿 / flume-additional

加入 Gitee
与超过 1200万 开发者一起发现、参与优秀开源项目,私有仓库也完全免费 :)
免费加入
克隆/下载
local2hdfs.conf 2.20 KB
一键复制 编辑 原始数据 按行查看 历史
user 提交于 2014-06-11 18:16 . 修改配置
# Name the components on this agent
a1.sources = r1
a1.sinks = k1 k2
a1.channels = c1
a1.sinkgroups = g1
# Describe/configure the source
a1.sources.r1.type = avro
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 41414
a1.sources.r1.interceptors = i3 i4
a1.sources.r1.interceptors.i3.type =com.fone.flumeExt.interceptor.RegexReplaceAllInterceptor$Builder
a1.sources.r1.interceptors.i3.regex =\\$#
##### limits=all or limits=first defalut : all
a1.sources.r1.interceptors.i3.limits=all
## use unicode string
a1.sources.r1.interceptors.i3.outcome=\\u0009
a1.sources.r1.interceptors.i4.type =com.fone.flumeExt.interceptor.DefaultAvroHeaderInterceptor$Builder
# sink group
a1.sinkgroups.g1.sinks = k1 k2
a1.sinkgroups.g1.processor.type =failover
a1.sinkgroups.g1.processor.priority.k1 = 10
a1.sinkgroups.g1.processor.priority.k2 = 5
a1.sinkgroups.g1.processor.maxpenalty = 12000
#hdfs sink
a1.sinks.k1.type = com.fone.flumeExt.sink.L2HSink.LocalToHDFSSink
a1.sinks.k1.local.directory =/data1/hadoop/flume/local
a1.sinks.k1.hdfs.directory = /user/hadoop/flumeagent/logs/dir2
a1.sinks.k1.local.middleDir = %{tablename}/%{fy}%{fm}%{fd}
a1.sinks.k1.local.filePrefix = %{tablename}.8.log.%{fy}-%{fm}-%{fd}
a1.sinks.k1.local.cronExpression = 0 */15 * * * ?
a1.sinks.k1.local.rollInterval =600
a1.sinks.k1.local.rollSize =0
a1.sinks.k1.local.rollCount =0
a1.sinks.k1.local.batchSize =1000
a1.sinks.k1.local.threadsPoolSize =15
a1.sinks.k1.local.useLocalTimeStamp = true
a1.sinks.k1.local.isKeep = false
a1.sinks.k1.local.round =true
# File Roll Sink
a1.sinks.k2.type =file_roll
a1.sinks.k2.sink.directory =/data1/hadoop/flume/failover
a1.sinks.k2.sink.rollInterval=600
a1.sinks.k2.sink.batchSize=10000
a1.sinks.k2.sink.serializer=avro_event
a1.channels.c1.type = memory
a1.channels.c1.capacity =100000
a1.channels.c1.transactionCapacity =10000
a1.channels.c1.byteCapacityBufferPercentage =5
a1.channels.c1.byteCapacity =8000000
a1.channels.c1.keep-alive =10
########################BDB JE############
#a1.channels.c1.type =com.fone.flumeExt.channel.bdb.JeSortedMapChannel
#a1.channels.c1.bdbdir = /home/hadoop/temp/flume/je
#a1.channels.c1.transactionCapacity =100
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
a1.sinks.k2.channel = c1
Java
1
https://gitee.com/atuchow/flume-additional.git
git@gitee.com:atuchow/flume-additional.git
atuchow
flume-additional
flume-additional
master

搜索帮助