5.flume实战（二）

需求：监控一个文件实时采集新增的数据并输出到控制台

简单理解就是：监控一个文件，只要这个文件有新的内容追加，就将它输出到控制台。

agent技术选型：exec source + memory channel + logger sink

下面写配置文件：

# example.conf: A single-node Flume configuration

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /usr/local/satori.txt

a1.sources.r1.shell = /bin/sh -c
a1.sources.r1.bind = localhost

a1.sources.r1.port = 44444

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

指定type为exec，然后添加一个command，我们输入tail -F file_path，表示跟踪文件,指定一个shell,可以看到配置文件改的不是很多，至于这里的参数exec，可以去flume官网上查看

新建一个配置文件，就叫exec-memory-logger.conf

输入：flume-ng agent --name a1 --conf $FLUME_HOME/conf --conf-file $FLUME_HOME/conf/exec-memory-logger.conf -Dflume.root.logger=INFO,console

我们重定向一些内容到satori.conf里面去

可以看到，已经实施监控了

但我们目前都是输出到控制台上面，意义不大，我们也可以写到hdfs里面。

此时需要指定，type为hdfs，同时指定hdfs的路径，这个是离线处理的。

实时处理的话，需要到kafka

巴特西

5.flume实战（二）

# example.conf: A single-node Flume configuration

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /usr/local/satori.txt

a1.sources.r1.shell = /bin/sh -c
a1.sources.r1.bind = localhost

a1.sources.r1.port = 44444

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

指定type为exec，然后添加一个command，我们输入tail -F file_path，表示跟踪文件,指定一个shell,可以看到配置文件改的不是很多，至于这里的参数exec，可以去flume官网上查看

输入：flume-ng agent --name a1 --conf $FLUME_HOME/conf --conf-file $FLUME_HOME/conf/exec-memory-logger.conf -Dflume.root.logger=INFO,console

最新文章

热门文章

巴特西

5.flume实战（二）

# example.conf: A single-node Flume configuration

# Name the components on this agenta1.sources = r1a1.sinks = k1a1.channels = c1

# Describe/configure the sourcea1.sources.r1.type = execa1.sources.r1.command = tail -F /usr/local/satori.txt

a1.sources.r1.shell = /bin/sh -ca1.sources.r1.bind = localhost

a1.sources.r1.port = 44444

# Describe the sinka1.sinks.k1.type = logger

# Use a channel which buffers events in memorya1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channela1.sources.r1.channels = c1a1.sinks.k1.channel = c1

指定type为exec，然后添加一个command，我们输入tail -F file_path，表示跟踪文件,指定一个shell,可以看到配置文件改的不是很多，至于这里的参数exec，可以去flume官网上查看

输入：flume-ng agent --name a1 --conf $FLUME_HOME/conf --conf-file $FLUME_HOME/conf/exec-memory-logger.conf -Dflume.root.logger=INFO,console

最新文章

热门文章

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /usr/local/satori.txt

a1.sources.r1.shell = /bin/sh -c
a1.sources.r1.bind = localhost

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1